Posts Tagged ‘solution’

Resize KVM .img QCOW Image file and Create new LVM partition and ext4 filesystem inside KVM Virtual Machine

Friday, November 10th, 2023

LVM-add-space-to-RHEL-Linux-on-KVM_Virtual_machine-howto

Part of migration project for a customer I'm working on is migration of a couple of KVM based Guest virtual machine servers. The old machines has a backup solution stratetegy using IBM's TSM and the new Machines should use the Cheaper solution adopted by the Customer company using the CommVault backup solution (an enterprise software thath is used for data backup and recovery not only to local Tape Library / Data blobs on central backup servers infra but also in Cloud infrastructure.

To install the CommVault software on the Redhat Linux-es, the official install documentation (prepared by the team who prepared the CommVault) infrastructure for the customer recommends to have a separate partition for the CommVault backups under /opt directory  (/opt/commvault) and the partition should be as a minumum at least 10 Gigabytes of size. 

Unfortunately on our new prepared KVM VM guest machines, it was forgotten to have the separate /opt of 10GB prepared in advanced. And we ended up with Virtual Machines that has a / (root directory) of 68GB size and a separate /var and /home LVM parititons. Thus to correct the issues it was required to find a way to add another separate LVM partition inside the KVM VirtualMachine.img (QCOW Image file). 

This seemed to be an easy task at first as that might be possible with simple .img partition mount with losetup command kpartx and simple lvreduce command in some way such as

# mount /dev/loop0 /mnt/test/

# kpartx -a /dev/loop0
# kpartx -l /dev/loop0
# ls -al /dev/mapper/*

… 

# lvreduce 

etc. however unfortunately kpartx though not returning error did not provided the new /dev/mapper devices to be used with LVM tools and this approach seems to not be possible on RHEL 8.8 as the kpartx couldn't list.

 

A colleague of mine Mr. Paskalev suggested that we can perhaps try to mount the partition with default KVM tool to mount .img partitions which is guestmount but unfortunately
with a command like:
 

# guestmount -a /kvm/VM.img -i –rw /mnt/test/

But unfortunately this mounted the filesystem in fuse filesystem and the LVM /dev/mapper of the VM can't be seen so we decided to abondon this method.

After some pondering with Dimitar Paskalev and Dimitar Hristov, thanks to joint efforts we found the way to do it, below are the steps we followed to succeed in creating new LVM ext4 partition required.
One would wonder how many system
 

1. Check enough space is available on the HV machine

 

The VMs are held under /kvm so in this case:

[root@hypervisor-host ~]# df -h|grep -i /kvm
/dev/mapper/vg00-vmprivate  206G   27G  169G  14% /kvm

 

2. Shutdown the running VM and make sure it is stopped
 

[root@hypervisor-host ~]# virsh shutdown vm-host

 

[root@hypervisor-host ~]# virsh list –all
 Id   Name       State
————————–
 4    lpdkv01f   running
 5    vm-host   shut off

 

3. Check current Space status of VM

 

[root@hypervisor-host ~]# qemu-img info /kvm/vm-host.img       
image: /kvm/vm-host.img
file format: qcow2
virtual size: 100 GiB (107374182400 bytes)
disk size: 8.62 GiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zlib
    lazy refcounts: true
    refcount bits: 16
    corrupt: false
    extended l2: false

 

4. Resize (extend VM) with whatever size you want    
 

[root@hypervisor-host ~]# qemu-img resize /kvm/vm-host.img +10G

 

5. Start VM    
 

[root@hypervisor-host ~]# virsh start vm-host


7. Check the LVM and block devices on HVs (not necessery but good for an overview)
 

[root@hypervisor-host ~]# pvs
  PV         VG   Fmt  Attr PSize   PFree 
  /dev/sda2  vg00 lvm2 a–  277.87g 19.87g
  
[root@hypervisor-host ~]# vgs
  VG   #PV #LV #SN Attr   VSize   VFree 
  vg00   1  11   0 wz–n- 277.87g 19.87g

 

[root@hypervisor-host ~]# lsblk 
NAME               MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                  8:0    0 278.9G  0 disk 
├─sda1               8:1    0     1G  0 part /boot
└─sda2               8:2    0 277.9G  0 part 
  ├─vg00-root      253:0    0    15G  0 lvm  /
  ├─vg00-swap      253:1    0     1G  0 lvm  [SWAP]
  ├─vg00-var       253:2    0     5G  0 lvm  /var
  ├─vg00-spool     253:3    0     2G  0 lvm  /var/spool
  ├─vg00-audit     253:4    0     3G  0 lvm  /var/log/audit
  ├─vg00-opt       253:5    0     2G  0 lvm  /opt
  ├─vg00-home      253:6    0     5G  0 lvm  /home
  ├─vg00-tmp       253:7    0     5G  0 lvm  /tmp
  ├─vg00-log       253:8    0     5G  0 lvm  /var/log
  ├─vg00-cache     253:9    0     5G  0 lvm  /var/cache
  └─vg00-vmprivate 253:10   0   210G  0 lvm  /vmprivate

  
8 . Check logical volumes on Hypervisor host
 

[root@hypervisor-host ~]# lvdisplay 
  — Logical volume —
  LV Path                /dev/vg00/swap
  LV Name                swap
  VG Name                vg00
  LV UUID                3tNa0n-HDVw-dLvl-EC06-c1Ex-9jlf-XAObKm
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:45 +0200
  LV Status              available
  # open                 2
  LV Size                1.00 GiB
  Current LE             256
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:1
   
  — Logical volume —
  LV Path                /dev/vg00/var
  LV Name                var
  VG Name                vg00
  LV UUID                JBerim-fxVv-jU10-nDmd-figw-4jVA-8IYdxU
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:45 +0200
  LV Status              available
  # open                 1
  LV Size                5.00 GiB
  Current LE             1280
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:2
   
  — Logical volume —
  LV Path                /dev/vg00/spool
  LV Name                spool
  VG Name                vg00
  LV UUID                nFlmp2-iXg1-tFxc-FKaI-o1dA-PO70-5Ve0M9
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:45 +0200
  LV Status              available
  # open                 1
  LV Size                2.00 GiB
  Current LE             512
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:3
   
  — Logical volume —
  LV Path                /dev/vg00/audit
  LV Name                audit
  VG Name                vg00
  LV UUID                e6H2OC-vjKS-mPlp-JOmY-VqDZ-ITte-0M3npX
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:46 +0200
  LV Status              available
  # open                 1
  LV Size                3.00 GiB
  Current LE             768
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:4
   
  — Logical volume —
  LV Path                /dev/vg00/opt
  LV Name                opt
  VG Name                vg00
  LV UUID                oqUR0e-MtT1-hwWd-MhhP-M2Y4-AbRo-Kx7yEG
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:46 +0200
  LV Status              available
  # open                 1
  LV Size                2.00 GiB
  Current LE             512
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:5
   
  — Logical volume —
  LV Path                /dev/vg00/home
  LV Name                home
  VG Name                vg00
  LV UUID                ehdsH7-okS3-gPGk-H1Mb-AlI7-JOEt-DmuKnN
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:47 +0200
  LV Status              available
  # open                 1
  LV Size                5.00 GiB
  Current LE             1280
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:6
   
  — Logical volume —
  LV Path                /dev/vg00/tmp
  LV Name                tmp
  VG Name                vg00
  LV UUID                brntSX-IZcm-RKz2-CP5C-Pp00-1fA6-WlA7lD
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:47 +0200
  LV Status              available
  # open                 1
  LV Size                5.00 GiB
  Current LE             1280
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:7
   
  — Logical volume —
  LV Path                /dev/vg00/log
  LV Name                log
  VG Name                vg00
  LV UUID                ZerDyL-birP-Pwck-yvFj-yEpn-XKsn-sxpvWY
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:47 +0200
  LV Status              available
  # open                 1
  LV Size                5.00 GiB
  Current LE             1280
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:8
   
  — Logical volume —
  LV Path                /dev/vg00/cache
  LV Name                cache
  VG Name                vg00
  LV UUID                bPPfzQ-s4fH-4kdT-LPyp-5N20-JQTB-Y2PrAG
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:48 +0200
  LV Status              available
  # open                 1
  LV Size                5.00 GiB
  Current LE             1280
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:9
   
  — Logical volume —
  LV Path                /dev/vg00/root
  LV Name                root
  VG Name                vg00
  LV UUID                mZr3p3-52R3-JSr5-HgGh-oQX1-B8f5-cRmaIL
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-08-07 13:47:48 +0200
  LV Status              available
  # open                 1
  LV Size                15.00 GiB
  Current LE             3840
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:0
   
  — Logical volume —
  LV Path                /dev/vg00/vmprivate
  LV Name                vmprivate
  VG Name                vg00
  LV UUID                LxNRWV-le3h-KIng-pUFD-hc7M-39Gm-jhF2Aj
  LV Write Access        read/write
  LV Creation host, time hypervisor-host, 2023-09-18 11:54:19 +0200
  LV Status              available
  # open                 1
  LV Size                210.00 GiB
  Current LE             53760
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  – currently set to     8192
  Block device           253:10

9. Check Hypervisor existing partitions and space
 

[root@hypervisor-host ~]# fdisk -l
Disk /dev/sda: 278.9 GiB, 299439751168 bytes, 584843264 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0581e6e2

Device     Boot   Start       End   Sectors   Size Id Type
/dev/sda1  *       2048   2099199   2097152     1G 83 Linux
/dev/sda2       2099200 584843263 582744064 277.9G 8e Linux LVM


Disk /dev/mapper/vg00-root: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-swap: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-var: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-spool: 2 GiB, 2147483648 bytes, 4194304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-audit: 3 GiB, 3221225472 bytes, 6291456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-opt: 2 GiB, 2147483648 bytes, 4194304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-home: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-tmp: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-log: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-cache: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/vg00-vmprivate: 210 GiB, 225485783040 bytes, 440401920 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

 

10. List block devices on VM
 

[root@vm-host ~]# lsblk 
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                 11:0    1 1024M  0 rom  
vda                252:0    0  100G  0 disk 
├─vda1             252:1    0    1G  0 part /boot
├─vda2             252:2    0   88G  0 part 
│ ├─vg00-root      253:0    0   68G  0 lvm  /
│ ├─vg00-home      253:2    0   10G  0 lvm  /home
│ └─vg00-var       253:3    0   10G  0 lvm  /var
├─vda3             252:3    0    1G  0 part [SWAP]
└─vda4             252:4    0   10G  0 part 

 

 

11. Create new LVM partition with fdisk or cfdisk
 

If there is no cfdisk new resized space with qemu-img could be setup with a fdisk, though I personally always prefer to use cfdisk

[root@vm-host ~]# fdisk /dev/vda
# > p (print)
# > m (manfile)
# > n
# … follow on screen instructions to select start and end blocks
# > t (change partition type)
# > select and set to 8e
# > w (write changes)

[root@vm-host ~]# cfdisk /dev/vda


Setup new partition from Free space as [ primary ] partition and Choose to be of type LVM


12. List partitions to make sure new LVM partition is present
 

[root@vm-host ~]# fdisk -l
Disk /dev/vda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe7b2d9fd

Device     Boot     Start       End   Sectors Size Id Type
/dev/vda1  *         2048   2099199   2097152   1G 83 Linux
/dev/vda2         2099200 186646527 184547328  88G 8e Linux LVM
/dev/vda3       186646528 188743679   2097152   1G 82 Linux swap / Solaris
/dev/vda4       188743680 209715199  20971520  10G 8e Linux LVM

The extra added 10 Giga is seen under /dev/vda4.
  — Physical volume —
  PV Name               /dev/vda4
  VG Name               vg01
  PV Size               10.00 GiB / not usable 4.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              2559
  Free PE               0
  Allocated PE          2559
  PV UUID               yvMX8a-sEka-NLA7-53Zj-fFdZ-Jd2K-r0Db1z
   
  — Physical volume —
  PV Name               /dev/vda2
  VG Name               vg00
  PV Size               <88.00 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              22527
  Free PE               0
  Allocated PE          22527
  PV UUID               i4UpGr-h9Cd-iKBu-KqEI-15vK-CGc1-DwRPj8
   
[root@vm-host ~]# 

 

13. List LVM Physical Volumes
 

[root@vm-host ~]# pvdisplay 
  — Physical volume —
  PV Name               /dev/vda2
  VG Name               vg00
  PV Size               <88.00 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              22527
  Free PE               0
  Allocated PE          22527
  PV UUID               i4UpGr-h9Cd-iKBu-KqEI-15vK-CGc1-DwRPj8

 


  
  Notice the /dev/vda4 is not seen in pvdisplay (Physical Volume display command) because not created yet, so lets create it.
 

14. Initialize new Physical Volume to be available for use by LVM
 

[root@vm-host ~]# pvcreate /dev/vda4


15. Inform the OS for partition table changes
 

If partprobe is not available as command on the host, below obscure command should do the trick.
 

[root@vm-host ~]# echo "- – -" | tee /sys/class/scsi_host/host*/scan

However usually, better to use partprobe to inform the Operating System of partition table changes

[root@vm-host ~]# partprobe


16. Use lsblk again to see the new /dev/vda4 LVM is listed into "vda" root block device
 

[root@vm-host ~]# 
[root@vm-host ~]# lsblk
NAME          MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0            11:0    1 1024M  0 rom  
vda           252:0    0  100G  0 disk 
├─vda1        252:1    0    1G  0 part /boot
├─vda2        252:2    0   88G  0 part 
│ ├─vg00-root 253:0    0   68G  0 lvm  /
│ ├─vg00-home 253:1    0   10G  0 lvm  /home
│ └─vg00-var  253:2    0   10G  0 lvm  /var
├─vda3        252:3    0    1G  0 part [SWAP]
└─vda4        252:4    0   10G  0 part 
[root@vm-host ~]# 


17. Create new Volume Group (VG) on /dev/vda4 block device
 

Before creating a new VG, list what kind of VG is on the machine to be sure the new created one will not be already present.
 

[root@vm-host ~]# vgdisplay 
  — Volume group —
  VG Name               vg00
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <88.00 GiB
  PE Size               4.00 MiB
  Total PE              22527
  Alloc PE / Size       22527 / <88.00 GiB
  Free  PE / Size       0 / 0   
  VG UUID               oyo1oY-saSm-0IKk-gZnf-Knwz-utO7-Aw8c60

vg00 is existing only, so we can use vg01 as a Volume Group name for the new volume group where the fresh 10GB LVM partition will lay

[root@vm-host ~]# vgcreate vg01 /dev/vda4
  Volume group "vg01" successfully created

 

18. Create new Logical Volume (LV) and extend it to occupy the full space available on Volume Group vg01

 

 

[root@vm-host ~]# lvcreate -n commvault -l 100%FREE vg01
  Logical volume "commvault" created.

  An alternative way to create the same LV is by running:

lvcreate -n commvault -L 10G vg01


19. Relist block devices with lsblk to make sure the new created Logical Volume commvault is really present and seen, in case of it missing re-run again partprobe cmd
 

[root@vm-host ~]# lsblk 
NAME               MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                 11:0    1 1024M  0 rom  
vda                252:0    0  100G  0 disk 
├─vda1             252:1    0    1G  0 part /boot
├─vda2             252:2    0   88G  0 part 
│ ├─vg00-root      253:0    0   68G  0 lvm  /
│ ├─vg00-home      253:1    0   10G  0 lvm  /home
│ └─vg00-var       253:2    0   10G  0 lvm  /var
├─vda3             252:3    0    1G  0 part [SWAP]
└─vda4             252:4    0   10G  0 part 
  └─vg01-commvault 253:3    0   10G  0 lvm  

 

As it is not mounted yet, the VG will be not seen in df free space but will be seen as a volume group with vgdispaly
 

[root@vm-host ~]# df -h
Filesystem                  Size  Used Avail Use% Mounted on
devtmpfs                    2.8G     0  2.8G   0% /dev
tmpfs                       2.8G   33M  2.8G   2% /dev/shm
tmpfs                       2.8G   17M  2.8G   1% /run
tmpfs                       2.8G     0  2.8G   0% /sys/fs/cgroup
/dev/mapper/vg00-root        67G  2.4G   61G   4% /
/dev/mapper/vg00-var        9.8G 1021M  8.3G  11% /var
/dev/mapper/vg00-home       9.8G   24K  9.3G   1% /home
/dev/vda1                   974M  242M  665M  27% /boot
tmpfs                       569M     0  569M   0% /run/user/0

 

[root@vm-host ~]# vgdisplay 
  — Volume group —
  VG Name               vg01
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <10.00 GiB
  PE Size               4.00 MiB
  Total PE              2559
  Alloc PE / Size       2559 / <10.00 GiB
  Free  PE / Size       0 / 0   
  VG UUID               nYP0tv-IbFw-fBVT-slBB-H1hF-jD0h-pE3V0S
   
  — Volume group —
  VG Name               vg00
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               <88.00 GiB
  PE Size               4.00 MiB
  Total PE              22527
  Alloc PE / Size       22527 / <88.00 GiB
  Free  PE / Size       0 / 0   
  VG UUID               oyo1oY-saSm-0IKk-gZnf-Snwz-utO7-Aw8c60
  


20. Create new ext4 filesystem on the just created vg01-commvault   
 

[root@vm-host ~]# mkfs.ext4 /dev/mapper/vg01-commvault 

[root@vm-host ~]# mkfs.ext4 /dev/mapper/vg01-commvault 
mke2fs 1.45.6 (20-Mar-2020)
Discarding device blocks: done                            
Creating filesystem with 2620416 4k blocks and 655360 inodes
Filesystem UUID: 1491d8b1-2497-40fe-bc40-5faa6a2b2644
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 


21. Mount vg01-commvault into /opt directory
 

[root@vm-host ~]# mkdir -p /opt/

[root@vm-host ~]# mount /dev/mapper/vg01-commvault /opt/


22. Check mount is present on VM guest OS
 

[root@vm-host ~]# mount|grep -i /opt
/dev/mapper/vg01-commvault on /opt type ext4 (rw,relatime)
[root@vm-host ~]# 

[root@vm-host ~]# df -h|grep -i opt
/dev/mapper/vg01-commvault  9.8G   24K  9.3G   1% /opt
[root@vm-host ~]# 
 

23. Add vg01-commvault to be auto mounted via /etc/fstab on next Virtual Machine reboot
 

[root@vm-host ~]# echo '/dev/mapper/vg01-commvault /opt         ext4            defaults        1        2' >> /etc/fstab

[root@vm-host ~]# rpm -ivh commvault-fs.Instance001-11.0.0-80.240.0.3589820.240.4083067.el8.x86_64.rpm

[root@vm-host ~]# systemctl status commvault
● commvault.Instance001.service – commvault Service
   Loaded: loaded (/etc/systemd/system/commvault.Instance001.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2023-11-10 15:13:59 CET; 27s ago
  Process: 9972 ExecStart=/opt/commvault/Base/Galaxy start direct -focus Instance001 (code=exited, status=0/SUCCESS)
    Tasks: 54
   Memory: 155.5M
   CGroup: /system.slice/commvault.Instance001.service
           ├─10132 /opt/commvault/Base/cvlaunchd
           ├─10133 /opt/commvault/Base/cvd
           ├─10135 /opt/commvault/Base/cvfwd
           └─10137 /opt/commvault/Base/ClMgrS

Nov 10 15:13:57 vm-host.ffm.de.int.atosorigin.com systemd[1]: Starting commvault Service…
Nov 10 15:13:58 vm-host.ffm.de.int.atosorigin.com Galaxy[9972]: Cleaning up /opt/commvault/Base/Temp …
Nov 10 15:13:58 vm-host.ffm.de.int.atosorigin.com Galaxy[9972]: Starting Commvault services for Instance001 …
Nov 10 15:13:59 vm-host.ffm.de.int.atosorigin.com Galaxy[9972]: [22B blob data]
Nov 10 15:13:59 vm-host.ffm.de.int.atosorigin.com systemd[1]: Started commvault Service.
[root@vm-host ~]# 

 

24. Install Commvault backup client RPM in new mounted LVM under /opt

[root@vm-host ~]#  rpm -ivh commvault.rpm

Fix Zabbix selinux caused permission issues on CentOS 7 Linux / cannot set resource limit: [13] Permission denied error solution

Tuesday, July 6th, 2021

zabbix-selinux-logo-fix-zabbix-permission-issues-when-running-on-ceontos-linux-change-selinux-to-permissive-howto.

If you have to install Zabbix client that has to communicate towards Zabbix server via a Zabbix Proxy you might be unpleasently surprised that it cannot cannot be start if the selinux mode is set to Enforcing.
Error message like on below screenshot will be displayed when starting proxy client with systemctl.

zabbix-proxy-cannot-be-started-due-to-selinux-permissions

In the zabbix logs you will see error  messages such as:
 

"cannot set resource limit: [13] Permission denied, CentOS 7"

 

29085:20160730:062959.263 Starting Zabbix Agent [Test host]. Zabbix 3.0.4 (revision 61185).
29085:20160730:062959.263 **** Enabled features ****
29085:20160730:062959.263 IPv6 support: YES
29085:20160730:062959.263 TLS support: YES
29085:20160730:062959.263 **************************
29085:20160730:062959.263 using configuration file: /etc/zabbix/zabbix_agentd.conf
29085:20160730:062959.263 cannot set resource limit: [13] Permission denied
29085:20160730:062959.263 cannot disable core dump, exiting…

 

Next step to do is to check whether zabbix is listed in selinux's enabled modules to do so run:
 

[root@centos ~ ]# semodules -l

…..
vhostmd    1.1.0
virt    1.5.0
vlock    1.2.0
vmtools    1.0.0
vmware    2.7.0
vnstatd    1.1.0
vpn    1.16.0
w3c    1.1.0
watchdog    1.8.0
wdmd    1.1.0
webadm    1.2.0
webalizer    1.13.0
wine    1.11.0
wireshark    2.4.0
xen    1.13.0
xguest    1.2.0
xserver    3.9.4
zabbix    1.6.0
zarafa    1.2.0
zebra    1.13.0
zoneminder    1.0.0
zosremote    1.2.0

 

[root@centos ~ ]# sestatus
# sestatusSELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

To get exact zabbix IDs that needs to be added as permissive for Selinux you can use ps -eZ like so:

[root@centos ~ ]# ps -eZ |grep -i zabbix
system_u:system_r:zabbix_agent_t:s0 1149 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1150 ?     00:04:28 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1151 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1152 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1153 ?     00:00:00 zabbix_agentd
system_u:system_r:zabbix_agent_t:s0 1154 ?     02:21:46 zabbix_agentd

As you can see zabbix is enabled and hence selinux enforcing mode is preventing zabbix client / server to operate and communicate normally, hence to make it work we need to change zabbix agent and zabbix proxy to permissive mode.

Setting selinux for zabbix agent and zabbix proxy to permissive mode

If you don't have them installed you might neet the setroubleshoot setools, setools-console and policycoreutils-python rpms packs (if you have them installed skip this step).

[root@centos ~ ]# yum install setroubleshoot.x86_64 setools.x86_64 setools-console.x86_64 policycoreutils-python.x86_64

Then to add zabbix service to become permissive either run

[root@centos ~ ]# semanage permissive –add zabbix_t

[root@centos ~ ]# semanage permissive -a zabbix_agent_t


In some cases you might also need in case if just adding the permissive for zabbix_agent_t try also :

setsebool -P zabbix_can_network=1

Next try to start zabbox-proxy and zabbix-agent systemd services 

[root@centos ~ ]# systemctl start zabbix-proxy.service

[root@centos ~ ]# systemctl start zabbix-agent.service

Hopefully all should report fine with the service checking the status should show you something like:

[root@centos ~ ]# systemctl status zabbix-agent
● zabbix-agent.service – Zabbix Agent
   Loaded: loaded (/usr/lib/systemd/system/zabbix-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-06-24 07:47:42 CEST; 1 weeks 5 days ago
 Main PID: 1149 (zabbix_agentd)
   CGroup: /system.slice/zabbix-agent.service
           ├─1149 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
           ├─1150 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
           ├─1151 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
           ├─1152 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
           ├─1153 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
           └─1154 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

Check the Logs finally to make sure all is fine with zabbix being allowed by selinux.

[root@centos ~ ]# grep zabbix_proxy /var/log/audit/audit.log

[root@centos ~ ]# tail -n 100 /var/log/zabbix/zabbix_agentd.log


If no errors are in and you receive and you can visualize the usual zabbix collected CPU / Memory / Disk etc. values you're good, Enjoy ! 🙂

Why du and df reporting different on a filesystem / How to fix inconsistency between used space on FS and disk showing full strangeness

Wednesday, July 24th, 2019

linux-why-du-and-df-shows-different-result-inconsincy-explained-filesystem-full-oddity

If you're a sysadmin on a large server environment such as a couple of hundred of Virtual Machines running Linux OS on either physical host or OpenXen / VmWare hosted guest Virtual Machine, you might end up sometimes at an odd case where some mounted partition mount point reports its file use different when checked with
df
cmd than when checked with du command, like for example:
 

root@sqlserver:~# df -hT /var/lib/mysql
Filesystem   Type  Size Used Avail Use% Mounted On
/dev/sdb5      ext4    19G  3,4G    14G  20% /var/lib/mysql

Here the '-T' argument is used to show us the filesystem.

root@sqlserver:~# du -hsc /var/lib/mysql
0K    /var/lib/mysql/
0K    total

 

1. Simple debug on what might be the root cause for df / du inconsistency reporting

 

Of course the basic thing to do when in that weird situation is to be totally shocked how this is possible and to investigate a bit what is the biggest first level sub-directories that eat up the space on the mounted location, with du:

 

# du -hkx –max-depth=1 /var/lib/mysql/|uniq|sort -n
4       /var/lib/mysql/test
8       /var/lib/mysql/ezmlm
8       /var/lib/mysql/micropcfreak
8       /var/lib/mysql/performance_schema
12      /var/lib/mysql/mysqltmp
24      /var/lib/mysql/speedtest
64      /var/lib/mysql/yourls
144     /var/lib/mysql/narf
320     /var/lib/mysql/webchat_plus
424     /var/lib/mysql/goodfaithair
528     /var/lib/mysql/moonman
648     /var/lib/mysql/daniel
852     /var/lib/mysql/lessn
1292    /var/lib/mysql/gallery

The given output is in Kilobytes so it is a little bit hard to read, if you're used to Mbytes instead, do

 

 # du -hmx –max-depth=1 /var/lib/mysql/|uniq|sort -n|less

 

I've also investigated on the complete /var directory contents sorted by size with:

 

 # du -akx ./ | sort -n
5152564    ./cache/rsnapshot/hourly.2/localhost
5255788    ./cache/rsnapshot/hourly.2
5287912    ./cache/rsnapshot
7192152    ./cache


Even after finding out the bottleneck dirs and trying to clear up a bit, continued facing that inconsistently shown in two commands and if you're likely to be stunned like me and try … to move some files to a different filesystem to free up space or assigned inodes with a hope that shown inconsitency output will be fixed as it might be caused  due to some kernel / FS caching ?? and this will eventually make the mounted FS to refresh …

But unfortunately, if you try it you'll figure out clearing up a couple of Megas or Gigas will make no difference in cmd output.

In my exact case /var/lib/mysql is a separate mounted ext4 filesystem, however same issue was present also on a Network Filesystem (NFS) and thus, my first thought that this is caused by a network failure problem or NFS bug turned to be wrong.

After further short investigation on the inodes on the Filesystem, it was clear enough inodes are available:
 

# df -i /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5      1221600  2562 1219038   1% /var/lib/mysql

 

So the filled inodes count assumed issue also has been rejected.
P.S. (if you're not well familiar with them read manual, i.e. – man 7 inode).
 

– Remounting the mounted filesystem

To make sure the filesystem shown inconsistency between du and df is not due to some hanging network mount or bug, first logical thing I did is to remount the filesytem showing different in size, in my case this was done with:
 

# mount -o remount,rw -t ext4 /var/lib/mysql

For machines with NFS remote mounted storage locations, used:

# mount -o remount,rw -t nfs /var/www


FS remount did not solved it so I continued to ponder what oddity and of course I thought of a workaround (in case if this issues are caused by kernel bug or OS lib issue) reboot might be the solution, however unfortunately restarting the VMs was not a wanted easy to do solution, thus I continued investigating what is wrong …

Next check of course was to check, what kind of network connections are opened to the affected hosts with:
 

# netstat -tupanl


Did not found anything that might point me to the reported different Megabytes issue, so next step was to check what is the situation with currently opened files by running processes on the weird df / du reported systems with lsof, and boom there I observed oddity such as multiple files

 

# lsof -nP | grep '(deleted)'

COMMAND   PID   USER   FD   TYPE DEVICE    SIZE NLINK  NODE NAME
mysqld   2588  mysql    4u   REG 253,17      52     0  1495 /var/lib/mysql/tmp/ibY0cXCd (deleted)
mysqld   2588  mysql    5u   REG 253,17    1048     0  1496 /var/lib/mysql/tmp/ibOrELhG (deleted)
mysqld   2588  mysql    6u   REG 253,17       777884290     0  1497 /var/lib/mysql/tmp/ibmDFAW8 (deleted)
mysqld   2588  mysql    7u   REG 253,17       123667875     0 11387 /var/lib/mysql/tmp/ib2CSACB (deleted)
mysqld   2588  mysql   11u   REG 253,17       123852406     0 11388 /var/lib/mysql/tmp/ibQpoZ94 (deleted)

 

Notice that There were plenty of '(deleted)' STATE files shown in memory an overall of 438:

 

# lsof -nP | grep '(deleted)' |wc -l
438


As I've learned a bit online about the problem, I found it is also possible to find deleted unlinked files only without any greps (to list all deleted files in memory files with lsof args only):

 

# lsof +L1|less


The SIZE field (fourth column)  shows a number of files that are really hard in size and that are kept in open on filesystem and in memory, totally messing up with the filesystem. In my case this is temp files created by MYSQLD daemon but depending on the server provided service this might be apache's www-data, some custom perl / bash script executed via a cron job, stalled rsync jobs etc.
 

2. Check all the list open files with the mysql / root user as part of the the server filesystem inconsistency debugging with:

 

– Grep opened files on server by user

# lsof |grep mysql
mysqld    1312                       mysql  cwd       DIR               8,21       4096          2 /var/lib/mysql
mysqld    1312                       mysql  rtd       DIR                8,1       4096          2 /
mysqld    1312                       mysql  txt       REG                8,1   20336792   23805048 /usr/sbin/mysqld
mysqld    1312                       mysql  mem       REG               8,21      24576         20 /var/lib/mysql/tc.log
mysqld    1312                       mysql  DEL       REG               0,16                 29467 /[aio]
mysqld    1312                       mysql  mem       REG                8,1      55792   14886933 /lib/x86_64-linux-gnu/libnss_files-2.28.so

 

# lsof | grep root
COMMAND    PID   TID TASKCMD          USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
systemd      1                        root  cwd       DIR                8,1       4096          2 /
systemd      1                        root  rtd       DIR                8,1       4096          2 /
systemd      1                        root  txt       REG                8,1    1489208   14928891 /lib/systemd/systemd
systemd      1                        root  mem       REG                8,1    1579448   14886924 /lib/x86_64-linux-gnu/libm-2.28.so

Other command that helped to track the discrepancy between df and du different file usage on FS is:
 

# du -hxa  / | egrep '^[[:digit:]]{1,1}G[[:space:]]*'
 

 

3. Fixing large files kept in memory filesystem problem


What is the real reason for ending up with this file handlers opened by running backgrounded programs on the Linux OS?
It could be multiple  but most likely it is due to exceeded server / client interactions or breaking up RAM or HDD drive with writing plenty of logs on the FS without ending keeping space occupied or Programming library bugs used by hanged service leaving the FH opened on storage.

What is the solution to file system files left in memory problem?

The best solution is to first fix custom script or hanged service and then if possible to simply restart the server to make the kernel / services reload or if this is not possible just restart the problem creation processes.

Once the process is identified like in my case this was MySQL on systemd enabled newer OS distros, just do:

 

 

# systemctl restart mysqld.service


or on older init.d system V ones:

# /etc/init.d/service restart


For custom hanged scripts being listed in ps axuwef you can grep the pid and do a kill -HUP (if the script is written in a good way to recognize -HUP and restart the sub-running process properly – BE EXTRA CAREFUL IF YOU'RE RESTARTING BROKEN SCRIPTS as this might cause your running service disruptions …).

# pgrep -l script.sh
7977 script.sh


# kill -HUP PID

 

Now finally this should either mitigate or at best case completely solve the reported disagreement between df and du, after which the calculated / reported disk space should be back to normal and show up approximately the same (note that size changes a bit as mysql service is writting data) constantly extending the size between the two checks.

 

# df -hk /var/lib/mysql; du -hskc /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5        19097172 3472744 14631296  20% /var/lib/mysql
3427772    /var/lib/mysql
3427772    total

 

What we learned?

What I've explained in this article is why and how it comes that 'zoombie' files reside on a filesystem
appearing to be eating disk space on a mounted local or network partition, giving strange inconsistent
reports, leading to system service disruptions and impossibility to have correctly shown information on used
disk space on mounted drive.

I went through with some standard logic on debugging service / filesystem / inode issues up explainat, that led me to the finding about deleted files being kept in filesystem and producing the filesystem strange sized / showing not correct / filled even after it was extended with tune2fs and was supposed to have extra 50GBs.

Finally it was explained shortly how to HUP / restart hanging script / service to fix it.

Some few good readings that helped to fix the issue:

What to do when du and df report different usage is here
df in linux not showing correct free space after file removal is here
Why do “df” and “du” commands show different disk usage?
 

Fix qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied – Finally a working solution and what causes the error

Wednesday, July 22nd, 2015

 

qmail-fixing-clamdscan-errors-and-qq-errors-qmail-binary-migration-few-things-to-check-out
I've lost a whole day and was angry and irritated after moving (migrating) a Working Qmail installation in a binary form from a Debian Lenny 5.0 to a Debian 7.0 Wheezy Linux. The whole migration exercise was quite of a crazy move and I can tell you it didn't worth the effort as I lost much more time than even if I went on installing the server following Thibs Great Qmail Tutorial.

Yes I know many would say why do you still bother with an old and unsupported Qmail Mail server and not go with Postfix, the logic is correct however the whole issue is the previous installation has a number of domains already running VirtualMail hosting using VPopMail, so migrating all the old mailboxes from Qmail to Postfix are not worthy IMHO. Plus I honestly love qmail for being so stable even today (even without support). After all most of Qmail is secure enough already 🙂
And to be honest I don't so much care about security as in the old days as I know NSA, does already have access to any server on planet 🙂

Almost always when a Qmail migration happens I end up swearing and sweating and generally getting crazy, but anyways …

The overall migration of binaries went quite OK I just copied every binary and all the related libraries from the old Debian 5.0 to Debian 7.0 and installed the following long list of perl deb binaries using apt-get:
 

dh-make-perl
libalgorithm-c3-perl
libalgorithm-diff-perl
libalgorithm-diff-xs-perl
libalgorithm-merge-perl
libapparmor-perl
libapt-pkg-perl
libarray-unique-perl
libclass-accessor-chained-perl
libclass-accessor-perl
libclass-c3-perl
libclass-c3-xs-perl
libclass-factory-util-perl
libclass-inspector-perl
libclass-isa-perl
libclass-load-perl
libclass-singleton-perl
libconfig-file-perl
libconvert-binhex-perl
libcrypt-openssl-bignum-perl
libcrypt-openssl-random-perl
libcrypt-openssl-rsa-perl
libcrypt-passwdmd5-perl
libcrypt-ssleay-perl
libdata-optlist-perl
libdata-section-perl
libdate-manip-perl
libdatetime-format-builder-perl
libdatetime-format-iso8601-perl
libdatetime-format-strptime-perl
libdatetime-locale-perl
libdatetime-perl
libdatetime-timezone-perl
libdbd-mysql-perl
libdbi-perl
libdevel-symdump-perl
libdigest-hmac-perl
libdigest-sha-perl
libdpkg-perl
libemail-address-perl
libemail-date-format-perl
libencode-detect-perl
libencode-locale-perl
libenv-sanctify-perl
liberror-perl
libexporter-lite-perl
libfcgi-perl
libfile-chdir-perl
libfile-fcntllock-perl
libfile-find-rule-perl
libfile-listing-perl
libfile-which-perl
libfont-afm-perl
libhtml-form-perl
libhtml-format-perl
libhtml-parser-perl
libhtml-tagset-perl
libhtml-template-perl
libhtml-tree-perl
libhttp-cookies-perl
libhttp-daemon-perl
libhttp-date-perl
libhttp-message-perl
libhttp-negotiate-perl
libhttp-server-simple-perl
libio-multiplex-perl
libio-socket-inet6-perl
libio-socket-ip-perl
libio-socket-ssl-perl
libio-string-perl
libio-stringy-perl
libip-country-perl
liblist-moreutils-perl
liblocale-gettext-perl
liblwp-mediatypes-perl
liblwp-protocol-https-perl
libmail-dkim-perl
libmail-sendmail-perl
libmail-spf-perl
libmailtools-perl
libmath-round-perl
libmime-tools-perl
libmodule-depends-perl
libmodule-implementation-perl
libmodule-runtime-perl
libmro-compat-perl
libnet-cidr-lite-perl
libnet-cidr-perl
libnet-daemon-perl
libnet-dns-perl
libnet-http-perl
libnet-ident-perl
libnet-ip-perl
libnet-server-perl
libnet-snmp-perl
libnet-ssleay-perl
libnetaddr-ip-perl
libnumber-compare-perl
libossp-uuid-perl
libpackage-deprecationmanager-perl
libpackage-stash-perl
libpackage-stash-xs-perl
libparams-classify-perl
libparams-util-perl
libparams-validate-perl
libparse-debcontrol-perl
libparse-debianchangelog-perl
libparse-syslog-perl
libpcre-ocaml-dev
libpcre3:amd64
libpcre3-dev
libpcrecpp0:amd64
libperl-dev
libperl5.14
libpod-coverage-perl
libregexp-assemble-perl
librpc-xml-perl
librrds-perl
libsoap-lite-perl
libsocket-perl
libsocket6-perl
libsoftware-license-perl
libsub-exporter-perl
libsub-install-perl
libsub-name-perl
libswitch-perl
libsys-hostname-long-perl
libsys-syslog-perl
libtask-weaken-perl
libterm-readkey-perl
libtest-distribution-perl
libtest-pod-coverage-perl
libtest-pod-perl
libtext-charwidth-perl
libtext-glob-perl
libtext-iconv-perl
libtext-template-perl
libtext-wrapi18n-perl
libtie-ixhash-perl
libtimedate-perl
libtry-tiny-perl
liburi-perl
libwww-mechanize-perl
libwww-perl
libwww-robotrules-perl
libxml-namespacesupport-perl
libxml-parser-perl
libxml-sax-base-perl
libxml-sax-expat-perl
libxml-sax-perl
libxml-simple-perl
libyaml-perl
libyaml-syck-perl
perl
perl-base
perl-doc
perl-modules
spamassassin
spf-tools-perl

 

I've also installed with apt-get daemontools and daemontools-run ucspi and some others to get all the necessery binaries qmail needs the whole list of apt installed packages is here

I've also copied all the old binaries from /usr/local/lib from server1 to server2, some others from /usr/local/share and /usr/share as well as /usr/lib/courier /usr/lib/courier-authlib /usr/local/libexec /usr/local/sbin also had to link a couple of libraries such as /usr/lib/libcrypto* , link  /usr/lib/libperl.so.5.10 to /usr/lib/libperl.so.5.14 and copy /usr/lib/libltdl.so.3 and few others which i don't exactly remember.

Well anyways once I've copied everything Qmail looked fined except I had a couple of permission issues and had to clean up and fix the queue with qfixq, I've also used qmail-scanner*/contrib/test_installation.sh script to test whether qmail-scanner was running fine, e.g.:
 

./test_installation.sh -doit


As well as

 

 

qmr_inst_check


script, thanks to which I've captured and resolved few of permission problems 

Finally I've stuck upon this shitty errors (appearing) in /var/log/syslog and /var/log/messages

 

 

Jul 21 22:04:19 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:08:27 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:08:38 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:11:17 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:16:09 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:19:15 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:38:59 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:42:33 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:43:49 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:46:05 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:50:40 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied
Jul 21 22:53:08 ns2 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied

 

 

 

There is plenty of things written about this:
 

qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[] cannot create /var/spool/qscan/tmp – Permission denied


But all written is too obscure and too old already somewhere between 2004 and 2010, I've been digging through Gentoo Forums, Fedora Debian and other Linux installs and everyone used to be pointing a permission issue with /var/spool/qscan/ said theoretically to be causing the error, however all looked perfectly fine with my /var/spool/qscan , e.g.:

 

 

roo@ns2:/usr/local/src# ls -ld /var/spool/qscan/
drwxr-s— 6 qscand qscand 4096 Jul 21 23:07 /var/spool/qscan/

 

 

 

 

ls -al /var/spool/qscan/
total 244904
drwxr-s— 6 qscand qscand      4096 Jul 21 23:07 .
drwxr-xr-x 4 root   root        4096 Jul 20 21:17 ..
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 archives
-rwxr-x— 1 qscand qscand      1434 Oct 12  2011 log-report.sh
-rw——- 1 qscand qscand 249731919 Jul 21 23:11 qmail-queue.log
-rw——- 1 qscand qscand    398225 Oct 28  2011 qmail-queue.log.1
-rw-rw—- 1 root   qscand        74 Jul 21 23:07 qmail-scanner-queue-version.txt
lrwxrwxrwx 1 root   qscand        16 Jul 21 23:07 qscan -> /var/spool/qscan
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 quarantine
-rw-r—– 1 root   qscand     12288 Jul 21 23:07 quarantine-events.db
-rw-r—– 1 qscand qscand     10443 Oct 12  2011 quarantine-events.txt
-rw-rw—- 1 qscand qscand    580033 Jul 21 23:07 quarantine.log
-rw-r—– 1 qscand qscand      2739 Oct 12  2011 settings_per_domain.txt
drwxr-x— 3 qscand qscand      4096 Jul 21 23:11 tmp
drwxrwx— 5 qscand qscand      4096 Oct 12  2011 working

 

Some suggested that /var/spool/qscan should be owned by qscand:clamav instead so I tried this but it didn't help,
others recommended adding clamav groupid into qscand's but this didn't help either:
 

root@ns2:/usr/local/src# grep -i clamav /etc/group
qscand:x:163:clamav,vpopmail
clamav:x:105:

 

Besides that I was getting also this shitty error:

 

Jul 21 20:05:42 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750194279012466] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:06:51 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750201179013125] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:15:42 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns214375025407906015] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:16:06 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750256579011980] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:18:54 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750273479014847] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 20:21:03 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750286379028491] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status -1/72057594037927935
Jul 21 22:07:47 vps186637 qmail-scanner-queue.pl: X-Qmail-Scanner-2.08st:[ns2143750926779010097] clamdscan: corrupt or unknown clamd scanner error or memory/resource/perms problem – exit status 512/2

 

All the time in logs, so I thought something might be wrong with clamdscan and followed and tried some suggestions from qms1.net as described here, however none of the fixes described there worked for me …

I also tried to reinstall clamav from source using a bit of modified version of this tutorial

This didn't help either … I saw some suggestions online that the permission issues are caused by some wrong clamd.conf and freshclamd.conf configuration options – failing clamdscan, but this didn't work either. I also tried to remove clamdscan and substitute it with clamscan  as a suggested workaround but this didn't work either …

I spend about 6 hours trying to catch what is causing this issues so finally I went on and re-installed bigger part of Qmail using Thibs tutorial over the old installation.

I've also tried in mean time multiple time to rebuild qmail scanner database with:
 

setuidgid qscand /var/qmail/bin/qmail-scanner-queue.pl -g


Played around with permissions in /etc/clamav e.g.

 

 

chown -R qscand:clamav /var/log/clamav /var/lib/clamav /var/run/clamav
chown qscand:qscand /etc/clamav/freshclam.conf

 

 

 

Created:

/etc/cron.daily/qmail-scanner with following content

 

cat /etc/cron.daily/qmail-scanner
setuidgid qscand /var/qmail/bin/qmail-scanner-queue.pl -z


However in /var/log/qmail/qmail-smtpd/current and /var/log/qmail-send/current I continously got:

Qmail 451 qq temporary problem (#4.3.0) error

Interestingly during looking for a solution to the 451 qq temporary problem and:

status: qmail-todo stop processing asap / status: exiting – I've stumbled to my own blog post here 🙂

 

Finally, I tried to reinstall qmail-scanner and in mean time update it to  Version: 2.11 – st – patch – 20130319
Just to realized something was wrong with suidperl, e.g. in Debian 7.0.* Wheezy perl-suid binary is no longer in repositories so only way to have suidperl there is either to re-compile perl from source manually which is too much work and I think in most cases not worthy the effort or to use a small suid-wrapper:
 

#include <unistd.h>
#include <errno.h>
 

main( int argc, char ** argv, char ** envp )
{
              if( setgid(getegid()) ) perror( "setgid" );
              if( setuid(geteuid()) ) perror( "setuid" );
              envp = 0; /* blocks IFS attack on non-bash shells */
              system( "/usr/bin/perl", argv, envp );
              perror( argv[0] );
              return errno;
}

 


Create it into a file lets say suid-wrapperc and compile the file with GNU C Compiler:
 

$ gcc -o suid-wrapper suid-wrapper.c


Then move the suid-wrapper produced binary into /usr/bin/suidperl e.g.

 

 

$ mv suid-wrapper /usr/bin/suidperl


Last you will need to edit /var/qmail/bin/qmail-scanner-queue.pl

 

 

 

 

# vim /var/qmail/bin/qmail-scaner-queue.pl


And substitute

 

 

 

 

#!/usr/bin/perl -T


with:

 

 

#!/usr/bin/suidperl


Note!!! that qmail-scanner-queue.pl permissions should be suid and owned by qscand:qscand as follows:

 

 

 

 

ls -al /var/qmail/bin/qmail-scanner-queue.pl
-rwsr-sr-x 1 qscand qscand 159727 Jul 21 23:11 /var/qmail/bin/qmail-scanner-queue.pl


Finally to resolve the error I had to restart qmail via qmailctl start / stop script:

 

 

 

 

root@ns2:/var/qmail/bin# qmailctl restart
Restarting qmail:
* Stopping qmail-smtpdssl.
* Stopping qmail-smtpd.
* Sending qmail-send SIGTERM and restarting.
* Restarting qmail-smtpd.
* Restarting qmail-smtpdssl.

 

Finally to test emails are sent and receiver properly I used good old mail command part of bsd-mailx deb package

 

# mail -s "testing 12345678" testemail1234@gmail.com
asdfadf
.
Cc:

 

 

I've also tested with plain telnet to verify no errors because often the mail command doesn't return (show) errors on email sent and errors are written only in /var/log/mail.log or /var/log/qmail/* logs

 

 

# telnet localhost 25
Trying 127.0.0.1…
Connected to localhost.
Escape character is '^]'.
220 servername.localdomain.tld ESMTP
mail from: <testmail@test.com>
250 ok
rcpt to: <nospam@test.com>
250 ok
data
354 go ahead
From: Test_sender <testmail@test.com>
To: Test_receiver <nospam@test.com>
Subject: Just a stupid SMTP test

 

Just a test !
.
250 ok 1279384489 qp 3711
quit
221 servername.localdomain.tld
Connection closed by foreign host.

One other thing which probably helped I did was:

# qmailctl doqueue

 

Thanks God this time, it worked out without any QQ errors 🙂 !
 

 

Ubuntu 9.04 (Jaunty) on Toshiba Satellite L40 14B ACPI problem ACPI loading problem and workaround

Thursday, September 10th, 2009

As you have noticed in my previous posts I'm playing a bit with Toshiba Satellite L40 14B updates this days. I've came across an ugly problem with ACPI on Ubuntu Jaunty. It's really a big issue because acpi stalls and keeps loading the system on 100% leaving it with 0% idle all the time. The solution I’ve found is suggested on the following Ubuntu bug launchpad . As it describes the solution comes to this:

1. rm -f /etc/acpi/events/asus-wireless-on

2. rm -f /etc/acpi/events/asus-wireless-2

the next two steps are added by meto prevent some problems caused by acpidupgrades.

3. touch /etc/acpi/events/asus-wireless-2

4. touch /etc/acpi/events/asus-wireless-on

Bill Gates explaining solution to over-population – depopulation through vacines and medical care system – Is Bill and Melinda Gates foundation really good?

Tuesday, June 26th, 2012

Bill-Gates-vaccine-depopulation-program-or-how-to-solve-the-over-population-with-vaccines

While checking some videos on the net I've came across information that Bill Gates bough 500 000 shares from Monsanto.

Gates is a well know for being Philantrophist, giving large amounts of charities aiming to fight the hunger in Africa through his and his life charity foundation Bill and Melinda Gates. This charity looks like something perfect on a first glimpse as of September 2011 he has donated $33 billions to the foundation which is trying to help Africans to fight hunger diseases, find cure for HIV etc.

The methodology to help the hungry and sick people in africa is done via distribution of food grown and vaccination and helping the "poor" people in Africa get some education. It all sounds like a very good deed and one can say "what a good man of honour" GAtes is.

Well this would have been true only if Gates didn't said it clear in TED's show that vacination is one of the ways which can be used to battle the over population and increased need for food, medical service and energy.

See the short video below:

Bill Gates suggests Depopulation Through Vaccines on a talk on TED show

I don't think it is too likely gates made an error in saying what he said on the presentation, obviously it was a clearly prepared presentation for the show and what he said was exactly what he meant. Now put aside the vacination of africa along with the likeness that a lot of latest drug medicine are probably injected in many unknowing Africans used as a guinea pig and you understand that this $33 Billion are given for eradication of large and testing of medicals on large population uneducated mass in Africa.

Not that I say vacination is necessery bad as one could think. From the few videos, I've seen with Gates rationalizing on various world problems one can come easily to the conclusion that Gates has a strong Utalitarian mindset. If you're not aware of Utilitarianism frightening philosophy see here

 

Fix “FAIL – Application at context path /application-name could not be started” deployment error in Tomcat manager

Thursday, October 1st, 2015

tomcat-manager-FAIL-Application-at-context-path-application-name-could-not-be-started-fix-solution-error

While deploying an environment called "Interim" which is pretty much like a testing Java application deployed from a Java EAR (Enterprise Archive) file from within a Tomcat Manager GUI web interface after stopping the application and trying to start it, the developers come across the error:

 

FAIL – Application at context path /application-name could not be started


The error puzzled me for a while until I checked the catalina.out I've seen a number of thrown Java Eceptions errors like:

Okt 01, 2015 10:48:46 AM org.springframework.web.context.ContextLoader initWebApplicationContext

Schwerwiegend: Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.sun.xml.ws.transport.http.servlet.SpringBinding#2' defined in ServletContex

t resource [/WEB-INF/pp-server-beans.xml]: Cannot create inner bean ‘(inner bean)’ of type [org.jvnet.jax_ws_commons.spring.SpringService] while setting bean property

'service'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#33': FactoryBean threw exception on

 object creation; nested exception is java.lang.OutOfMemoryError: PermGen space

I've googled a bit about the error:

"FAIL – Application at context path /application-name could not be started"

and come across this Stackoverflow thread and followed suggested solution to fix web.xml tag closing error but it seems there was no such error in my case, I then also tried solution suggested by this thread (e.g. adding in logging.properties) file:
 

org.apache.catalina.core.ContainerBase.[Catalina].level = INFO
org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

unfortunately this helped neither to solve the error when it is tried to be started from tomcat manager.

After asking for help a colleague Kostadin, he pointed me to take a closer look in the error (which is a clear indication) that the reserved space is not enough (see below err):
 

java.lang.OutOfMemoryError: PermGen space

And he pointed me then to Solution (which is to modify the present tomcat setenv.sh) settings which looked like this:

# Heap size settings

export JAVA_OPTS="-Xms2048M -Xmx2048M"

 

# SSCO test page parameter

export JAVA_OPTS="$JAVA_OPTS -DTS1A_TESTSEITE_CONFIG_PATH=test-myapplication.com"

# Default garbage collector settings

export JAVA_OPTS="$JAVA_OPTS -XX:MaxPermSize=128M"

 

# Aggressive garbage collector settings.

# Do not use! For testing purposes only!

#export JAVA_OPTS="$JAVA_OPTS -Xss128k -XX:ParallelGCThreads=20 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:+AggressiveOpts -XX:MaxPermSize=128M"

 

####### DO NOT CHANGE BELOW HERE #######

# Disable X11 usage

unset DISPLAY

export JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

 

# Garbage collection log settings

export JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/web/tomcat/current/logs/gc.log -XX:-TraceClassUnloading"

 

# Enable JMX console

export JAVA_OPTS="$JAVA_OPTS -Dcom.sun.management.jmxremote"

 

 

 

 

 

 

The solution is to to add following two options to export JAVA_OPTS (above options):

-XX:PermSize=512m -XX:MaxPermSize=512m


After modifications, my new line within setenv.sh looked like so:

 

JAVA_OPTS="-Xms2048M -Xmx2048M -XX:PermSize=512m -XX:MaxPermSize=512m"


Finally to make new JAVA_OPTS settings, I've restarted Tomcat with:

 

cd /web/tomcat/bin/
./shutdown.sh
sleep 5; ./startup.sh


And hooray it works fine thanks God! 🙂

Stop contact form spam emails in Joomla, Disable “E-mail a copy of this message to your own address.” in Joomla

Friday, April 11th, 2014

email-copy-of-this-message-to-your-own-address_Contact_email_form
If you happen to have installed Joomla based website and setup a contact form and everything worked fine until recently but suddenly your server starts mysteriously acting as a spam relay – even though email server is perfectly secured against spam.
You probably have some issue with a website email contact form hacked or some vulnerability which allowed hackers to upload spammer php script.

I have a website based on Joomla and just until recently everything was okay until I noticed there are tons of spam flying out from my Qmail mail server (which is configured to check spam with Spamassassin has Bayesian Filtering, Distributed Checksum Claring House, Python Razor and plenty of custom anti-spam rules.

It was just yesterday I ended into that situation, then after evaluating all the hosted website, I've realized Spam issues are caused by an Old Joomla Website Contact form!

There were two issues in the form

in the contact form you have the field with a tick:

1. Well Known Joomla Form Vulnerability
Currently all Joomla (including 1.5.22 and 1.6 versions) are vulnerable to a serious spam relay problem as described in the official Joomla site.

There is a quick dirty workaround fix to contact form vulnerability –  disable a Joomla Comonent in ../joomla/components/com_mailto/

To disable it I had to:

cd /var/www/joomla/components
mv com_mailto com_mailtoNOT_USED

Above solution was described under a post resolve joomla spam relay earlier by Anatoliy Dimitrov (after checking closely the website it happened he is a colleague at HP 🙂 )

2. Second issue causing high amount of spam sent over the email server
was: "E-mail a copy of this message to your own address." contact form tick, which was practically enabling any Spammer with a list to inect emails and spam via the form sending copies to any email out on the internet!

You would definitely want to disable  "E-mail a copy of this message to your own address."
I wonder why ever any Joomla developer came up with this "spam form"?? 

joomla-disable-email-copy-of-this-message-to-your-own-address

Here is the solution to this:

1. Login to Joomla Admin with admin account
2. Goto Components -> Contacts -> Contacts
3. Click on the relevant Contact form
4. Under Contact Parameters go to Email Parameters
5. Change field E-mail Copy from Show to Hide and click Apply button

And Hooray the E-mail a copy of this message to your own address will be gone from contact form! 🙂

I've seen already plenty of problematic hacked servers and scripts before with Joomla in my last job in International University College – where joomla was heavy used, but I never experienced Joomla Security issues myself 'till know, in future I'm planning to never ever use joomla. Though it is an easy CMS system to setup a website its quite complicated to learn the menus – I remember when creating the problematic website it took me days until I properly setup all the menus and find all joomla components … besides these there is no easy way to migrate between different versions major releases in Joomla like in Wordperss, I guess this Mail Security Issue absolutely convinced me to quit using that piece of crap in future.

In mean Time another very serious Apache security flaw leaked on the Internet just few days ago – The OpenSSL Hearbleed Bug. Thanksfully I'm not running SSL anywhere on my website but many systems are affecting making most of your SSL communication with your Internet banking, E-mail etc. in danger. If you're running Apache with SSL make sure you test it for this vulnerability. Here is description of Heartbleed SSL Critical Vulnerability.

heartbleed_ssl_remote_vulnerability_logo

"The Heartbleed Bug is a serious vulnerability in the popular OpenSSL cryptographic software library. This weakness allows stealing the information protected, under normal conditions, by the SSL/TLS encryption used to secure the Internet. SSL/TLS provides communication security and privacy over the Internet for applications such as web, email, instant messaging (IM) and some virtual private networks (VPNs).

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content. This allows attackers to eavesdrop on communications, steal data directly from the services and users and to impersonate services and users."

11

 

ping “General Failure” no internet connection Windows 7 on HP work computer – Reasons for general failure and solution

Monday, May 26th, 2014

windows-7-general-failure-error-fix-on-hp-workbook
Out of a sudden today after running Malware Bytes – Free Anti-Malware & Internet Security Software, and after it found some Malware (Pup.Optional.Opencandy) and removed it it WI-FI internet on my work computer HP Elitebook 8470p mysteriously stopped working.

That's quite nasty because today I'm working from Home – well known among Hewlett Packard employees under WFH abbreviation. I couldn't connect normally to my home Access Point and tried pinging Google from command line just to get an error:
 

Transmit Failed: General Failure


and first I thought it is a wi-fi router related problem and restarted my WIFI RouterD-Link DI524. As I could normally connect to the WIFI and I see there is an internet IP assigned running:

ipconfig /all
...
Wireless LAN adapter Wireless Network Connection:
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Intel(R) Centrino(R) Ultimate-N 6300 AGN
Physical Address. . . . . . . . . : 3C-A9-F4-4C-E7-98
DHCP Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::5d2f:97b8:9e1a:2b13%63(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.2.159(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Lease Obtained. . . . . . . . . . : May 23, 2014 14:19:01 PM
Lease Expires . . . . . . . . . . : May 30, 2014 14:32:49 PM
Default Gateway . . . . . . . . . : 192.168.2.1
DHCP Server . . . . . . . . . . . : 192.168.2.1
DHCPv6 IAID . . . . . . . . . . . : 1094494708
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-19-CB-1A-5D-A4-5D-36-5A-EB-84
DNS Servers . . . . . . . . . . . : 8.8.8.8
192.168.2.1
NetBIOS over Tcpip. . . . . . . . : Enabled

As you see in above output I have notebook IP, default gateway and DNS IP assigned – i.e. all seems fine, so as I got General Failure from pinging the Internet in order to make sure my Linux router is not the bottleneck I tried pinging Default GW

C:UsersGeorgi> ping 192.168.2.1 Pinging 192.168.2.1 with 32 bytes of data:
PING: transmit failed. General failure.
PING: transmit failed. General failure.
PING: transmit failed. General failure.
PING: transmit failed. General failure.
Ping statistics for 192.168.2.1:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)

Just to realize I'm continually getting General Failure error

I tried trying to renew IPs, to make sure there is no some kind of IP assignment conflict with my other WIFI connected devices,reflushing DNS and resetting WinSock stack, hoping this could help:

> ipconfig /renew
> ipconfig /flushdns
> netsh winsock reset

Then I tried restarting the PC a couple of times, but unfortunately none of those helped the shitty error:
 

PING: Transmit failed. General Failure

continued …

I was totally out of idea .. and red some people managed to fix the issue after booting Windows into Safe mode with Networking. I tried booting in Safe mode, but as Hard Drive data is encrypted with Bitlocker encryption I was asked about some kinf of Serial Key – which I don't have at hand – hence I couldn't boot into Safe mode …

Here is moment to say even loopback device was returning "General Failure"!

I tried even connecting the laptop directly into my homelan with UTP cable, but though everything got connected, there was no local network and internet. I tried even connecting via Vivacom's mobile network 3G modem and even there I got the "General Failure" error …

Running out of options, I decided it might be that Malware Bytes broke something during Malware removal hence I put out back Quarantined Malware files – but this didn't solve it either.

solution-to-no-internet-general-failure-ping-error-firewall-off-screenshot.png

Finally I found this post and this thread talking that reason for "General Failure" might be firewall related. After checking my firewall settings in Windows Firewall and Advanced Security, surprisingly I realized everything related to firewall – e.g. Default Profile, Inbound, Outbound connections are Turned off!!!

windows-firewall-off-reason-for-general-failure-no-internet

I switched everything back and my Internet and local connection came back! THANKS GOD! Pfuu, now I can continue working. It seems HP work computers are patched with software / configured to not allow Internet connection in case if Firewall is Off. If you happen to be an HP Employee and you get the PING: Transmit failed. General Failure, be sure the first place to check is whether Windows Firewall is enabled? – if not enable it and this will solve your connectivity issues. Cheers ! 🙂