Initialize Archives - ☩ Walking in Light with Christ - Faith, Computing, Diary ☩ Walking in Light with Christ

Posts Tagged ‘Initialize’

Resize KVM .img QCOW Image file and Create new LVM partition and ext4 filesystem inside KVM Virtual Machine

Friday, November 10th, 2023

LVM-add-space-to-RHEL-Linux-on-KVM_Virtual_machine-howto

Part of migration project for a customer I'm working on is migration of a couple of KVM based Guest virtual machine servers. The old machines has a backup solution stratetegy using IBM's TSM and the new Machines should use the Cheaper solution adopted by the Customer company using the CommVault backup solution (an enterprise software thath is used for data backup and recovery not only to local Tape Library / Data blobs on central backup servers infra but also in Cloud infrastructure.

To install the CommVault software on the Redhat Linux-es, the official install documentation (prepared by the team who prepared the CommVault) infrastructure for the customer recommends to have a separate partition for the CommVault backups under /opt directory (/opt/commvault) and the partition should be as a minumum at least 10 Gigabytes of size.

Unfortunately on our new prepared KVM VM guest machines, it was forgotten to have the separate /opt of 10GB prepared in advanced. And we ended up with Virtual Machines that has a / (root directory) of 68GB size and a separate /var and /home LVM parititons. Thus to correct the issues it was required to find a way to add another separate LVM partition inside the KVM VirtualMachine.img (QCOW Image file).

This seemed to be an easy task at first as that might be possible with simple .img partition mount with losetup command kpartx and simple lvreduce command in some way such as

# mount /dev/loop0 /mnt/test/

# kpartx -a /dev/loop0
# kpartx -l /dev/loop0
# ls -al /dev/mapper/*

…

# lvreduce

etc. however unfortunately kpartx though not returning error did not provided the new /dev/mapper devices to be used with LVM tools and this approach seems to not be possible on RHEL 8.8 as the kpartx couldn't list.

A colleague of mine Mr. Paskalev suggested that we can perhaps try to mount the partition with default KVM tool to mount .img partitions which is guestmount but unfortunately
with a command like:

# guestmount -a /kvm/VM.img -i –rw /mnt/test/

But unfortunately this mounted the filesystem in fuse filesystem and the LVM /dev/mapper of the VM can't be seen so we decided to abondon this method.

After some pondering with Dimitar Paskalev and Dimitar Hristov, thanks to joint efforts we found the way to do it, below are the steps we followed to succeed in creating new LVM ext4 partition required.
One would wonder how many system

1. Check enough space is available on the HV machine

The VMs are held under /kvm so in this case:

[root@hypervisor-host ~]# df -h|grep -i /kvm
/dev/mapper/vg00-vmprivate 206G 27G 169G 14% /kvm

2. Shutdown the running VM and make sure it is stopped

[root@hypervisor-host ~]# virsh shutdown vm-host

[root@hypervisor-host ~]# virsh list –all
Id Name State
————————–
4 lpdkv01f running
5 vm-host shut off

3. Check current Space status of VM

[root@hypervisor-host ~]# qemu-img info /kvm/vm-host.img
image: /kvm/vm-host.img
file format: qcow2
virtual size: 100 GiB (107374182400 bytes)
disk size: 8.62 GiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: true
refcount bits: 16
corrupt: false
extended l2: false

4. Resize (extend VM) with whatever size you want

[root@hypervisor-host ~]# qemu-img resize /kvm/vm-host.img +10G
…

5. Start VM

[root@hypervisor-host ~]# virsh start vm-host
…

7. Check the LVM and block devices on HVs (not necessery but good for an overview)

[root@hypervisor-host ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 vg00 lvm2 a– 277.87g 19.87g

[root@hypervisor-host ~]# vgs
VG #PV #LV #SN Attr VSize VFree
vg00 1 11 0 wz–n- 277.87g 19.87g

[root@hypervisor-host ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 278.9G 0 disk
├─sda1 8:1 0 1G 0 part /boot
└─sda2 8:2 0 277.9G 0 part
├─vg00-root 253:0 0 15G 0 lvm /
├─vg00-swap 253:1 0 1G 0 lvm [SWAP]
├─vg00-var 253:2 0 5G 0 lvm /var
├─vg00-spool 253:3 0 2G 0 lvm /var/spool
├─vg00-audit 253:4 0 3G 0 lvm /var/log/audit
├─vg00-opt 253:5 0 2G 0 lvm /opt
├─vg00-home 253:6 0 5G 0 lvm /home
├─vg00-tmp 253:7 0 5G 0 lvm /tmp
├─vg00-log 253:8 0 5G 0 lvm /var/log
├─vg00-cache 253:9 0 5G 0 lvm /var/cache
└─vg00-vmprivate 253:10 0 210G 0 lvm /vmprivate

8 . Check logical volumes on Hypervisor host

[root@hypervisor-host ~]# lvdisplay
— Logical volume —
LV Path /dev/vg00/swap
LV Name swap
VG Name vg00
LV UUID 3tNa0n-HDVw-dLvl-EC06-c1Ex-9jlf-XAObKm
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:45 +0200
LV Status available
# open 2
LV Size 1.00 GiB
Current LE 256
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:1

— Logical volume —
LV Path /dev/vg00/var
LV Name var
VG Name vg00
LV UUID JBerim-fxVv-jU10-nDmd-figw-4jVA-8IYdxU
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:45 +0200
LV Status available
# open 1
LV Size 5.00 GiB
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:2

— Logical volume —
LV Path /dev/vg00/spool
LV Name spool
VG Name vg00
LV UUID nFlmp2-iXg1-tFxc-FKaI-o1dA-PO70-5Ve0M9
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:45 +0200
LV Status available
# open 1
LV Size 2.00 GiB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:3

— Logical volume —
LV Path /dev/vg00/audit
LV Name audit
VG Name vg00
LV UUID e6H2OC-vjKS-mPlp-JOmY-VqDZ-ITte-0M3npX
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:46 +0200
LV Status available
# open 1
LV Size 3.00 GiB
Current LE 768
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:4

— Logical volume —
LV Path /dev/vg00/opt
LV Name opt
VG Name vg00
LV UUID oqUR0e-MtT1-hwWd-MhhP-M2Y4-AbRo-Kx7yEG
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:46 +0200
LV Status available
# open 1
LV Size 2.00 GiB
Current LE 512
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:5

— Logical volume —
LV Path /dev/vg00/home
LV Name home
VG Name vg00
LV UUID ehdsH7-okS3-gPGk-H1Mb-AlI7-JOEt-DmuKnN
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:47 +0200
LV Status available
# open 1
LV Size 5.00 GiB
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:6

— Logical volume —
LV Path /dev/vg00/tmp
LV Name tmp
VG Name vg00
LV UUID brntSX-IZcm-RKz2-CP5C-Pp00-1fA6-WlA7lD
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:47 +0200
LV Status available
# open 1
LV Size 5.00 GiB
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:7

— Logical volume —
LV Path /dev/vg00/log
LV Name log
VG Name vg00
LV UUID ZerDyL-birP-Pwck-yvFj-yEpn-XKsn-sxpvWY
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:47 +0200
LV Status available
# open 1
LV Size 5.00 GiB
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:8

— Logical volume —
LV Path /dev/vg00/cache
LV Name cache
VG Name vg00
LV UUID bPPfzQ-s4fH-4kdT-LPyp-5N20-JQTB-Y2PrAG
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:48 +0200
LV Status available
# open 1
LV Size 5.00 GiB
Current LE 1280
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:9

— Logical volume —
LV Path /dev/vg00/root
LV Name root
VG Name vg00
LV UUID mZr3p3-52R3-JSr5-HgGh-oQX1-B8f5-cRmaIL
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-08-07 13:47:48 +0200
LV Status available
# open 1
LV Size 15.00 GiB
Current LE 3840
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:0

— Logical volume —
LV Path /dev/vg00/vmprivate
LV Name vmprivate
VG Name vg00
LV UUID LxNRWV-le3h-KIng-pUFD-hc7M-39Gm-jhF2Aj
LV Write Access read/write
LV Creation host, time hypervisor-host, 2023-09-18 11:54:19 +0200
LV Status available
# open 1
LV Size 210.00 GiB
Current LE 53760
Segments 1
Allocation inherit
Read ahead sectors auto
– currently set to 8192
Block device 253:10

9. Check Hypervisor existing partitions and space

[root@hypervisor-host ~]# fdisk -l
Disk /dev/sda: 278.9 GiB, 299439751168 bytes, 584843264 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0581e6e2

Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 2099199 2097152 1G 83 Linux
/dev/sda2 2099200 584843263 582744064 277.9G 8e Linux LVM

Disk /dev/mapper/vg00-root: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-swap: 1 GiB, 1073741824 bytes, 2097152 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-var: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-spool: 2 GiB, 2147483648 bytes, 4194304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-audit: 3 GiB, 3221225472 bytes, 6291456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-opt: 2 GiB, 2147483648 bytes, 4194304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-home: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-tmp: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-log: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-cache: 5 GiB, 5368709120 bytes, 10485760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/vg00-vmprivate: 210 GiB, 225485783040 bytes, 440401920 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

10. List block devices on VM

[root@vm-host ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 100G 0 disk
├─vda1 252:1 0 1G 0 part /boot
├─vda2 252:2 0 88G 0 part
│ ├─vg00-root 253:0 0 68G 0 lvm /
│ ├─vg00-home 253:2 0 10G 0 lvm /home
│ └─vg00-var 253:3 0 10G 0 lvm /var
├─vda3 252:3 0 1G 0 part [SWAP]
└─vda4 252:4 0 10G 0 part

11. Create new LVM partition with fdisk or cfdisk

If there is no cfdisk new resized space with qemu-img could be setup with a fdisk, though I personally always prefer to use cfdisk

[root@vm-host ~]# fdisk /dev/vda
# > p (print)
# > m (manfile)
# > n
# … follow on screen instructions to select start and end blocks
# > t (change partition type)
# > select and set to 8e
# > w (write changes)

[root@vm-host ~]# cfdisk /dev/vda

Setup new partition from Free space as [ primary ] partition and Choose to be of type LVM

12. List partitions to make sure new LVM partition is present

[root@vm-host ~]# fdisk -l
Disk /dev/vda: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe7b2d9fd

Device Boot Start End Sectors Size Id Type
/dev/vda1 * 2048 2099199 2097152 1G 83 Linux
/dev/vda2 2099200 186646527 184547328 88G 8e Linux LVM
/dev/vda3 186646528 188743679 2097152 1G 82 Linux swap / Solaris
/dev/vda4 188743680 209715199 20971520 10G 8e Linux LVM

The extra added 10 Giga is seen under /dev/vda4.
— Physical volume —
PV Name /dev/vda4
VG Name vg01
PV Size 10.00 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 2559
Free PE 0
Allocated PE 2559
PV UUID yvMX8a-sEka-NLA7-53Zj-fFdZ-Jd2K-r0Db1z

— Physical volume —
PV Name /dev/vda2
VG Name vg00
PV Size <88.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 22527
Free PE 0
Allocated PE 22527
PV UUID i4UpGr-h9Cd-iKBu-KqEI-15vK-CGc1-DwRPj8

[root@vm-host ~]#

13. List LVM Physical Volumes

[root@vm-host ~]# pvdisplay
— Physical volume —
PV Name /dev/vda2
VG Name vg00
PV Size <88.00 GiB / not usable 3.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 22527
Free PE 0
Allocated PE 22527
PV UUID i4UpGr-h9Cd-iKBu-KqEI-15vK-CGc1-DwRPj8

Notice the /dev/vda4 is not seen in pvdisplay (Physical Volume display command) because not created yet, so lets create it.

14. Initialize new Physical Volume to be available for use by LVM

[root@vm-host ~]# pvcreate /dev/vda4

15. Inform the OS for partition table changes

If partprobe is not available as command on the host, below obscure command should do the trick.

[root@vm-host ~]# echo "- – -" | tee /sys/class/scsi_host/host*/scan

However usually, better to use partprobe to inform the Operating System of partition table changes

[root@vm-host ~]# partprobe

16. Use lsblk again to see the new /dev/vda4 LVM is listed into "vda" root block device

[root@vm-host ~]#
[root@vm-host ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 100G 0 disk
├─vda1 252:1 0 1G 0 part /boot
├─vda2 252:2 0 88G 0 part
│ ├─vg00-root 253:0 0 68G 0 lvm /
│ ├─vg00-home 253:1 0 10G 0 lvm /home
│ └─vg00-var 253:2 0 10G 0 lvm /var
├─vda3 252:3 0 1G 0 part [SWAP]
└─vda4 252:4 0 10G 0 part
[root@vm-host ~]#

17. Create new Volume Group (VG) on /dev/vda4 block device

Before creating a new VG, list what kind of VG is on the machine to be sure the new created one will not be already present.

[root@vm-host ~]# vgdisplay
— Volume group —
VG Name vg00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size <88.00 GiB
PE Size 4.00 MiB
Total PE 22527
Alloc PE / Size 22527 / <88.00 GiB
Free PE / Size 0 / 0
VG UUID oyo1oY-saSm-0IKk-gZnf-Knwz-utO7-Aw8c60

vg00 is existing only, so we can use vg01 as a Volume Group name for the new volume group where the fresh 10GB LVM partition will lay

[root@vm-host ~]# vgcreate vg01 /dev/vda4
Volume group "vg01" successfully created

18. Create new Logical Volume (LV) and extend it to occupy the full space available on Volume Group vg01

[root@vm-host ~]# lvcreate -n commvault -l 100%FREE vg01
Logical volume "commvault" created.

An alternative way to create the same LV is by running:

# lvcreate -n commvault -L 10G vg01

19. Relist block devices with lsblk to make sure the new created Logical Volume commvault is really present and seen, in case of it missing re-run again partprobe cmd

[root@vm-host ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 1024M 0 rom
vda 252:0 0 100G 0 disk
├─vda1 252:1 0 1G 0 part /boot
├─vda2 252:2 0 88G 0 part
│ ├─vg00-root 253:0 0 68G 0 lvm /
│ ├─vg00-home 253:1 0 10G 0 lvm /home
│ └─vg00-var 253:2 0 10G 0 lvm /var
├─vda3 252:3 0 1G 0 part [SWAP]
└─vda4 252:4 0 10G 0 part
└─vg01-commvault 253:3 0 10G 0 lvm

As it is not mounted yet, the VG will be not seen in df free space but will be seen as a volume group with vgdispaly

[root@vm-host ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.8G 0 2.8G 0% /dev
tmpfs 2.8G 33M 2.8G 2% /dev/shm
tmpfs 2.8G 17M 2.8G 1% /run
tmpfs 2.8G 0 2.8G 0% /sys/fs/cgroup
/dev/mapper/vg00-root 67G 2.4G 61G 4% /
/dev/mapper/vg00-var 9.8G 1021M 8.3G 11% /var
/dev/mapper/vg00-home 9.8G 24K 9.3G 1% /home
/dev/vda1 974M 242M 665M 27% /boot
tmpfs 569M 0 569M 0% /run/user/0

[root@vm-host ~]# vgdisplay
— Volume group —
VG Name vg01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size <10.00 GiB
PE Size 4.00 MiB
Total PE 2559
Alloc PE / Size 2559 / <10.00 GiB
Free PE / Size 0 / 0
VG UUID nYP0tv-IbFw-fBVT-slBB-H1hF-jD0h-pE3V0S

— Volume group —
VG Name vg00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 4
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 3
Open LV 3
Max PV 0
Cur PV 1
Act PV 1
VG Size <88.00 GiB
PE Size 4.00 MiB
Total PE 22527
Alloc PE / Size 22527 / <88.00 GiB
Free PE / Size 0 / 0
VG UUID oyo1oY-saSm-0IKk-gZnf-Snwz-utO7-Aw8c60

20. Create new ext4 filesystem on the just created vg01-commvault

[root@vm-host ~]# mkfs.ext4 /dev/mapper/vg01-commvault

[root@vm-host ~]# mkfs.ext4 /dev/mapper/vg01-commvault
mke2fs 1.45.6 (20-Mar-2020)
Discarding device blocks: done
Creating filesystem with 2620416 4k blocks and 655360 inodes
Filesystem UUID: 1491d8b1-2497-40fe-bc40-5faa6a2b2644
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

21. Mount vg01-commvault into /opt directory

[root@vm-host ~]# mkdir -p /opt/

[root@vm-host ~]# mount /dev/mapper/vg01-commvault /opt/

22. Check mount is present on VM guest OS

[root@vm-host ~]# mount|grep -i /opt
/dev/mapper/vg01-commvault on /opt type ext4 (rw,relatime)
[root@vm-host ~]#

[root@vm-host ~]# df -h|grep -i opt
/dev/mapper/vg01-commvault 9.8G 24K 9.3G 1% /opt
[root@vm-host ~]#

23. Add vg01-commvault to be auto mounted via /etc/fstab on next Virtual Machine reboot

[root@vm-host ~]# echo '/dev/mapper/vg01-commvault /opt ext4 defaults 1 2' >> /etc/fstab

[root@vm-host ~]# rpm -ivh commvault-fs.Instance001-11.0.0-80.240.0.3589820.240.4083067.el8.x86_64.rpm
…

[root@vm-host ~]# systemctl status commvault
● commvault.Instance001.service – commvault Service
Loaded: loaded (/etc/systemd/system/commvault.Instance001.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2023-11-10 15:13:59 CET; 27s ago
Process: 9972 ExecStart=/opt/commvault/Base/Galaxy start direct -focus Instance001 (code=exited, status=0/SUCCESS)
Tasks: 54
Memory: 155.5M
CGroup: /system.slice/commvault.Instance001.service
├─10132 /opt/commvault/Base/cvlaunchd
├─10133 /opt/commvault/Base/cvd
├─10135 /opt/commvault/Base/cvfwd
└─10137 /opt/commvault/Base/ClMgrS

Nov 10 15:13:57 vm-host.ffm.de.int.atosorigin.com systemd[1]: Starting commvault Service…
Nov 10 15:13:58 vm-host.ffm.de.int.atosorigin.com Galaxy[9972]: Cleaning up /opt/commvault/Base/Temp …
Nov 10 15:13:58 vm-host.ffm.de.int.atosorigin.com Galaxy[9972]: Starting Commvault services for Instance001 …
Nov 10 15:13:59 vm-host.ffm.de.int.atosorigin.com Galaxy[9972]: [22B blob data]
Nov 10 15:13:59 vm-host.ffm.de.int.atosorigin.com systemd[1]: Started commvault Service.
[root@vm-host ~]#

24. Install Commvault backup client RPM in new mounted LVM under /opt

[root@vm-host ~]# rpm -ivh commvault.rpm
…

Tags: command, Creating, Device Boot Start End Sectors Size Id Type, img, Initialize, kvm, LVM, mapper, mount dev, optimal, partition, Redhat Linux-es, sector size, separate, solution, var, virtual machines
Posted in Linux, Linux and FreeBSD Desktop, System Administration | No Comments »

How to install and configure AIDE ( Advanced Intrusion Detection Environment ) on Debian GNU / Linux 11 to monitor files for changes

Thursday, March 9th, 2023

How to install and configure AIDE ( Advanced Intrusion Detection Environment ) on Debian GNU / Linux 11 to monitor files for changes

Having a intrusion detection system is essential to keeping a server security to good level and being compliant with PCI (Payment Card Industry) DSS Standards. It is a great thing for the sake to protect oneself from hackers assaults.

There is plenty of Intrusion Detection systems available all around since many years, in the past one of main ones for Linux as older system administrators should remember was Tripwire – integrity tool for monitoring and alerting on specific file change(s) on a range of systems.

Tripwire is still used today but many today prefer to use AIDE that is a free software replacement for Tripwire under GPL (General Public License), that is starting to become like a "standard" for many Unix-like systems as an inexpensive baseline control and rootkit detection system.

In this article I'll explain shortly how to Install / Configure and Use AIDE to monitor, changes with files on the system.

But before proceeding it is worthy to mention on some of the alternatives companies and businesses choose to as an IDS (Intrusion Detection Systems), that is useful to give a brief idea of the sysadmins that has to deal with Security, on what is some of the main Intrusion Detection Systems adopted on UNIX OSes today:

Samhain

An integrity checker and host intrusion detection system that can be used on single hosts as well as large, UNIX-based networks. It supports central monitoring as well as powerful (and new) stealth features to run undetected in memory, using steganography. Samhain is an open-source multiplatform application for POSIX systems (Unix, Linux, Cygwin/Windows).
OSSEC
OSSEC uses a centralized, cross-platform architecture allowing multiple systems to be monitored and managed.
Snort
IDS which has the capabilities to prevent attacks. By taking a particular action based on traffic patterns, it can become an intrusion prevention system (IPS). – written in Pure C.
Zeek (Bro)
Zeek helps to perform security monitoring by looking into the network's activity. It can find suspicious data streams. Based on the data, it alert, react, and integrate with other tools – written in C++.
Maltrail (Maltrail monitors for traffic on the network that might indicate system compromise or other bad behavior. It is great for intrusion detection and monitoring. – written in Python).

1. Install aide deb package

# apt -y install aide
…

root@haproxy2:~# aide -v
Aide 0.17.3

Compiled with the following options:

WITH_MMAP
WITH_PCRE
WITH_POSIX_ACL
WITH_SELINUX
WITH_XATTR
WITH_CAPABILITIES
WITH_E2FSATTRS
WITH_ZLIB
WITH_MHASH
WITH_AUDIT

Default config values:
config file: <none>
database_in: <none>
database_out: <none>

Available hashsum groups:
md5: yes
sha1: yes
sha256: yes
sha512: yes
rmd160: yes
tiger: yes
crc32: yes
crc32b: yes
haval: yes
whirlpool: yes
gost: yes
stribog256: no
stribog512: no

Default compound groups:
R: l+p+u+g+s+c+m+i+n+md5+acl+selinux+xattrs+ftype+e2fsattrs+caps
L: l+p+u+g+i+n+acl+selinux+xattrs+ftype+e2fsattrs+caps
>: l+p+u+g+i+n+acl+S+selinux+xattrs+ftype+e2fsattrs+caps
H: md5+sha1+rmd160+tiger+crc32+haval+gost+crc32b+sha256+sha512+whirlpool
X: acl+selinux+xattrs+e2fsattrs+caps

2. Prepare AIDE configuration and geenrate (initialize) database

Either you can use the default AIDE configuration which already has a preset rules for various files and directories to be monitored,
or you might add up additional ones.

For details on configuration of aide.conf accepted options "man aide.conf"

The rules and other configurations resides lays under /etc/aide/ directory

The AIDE database is located under /var/lib/aide

root@server:~# ls -al /var/lib/aide/
общо 33008
drwxr-xr-x 2 root root 4096 9 мар 12:38 ./
drwxr-xr-x 27 root root 4096 9 мар 12:01 ../
-rw——- 1 root root 16895467 9 мар 16:03 aide.db
-rw——- 1 root root 16895467 9 мар 18:49 aide.db.new

Also, details about major setting rules config regarding how AIDE will run via cronjob as with most debian services are into /etc/default/aide

Default aide.conf config is in /etc/aide/aide.conf if you need custom stuff to do with it simply edit it.

Here is an Example:
Lets say you want to omit some directory to not be monitored by aide, which would otherwise do, i.e.
omit /var/log/* from monitoring

# At the end of file /etc/aide/aide.conf

add:

!/var/log
!/home/
!/var/lib
!/proc

Initialize the aide database first time

Run aideinit command, aideinit will create a new baseline database – /var/lib/aide/aide.db.new (a baseline)
Note that, /var/lib/aide/aide.db is the old database that aide uses to check against for any changes of files / directories on the configured monitored filesystem objects.

root@server:~# aideinit
Running aide –init…

debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
Start timestamp: 2023-03-09 12:06:16 +0200 (AIDE 0.17.3)
AIDE initialized database at /var/lib/aide/aide.db.new

Number of entries: 66971

—————————————————
The attributes of the (uncompressed) database(s):
—————————————————

/var/lib/aide/aide.db.new
SHA256 : nVrYljiBFM/KaKCTjbaJtR2w6N8vc8qN
DPObbo2UMVo=
SHA512 : S1ZNB0DCqb4UTmuqaalTgiQ3UAltTOzO
YNfEQJldp32q5ahplBo4/65uwgtGusMy
rJC8nvxvYmh+mq+16kfrKA==
RMD160 : xaUnfW1+/DJV/6FEm/nn1k1UKOU=
TIGER : nGYEbX281tsQ6T21VPx1Hr/FwBdwF4cK
CRC32 : fzf7cg==
HAVAL : yYQw/87KUmRiRLSu5JcEIvBUVfsW/G9H
tVvs6WqL/0I=
WHIRLPOOL : 6b5y42axPjpUxWFipUs1PtbgP2q0KJWK
FwFvAGxHXjZeCBPEYZCNkj8mt8MkXBTJ
g83ZELK9GQBPLea7UF3tng==
GOST : sHAzx7hkr5H3q8TCSGCKjndEiZgcvCEL
E45qcRb25tM=

End timestamp: 2023-03-09 12:38:30 +0200 (run time: 32m 14s)

Be patient now, go grab a coffee / tea or snack as the command might take up to few minutes for the aide to walk through the whole monitored filesystems and built its database.

root@server:~# echo cp /var/lib/aide/aide.db{.new,}
cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

root@server:~# cp /var/lib/aide/aide.db{.new,}

root@server:~# aide –check –config /etc/aide/aide.conf

Start timestamp: 2023-03-09 13:01:32 +0200 (AIDE 0.17.3)
AIDE found differences between database and filesystem!!

Summary:
Total number of entries: 66972
Added entries: 1
Removed entries: 0
Changed entries: 7

—————————————————
Added entries:
—————————————————

f+++++++++++++++++: /var/lib/aide/aide.db

—————————————————
Changed entries:
—————————————————

d =…. mc.. .. . : /etc/aide
d =…. mc.. .. . : /root
f <…. mci.H.. . : /root/.viminfo
f =…. mc..H.. . : /var/lib/fail2ban/fail2ban.sqlite3
d =…. mc.. .. . : /var/lib/vnstat
f =…. mc..H.. . : /var/lib/vnstat/vnstat.db
f >b… mc..H.. . : /var/log/sysstat/sa09

—————————————————
Detailed information about changes:
—————————————————

Directory: /etc/aide
Mtime : 2023-03-09 12:04:03 +0200 | 2023-03-09 12:51:11 +0200
Ctime : 2023-03-09 12:04:03 +0200 | 2023-03-09 12:51:11 +0200

Directory: /root
Mtime : 2023-03-09 12:06:13 +0200 | 2023-03-09 12:51:11 +0200
Ctime : 2023-03-09 12:06:13 +0200 | 2023-03-09 12:51:11 +0200

File: /root/.viminfo
Size : 18688 | 17764
Mtime : 2023-03-09 12:06:13 +0200 | 2023-03-09 12:51:11 +0200
Ctime : 2023-03-09 12:06:13 +0200 | 2023-03-09 12:51:11 +0200
Inode : 133828 | 133827
SHA256 : aV54gi33aA/z/FuBj2ZioU2cTa9H16TT | dnFdLVQ/kx3UlTah09IgEMrJ/aYgczHe
TzkLSxBDSB4= | DdxDAmPOSAM=

…

3. Test aide detects file changes

Create a new file and append some text and rerun the aide check

root@server:~# touch /root/test.txt
root@server:~# echo aaa > /root/test.txt
root@server:~# aide –check –config /etc/aide/aide.conf

Start timestamp: 2023-03-09 13:07:21 +0200 (AIDE 0.17.3)
AIDE found differences between database and filesystem!!

Summary:
Total number of entries: 66973
Added entries: 2
Removed entries: 0
Changed entries: 7

—————————————————
Added entries:
—————————————————

f+++++++++++++++++: /root/test.txt
f+++++++++++++++++: /var/lib/aide/aide.db

—————————————————
Changed entries:
—————————————————

d =…. mc.. .. . : /etc/aide
d =…. mc.. .. . : /root
f <…. mci.H.. . : /root/.viminfo
f =…. mc..H.. . : /var/lib/fail2ban/fail2ban.sqlite3
d =…. mc.. .. . : /var/lib/vnstat
f =…. mc..H.. . : /var/lib/vnstat/vnstat.db
f >b… mc..H.. . : /var/log/sysstat/sa09

….

The same command can be shortened for the lazy typist:

root@server:~# aide -c /etc/aide/aide.conf -C

The command will basically try to check the deviation between the AIDE database and the filesystem.

4. Limiting AIDES Integrity Checks to Specific Files / Directories

In order to limit the integrity checks to a specific entries for example /etc, pass the –limit REGEX option to AIDE check command where REGEX is the entry to check.

For example, check and update the database entries matching /etc, you would run aide command as shown below;

root@server:~# aide -c /etc/aide/aide.conf –limit /etc –check

AIDE found differences between database and filesystem!!
Limit: /etc

Summary:
Total number of entries: 66791
Added entries: 0
Removed entries: 0
Changed entries: 2

—————————————————
Changed entries:
—————————————————

d =…. mc.. .. . : /etc/aide
d =…. mc.. .. . : /etc/default

—————————————————
Detailed information about changes:
—————————————————

Directory: /etc/aide
Mtime : 2023-03-09 15:59:53 +0200 | 2023-03-09 16:43:03 +0200
Ctime : 2023-03-09 15:59:53 +0200 | 2023-03-09 16:43:03 +0200

Directory: /etc/default
Mtime : 2023-03-09 12:06:13 +0200 | 2023-03-09 18:42:12 +0200
Ctime : 2023-03-09 12:06:13 +0200 | 2023-03-09 18:42:12 +0200

—————————————————
The attributes of the (uncompressed) database(s):
—————————————————

/var/lib/aide/aide.db
SHA256 : sjCxyIkr0nC/gTkNmn7DNqAQWttreDF6
vSUV4jBoFY4=
SHA512 : vNMpb54qxrbOk6S1Z+m9r0UwGvRarkWY
0m50TfMvGElfZWR1I3SSaeTdORAZ4rQe
17Oapo5+Sc0E2E+STO93tA==
RMD160 : anhm5E6UlKmPYYJ4WYnWXk/LT3A=
TIGER : 5e1wycoF35/ABrRf7FNypZ45169VTuV4
CRC32 : EAJlFg==
HAVAL : R5imONWRYgNGEfhBTc096K+ABnMFkMmh
Hsqe9xt20NU=
WHIRLPOOL : c6zySLliXNgnOA2DkHUdLTCG2d/T18gE
4rdAuKaC+s7gqAGyA4p2bnDHhdd0v06I
xEGY7YXCOXiwx8BM8xHAvQ==
GOST : F5zO2Ovtvf+f7Lw0Ef++ign1znZAQMHM
AApQOiB9CqA=

End timestamp: 2023-03-09 20:02:18 +0200 (run time: 1m 32s)

5. Add the modified /root/test.txt to AIDE list of known modified files database

root@server:~# aide –update –config /etc/aide/aide.
ERROR: cannot open config file '/etc/aide/aide.': No such file or directory

root@server:~# aide –update –config /etc/aide/aide.conf

Start timestamp: 2023-03-09 18:45:17 +0200 (AIDE 0.17.3)
AIDE found differences between database and filesystem!!
New AIDE database written to /var/lib/aide/aide.db.new

Summary:
Total number of entries: 66791
Added entries: 0
Removed entries: 0
Changed entries: 8

—————————————————
Changed entries:
—————————————————

d =…. mc.. .. . : /etc/aide
d =…. mc.. .. . : /etc/default
d =…. mc.. .. . : /root
f >…. mci.H.. . : /root/.viminfo
f >…. mci.H.. . : /root/test.txt
f =…. mc..H.. . : /var/lib/fail2ban/fail2ban.sqlite3
d =…. mc.. .. . : /var/lib/vnstat
f =…. mc..H.. . : /var/lib/vnstat/vnstat.db

—————————————————
Detailed information about changes:
—————————————————

Directory: /etc/aide
Mtime : 2023-03-09 15:59:53 +0200 | 2023-03-09 16:43:03 +0200
Ctime : 2023-03-09 15:59:53 +0200 | 2023-03-09 16:43:03 +0200

Directory: /etc/default
Mtime : 2023-03-09 12:06:13 +0200 | 2023-03-09 18:42:12 +0200
Ctime : 2023-03-09 12:06:13 +0200 | 2023-03-09 18:42:12 +0200

Directory: /root
Mtime : 2023-03-09 15:59:53 +0200 | 2023-03-09 18:44:34 +0200
Ctime : 2023-03-09 15:59:53 +0200 | 2023-03-09 18:44:34 +0200

File: /root/.viminfo
Size : 16706 | 16933
Mtime : 2023-03-09 15:59:53 +0200 | 2023-03-09 18:44:34 +0200
Ctime : 2023-03-09 15:59:53 +0200 | 2023-03-09 18:44:34 +0200
Inode : 136749 | 133828
SHA256 : KMHGoMVJo10BtafVrWIOLt3Ht9gK8bc+ | rrp8S3VftzZzvjBP1JC+PBpODv9wPKGw
9uHh/z7iJWA= | TA+hyhTiY+U=
SHA512 : ieDHy7ObSTfYm5d8DtYcHKxHya13CS65 | PDAJjyZ39uU3kKFo2lHBduTqxMDq4i01
ObMYIRAre6IgvLslEs0ZodQFyrczMyRt | 1Kvm/h6xzFhHtFgjidtcemG8wDcjtfNF
+d6SrW0gn3skKn2B7G09eQ== | Z7LO230fgGeO7UepqtxZjQ==
RMD160 : nUgg/G4zsVGKzVmmrqltuYUDvtg= | jj61KAFONK92mj+u66RDJmxFhmI=
TIGER : 3vPSOrla5k+k2br1E2ES4eNiSZ2novFX | mn4kNCzd8SQr2ID2VSe4f4l0ta7pO/xo
CRC32 : NDnMgw== | AyzVUQ==
HAVAL : Q9/KozxRiPbLEkaIfnBUZdEWftaF52Mw | 6jADKV6jg7ZVr/A/oMhR4NXc8TO1AOGW
7tiR7DXhl0o= | NrYe+j6UcO0=
WHIRLPOOL : vB/ZMCul4hN0aYd39gBu+HmZT/peRUI8 | mg6c1lYYVNZcy4mVzGojwraim8e3X2/R
KDkaslNb8+YleoFWx0mbhAbkGurc0+jh | urVvEmbsgTuUCJOuf9+OrEACiF0fbe/x
YPBviZIKcxUbTc2nGthTWw== | t+BXnSQWk08OL9EI6gMGqA==
GOST : owVGTgU9BH3b0If569wQygw3FAbZIZde | ffx29GV2jaCB7XzuNjdiRzziIiZYnbi3
eAfQfzlRPGY= | Ar7jyNMUutk=

File: /root/test.txt
Size : 4 | 8
Mtime : 2023-03-09 13:07:12 +0200 | 2023-03-09 18:44:34 +0200
Ctime : 2023-03-09 13:07:12 +0200 | 2023-03-09 18:44:34 +0200
Inode : 133828 | 136751
SHA256 : F+aC8GC1+OR+oExcSFWQiwpa1hICImD+ | jUIZMGfiMdAlWFHu8mmmlml4qAGNQNL5
UOEeywzAq3Y= | 6NhzJ1sYFZE=
SHA512 : d+UmFKFBzvGadt5hk+nIRbjP//7PSXNl | ixn20lcEMDEtsJo3hO90Ea/wHWLCHcrz
Pl16XRIUUPq2FCiQ4PeUcVciukJX7ijL | seBWunbBysY0z3BWcfgnN2vH05WfRfvA
D045ZvGOEcnmL6a6vwp0jw== | QiNtQS1tStuEdB3Voq54zQ==
RMD160 : I6waxKN3rMx4WTz4VCUQXoNoxUg= | urTh1j1t3UHchnJGnBG4lUZnjI4=
TIGER : cwUYgfKHcJnWXcA0pr/OKuxuoxh+b9lA | prstKqCfMXL39aVGFPA0kX4Q9x7a+hUn
CRC32 : UD78Dw== | zoYiEA==
HAVAL : bdbKR9LvPgsYClViKiHx48fFixfIL/jA | ZdpdeMhw4MvKBgWsM4EeyUgerO86Rt82
F3tjdc2Gm8Y= | W94fJFRWbrM=
WHIRLPOOL : OLP0Y4oKcqW2yEvme8z419N1KE4TB9GJ | Xk8Ujo3IU2SzSqbJFegq7p1ockmrnxJF
biHn/9XgrBz4fQiDJ8eHpx+0exA9hXmY | R3Rfstd1jWSwLFNTEwfbRRw+TARtRK50
EbbakMJJdzLt1ipKWiV9gg== | iWJeHLsD5dZ+CzV0tf4sUg==
GOST : ystISzoeH/ZznYrrXmxe4rwmybWMpGuE | GhMWNxg7Is0svJ+5LP+DVWbgt+CDQO+3
0PzRnVEqnR8= | 08dwBuVAwB8=

File: /var/lib/fail2ban/fail2ban.sqlite3
Mtime : 2023-03-09 15:55:01 +0200 | 2023-03-09 18:45:01 +0200
Ctime : 2023-03-09 15:55:01 +0200 | 2023-03-09 18:45:01 +0200
SHA256 : lLilXNleqSgHIP1y4o7c+oG5XyUPGzgi | NCJJ2H6xgCw/NYys1LMA7hOWwoOoxI8Y
RHYH+zvlAL4= | 4SJygfqEioE=
SHA512 : iQj2pNT4NES4fBcujzdlEEGZhDnkhKgc | ClQZ5HMOSayUNb//++eZc813fiMJcXnj
QDlGFSAn6vi+RXesFCjCABT7/00eEm5/ | vTGs/2tANojoe6cqpsT/LaJ3QZXpmrfh
ILcaqlQtBSLJgHjMQehzdg== | syVak1I4n9yg8cDKEkZUvw==
RMD160 : Xg4YU8YI935L+DLvkRsDanS4DGo= | SYrQ27n+/1fvIZ7v+Sar/wQHulI=
TIGER : 2WhhPq9kuyeNJkOicDTDeOeJB8HR8zZe | o1LDZtRclri2KfZBe5J3D4YhM05UaP4E
CRC32 : NQmi4A== | tzIsqg==
HAVAL : t1ET+84+8WgfwqlLy4R1Qk9qGZQRUbJI | MwVnjtM3dad/RuN2BfgsySX2DpfYq4qi
z2J0ROGduXc= | H1pq6RYsA6o=
WHIRLPOOL : xKSn71gFIVhk5rWJIBaYQASl0V+pGn+3 | m5LEXfhBbhWFg/d8CFJhklOurmRSkDSG
N85R0tiCKsTZ2+LRkxDrzcVQdss2k8+z | LC/vICnbEWzLwrCuMwBi1/e5wDNIY8gK
oqExhoXtPsMaREjpCugd3Q== | mvGn40x+G4cCYNZ6lGT9Zg==
GOST : WptpUlfooIlUjzDHU8XGuOU2waRud5SR | i6K4COXU0nyZ1mL3ZBuGUPz/ZXTj8KKQ
E/tnoBqk+q0= | L6VNyS8/X2Y=

Directory: /var/lib/vnstat
Mtime : 2023-03-09 16:00:00 +0200 | 2023-03-09 18:45:01 +0200
Ctime : 2023-03-09 16:00:00 +0200 | 2023-03-09 18:45:01 +0200

File: /var/lib/vnstat/vnstat.db
Mtime : 2023-03-09 16:00:00 +0200 | 2023-03-09 18:45:00 +0200
Ctime : 2023-03-09 16:00:00 +0200 | 2023-03-09 18:45:00 +0200
SHA256 : X/lnJuuSo4jX4HRzxMBodnKHAjQFvugi | oqtY3HTNds/qDNFCRAEsfN5SuO0U5LRg
2sh2c0u69x8= | otc5z1y+eGY=
SHA512 : U/g8O6G8cuhsqCUCbrElxgiy+naJKPkI | y+sw4LX8mlDWkRJMX38TsYSo1DQzxPOS
hG7vdH9rBINjakL87UWajT0s6WSy0pvt | 068otnzw2FSSlM5X5j5EtyJiY6Hd5P+A
ALaTcDFKHBAmmFrl8df2nQ== | jFiWStMbx+dQidXYZ4XFAw==
RMD160 : F6YEjIIQu2J3ru7IaTvSemA9e34= | bmVSaRKN2qU7qpEWkzfXFoH4ZK4=
TIGER : UEwLoeR6Qlf2oOI58pUCEDaWk0pHDkcY | 0Qb4nUqe3cKh/g5CQUnOXGfjZwJHjeWa
CRC32 : Bv3/6A== | jvW6mg==
HAVAL : VD7tjHb8o8KTUo5xUH7eJEmTWgB9zjft | rumfiWJvy/sTK/09uj7XlmV3f7vj6KBM
kOkzKxFWqqU= | qeOuKvu0Zjc=
WHIRLPOOL : wR0qt8u4N8aQn8VQ+bmfrxB7CyCWVwHi | FVWDRE3uY6qHxLlJQLU9i9QggLW+neMj
ADHpMTUxBEKOpOBlHTWXIk13qYZiD+o/ | Wt+Dj9Rz92BG9EomgLUgUkxfiVFO8cMq
XtzTB4rMbxS4Z5PAdC/07A== | WaR/KKq3Z7R8f/50tc9GMQ==
GOST : l3ibqMkHMSPpQ+9ok51/xBthET9+JQMd | qn0GyyCg67KRGP13At52tnviZfZDgyAm
OZtiFGYXmgU= | c82NXSzeyV0=

—————————————————
The attributes of the (uncompressed) database(s):
—————————————————

/var/lib/aide/aide.db
SHA256 : sjCxyIkr0nC/gTkNmn7DNqAQWttreDF6
vSUV4jBoFY4=
SHA512 : vNMpb54qxrbOk6S1Z+m9r0UwGvRarkWY
0m50TfMvGElfZWR1I3SSaeTdORAZ4rQe
17Oapo5+Sc0E2E+STO93tA==
RMD160 : anhm5E6UlKmPYYJ4WYnWXk/LT3A=
TIGER : 5e1wycoF35/ABrRf7FNypZ45169VTuV4
CRC32 : EAJlFg==
HAVAL : R5imONWRYgNGEfhBTc096K+ABnMFkMmh
Hsqe9xt20NU=
WHIRLPOOL : c6zySLliXNgnOA2DkHUdLTCG2d/T18gE
4rdAuKaC+s7gqAGyA4p2bnDHhdd0v06I
xEGY7YXCOXiwx8BM8xHAvQ==
GOST : F5zO2Ovtvf+f7Lw0Ef++ign1znZAQMHM
AApQOiB9CqA=

/var/lib/aide/aide.db.new
SHA256 : QRwubXnz8md/08n28Ek6DOsSQKGkLvuc
gSZRsw6gRw8=
SHA512 : 238RmI1PHhd9pXhzcHqM4+VjNzR0es+3
6eiGNrXHAdDTz7GlAQQ4WfKeQJH9LdyT
1r5ho/oXRgzfa2BfhKvTHg==
RMD160 : GJWuX/nIPY05gz62YXxk4tWiH5I=
TIGER : l0aOjXlM4/HjyN9bhgBOvvCYeqoQyjpw
CRC32 : KFz6GA==
HAVAL : a//4jwVxF22URf2BRNA612WOOvOrScy7
OmI44KrNbBM=
WHIRLPOOL : MBf+NeXElUvscJ2khIuAp+NDu1dm4h1f
5tBQ0XrQ6dQPNA2HZfOShCBOPzEl/zrl
+Px3QFV4FqD0jggr5sHK2g==
GOST : EQnPh6jQLVUqaAK9B4/U4V89tanTI55N
K7XqZR9eMG4=

End timestamp: 2023-03-09 18:49:51 +0200 (run time: 4m 34s)

6. Substitute old aide database with the new that includes the modified files

As you see AIDE detected the changes in /root/test.txt

To apply the changes be known by AIDE for next time (e.g. this file was authorized and supposed to be written there) simply move the new generated database
to current aide database.

# copy generated DB to master DB
root@dlp:~# cp -p /var/lib/aide/aide.db.new /var/lib/aide/aide.db

7. Check once again to make sure recently modified files are no longer seen as changed by AIDE

Recheck again the database to make sure the files you wanted to omit are no longer mentioned as changed

root@server:~# aide –check –config /etc/aide/aide.conf
Start timestamp: 2023-03-09 16:23:05 +0200 (AIDE 0.17.3)
AIDE found differences between database and filesystem!!

Summary:
Total number of entries: 66791
Added entries: 0
Removed entries: 0
Changed entries: 3

—————————————————
Changed entries:
—————————————————

f =…. mc..H.. . : /var/lib/fail2ban/fail2ban.sqlite3
d =…. mc.. .. . : /var/lib/vnstat
f =…. mc..H.. . : /var/lib/vnstat/vnstat.db

—————————————————
Detailed information about changes:
—————————————————

File: /var/lib/fail2ban/fail2ban.sqlite3
Mtime : 2023-03-09 15:55:01 +0200 | 2023-03-09 16:25:02 +0200
Ctime : 2023-03-09 15:55:01 +0200 | 2023-03-09 16:25:02 +0200
SHA256 : lLilXNleqSgHIP1y4o7c+oG5XyUPGzgi | MnWXC2rBMf7DNJ91kXtHXpM2c2xxF60X
RHYH+zvlAL4= | DfLUQLHiSiY=
SHA512 : iQj2pNT4NES4fBcujzdlEEGZhDnkhKgc | gxHVBxhGTKi0TjRE8/sn6/gtWsRw7Mfy
QDlGFSAn6vi+RXesFCjCABT7/00eEm5/ | /wCfPlDK0dkRZEbr8IE2BNUhBgwwocCq
ILcaqlQtBSLJgHjMQehzdg== | zuazTy4N4x6X8bwOzRmY0w==
RMD160 : Xg4YU8YI935L+DLvkRsDanS4DGo= | +ksl9kjDoSU9aL4tR7FFFOK3mqw=
TIGER : 2WhhPq9kuyeNJkOicDTDeOeJB8HR8zZe | 9cvXZNbU+cp5dA5PLiX6sGncXd1Ff5QO
CRC32 : NQmi4A== | y6Oixg==
HAVAL : t1ET+84+8WgfwqlLy4R1Qk9qGZQRUbJI | aPnCrHfmZAUm7QjROGEl6rd3776wO+Ep
z2J0ROGduXc= | s/TQn7tH1tY=
WHIRLPOOL : xKSn71gFIVhk5rWJIBaYQASl0V+pGn+3 | 9Hu6NBhz+puja7uandb21Nt6cEW6zEpm
N85R0tiCKsTZ2+LRkxDrzcVQdss2k8+z | bTsq4xYA09ekhDHMQJHj2WpKpzZbA+t0
oqExhoXtPsMaREjpCugd3Q== | cttMDX8J8M/UadqfL8KZkQ==
GOST : WptpUlfooIlUjzDHU8XGuOU2waRud5SR | WUQfAMtye4wADUepBvblvgO+vBodS0Ej
E/tnoBqk+q0= | cIbXy4vpPYc=

Directory: /var/lib/vnstat
Mtime : 2023-03-09 16:00:00 +0200 | 2023-03-09 16:25:01 +0200
Ctime : 2023-03-09 16:00:00 +0200 | 2023-03-09 16:25:01 +0200

File: /var/lib/vnstat/vnstat.db
Mtime : 2023-03-09 16:00:00 +0200 | 2023-03-09 16:25:01 +0200
Ctime : 2023-03-09 16:00:00 +0200 | 2023-03-09 16:25:01 +0200
SHA256 : X/lnJuuSo4jX4HRzxMBodnKHAjQFvugi | N1lzhV3+tkDBud3AVlmIpDkU1c3Rqhnt
2sh2c0u69x8= | YqE8naDicoM=
SHA512 : U/g8O6G8cuhsqCUCbrElxgiy+naJKPkI | +8B9HvHhOp1C/XdlOORjyd3J2RtTbRBF
hG7vdH9rBINjakL87UWajT0s6WSy0pvt | b0Moo2Gj+cIxaMCu5wOkgreMp6FloqJR
ALaTcDFKHBAmmFrl8df2nQ== | UH4cNES/bAWtonmbj4W7Vw==
RMD160 : F6YEjIIQu2J3ru7IaTvSemA9e34= | 8M6TIOHt0NWgR5Mo47DxU28cp+4=
TIGER : UEwLoeR6Qlf2oOI58pUCEDaWk0pHDkcY | Du9Ue0JA2URO2tiij31B/+663OaWKefR
CRC32 : Bv3/6A== | v0Ai4w==
HAVAL : VD7tjHb8o8KTUo5xUH7eJEmTWgB9zjft | XA+vRnMNdVGFrO+IZtEA0icunWqBGaCf
kOkzKxFWqqU= | leR27LN4ejc=
WHIRLPOOL : wR0qt8u4N8aQn8VQ+bmfrxB7CyCWVwHi | HG31dNEEcak2zZGR24W7FDJx8mh24MaJ
ADHpMTUxBEKOpOBlHTWXIk13qYZiD+o/ | BQNhqkuS6R/bmlhx+P+eQ/JimwPAPOaM
XtzTB4rMbxS4Z5PAdC/07A== | xWG7cMETIXdT9sUOUal8Sw==
GOST : l3ibqMkHMSPpQ+9ok51/xBthET9+JQMd | y6Ek/TyAMGV5egkfCu92Y4qqk1Xge8c0
OZtiFGYXmgU= | 3ONXRveOlr0=

—————————————————
The attributes of the (uncompressed) database(s):
—————————————————

/var/lib/aide/aide.db
SHA256 : sjCxyIkr0nC/gTkNmn7DNqAQWttreDF6
vSUV4jBoFY4=
SHA512 : vNMpb54qxrbOk6S1Z+m9r0UwGvRarkWY
0m50TfMvGElfZWR1I3SSaeTdORAZ4rQe
17Oapo5+Sc0E2E+STO93tA==
RMD160 : anhm5E6UlKmPYYJ4WYnWXk/LT3A=
TIGER : 5e1wycoF35/ABrRf7FNypZ45169VTuV4
CRC32 : EAJlFg==
HAVAL : R5imONWRYgNGEfhBTc096K+ABnMFkMmh
Hsqe9xt20NU=
WHIRLPOOL : c6zySLliXNgnOA2DkHUdLTCG2d/T18gE
4rdAuKaC+s7gqAGyA4p2bnDHhdd0v06I
xEGY7YXCOXiwx8BM8xHAvQ==
GOST : F5zO2Ovtvf+f7Lw0Ef++ign1znZAQMHM
AApQOiB9CqA=

End timestamp: 2023-03-09 16:27:33 +0200 (run time: 4m 28s)

As you can see there are no new added entries for /root/test.txt and some other changed records for vnstat service as well as fail2ban ones, so the Intrusion detection system works just as we expected it.

8. Configure Email AIDE changed files alerting Email recipient address

From here on aide package has set its own cron job which is automatically doing the check operation every day and any new file modifications will be captured and alerts sent to local root@localhost mailbox account, so you can check it out later with mail command.

If you want to sent the Email alert for any files modifications occured to another email, assuming that you have a locally running SMTP server with a mail relay to send to external mails, you can do it via /etc/default/aide via:

MAILTO=root

For example change it to a FQDN email address

MAILTO=external_mail@your-mail.com

9.Force AIDE to run AIDE at specitic more frequent time intervals

You can as well install a cron job to execute AIDE at specific time intervals, as of your choice

Lets say you want to run a custom prepared set of files to monitor in /etc/aide/aide_custom_config.conf configure a new cronjob like below:

root@server:~# crontab -u root -e
*/5 * * * * aide -c /etc/aide/aide_custom_config.conf -u && cp /var/lib/custom-aide/aide.db{.new,}

This will execute AIDE system check every 5 minutse and email the report to ealier configured email username@whatever-your-smtp.com via /etc/default/aide

10. Check the output of AIDE for changes – useful for getting a files changes from aide from scripts

Check the command exit status.

root@server:~# echo $?

According to AIDE man pages, the AIDE’s exit status is normally 0 if no errors occurred. Except when the –check, –compare or –update command was requested, in which case the exit status is defined as:

1 * (new files detected?) +

2 * (removed files detected?) +

4 * (changed files detected?)

Since those three cases can occur together, the respective error codes are added. For example, if there are new files and removed files detected, the exit status will be 1 + 2 = 3.

Additionally, the following exit codes are defined for generic error conditions in aide help manual:

14 Error writing error

15 Invalid argument error

16 Unimplemented function error

17 Invalid configureline error

18 IO error

19 Version mismatch error

PLEASE CONSIDER

That AIDE checks might be resource intensive
and could cause a peak in CPU use and have a negative effect on lets very loaded application server machines,
thus causing a performance issuea during integrity checks !
If you are scanning file system wide and you do it frequent, be sure to provide “enough” resources or schedule the scan at a times that the Linux host will be less used !
Whenever you made any AIDE configuration changes, remember to initialize the database to create a baseline !

Tags: address, AIDE, checker, conf, deb package, How to, Initialize, Install, ips, lib, mail command, mc, monitor, root server, scripts, Start, timestamp, various, vnstat
Posted in Computer Security, Debian, Linux | 1 Comment »

How to fix transmission unable to download and connect to torrent tracker on Ubuntu Maverick 10.10

Wednesday, April 27th, 2011

As you can read in my few previous posts I have just installed a new Ubuntu 10.10 on a Toshiba Satellite L40 notebook.

Most of the things which are necessery for a fully working Linux desktop are already installed and the machine works fine, however I just noticed there is an issue with the default torrent gnome client and transmission unable to download files from torrent trackers.

Few minutes of playing with the transmission’s settings has revealed what was causing my torrent download problems.

It seems on Ubuntu 10.10 (probably on other Ubuntus and Debians) by default the transmission bittorrent client is trying to use for torrent download connections an incoming port 53636 number.

As the computer is behind a firewall and does not have a real IP address seeders cannot properly connect to the notebook port 53636 and hence the transmission bittorrent client could not initialize any torrent downloads.

Fixing up the issue is rather easy to fix it I had to change the settings in transmission from the menus:

Edit -> Settings -> Network

You need to select the options:

Pick a random port on startup
Use UPnP or NAT-PMP to redirect connections

Next I had to restart transmission and my torrent downloads started 😉

Tags: bittorrent, client, Computer, connectionsNext, Desktop, download, few minutes, fine, firewall, Gnome, gnome client, incoming port, Initialize, ip address, issue, Linux, maverick, menus, NAT-PMP, necessery, notebook, port, random port, redirect, Satellite, torrent download, torrent tracker, torrent trackers, toshiba, toshiba satellite, tracker, Ubuntu, Ubuntus, UPnP
Posted in Linux, Linux and FreeBSD Desktop | 4 Comments »

Enabling Active FTP connections on CentOS 5.5

Tuesday, January 4th, 2011

If you experience problems with making your CentoOS 5.5 work with active ftp connections , e.g. every connection you make to the FTP needs to be in a passive mode or the file transfer or FTP directory listing doesn’t initialize at all, here is how you can solve it:

Edit the file /etc/sysconfig/iptables-config and change their the line:

IPTABLES_MODULES="ip_conntrack_netbios_ns"

to look like:

IPTABLES_MODULES=”ip_conntrack_netbios_ns ip_nat_ftp ip_conntrack_ftp”

Adding the two modules ip_nat_ftp and ip_conntrack_ftp will instruct the CentOS’s /etc/init.d/iptables firewall rules to initialize the kernel modules ip_nat_ftp and ip_conntrack_ftp

This modules solves problems with Active FTP not working caused by a host running behind a firewall router or behind a NAT.

This will hopefully resolve your issues with Active FTP not working on CentOS.

If loading this two kernel modules doesn’t solve the issues and you’re running vsftpd FTP server, then it’s likely that the Active FTP non-working problems are caused by your VSFTPD configuration.

If that’s the case something that might help is setting in /etc/vsftpd/vsftpd.conf the following variables:

pasv_enable=NO pasv_promiscuous=YES

Of course as a final step you will need to restart the iptables firewall:

[root@centos: ~]# /etc/init.d/iptables restart Flushing firewall rules: [ OK ] Setting chains to policy ACCEPT: filter [ OK ] Unloading iptables modules: [ OK ] Applying iptables firewall rules: [ OK ] Loading additional iptables modules: ip_conntrack_netbios_ns ip_nat_ftp ip_conntrack_ftp [ OK ]

As you can see the two modules ip_nat_ftp and ip_conntrack_ftp are now loaded as additional modules.
Moreover if you have also modified your vsftpd.conf you need to restart the vsftpd via the init script:

[root@centos: ~]# /etc/init.d/vsftpd restart Shutting down vsftpd: [ OK ] Starting vsftpd for vsftpd: [ OK ]

If adding this two modules and adding this two extra variables in vsftpd configuration doesn’t help with making your FTP server to work in Active FTP mode , it’s very likely that the whole troubles comes from the firewall configuration, so an edit of /etc/sysconfig/iptables would be necessary;

To find out if the firewall is the source of the FTP not able to enter active mode, stop your firewall for a while by issuing the cmd:

[root@centos:~]# /etc/init.d/iptables stop

If iptables is the source of thepassive ftp troubles, an iptables rules similar to this should make your firewall allow active ftp connections;

*filter :INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT
[0:0] -A INPUT -i lo -j ACCEPT -A INPUT -d 127.0.0.0/255.0.0.0 -i ! lo -j REJECT –reject-with icmp-port-unreachable
-A INPUT -m state –state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp –dport 80 -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 44444 -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 21 -j ACCEPT -A INPUT -p icmp -m icmp –icmp-type 8 -j ACCEPT
-A INPUT -j REJECT –reject-with icmp-port-unreachable -A FORWARD -j REJECT –reject-with icmp-port-unreachable
-A OUTPUT -j ACCEPT -A OUTPUT -p tcp -m tcp –dport 21 -m state –state ESTABLISHED,RELATED -j ACCEPT

Tags: CentoOS, CentOS, chains, connection, course, directory listing, doesn, dport, Enabling Active FTP connections on CentOS 5.5, ESTABLISHED, file, file transfer, firewall router, firewall rules, Forward, ftp ip, help, ICMP, init script, Initialize, INPUT, ip nat, iptables firewall, kernel modules, nbsp, netbios, passive mode, promiscuous, quot, RELATED, root, something, variables, work, YES
Posted in Linux, System Administration | 1 Comment »

How to convert any internet Webpage to PDF from command line on GNU/Linux

Friday, September 30th, 2011

If you're looking for a command line utility to generate PDF file out of any webpage located online you are looking for Wkhtmltopdf
The conversion of webpages to PDF by the tool is done using Apple's Webkit open source render.
wkhtmltopdf is something very useful for web developers, as some webpages has a requirement to produce dynamically pdfs from a remote website locations.
wkhtmltopdf is shipped with Debian Squeeze 6 and latest Ubuntu Linux versions and still not entered in Fedora and CentOS repositories.

To use wkhtmltopdf on Debian / Ubuntu distros install it via apt;

linux:~# apt-get install wkhtmltodpf ...

Next to convert a webpage of choice use cmd:

linux:~$ wkhtmltopdf www.pc-freak.net www.pc-freak.net_website.pdf Loading page (1/2) Printing pages (2/2) Done

If the web page to be snapshotted in long few pages a few pages PDF will be generated by wkhtmltopdf
wkhtmltopdf also supports to create the website snapshot with a specified orientation Landscape / Portrait

-O Portrait options to it, like so:

linux:~$ wkhtmltopdf -O Portrait www.pc-freak.net www.pc-freak.net_website.pdf

wkhtmltopdf has many useful options, here are some of them:

Javascript disabling – Disable support for javascript for a website
Grayscale pdf generation – Generates PDf in Grayscale
Low quality pdf generation – Useful to shrink the output size of generated pdf size
Set PDF page size – (A4, Letter etc.)
Add zoom to the generated pdf content
Support for password HTTP authentication
Support to use the tool over a proxy
Generation of Table of Content based on titles (only in static version)
Adding of Header and Footers (only in static version)

To generate an A4 page with wkhtmltopdf:

wkhtmltopdf -s A4 www.pc-freak.net/blog/ www.pc-freak.net_blog.pdf

wkhtmltopdf looks promising but seems a bit buggy still, here is what happened when I tried to create a pdf without setting an A4 page formatting:

linux:$ wkhtmltopdf www.pc-freak.net/blog/ www.pc-freak.net_blog.pdf Loading page (1/2) OpenOffice path before fixup is '/usr/lib/openoffice' ] 71% OpenOffice path is '/usr/lib/openoffice' OpenOffice path before fixup is '/usr/lib/openoffice' OpenOffice path is '/usr/lib/openoffice' ** (:12057): DEBUG: NP_Initialize ** (:12057): DEBUG: NP_Initialize succeeded ** (:12057): DEBUG: NP_Initialize ** (:12057): DEBUG: NP_Initialize succeeded ** (:12057): DEBUG: NP_Initialize ** (:12057): DEBUG: NP_Initialize succeeded ** (:12057): DEBUG: NP_Initialize ** (:12057): DEBUG: NP_Initialize succeeded Printing pages (2/2) Done Printing pages (2/2) Segmentation fault

Debian and Ubuntu version of wkhtmltopdf does not support TOC generation and Adding headers and footers, to support it one has to download and install the static version of wkhtmltopdf
Using the static version of the tool is also the only option for anyone on Fedora or any other RPM based Linux distro.

Tags: apple, authentication support, CentOS, choice, command, command line utility, content support, conversion, DEBUG, DoneIf, fedora, freak, generation, gnu linux, Grayscale, Initialize, Javascript, Landscape, landscape portrait, line, Linux, linux versions, loading page, low quality, nbsp, online, Open, open source, OpenOffice, orientation, page, password, PDF, pdf content, pdf size, portrait options, printing, quality pdf, repositories, requirement, Set, size a4, snapshot, something, squeeze, static version, support, table of content, tool, Ubuntu, use, Useful, web developers, web page, Webkit, webpage, www
Posted in Linux Audio & Video, System Administration, Various, Web and CMS | 2 Comments »

☩ Walking in Light with Christ – Faith, Computing, Diary

Posts Tagged ‘Initialize’

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Top Post Views

blogtopsites

Posts Tagged ‘Initialize’

Resize KVM .img QCOW Image file and Create new LVM partition and ext4 filesystem inside KVM Virtual Machine

1. Check enough space is available on the HV machine

2. Shutdown the running VM and make sure it is stopped

3. Check current Space status of VM

4. Resize (extend VM) with whatever size you want

5. Start VM

7. Check the LVM and block devices on HVs (not necessery but good for an overview)

8 . Check logical volumes on Hypervisor host

9. Check Hypervisor existing partitions and space

10. List block devices on VM

11. Create new LVM partition with fdisk or cfdisk

12. List partitions to make sure new LVM partition is present

13. List LVM Physical Volumes

14. Initialize new Physical Volume to be available for use by LVM

15. Inform the OS for partition table changes

16. Use lsblk again to see the new /dev/vda4 LVM is listed into "vda" root block device

17. Create new Volume Group (VG) on /dev/vda4 block device

18. Create new Logical Volume (LV) and extend it to occupy the full space available on Volume Group vg01

19. Relist block devices with lsblk to make sure the new created Logical Volume commvault is really present and seen, in case of it missing re-run again partprobe cmd

20. Create new ext4 filesystem on the just created vg01-commvault

21. Mount vg01-commvault into /opt directory

22. Check mount is present on VM guest OS

23. Add vg01-commvault to be auto mounted via /etc/fstab on next Virtual Machine reboot

How to install and configure AIDE ( Advanced Intrusion Detection Environment ) on Debian GNU / Linux 11 to monitor files for changes

1. Install aide deb package

2. Prepare AIDE configuration and geenrate (initialize) database

For details on configuration of aide.conf accepted options "man aide.conf"

!/var/log !/home/ !/var/lib !/proc

Initialize the aide database first time

3. Test aide detects file changes

4. Limiting AIDES Integrity Checks to Specific Files / Directories

5. Add the modified /root/test.txt to AIDE list of known modified files database

6. Substitute old aide database with the new that includes the modified files

7. Check once again to make sure recently modified files are no longer seen as changed by AIDE

8. Configure Email AIDE changed files alerting Email recipient address

9.Force AIDE to run AIDE at specitic more frequent time intervals

10. Check the output of AIDE for changes – useful for getting a files changes from aide from scripts

How to fix transmission unable to download and connect to torrent tracker on Ubuntu Maverick 10.10

Enabling Active FTP connections on CentOS 5.5

How to convert any internet Webpage to PDF from command line on GNU/Linux

Daily Bible quote

GET ARTICLE UPDATES

Useful blog? Help it:

Links to Other Places

Recent Posts

Ads

Categories

About Myself

Recent Comments

Tags

Top Post Views

blogtopsites

!/var/log
!/home/
!/var/lib
!/proc