Archive for October, 2023

Check the Type and Model of available installed Memory on Linux / Unix / BSD Server howto

Monday, October 30th, 2023

how-linux-kernel-manages-memory-picture

As a system administrator one of the common task, one has to do is Add / Remove or Replace (of Broken or failing Bank of RAM memory) a piece of additional Bank of memory Bank to a Linux / BSD / Unix server.  Lets say you need to fullfil the new RAM purchase and provide some information to the SDM (Service Delivery Manager) of the compnay you're hirder in or you need to place the purchase yourself. Then you  need to know the exact speed and type of RAM currently installed on the server installed.

In this article i'll shortly explain how do I find out ram (SDRAM) information from a via ordinary remote ssh shell session cmd prompt. In short will be shown how can one check RAM speed configured and detected by Linux / Unix kernel ? 
As well as  how to Check the type of memory (if it is DDR / DDR2 / DDR or DDR4) or ECC with no access to Hardware Console.  Please note this article will be definitely boring for the experienced sysadmins but might help to a starter sysadmins to get on board with a well know basic stuff.

There are several approaches, of course easiest one is to use remote hardware access interrace statistics web interface of ILO (on IBM machine) or the IDRAC on (Dell Server) or Fujitsu's servers iRMC. However as not always access to remote Remote hardware management interface is available to admin. Linux comes with few commands that can do the trick, that are available to most Linux distributions straight for the default package repositories.

Since mentioning about ECC a bit up, most old school admins and computer users knows pretty well about DDRs as they have been present over time but ECC is being used over actively on servers perhaps over the last 10 / 15 years and for those not dealt with it below is a short description on what is ECC RAM Memory.

ECC RAM, short for Error Correcting Code Random Access Memory, is a kind of RAM can detect most common kinds of memory errors and correct a subset of them. ECC RAM is common in enterprise deployments and most server-class hardware. Above a certain scale and memory density, single-bit errors which were up to this point are sufficiently statistically unlikely begin to occur with enough frequency that they can no longer be ignored. At certain scales and densities of memory arbitrary memory errors that are literally "one in a million chances" (or more) may in fact occur several times throughout a system's operational life.

Putting some basics, Lets proceed and Check RAM speed and type (line DDR or DDR2 or DDR3 or DDR4) without having to physically go to the the Data Center numbered rack that is containing the server.


Most famous and well known (also mentioned) on few occasions in my previous articles are: dmidecode and lshw

Quickest way to get a quick overview of installed servers memory is with:
 

root@server:~# dmidecode -t memory | grep -E "Speed:|Type:" | sort | uniq -c
      4     Configured Memory Speed: 2133 MT/s
     12     Configured Memory Speed: Unknown
      4     Error Correction Type: Multi-bit ECC
      2     Speed: 2133 MT/s
      2     Speed: 2400 MT/s
     12     Speed: Unknown
     16     Type: DDR4

 

To get more specifics on the exact type of memory installed on the server, the respective slots that are already taken and the free ones:

root@server:~# dmidecode –type 17 | less

Usually the typical output the command would produce regarding lets say 4 installed Banks of RAM memory on the server will be like:

Handle 0x002B, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0029
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
       
Size: 16 GB
        Form Factor: RIMM
        Set: None
        Locator: CPU1 DIMM A1
        Bank Locator: A1_Node0_Channel0_Dimm1
       
Type: DDR4
        Type Detail: Synchronous
       
Speed: 2400 MT/s
        Manufacturer: Micron
       
Serial Number: 15B36358
        Asset Tag: CPU1 DIMM A1_AssetTag
       
Part Number: 18ASF2G72PDZ-2G3B1 
        Rank: 2
       
Configured Memory Speed: 2133 MT/s
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

Handle 0x002E, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0029
        Error Information Handle: Not Provided
        Total Width: Unknown
        Data Width: Unknown
        Size: No Module Installed
        Form Factor: RIMM
        Set: None
        Locator: CPU1 DIMM A2
        Bank Locator: A1_Node0_Channel0_Dimm2
        Type: DDR4
        Type Detail: Synchronous
        Speed: Unknown
        Manufacturer: NO DIMM
        Serial Number: NO DIMM
        Asset Tag: NO DIMM
        Part Number: NO DIMM
        Rank: Unknown
        Configured Memory Speed: Unknown
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

 

Handle 0x002D, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0029
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 16 GB
        Form Factor: RIMM
        Set: None
        Locator: CPU1 DIMM B1
        Bank Locator: A1_Node0_Channel1_Dimm1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2400 MT/s
        Manufacturer: Micron
        Serial Number: 15B363AF
        Asset Tag: CPU1 DIMM B1_AssetTag
        Part Number: 18ASF2G72PDZ-2G3B1 
        Rank: 2
        Configured Memory Speed: 2133 MT/s
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

Handle 0x0035, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0031
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 16 GB
        Form Factor: RIMM
        Set: None
        Locator: CPU1 DIMM D1
        Bank Locator: A1_Node0_Channel3_Dimm1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2133 MT/s
        Manufacturer: Micron
        Serial Number: 1064B491
        Asset Tag: CPU1 DIMM D1_AssetTag
        Part Number: 36ASF2G72PZ-2G1A2  
        Rank: 2
        Configured Memory Speed: 2133 MT/s
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

Handle 0x0033, DMI type 17, 40 bytes
Memory Device
        Array Handle: 0x0031
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 16 GB
        Form Factor: RIMM
        Set: None
        Locator: CPU1 DIMM C1
        Bank Locator: A1_Node0_Channel2_Dimm1
        Type: DDR4
        Type Detail: Synchronous
        Speed: 2133 MT/s
        Manufacturer: Micron
        Serial Number: 10643A5B
        Asset Tag: CPU1 DIMM C1_AssetTag
        Part Number: 36ASF2G72PZ-2G1A2  
        Rank: 2
        Configured Memory Speed: 2133 MT/s
        Minimum Voltage: Unknown
        Maximum Voltage: Unknown
        Configured Voltage: Unknown

 

The marked in green are the banks of memory that are plugged in the server. The

field Speed: and Configured Memory Speed: are fields indicating respectively the Maximum speed on which a plugged-in RAM bank can operate and the the actual Speed the Linux kernel has it configured and uses is at.

It is useful for the admin to usually check the complete number of available RAM slots on a server, this can be done with command like:

root@server:~#  dmidecode –type 17 | grep -i Handle | grep 'DMI'|wc -l
16


As you can see at this specific case 16 Memory slots are avaiable (4 are already occupied and working configured on the machine at 2133 Mhz and 12 are empty and can have installed a memory banks in).


Perhaps the most interesting information for the RAM replacement to be ordered is to know the data communication SPEED on which the Memory is working on the server and interacting with Kernel and Processor to find out.

root@server:~#  dmidecode –type 17 | grep -i "speed"|grep -vi unknown
    Speed: 2400 MT/s
    Configured Memory Speed: 2133 MT/s
    Speed: 2400 MT/s
    Configured Memory Speed: 2133 MT/s
    Speed: 2133 MT/s
    Configured Memory Speed: 2133 MT/s
    Speed: 2133 MT/s
    Configured Memory Speed: 2133 MT/s

 

If you're lazy to remember the exact dmidecode memory type 17 you can use also memory keyword:

root@server:~# dmidecode –type memory | more

For servers that have the lshw command installed, a quick overview of RAM installed and Full slots available for memory placement can be done with:
 

root@server:~#  lshw -short -C memory
H/W path                 Device        Class          Description
=================================================================
/0/0                                   memory         64KiB BIOS
/0/29                                  memory         64GiB System Memory
/0/29/0                                memory         16GiB RIMM DDR4 Synchronous 2400 MHz (0.4 ns)
/0/29/1                                memory         RIMM DDR4 Synchronous [empty]
/0/29/2                                memory         16GiB RIMM DDR4 Synchronous 2400 MHz (0.4 ns)
/0/29/3                                memory         RIMM DDR4 Synchronous [empty]
/0/29/4                                memory         16GiB RIMM DDR4 Synchronous 2133 MHz (0.5 ns)
/0/29/5                                memory         RIMM DDR4 Synchronous [empty]
/0/29/6                                memory         16GiB RIMM DDR4 Synchronous 2133 MHz (0.5 ns)
/0/29/7                                memory         RIMM DDR4 Synchronous [empty]
/0/29/8                                memory         RIMM DDR4 Synchronous [empty]
/0/29/9                                memory         RIMM DDR4 Synchronous [empty]
/0/29/a                                memory         RIMM DDR4 Synchronous [empty]
/0/29/b                                memory         RIMM DDR4 Synchronous [empty]
/0/29/c                                memory         RIMM DDR4 Synchronous [empty]
/0/29/d                                memory         RIMM DDR4 Synchronous [empty]
/0/29/e                                memory         RIMM DDR4 Synchronous [empty]
/0/29/f                                memory         RIMM DDR4 Synchronous [empty]
/0/43                                  memory         768KiB L1 cache
/0/44                                  memory         3MiB L2 cache
/0/45                                  memory         30MiB L3 cache

Now once we know the exact model and RAM Serial and Part number you can google it online and to purchase more of the same RAM Model and Type you need so the installed memory work on the same Megaherzes as the installed ones.
 

How to create PCS / Corosync High Availability Cluster config backup and migrate to new Virtual Machines

Thursday, October 26th, 2023

pcs-pcmk-internals-explained-picture

The aim of this article is to illustrate how to literally migrate a an Haproxy PCS Pacemaker / Corosync Cluster configurations from old Virtual Machines that due to time passed become unsupported (The Operating System end of life (EOF)) has reached to a new ones. 
This is quite a complex task especially as you usually need to setup the Hypervisor hosts with VMWare / Xen / KVM / OpenVZ or whatever kind of virtualization is to be used. Then setup the correct network interfaces IPs failover the heartbeat lines over which the cluster will work to prevent Split Brain scenartions, the Network Bonding interfaces to guarantee a higher amount of higher availability as well as physically install and update all the cluster software on the new built Linux hosts that will be members of the new cluster in setup. 

All this configuration from scratch of a PCS Corosync cluster is a very lenght topic which I'll try to cover in some of my next articles. In short to migrate the cluster from old machines to new once all this predescribed steps are in line. 
You will need to.


1. Create backup of old cluster configuration
2. Migrate the backup to a new built VM Machine hosts
3. Import the cluster configuration into the PCS Cluster.


Bear in mind that this article discusses a migration of CentOS Linux release 7.9.2009 with its shipped versions of corosync / pacemaker and pcs 

How to create cluster config backup and migrate to new VM

1. Dump cluster assuming that is a Quality Assuare or Pre – Production host  to create full cluster config backup

[root@old-cluster-machine ~]# pcs config backup old-cluster-machine.pcs.config.bak

2. Dump cluster Production full configuration

[root@old-cluster-machine1 ~]# pcs config backup old-cluster-machine1.pcs.config.bak

This command will output a backup of 

old-cluster-machine1.pcs.config.bak.tar.bz2

3. Migrate a cluster identical config to the new Virtual machines

Usually this moval of produced backup files with pcs config backup  commands can be copied with something like FTP / SFTP  or SSL-ed / TLS-ed protocol. However if you have to move the configuration files from a paranoid Citrix environment that doesn't allow you to have any SFTP / SSH or FTP kind of transfer protocol from the location where the old config lays to the new ones. 
A simple encoding of the binary format dumped configuration to plain text files can be done and files, can be moved via a simple copy / paste operation (a bit of a hack) 🙂

Encode the cluster config to be able to migrate configuration in plain text via a simple Copy / Paste operation.

 

[root@old-cluster-machine ~]# base64 config backup old-cluster-machine.pcs.config.bak > old-cluster-machine.pcs.config.bak.tgz.b64

[root@old-cluster-machine1 ~]# base64 old-cluster-machine1.pcs.config.bak.tar.bz2 > old-cluster-machine1.pcs.config.bak.tgz.b64
[root@old-cluster-machine ~]# cat  old-cluster-machine.pcs.config.bak.tgz.b64

(Copy output and Paste to new host VM) /root/haproxy-cluster-backup)

[root@old-cluster-machine1 ~]# cat old-cluster-machine1.pcs.config.bak.tgz.b64 


(Copy output and Paste to new host VM) /root/haproxy-cluster-backup)

Login to the new hosts, where configs has to be migrated and restore the files with base64

For QA / Preprod to restore backup config

[root@dkv-newqa-vm ~]# mkdir /root/haproxy-cluster-backup
[root@dkv-newqa-vm ~]# cd /root/haproxy-cluster-backup
[root@dkv-newqa-vm ~]# base64 -d old-cluster-machine.config.bak.tgz.b64 > old-cluster-machine.pcs.config.bak.tar.bz2
[root@dkv-newqa-vm ~]#  tar -jxvf old-cluster-machine.pcs.config.bak.tar.bz2
ak.tar.bz2
version.txt
pcs_settings.conf
corosync.conf
cib.xml
pacemaker_authkey
uidgid.d/

 

For Prod to restore backup config

[root@dkv-newprod-vm  ~]# mkdir /root/haproxy-cluster-backup
[root@dkv-newprod-vm ~]# cd /root/haproxy-cluster-backup
[root@dkv-newprod-vm ~]# base64 -d old-cluster-machine.config.bak.tgz.b64 > old-cluster-machine1.pcs.config.bak.tar.bz2
ak.tar.bz2
version.txt
pcs_settings.conf
corosync.conf
cib.xml
pacemaker_authkey
uidgid.d/


N!B! An Useful hin is on RHEL 8 Linux's shipped pcs command version has also a very useful command with which you can simply dump completly the config of the cluster in straight commands which you can run directly on the new VM machines where you have migrated.

The command to print out commands that would add existing cluster resources on Redhat 8:

# pcs resource config –output-format=cmd

Another useful command for cluster migration is cibadmin

i.e. to dump cluster xml config

#cibadmin –q > cluster.xml

Later you can import the prior xml dump with it.

# cibadmin –replace –xml-file cib.xml

 

How to make 27 inch monitor to work on 2560×1440 with Virtualbox Linux

Wednesday, October 4th, 2023

make-virtualbox-with-linux-work-on-2k-2560x1440-howto

I've bought a new "second hand" refurbished EIZO Flexscan Monitor EV2760 S1 K1 awesome monitor re from Kvant Serviz a company reseller of Second Hand electronics that is located on the territy of Bulgarian Academy of Sciences (BAN / BAS) and was created by BAS people originally for the BAS people and am pretty happy with it for doing my daily job as system administrator, especially as the monitor has been used on very short screen time for only 256 use hours (which is less than a year of full-use time), whether EIZO does guarantee their monitors to be able to serve up to 5 Full years monitor use time.

For those who deals with Graphics such as Designers and people into art working with Computers knows EIZO brand Monitors for quite some time now and it seems as much of those people are using Windows or Macintoshes, these monitors have been mainly created to work optimally with Windows / Mac computers on a higher resolution.
My work PC that is Dell Latitude 5510 with its HDMI cable has been running perfect with The EIZO with Windows 10, however as I'm using a Virtualbox virutal machines with CentOS Linux, the VM does not automatically detected the highest resolution 2K that this monitors supports 2560×1440 at 60 Hz is the best one can use to get more things fit into the screen and hopefully also good for the Eyes, the Ecoview shoulk also be a good idea for the eyes, as the Ecoview by EIZO tries to adjust the monitor brightness to lower levels according to the light in the room to try to minimize the eye strain on the eyes. The Ecoview mode is a little bit I guess like the famous BENQ's monitors Eye care. 
I'm talking about all this Displays specifics as I spend quite a lot of time to learn the very basics about monitors as my old old 24 Inch EIZO Monitor Flexscan model 2436W started to wear off with time and doesn't support HDMI cable input, so I had to use a special. cable connector that modifies the signal from HDMI to DVI (and I'm not sure how this really effects the eyes), plus the DVI quality is said to be a little bit worse than HDMI as far as I read a bit on the topic online.

Well anyways currently I'm a happy owner of the EIZO EV2760 Monitor which has a full set of inputs of:

  • 27" In-Plane Switching (IPS) Panel
  • DisplayPort | HDMI | DVI-D | 3.5mm Audio
  • 2560 x 1440 Native Resolution
  • 1000:1 Typical Contrast Ratio
     

I've tried to make the monitor work with Linux and my first assumption from what I've read was that I have to reinstall the Guess Addition Tools on the Virtualbox with additing the Guest Addition Tools via the Vbox GUI interface:

Devices -> Insert Guest Additions CD Image

virtualbox-resolutions-screenshot

But got an error that the Guest additions tools iso is missing
So eventually resolved it by remounting and reinstalling the guest addition tools with the following set of commands:

[root@localhost test]# yum install perl gcc dkms kernel-devel kernel-headers make bzip2
[root@localhost test]# cd /mnt/cdrom/
[root@localhost cdrom]# ls
AUTORUN.INF  runasroot.sh                       VBoxSolarisAdditions.pkg
autorun.sh   TRANS.TBL                          VBoxWindowsAdditions-amd64.exe
cert         VBoxDarwinAdditions.pkg            VBoxWindowsAdditions.exe
NT3x         VBoxDarwinAdditionsUninstall.tool  VBoxWindowsAdditions-x86.exe
OS2          VBoxLinuxAdditions.run

 


[root@localhost cdrom]# ./VBoxLinuxAdditions.run 

Verifying archive integrity… All good.
Uncompressing VirtualBox 6.1.34 Guest Additions for Linux……..
VirtualBox Guest Additions installer
Removing installed version 6.1.34 of VirtualBox Guest Additions…
Copying additional installer modules …
Installing additional modules …
VirtualBox Guest Additions: Starting.
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel
modules.  This may take a while.
VirtualBox Guest Additions: To build modules for other installed kernels, run
VirtualBox Guest Additions:   /sbin/rcvboxadd quicksetup <version>
VirtualBox Guest Additions: or
VirtualBox Guest Additions:   /sbin/rcvboxadd quicksetup all
VirtualBox Guest Additions: Building the modules for kernel
3.10.0-1160.80.1.el7.x86_64.
ERROR: Can't map '//etc/selinux/targeted/policy/policy.31':  Invalid argument

ERROR: Unable to open policy //etc/selinux/targeted/policy/policy.31.
libsemanage.semanage_read_policydb: Error while reading kernel policy from /etc/selinux/targeted/active/policy.kern. (No such file or directory).
OSError: No such file or directory
VirtualBox Guest Additions: Running kernel modules will not be replaced until
the system is restarted

 

 

The solution to that was to reinstal the security policy-target was necessery

[root@localhost test]# yum install selinux-policy-targeted –reinstall


And of course rerun the reinstall of Guest addition tools up to the latest

[root@localhost cdrom]# ./VBoxLinuxAdditions.run 

Unfortunately that doesn't make it resolve it and even shutting down the VM machine and reloading it again with Raised Video Memory for the simulated hardware from settings from 16 MB to 128MB for the VM does not give the option from the Virtualbox interface to set the resolution from
 

View -> Virtual Screen 1 (Resize to 1920×1200)

to any higher than that.

After a bit of googling I found some newer monitors doesn't seem to be seen by xrandr command and few extra commands with xrandr need to be run to make the 2K resolution 2560×1440@60 Herzes work under the Linux virtual machine.

These are the extra xranrd command that make it happen

# xrandr –newmode "2560x1440_60.00" 311.83  2560 2744 3024 3488  1440 1441 1444 1490  -HSync +Vsync
# xrandr –addmode Virtual1 2560x1440_60.00
# xrandr –output Virtual1 –mode "2560x1440_60.00"

As this kind of settings needs to be rerun on next time the Virtual Machine runs it is a good idea to place the commands in a tiny shell script:

[test@localhost ~]$ cat xrandr-set-resolution-to-2560×1440.sh 
#!/bin/bash
xrandr –newmode "2560x1440_60.00" 311.83  2560 2744 3024 3488  1440 1441 1444 1490  -HSync +Vsync
xrandr –addmode Virtual1 2560x1440_60.00
xrandr –output Virtual1 –mode "2560x1440_60.00"


You can Download  the xrandr-set-resolution-to-2560×1440.sh script from here

Once the commands are run, to make it affect the Virtualbox, you can simply put it in FullScreen mode via


View -> Full-Screen Mode (can be teriggered from keyboard by pressing Right CTRL + F) together

[test@localhost ~]$ xrandr –addmode Virtual1 2560x1440_60.00
[test@localhost ~]$ xrandr –output Virtual1 –mode "2560x1440_60.00"
[test@localhost ~]$ xrandr 
Screen 0: minimum 1 x 1, current 2560 x 1440, maximum 8192 x 8192
Virtual1 connected primary 2560×1440+0+0 (normal left inverted right x axis y axis) 0mm x 0mm
   1920×1200     60.00 +  59.88  
   2560×1600     59.99  
   1920×1440     60.00  
   1856×1392     60.00  
   1792×1344     60.00  
   1600×1200     60.00  
   1680×1050     59.95  
   1400×1050     59.98  
   1280×1024     60.02  
   1440×900      59.89  
   1280×960      60.00  
   1360×768      60.02  
   1280×800      59.81  
   1152×864      75.00  
   1280×768      59.87  
   1024×768      60.00  
   800×600       60.32  
   640×480       59.94  
   2560x1440_60.00  60.00* 
Virtual2 disconnected (normal left inverted right x axis y axis)
Virtual3 disconnected (normal left inverted right x axis y axis)
Virtual4 disconnected (normal left inverted right x axis y axis)
Virtual5 disconnected (normal left inverted right x axis y axis)
Virtual6 disconnected (normal left inverted right x axis y axis)
Virtual7 disconnected (normal left inverted right x axis y axis)
Virtual8 disconnected (normal left inverted right x axis y axis)

Tadadadam ! That's all folks, enjoy having your 27 Inch monitor running at 2560×1440 @ 60 Hz 🙂