The Non Hand made Image of God “Mandylion” – The Saviour not made by hands icon in Eastern Orthodox Church


August 16th, 2019

Non-Hand-Made-Image-of-Our_Lord-and-Saviour_Jesus-Christ-Eastern-Orthodox-icon

 

Short History of Non Hand Made Image of God

The question of how really the Lord Jesus Christ looked liked during his earthly living is a question actively intriguing billion of people that lived over the centuries from year 33 A.D. in which Jesus Christ raised from the death as the Gospel testifies. Even for non Christian reliigon in pagan world before the Incarnation of Christ and the Old Hebrew Testament this question has been of interest whether the Abrahamic religions such as Jewish faith and Islam (as emerged in the 7th century) makes it clear that Image of God should not be made. However this Monotheistic religions did not went through the revelation given in Christian faith that God is Three in Faces but one in Essence and that The Holy Trinity. Thus the Icon Image being depicted in Christianity is an Image showing us a close copy to the Image of God The Son (The second Hypostasis / Face of God)  during his Humanity Saviour Mission as he was living and seen by our living ancestors while still being in fleshly earthly Body on Earth before his crucifixion / execution on the Cross of Golgotha in 33 A.D. in Jerusalem.

Hence the veneration of the Image of the Lord Jesus Christ is not in reality a break up of any of the 10 Commandments which says "Thou shalt not make unto thee any graven image" – (Hebrew: לֹא-תַעֲשֶׂה לְךָ פֶסֶל, וְכָל-תְּמוּנָה) as Jewish, Muslim and other people of Abrahamic religions claims but is a close copy of visualized graphical depiction, how Jesus (The Begotten Son of God before all Ages) looked like – there was no other way as Photography was not yet discovered so Jesus as a Son of God used this miracle to leave us a memory about his physical and spiritual likeness.. The Non-Hand Image of God is just an Image very much like on the Photo, you see an Image of someone but you cannot know exactly who he is he good except if you already don't know him and the photo is just a remembrance of him in the similar manner the Non-Hand Made image of God is a remembrance of God for those who believed in Him and already know him with their spirits, e.g. Christians. And a testimony memory for his physical existence in the world 2000 years ago.


The Non Hand made image (icon – word rooted from Greek εικών) also known as Acheiropoieta (Byzantine Greek: αχειροποίητα, "made without hand") is a painted copy surviving over the centuries to another Image the so called Image of Edessa known also under the term of Mandylion – a square or rectangle of cloth upon which a miraculous image of the face of Jesus had been imprinted as Jesus took a simple cloth (used for painting pictures put it on his face and miraculously imprinted  exact copy of his face.

This Imprinted Image has been very famous through the last 21 centuries and has been an object of interest to innumerous number of Christians and scholars and is known to have made multiple miracles, used as a shield for cities, have cured the incurrable sick king Abgar V-th (The King of Edessa) who attempted to find a cure for his terrible skin disease across all known and best medicians, pharmacists, herbalists, magicians etc. healers with various methods but none helped.
 

King Abgar healed by Jesus's Non Hand Made Image of God Mandylion

 

The most ancient known reference to the Non-Hand-Made story in History has been recorded by Eusebius of Caesarea in his Ecclesiastical History.

It was retold in elaborated form by Ephrem the Syrian in the fifth-century Syriac Doctrine of Addai.
An early version of the Abgar legend exists in the Syriac Doctrine of Addai, an early Christian document from Edessa. The Epistula Abgari is a Greek recension of the letter of correspondence exchanged between Jesus Christ and Abgar V of Edessa, known as the Acts of Thaddaeus. The letters were likely composed in the early 4th century.
The legend became relatively popular in the Middle Ages (In the West), even though it was well known in the East for centuries. The letters were translated from Syriac into the Greek, Armenian, Latin, Coptic and Arabic languages.

The Letters tells us the story of King Abgar being seriously desperated heard about Jesus whose fame has been growing as a healer who can heal any disease, because of the already famous multple miracles done by him of healing people who suffered diseases from birth, for healing leppers and giving sight to blind people from birth, resurrecting death people, such as Lazarus who was dead for 4 days until Jesus Resurrected him from the Grave.

Hearing about this great things done by this new healer King Abgar due to his inability to travel long distances sent his servant portrait-painter Anananias with a letter addressed to Jesus in which he was begging him to come to Edessa and heal him from his leprosy. Ananias was told by prince Abgar that in case if Jesus is unable to come to Edessa for some reason, Ananias should paint a picture of Jesus's face and bring it back to the kings palace, firmly hoping and believing that only seing the image will immediately heal him.

The servent went to search for Jesus and found him, preaching the Good news of Salvation being fulfilled in himself and stayed, kept himself on a side and tried to paint Jesus face on a number of times as each time he started painting the face of Jesus in a very short while the face has changed so he couldn't complete it and have to start again.
After some time Jesus has called him took the cloth (napkin) and placed the cloth on his face and an image depicting has face has imprinted on the cloth.

The Image of Christ on the cloth was brought to the King and as Abgar kissed the Image on the napkin with Love and Faith and the sickness disappeared as he was completely cured, only a small part of his face was still having the trace of the disease. After this glorious Miracle King Abgar ordered that the idols on the Entry Gate of the City that were believed to protect the city from Evil should be and in their place the Non-Hand made image of Jesus Christ to be put on top of the city Entry Gate having the Image stuck onto wood, surrounded with a gold frame and ornamented with pearls. Prince Abgar also wrote above the icon on the gateway:
 

"O Christ our God, no-one who hopes in Thee will be put to shame" !

 

, to let everyone know about the Great power of the one who is depicted on the Mandilyon cloth, just as well as a protection sign for the city from all evil and invaders as he was convinced the image of Christ had the power to not only heal him but had the Power to protect his city from the numerous barbaryan raids that were so common during Ist century.

In the coming centuries The Non Hand Made Image of God have been said to have cured multiple people from all kind of diseases.

Abgar-with-image-of-Edessa10thcentury-Monastery-Saint_Ekaterina-Sinai

Abgar receiving the Non Hand Made Image of Christ cloth icon – Saint Catherin (Ekaterina) Monastery Sinai 10th century

According to the eastern Tradition, Jesus has said to the survent to convey message to the the King that this Image of Him will cure most of the disease of the king but the complete healing will be completed by Jesus's desciple Thaddaeus (who was one of the 70th Apostles). This letter was brought to King Abgar too. The Orthodox Tradition believed in the Church is that the words of Jesus fulfilled, King Abgar biggest part of body lepper buds, have been healed and only a small part on his face remaimed and this was also healed by disciple Thaddaeus who after the Crucifix and Resurrection of Christ in the 3rd day after his execution went to the King and have preached the Good news of Gospel that Christ has resurrected and asked the king to repent and shortly baptized him in the New Christian faith In the Name of the Father, The Son and The Holy Spirit. Immediately after Baptism King Abgar become completely healed. More on the Story for the Image Not Made By Hands can be red here.

 

08.21_saint_ap_Thaddeus_rex_Abgar_ikona_Sinai_Sv_Ekaterina
Saint Apostle Thaddeus arriving to King Abgar V of Eddessa and healing – icon X-th Saint Catherin's Monastery Sinai

The Jesus Image of Edessa – a Syrian city in upper Mesopotamia on the Banks of Euphrates  (today city of Urfa in Turkey), painted copy has been made on number of occasions through the history before the original clothed image disappeared mysteriously in the 13th century as it was stolen by the Roman Catholic Crusaders during IV-th Crusade whose goal was the liberation of the Holy Lands (Jerusalem and the other biblical cities from the Muslim) in year 1204.  After the hungry and and mad crusaders destroed (sacked out) Constantinople they took the Mandylion have reappeared as a relic in King Louis IX of France's Sainte-Chapelle Church in Paris and has been said to have been there until its total disappearance in the French Revolion (during which many holy relics were stolen by revolutionists, many of whose are known to have been Masons as well as many relics were destroyed).

This Non-Hand Made image of Jesus Christ known in Church Slavonic under the term (Убрус / Ubrus) is considered the first icon ("image"). In the Eastern Orthodox Church (Bulgarian, Russian, Greek, Serbian, Macedonian, Moldovian, Syrian, Antiochian, Jerusalem's Churches) and the rest of Oriental Orthodox Churches (Armenian, Copts, Syriacs, Jacobites etc.) according to Church Tradition.

Christos_Acheiropoietos-Non-hand-made-image-of-Jesus-Christ-given-to-King-Abgar

Christos Acheiropoietos (Non-hand-made-image)  of Jesus Christ given to King Abgar of Edessa
(Novgorodian Russian Icon circa 1100)

 

The Eastern Orthodox Church observes a feast for this icon on August 16 (August 29 in N.S.), which commemorates its translation from Edessa to Constantinople and is being observed with a Church service Holy Liturgy.

Ancha_Icon_of_the_Savior_(Art_Museum_of_Georgia,_Tbilisi)

Ancha Icon of the Savior ანჩისხატი (Traditionally considered to have been the tile Kiramidion a tile copy of the Mandylion) – "holy tile" imprinted with the face of Jesus Christ miraculously transferred by contact with the Image of Edessa (Mandylion).

Ancha icon is dated to the 6th-7th century, it was covered with a chased silver riza and partly repainted in the following centuries. The icon derives its name from the Georgian monastery of Ancha in what is now Turkey, whence it was brought to Tbilisi in 1664.

Holy Face of Genoa

Mandylion image is also point of high interest in the Roman Catholic Church too as well as to some more conservative Protestant denominations (though perhaps most of them are rejecting that story as a pious myth.).
In Western Tradtion the Ubrus's copy there is the famous Holy Face of Genoa (which is a donation by the Byzantine Emperor John V Paleologos to Doge of Genos Leonardo Montaldo in the 14th century it was carefully studied by Colette Dufour Bozzo that the image is on cloth that is on a wooden board (the wooden board is a typical plane on which icons are being painted) and is dated also to 14th century.

Jesus_Christ_Face_Holy-face-of-Genoa-_Genes

Holy face of Genoa 14th Century with face made more visible

 

Holy Face of San Silvestro

 

Holy Face of San Silvestro (Saint Silvester) image was kept in Rome's Church San Silvestro in Capite and is now
kept in Matilda chapel Vatican Palace. Its earliest existence known is from 1517 (at that year the nuns were forbidden to exhibit it) as people often has mixed it up with the Veronica Veil (another famous Jesus cloth face imprint often called Volto Santo – holy face).

The_San_Silvestro-image-imprint-of-Jesus-non-hand-made-Mandylion_visage

 

Veronica Veil

 

The Western Roman Catholic tradition recounts that Saint Veronica from Jerusalem encountered Jesus along the Via Dolorosa on the way to Calvary. When she paused to wipe the blood and sweat (Latin sudor) off his face with her veil, his image was imprinted on the cloth. The Veronica Veil tradition however is legendary and not accepted in Eastern Orthodox Churches as this tradition is quite new first to occur somewhere in the middle ages, like in the 16th century and was not known at all in Christiandome prior to that.

The act of Saint Veronica wiping the face of Jesus with her veil is celebrated in the sixth Station of the Cross in many Anglican, Catholic, Lutheran rites, Methodist and Western Orthodox churches.

 

Image of the Saviour Other traditional Orthodox Copy icons

It is notable to mention few very famous Russian iconography interpretation, many of the interpretations are taken by the Russian Iconography from the sample Novgorodian Russian Icon which of itself has been an exact copy, the Russian and Greek monks have made of the Origianl Non Hand Made Cloth image that has been on top of Byzantine Eastern Empire Capital Constantinople's city hanging on the entry Gate.

Saviour-Wet-Beard-Spas_Mokraya_Brada_ikona-Russian-icon


Спас Мокрая Брада (Spas Mokraja Brada – The Saviour Wet Beard) Russian Orthodox Icon 16th Century

Even today, many of the Eastern Orthodox Church are having a copy interpretation of the icon hanging on top of the Dveri (The Inner part Alter Church Doors) – see below picture for reference:

Ubrus-Mandylion-Non-Hand-Made-icon-Hanging-upper-to-Church-Alter-Walls-Dveri-Eastern-Orthodox-Church-Vlaherna-Tsarigrad-Istanbul-ex-Konstantinopol

Non-Hand Made Image copy of the Saviour Jesus Christ hanging on top of Church Alter Walls (Dveri) on top of the priest head

Image-of-the-Saviour-traditional-Orthodox-Iconography-Simon_Ushakov_Nerukotvorniy

 

The Mandylion by Simeon Ushakov – year 1658.

As well as the bit newer but very beautiful Russian Iconograph interpretation  of  Image of Saviour from Harkov.

Harkovskij-Spas-Saviour-Orthodox-icon-18th-century-Harkov

Harkovskij Spas icon Harkov Saviour from 18th century Russian Orthodox Icon

Helpful Hints For Starting A Small WordPress Website or Ecomerce Business


August 14th, 2019

hints-for-starting-wordpress-site

Wordpress is the web application collection of PHP program behind thirty four percent (43%) of the internet’s websites, and fifteen percent (50%) of the top one hundred websites in the world, so if you’re considering it for your website then you’re perhaps thinking in the right direction. Small start-up projects a community website or even a small personal owned blog or mid to even large business presentation site  can benefit greatly from setting up their Web Platrform or Ecommerce shops on a WordPress website platform (that of itself depends just on a small number of technologies such as a Linux server with a Web Server installed on it to serve PHP as well as some kind of Linux host installed Database  backend engine such as MYSQL / PostgreSQL etc. …

But if you really want to create a successful ecommerce website on WordPress, that can seem a little intimidating at first as the general complexity to start up with WordPress looks very scary in the beginning. However in this article I’ll point to fewhelpful hints should get you off on the right foot, and make your entry into the world of Wodpress / WP Ecommerce a little easier and less scary.

This article is to be less technical than expected and in that will contrast slightly with many of the articles on this blog, the target audience is more of Web Marketing Manager or a Start-up Search Engine Optimization person at a small personal project or employed in the big bad corporate world.This is no something new that is going to be outlined in this article but a general rules that are known for the professional SEO Gurus but is most likely to be helpful for the starting persons.

If you happen to be one of these you should know you have to follow a set of well known rules on the website structure text, descriptions, text, orientation, ordering of menus and data etc. in order to have the WordPress based website running at full speed attracting more visitors to your site.
 

Photos
 

 

Importance of Photos on a Webiste
Although the text for your website is very important – more on that later – when a user first opens up your website in their browser, their eyes are going to be caught by the images that you have laid out on your website. Not using images is a big mistake, since it bores users’ eyes and makes your website seem amateur and basic, but using low quality images or irrelevant images can also harm your chances of appearing authentic to a user (yes here on this blog there are some of this low quality pictures but this is due to fact this website is more of information blog and not ecommerce. Thus at best case always make sure that you find the best, high-quality images for your website – make sure that you have the correct rights to use the images as well (as copyright infrignmenets) could cause you even a law suits ending in hundred or thousand dollar fines or even if this doesn't happen any publicity of such would reduce your website indexing rating. The images placed should always be relevant to your website. If you find a breath-taking sunset or tech-gadget picture, that’s great, but maybe not for your healthy food ecommerce store, but for your personal ranting or describing a personal experience.

 

Product Photos


Assuming that sooner or later even if you have a community website you will want to monerize it to bring back to yourself in material form at least part of the many years effort to bring the site to the web rank gained.
Leading on from that point, you’re going to be selling or advertise items – that’s the whole point of ecommerce. But users often find ads / online shopping frustrating due to not being able to properly see and understand what they’re buying before they make their purchase. This can lead to ‘buyer’s remorse’, and, consequently, refunds galore, which is not what you want. Make sure that images of your products are always available and of a high quality – investing in a fairly high quality camera might be a good idea – and consider many pictures for different angles or even rotating images so that the user can decide for themself which angle they want to look at.

 

Engaging Descriptions


“I can guarantee that you can’t remember the last five product descriptions you read – not even word-for-word, but the general ideas and vocabulary used will have been tossed into your short-term memory and forgotten in an instant. This is where your website can shine, and become better than ninety percent of those lingering on the internet,” Matthew Kelly, a project manager at WriteMyX and NextCoursework, suggests, “since putting effort into writing your product descriptions and making them lively and engaging will make your website memorable, and your subscribers will turn helpfully soon loyal customers will be more likely to come back time and time again and become repeat business, as well as mention you to their friends (social mounth to mouth marketing) and that way working as free advertising for you and making your website incredibly effective.”

 

Mobile-Friendly

 

Which device is most used to check email Laptop / PC or Mobile statistics as of year 2019

These days with the bloom of Mobile Devices that are currently overrunning the user of normal Desktop PCs, Laptops and Tablets and this trend is likely to stay and even increase, “If your website isn’t mobile-friendly in this day and age, then you won’t get anywhere with it.” Anne Baker, a marketer at BritStudent and Australia2Write, states. “Most people use their phones when they access websites, especially when they go shopping on the internet.

Statistics on user stay (secs / mins) stay on a website from Desktop PC and Mobile devices

On WordPress, this means finding a more recent theme – an older theme, maybe four-five years old, will probably not support mobile, and you just can’t afford to lose out on the mobile market.” In short, find yourself a mobile-friendly theme or install the right WordPress Pluguin that will enable you to have a Mobile Friendly theme in case if blog is accessed from a Mobile Dev or many of your customers will become frustrated with the badly formatted ‘mobile’ version of your website that they end up using, which might be for instance meant for a much larger screen. It can also ruin the atmosphere (experience) created at the accessed user site and have negative impact on your audience opion of your site or business. This is even more the case  if your website or webapp is targetting to be modern and keeping with the times – or especially if it deals with IT and electronics (where the competition is huge)!

 

Registration

 

Registration Ecommerce website

Registration form (Sign Up) on a website and the overall business cycle idea behind web product or business is of critical importance as this is the point that will guarantee intimidation with the customer, failing to have the person be engaged will quickly make your website rank lower and your producs less wanted. The general rule here is to make your registration be an easy (to orientate for the user) and be present on a very visible place on the site.

Registration steps should be as less as possible as this might piss off the user and repel him out of the site before the registration is completed. Showing oportunity to register with a Pop-Up window (while the user clicks on a place showing interest for the produce might be useful in some cases but generally might also push the user back so if you decide to implement it do it with a lot of care (beware of too much aggressive marketing on our site).

An example


The registration process should be as intimidating as possible to leave joy in the user that might later return and log in to your site or ecommerce platform, e.g. be interested to stay for a longer time. The marketing tactic aiming to make the user stay for a longer time on the website (dragging his attention / interest to stuff)  is nothing new by the way as it is well known marketing rule integrated in every supermarket you buy groceries, where all is made to keep you in the shop for as longer as possible. Research has shown that spending longer time within the supermarket makes the user buy more.

 

Returning customers can be intimidated with membership or a free gift (be it even virtual picture gift – free email whatever) or information store place could be given or if products are sold, registration will be obligatory to make them use their payment method or delivery address on next login to easify the buy out process. But if registration is convoluted and forced (e.g. user is somehow forced to become meber) then many customers will turn away and find another website for their shopping needs. Using a method like Quora’s ‘login to see more’ in that case might be a good idea even though for me this is also a very irritating and irritating – this method however should never be used if you run a ecommerce selling platform, on ecommerce site gatekeeping will only frustrate customers. Login is good to be implmeneted as a popup option (and not taking too much of the screen). Sign up and Login should be simplistic and self-explanatory – always not required but optioned and user should get the understanding of the advantage to be a member of the website if possible before the sign up procedure. Then, customers are more likely to sign up and won’t feel like they’ve been pushed into the decision – or pushed away, as the case may be.

Katrina Hatchett works as a lifestyle blogger at both Academic Brits and Assignment Help, due to a love of literature and writing, which she has had since youth. Throughout her career, she has become involved with many projects, such as writing for the PhD Kingdom blog.

Howto debug and remount NFS hangled filesystem on Linux


August 12th, 2019

nfsnetwork-file-system-architecture-diagram

If you're using actively NFS remote storage attached to your Linux server it is very useful to get the number of dropped NFS connections and in that way to assure you don't have a remote NFS server issues or Network connectivity drops out due to broken network switch a Cisco hub or other network hop device that is routing the traffic from Source Host (SRC) to Destination Host (DST) thus, at perfect case if NFS storage and mounted Linux Network filesystem should be at (0) zero dropped connectios or their number should be low. Firewall connectivity between Source NFS client host and Destination NFS Server and mount should be there (set up fine) as well as proper permissions assigned on the server, as well as the DST NFS should be not experiencing I/O overheads as well as no DNS issues should be present (if NFS is not accessed directly via IP address).
In below article which is mostly for NFS novice admins is described shortly few of the nuances of working with NFS.
 

1. Check nfsstat and portmap for issues

One indicator that everything is fine with a configured NFS mount is the number of dropped NFS connections
or with a very low count of dropped connections, to check them if you happen to administer NFS

nfsstat

 

linux:~# nfsstat -o net
Server packet stats:
packets    udp        tcp        tcpconn
0          0          0          0  


nfsstat is useful if you have to debug why occasionally NFS mounts are getting unresponsive.

As NFS is so dependent upon portmap service for mapping the ports, one other point to check in case of Hanged NFSes is the portmap service whether it did not crashed due to some reason.

 

linux:~# service portmap status
portmap (pid 7428) is running…   [portmap service is started.]

 

linux:~# ps axu|grep -i rpcbind
_rpc       421  0.0  0.0   6824  3568 ?        Ss   10:30   0:00 /sbin/rpcbind -f -w


A useful commands to debug further rcp caused issues are:

On client side:

 

rpcdebug -m nfs -c

 

On server side:

 

rpcdebug -m nfsd -c

 

It might be also useful to check whether remote NFS permissions did not changed with the good old showmount cmd

linux:~# showmount -e rem_nfs_server_host


Also it is useful to check whether /etc/exports file was not modified somehow and whether the NFS did not hanged due to attempt of NFS daemon to reload the new configuration from there, another file to check while debugging is /etc/nfs.conf – are there group / permissions issues as well as the usual /var/log/messages and the kernel log with dmesg command for weird produced NFS client / server or network messages.

nfs-utils disabled serving NFS over UDP in version 2.2.1. Arch core updated to 2.3.1 on 21 Dec 2017 (skipping over 2.2.1.) If UDP stopped working then, add udp=y under [nfsd] in /etc/nfs.conf. Then restart nfs-server.service.

If the remote NFS server is running also Linux it is useful to check its /etc/default/nfs-kernel-server configuration

At some stall cases it might be also useful to remount the NFS (but as there might be a process on the Linux server) trying to read / write data from the remote NFS mounted FS it is a good idea to check (whether a process / service) on the server is not doing I/O operations on the NFS and if such is existing to kill the process in question with fuser
 

linux:~# fuser -k [mounted-filesystem]
 

 

2. Diagnose the problem interactively with htop


    Htop should be your first port of call. The most obvious symptom will be a maxed-out CPU.
    Press F2, and under "Display options", enable "Detailed CPU time". Press F1 for an explanation of the colours used in the CPU bars. In particular, is the CPU spending most of its time responding to IRQs, or in Wait-IO (wio)?
 

3. Get more extensive Mount info with mountstats

 

nfs-utils package contains mountstats command which is very useful in debugging further the issues identified

$ mountstats
Stats for example:/tank mounted on /tank:
  NFS mount options: rw,sync,vers=4.2,rsize=524288,wsize=524288,namlen=255,acregmin=3,acregmax=60,acdirmin=30,acdirmax=60,soft,proto=tcp,port=0,timeo=15,retrans=2,sec=sys,clientaddr=xx.yy.zz.tt,local_lock=none
  NFS server capabilities: caps=0xfbffdf,wtmult=512,dtsize=32768,bsize=0,namlen=255
  NFSv4 capability flags: bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x803,acl=0x3,sessions,pnfs=notconfigured
  NFS security flavor: 1  pseudoflavor: 0

 

NFS byte counts:
  applications read 248542089 bytes via read(2)
  applications wrote 0 bytes via write(2)
  applications read 0 bytes via O_DIRECT read(2)
  applications wrote 0 bytes via O_DIRECT write(2)
  client read 171375125 bytes via NFS READ
  client wrote 0 bytes via NFS WRITE

RPC statistics:
  699 RPC requests sent, 699 RPC replies received (0 XIDs not found)
  average backlog queue length: 0

READ:
    338 ops (48%)
    avg bytes sent per op: 216    avg bytes received per op: 507131
    backlog wait: 0.005917     RTT: 548.736686     total execute time: 548.775148 (milliseconds)
GETATTR:
    115 ops (16%)
    avg bytes sent per op: 199    avg bytes received per op: 240
    backlog wait: 0.008696     RTT: 15.756522     total execute time: 15.843478 (milliseconds)
ACCESS:
    93 ops (13%)
    avg bytes sent per op: 203    avg bytes received per op: 168
    backlog wait: 0.010753     RTT: 2.967742     total execute time: 3.032258 (milliseconds)
LOOKUP:
    32 ops (4%)
    avg bytes sent per op: 220    avg bytes received per op: 274
    backlog wait: 0.000000     RTT: 3.906250     total execute time: 3.968750 (milliseconds)
OPEN_NOATTR:
    25 ops (3%)
    avg bytes sent per op: 268    avg bytes received per op: 350
    backlog wait: 0.000000     RTT: 2.320000     total execute time: 2.360000 (milliseconds)
CLOSE:
    24 ops (3%)
    avg bytes sent per op: 224    avg bytes received per op: 176
    backlog wait: 0.000000     RTT: 30.250000     total execute time: 30.291667 (milliseconds)
DELEGRETURN:
    23 ops (3%)
    avg bytes sent per op: 220    avg bytes received per op: 160
    backlog wait: 0.000000     RTT: 6.782609     total execute time: 6.826087 (milliseconds)
READDIR:
    4 ops (0%)
    avg bytes sent per op: 224    avg bytes received per op: 14372
    backlog wait: 0.000000     RTT: 198.000000     total execute time: 198.250000 (milliseconds)
SERVER_CAPS:
    2 ops (0%)
    avg bytes sent per op: 172    avg bytes received per op: 164
    backlog wait: 0.000000     RTT: 1.500000     total execute time: 1.500000 (milliseconds)
FSINFO:
    1 ops (0%)
    avg bytes sent per op: 172    avg bytes received per op: 164
    backlog wait: 0.000000     RTT: 2.000000     total execute time: 2.000000 (milliseconds)
PATHCONF:
    1 ops (0%)
    avg bytes sent per op: 164    avg bytes received per op: 116
    backlog wait: 0.000000     RTT: 1.000000     total execute time: 1.000000 (milliseconds)


nfs-utils disabled serving NFS over UDP in version 2.2.1. Arch core updated to 2.3.1 on 21 Dec 2017 (skipping over 2.2.1.) If UDP stopped working then, add udp=y under [nfsd] in /etc/nfs.conf. Then restart nfs-server.service.
 

4. Check for firewall issues
 

If all fails make sure you don't have any kind of firewall issues. Sometimes firewall changes on remote server or somewhere in the routing servers might lead to stalled NFS mounts.

 

To use properly NFS as you should know as a minimum you need to have opened as ports is Port 111 (TCP and UDP) and 2049 (TCP and UDP) on the NFS server (side) as well as any traffic inspection routers on the road from SRC (Linux client host) and NFS Storage destination DST server.

There are also ports for Cluster and client status (Port 1110 TCP for the former, and 1110 UDP for the latter) as well as a port for the NFS lock manager (Port 4045 TCP and UDP) but having this opened or not depends on how the NFS is configured. You can further determine which ports you need to allow depending on which services are needed cross-gateway.
 

5. How to Remount a Stalled unresponsive NFS filesystem mount

 

At many cases situation with remounting stalled NFS filesystem is not so easy but if you're lucky a standard mount and remount should do the trick.

Most simple way to remout the NFS (once you're sure this might not disrupt any service) – don't blame me if you break something is with:
 

umount -l /mnt/NFS_mnt_point
mount /mnt/NFS_mnt_point


Note that the lazy mount (-l) umount opt is provided here as very often this is the only way to unmount a stalled NFS mount.

Sometimes if you have a lot of NFS mounts and all are inacessible it is useful to remount all NFS mounts, if the remote NFS is responsive this should be possible with a simple for bash loop:

for P in $(mount | awk '/type nfs / {print $3;}'); do echo $P; echo "sudo umount $P && sudo mount $P" && echo "ok :)"; done


If you cd /mnt/NFS_mnt_point and try ls and you get

$ ls
.: Stale File Handle

 

You will need to unmount the FS with forceful mount flag

umount -f /mnt/NFS_mnt_point
 

Sum it up


In this article, I've shown you a few simple ways to debug what is wrong with a Stalled / Hanged NFS filesystem present on a NFS server mounted on a Linux client server.
Above was explained the common issues caused by NFS portmap (rpcbind) dependency, how to its status is fine, some further diagnosis with htop and mountstat was pointed. I've pointed the minimum amount of TCP / UDP ports 2049 and 111 that needs to be opened for the NFS communication to work and finally explained on how to remount a stalled NFS single or all attached mount on a NFS client to restore to normal operations.
As NFS is a whole ocean of things and the number of ways it is used are too extensive this article is just a general info useful for the NFS dummy admin for more robust configs read some good book on NFS such as Managing NFS and NIS, 2nd Edition – O'Reilly Media and for Kernel related NFS debugging make sure you check as a minimum ArchLinux's NFS troubleshooting guide and sourceforge's NFS Troubleshoting and Optimizing NFS Performance guides.

 

How to View and Delete NetApp Storage qtree, Get statistics about Filer Volume Read / Writes operations and delete and show mounted volumes


August 2nd, 2019

how-to-delete-volume-qtree-snapmirror-view-netapp-volume-qtree-and-and-view-netapp-cluster-device-statistics-NetAppLogo

I've had recently the trivial decomissioning task to delete some NetApp Storage qtrees on some of the SAP Hana Enterprise Cloud NetApp filers.
If it is first time you heard of NetApp is a hybrid cloud data services and data management (ranked in the Fortune 500 companies).

Netapps are hybrid cloud data services for management of applications and data across cloud and on-premises environments and are a de-facto standard for Data storage on many of the existing Internet Clouds and Large Corporatons that Stores many Pentabytes of Data.

The Netapp storage devices are a kinda of proprietary Clustered version of the Small business NAS storage Solution FreeNAS (which of itself is a Free FreeBSD based Data Storage OS – The #1 Storage OS).
NetApps allow plenty of things to do such as Data Mirroring (Data Backups), Data Syncing, SpapMirroring, SnapVault and many, many more custom Data revolutionary solutions such as StorageGrid.

NetApp supports integration with Kubernetes, Docker, Oracle / SAP DB, Citrix, Xen, KVM as well as multiple cloud environments such as AWS, Azure, OpenStack and has even integration with some CI/CD DevOps data provisioning – i.e. Jenkins.

In this small article, I'll show you how a Volume / Qtree on a NetApp filer could be viewed, mounted, unmounted, deleted. I'll also show you how to get statistics, while logged in remotely to the NetApp console and finally how to view and delete a NetApp configured snapmirror.

 

View NetApp Qtree

 

Here is how to view the Storage Qtree:

netapp> qtree show -vserver netapp01fv018 -volume VOL_OS_MIG -qtree bck_01v046485_20190108

To view the file content existing on the Storage server from the Linux bost next step to do is mount it with regular mount:

linux-host:~# mount netappfiler01fv018:/VOL_OS_MIG/ bck_01v035527_20190108 /mnt/test

 

Delete the Qtree from NetApp (Storage) Filer

Become administrator on the device

Once assured the content could go on to delete the qtree, it is necessery to become superuser (root) on the NetApp device, to do so, I hed to type:

 

netapp> set -privilege advanced

 

Then to delete the unneded volume previously used for transferring system update files, when logged in via SSH to the NetApp device – ONTAP Proprietary Operating system :

 

netapp> qtree delete -vserver netapp01fv018 -volume VOL_OS_MIG -force -qtree bck_01v035527_20190108


Note that this command will return back a job ID
assigned until operation is completed, to check the status of completion of generated JOB that is backgrounded, I've used command:

netapp> job show 53412

If all is okay you should get a Status of Success otherwise, if you get failed status you have to debug further what's causing it.
 

How to view existing export polcities and remove them

 

If you don't want to delete the qtree or volume but want to prevent a certain Linux server / application to not have access to it, it is useful to view existing export policy for a qtree.
 

netappfc001::> qtree show -exports -volume vol1_vmspace_netapp01v000885 -qtree q_01v002131
                                                   Is Export
Vserver    Volume        Qtree        Policy Name  Policy Inherited
———- ————- ———— ———— —————–
netapp01fv001 vol1_vmspace_netapp01v000885
                         q_01v002131
                                      vol1_vmspace_netapp01v000885.exports
                                                   true

 


To remove then export policy (to not exist at all), this is how:

 

 

netapp> volume qtree modify -vserver hec01fv018 -qtree-path /vol/volume_name/qtree_name -export-policy ""

 

I've also found the following volume qtree commands NetApp ONTAP documentation page helpful to read and recommend to anyone that wants to learn more.
 

How to delete a NetApp Volume if it is not used anymore

To delete unsed netapp volume, you have to do 3 things.
1. Unmount the volume
2. Put it offline
3. Delete it

to do so run below 3 cmds:

 

netapp> volume unmount -vserver vserver_name -volume volume_name
netapp> volume offline -vserver vserver_name volume_name
netapp> volume delete -vserver vserver_name volume_name

 

Show mounted Volume junctions (Get Extra Storage Volume information)

 

netapp> volume show -vserver netapp01fv004 -junction
netapp> volume show -vserver netapp01fv004 -volume MUFCF01_BACKUP

 

How to delete a Configured SnapMirror

What is a snapmirror?

 

Recovery-Scenario-Restore-Changes-To-Recovery-site-snapmirror-diagram

SnapMirror is a feature of Data ONTAP that enables you to replicate data. SnapMirror enables you to replicate data from specified source volumes or qtrees to specified destination volumes or qtrees, respectively. You need a separate license to use SnapMirror.

You can use SnapMirror to replicate data within the same storage system or with different storage systems.

After the data is replicated to the destination storage system, you can access the data on the destination to perform the following actions:

  • You can provide users immediate access to mirrored data in case the source goes down.
  • You can restore the data to the source to recover from disaster, data corruption (qtrees only), or user error.
  • You can archive the data to tape.
  • You can balance resource loads.
  • You can back up or distribute the data to remote sites.

 

netapp> snapmirror show -destination-path netapp02fv001:vol1_MUF_PS1_DR

 

netapp> snapmirror delete -destination-path netapp02fv001:vol1_MUF_PS1_DR -force
Operation succeeded: snapmirror delete for the relationship with destination "hec02fv001:vol1_MUF_PS1_DR".
 

If the snapmirror deletion gets scheduled you can use snapmirror status command to check status:
 

netapp> snapmirror status MUF_PS1_PRD
Snapmirror is on.

 

How to telnet from NetApp Storage to another one / check status of configured SMTPs for NetApp Cluster (filer)

 

 

You can use the autosupport and options autosupport commands to change or view AutoSupport configuration, display information about past AutoSupport messages, and send or resend an AutoSupport message.

For example if NetApp Filers have configured SMTP or SMTPs servers or other Proxy Configurations to pass on traffic from DMZ-ed network to external Internet resources or Relay servers this command will provide information on the Connection status of this remote services.

 

rows 0
set diag
node show

autosupport check show
systemshell -node netapp01f0018 -c telnet
autosupport show -fields proxy-url
systemshell -node netappf0018 -c telnet  147.204.148.38 80

netapp09fc001::*> systemshell -node  netapp08f0013 -c telnet  8080
  (system node systemshell)
Trying 100.127.20.4


node show – will provide information about configured nodes
rows 0 – will set the output print rows how they will be displayed
set – diag sets the device in diagnostic state

As you can see you can use the systemshell netapp command to try out telnet connections from the Configured NetApp logged in Source to any remote destination to make sure the set Proxy or SMTP is properly reachable.

How to get Statistics about NetApp existing volume Read / Write operations

 

On Netapp side issue:

netapp> statistics volume show -interval 5 -iterations 1 -max 25 -vserver netapp01fv004 -volume MUFCF01_BACKUP

For people starting up with NetApps, it is very useful to get a in-depth read on quick and dirty –  Netapp Commandline CheatSheet (for simplicity I've stored it in netapp-commands-cheatsheat.txt formatted file here ).

Conclusion

NetApp storages are used in many Governments and Large Corporations and for critical applications with SLAs forfeits for million bucks, mostly for applications and Database storage that are of a very large scale and too critical to be handled by the conventional storage computing of simple RAIDS 1,2,3,5,6 etc. / LVM and so on. ONTAP and NetApp Filers and Filer Clusters, are easy to maintain but due to its high number of features, not many NetApp Storage / Backup system administrators have the knowledge how to take a good advantage of this beasts. Thus finally, my even small experience with them shows that even simple things as critical errors are not handled properly at least that was my experience as a SAP consultant with SAP Hana Enterprise Cloud (HEC) and their HANA Converged Cloud where, main storage. 
This article's goal was pretty simple to guide the user on a minimum set of commands for simple qtree / volume / snapmirror view and removal decomissioning tasks. NetApps Clusters are a whole ocean of stuff and knowledge so before doing anything complex, if you're not sure what you're doing always consult a NetApp storage sysadmin as some of this animals features looks easy for the common general sysadmin but not are not so.
 

Why du and df reporting different on a filesystem / How to fix inconsistency between used space on FS and disk showing full strangeness


July 24th, 2019

linux-why-du-and-df-shows-different-result-inconsincy-explained-filesystem-full-oddity

If you're a sysadmin on a large server environment such as a couple of hundred of Virtual Machines running Linux OS on either physical host or OpenXen / VmWare hosted guest Virtual Machine, you might end up sometimes at an odd case where some mounted partition mount point reports its file use different when checked with
df
cmd than when checked with du command, like for example:
 

root@sqlserver:~# df -hT /var/lib/mysql
Filesystem   Type  Size Used Avail Use% Mounted On
/dev/sdb5      ext4    19G  3,4G    14G  20% /var/lib/mysql

Here the '-T' argument is used to show us the filesystem.

root@sqlserver:~# du -hsc /var/lib/mysql
0K    /var/lib/mysql/
0K    total

 

1. Simple debug on what might be the root cause for df / du inconsistency reporting

 

Of course the basic thing to do when in that weird situation is to be totally shocked how this is possible and to investigate a bit what is the biggest first level sub-directories that eat up the space on the mounted location, with du:

 

# du -hkx –max-depth=1 /var/lib/mysql/|uniq|sort -n
4       /var/lib/mysql/test
8       /var/lib/mysql/ezmlm
8       /var/lib/mysql/micropcfreak
8       /var/lib/mysql/performance_schema
12      /var/lib/mysql/mysqltmp
24      /var/lib/mysql/speedtest
64      /var/lib/mysql/yourls
144     /var/lib/mysql/narf
320     /var/lib/mysql/webchat_plus
424     /var/lib/mysql/goodfaithair
528     /var/lib/mysql/moonman
648     /var/lib/mysql/daniel
852     /var/lib/mysql/lessn
1292    /var/lib/mysql/gallery

The given output is in Kilobytes so it is a little bit hard to read, if you're used to Mbytes instead, do

 

 # du -hmx –max-depth=1 /var/lib/mysql/|uniq|sort -n|less

 

I've also investigated on the complete /var directory contents sorted by size with:

 

 # du -akx ./ | sort -n
5152564    ./cache/rsnapshot/hourly.2/localhost
5255788    ./cache/rsnapshot/hourly.2
5287912    ./cache/rsnapshot
7192152    ./cache


Even after finding out the bottleneck dirs and trying to clear up a bit, continued facing that inconsistently shown in two commands and if you're likely to be stunned like me and try … to move some files to a different filesystem to free up space or assigned inodes with a hope that shown inconsitency output will be fixed as it might be caused  due to some kernel / FS caching ?? and this will eventually make the mounted FS to refresh …

But unfortunately, if you try it you'll figure out clearing up a couple of Megas or Gigas will make no difference in cmd output.

In my exact case /var/lib/mysql is a separate mounted ext4 filesystem, however same issue was present also on a Network Filesystem (NFS) and thus, my first thought that this is caused by a network failure problem or NFS bug turned to be wrong.

After further short investigation on the inodes on the Filesystem, it was clear enough inodes are available:
 

# df -i /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5      1221600  2562 1219038   1% /var/lib/mysql

 

So the filled inodes count assumed issue also has been rejected.
P.S. (if you're not well familiar with them read manual, i.e. – man 7 inode).
 

– Remounting the mounted filesystem

To make sure the filesystem shown inconsistency between du and df is not due to some hanging network mount or bug, first logical thing I did is to remount the filesytem showing different in size, in my case this was done with:
 

# mount -o remount,rw -t ext4 /var/lib/mysql

For machines with NFS remote mounted storage locations, used:

# mount -o remount,rw -t nfs /var/www


FS remount did not solved it so I continued to ponder what oddity and of course I thought of a workaround (in case if this issues are caused by kernel bug or OS lib issue) reboot might be the solution, however unfortunately restarting the VMs was not a wanted easy to do solution, thus I continued investigating what is wrong …

Next check of course was to check, what kind of network connections are opened to the affected hosts with:
 

# netstat -tupanl


Did not found anything that might point me to the reported different Megabytes issue, so next step was to check what is the situation with currently opened files by running processes on the weird df / du reported systems with lsof, and boom there I observed oddity such as multiple files

 

# lsof -nP | grep '(deleted)'

COMMAND   PID   USER   FD   TYPE DEVICE    SIZE NLINK  NODE NAME
mysqld   2588  mysql    4u   REG 253,17      52     0  1495 /var/lib/mysql/tmp/ibY0cXCd (deleted)
mysqld   2588  mysql    5u   REG 253,17    1048     0  1496 /var/lib/mysql/tmp/ibOrELhG (deleted)
mysqld   2588  mysql    6u   REG 253,17       777884290     0  1497 /var/lib/mysql/tmp/ibmDFAW8 (deleted)
mysqld   2588  mysql    7u   REG 253,17       123667875     0 11387 /var/lib/mysql/tmp/ib2CSACB (deleted)
mysqld   2588  mysql   11u   REG 253,17       123852406     0 11388 /var/lib/mysql/tmp/ibQpoZ94 (deleted)

 

Notice that There were plenty of '(deleted)' STATE files shown in memory an overall of 438:

 

# lsof -nP | grep '(deleted)' |wc -l
438


As I've learned a bit online about the problem, I found it is also possible to find deleted unlinked files only without any greps (to list all deleted files in memory files with lsof args only):

 

# lsof +L1|less


The SIZE field (fourth column)  shows a number of files that are really hard in size and that are kept in open on filesystem and in memory, totally messing up with the filesystem. In my case this is temp files created by MYSQLD daemon but depending on the server provided service this might be apache's www-data, some custom perl / bash script executed via a cron job, stalled rsync jobs etc.
 

2. Check all the list open files with the mysql / root user as part of the the server filesystem inconsistency debugging with:

 

– Grep opened files on server by user

# lsof |grep mysql
mysqld    1312                       mysql  cwd       DIR               8,21       4096          2 /var/lib/mysql
mysqld    1312                       mysql  rtd       DIR                8,1       4096          2 /
mysqld    1312                       mysql  txt       REG                8,1   20336792   23805048 /usr/sbin/mysqld
mysqld    1312                       mysql  mem       REG               8,21      24576         20 /var/lib/mysql/tc.log
mysqld    1312                       mysql  DEL       REG               0,16                 29467 /[aio]
mysqld    1312                       mysql  mem       REG                8,1      55792   14886933 /lib/x86_64-linux-gnu/libnss_files-2.28.so

 

# lsof | grep root
COMMAND    PID   TID TASKCMD          USER   FD      TYPE             DEVICE   SIZE/OFF       NODE NAME
systemd      1                        root  cwd       DIR                8,1       4096          2 /
systemd      1                        root  rtd       DIR                8,1       4096          2 /
systemd      1                        root  txt       REG                8,1    1489208   14928891 /lib/systemd/systemd
systemd      1                        root  mem       REG                8,1    1579448   14886924 /lib/x86_64-linux-gnu/libm-2.28.so

Other command that helped to track the discrepancy between df and du different file usage on FS is:
 

# du -hxa  / | egrep '^[[:digit:]]{1,1}G[[:space:]]*'
 

 

3. Fixing large files kept in memory filesystem problem


What is the real reason for ending up with this file handlers opened by running backgrounded programs on the Linux OS?
It could be multiple  but most likely it is due to exceeded server / client interactions or breaking up RAM or HDD drive with writing plenty of logs on the FS without ending keeping space occupied or Programming library bugs used by hanged service leaving the FH opened on storage.

What is the solution to file system files left in memory problem?

The best solution is to first fix custom script or hanged service and then if possible to simply restart the server to make the kernel / services reload or if this is not possible just restart the problem creation processes.

Once the process is identified like in my case this was MySQL on systemd enabled newer OS distros, just do:

 

 

# systemctl restart mysqld.service


or on older init.d system V ones:

# /etc/init.d/service restart


For custom hanged scripts being listed in ps axuwef you can grep the pid and do a kill -HUP (if the script is written in a good way to recognize -HUP and restart the sub-running process properly – BE EXTRA CAREFUL IF YOU'RE RESTARTING BROKEN SCRIPTS as this might cause your running service disruptions …).

# pgrep -l script.sh
7977 script.sh


# kill -HUP PID

 

Now finally this should either mitigate or at best case completely solve the reported disagreement between df and du, after which the calculated / reported disk space should be back to normal and show up approximately the same (note that size changes a bit as mysql service is writting data) constantly extending the size between the two checks.

 

# df -hk /var/lib/mysql; du -hskc /var/lib/mysql
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/sdb5        19097172 3472744 14631296  20% /var/lib/mysql
3427772    /var/lib/mysql
3427772    total

 

What we learned?

What I've explained in this article is why and how it comes that 'zoombie' files reside on a filesystem
appearing to be eating disk space on a mounted local or network partition, giving strange inconsistent
reports, leading to system service disruptions and impossibility to have correctly shown information on used
disk space on mounted drive.

I went through with some standard logic on debugging service / filesystem / inode issues up explainat, that led me to the finding about deleted files being kept in filesystem and producing the filesystem strange sized / showing not correct / filled even after it was extended with tune2fs and was supposed to have extra 50GBs.

Finally it was explained shortly how to HUP / restart hanging script / service to fix it.

Some few good readings that helped to fix the issue:

What to do when du and df report different usage is here
df in linux not showing correct free space after file removal is here
Why do “df” and “du” commands show different disk usage?
 

How to make Samba smbfs / cifs mount share location with user / pass credentials authenticate via file stored credentials


July 19th, 2019

how-to-use-username-and-password-to-authenticate-to-samba-share-server-or-linux-share-server-linux-samba-logo
That's pretty trivial and perhaps if you had to manage samba server or cifs on a Linux host you already know it but for beginners, that might be interesting.

So in this short article I will explain how to make configure smbfs / cifs authentication from Linux host A client to Linux host B server running smbd and nmbd samba server (which is the smfs / cifs share server) by using external authentication file for either mount command or if /etc/fstab used to automatically authenticate using a preconfigured mount saba share via /etc/fstab.

Before you start to do anything with samba on Linux host A client machine, you will need as a minimum to have installed cifs-utils or smbfs (assuming you're on Debian Linux like you can check with dpkg -l and if missing install it via:

 

 

apt-get install cifs-utils

 

Or on older systems or for smbfs support

 

apt-get install smbfs

 

The general mount smbfs share command without specified external credentials file would look like so:

 

mount //mynetworksharename/ /shares/data -o username=myusername, password=mypassword


So how to use external auth file to prevent samba shares  users and passwords to not be stored in root user history all the time?

To do so it is pretty straight forward all you need to do is to create a single user / pass credentials variable defined lets say to file called .smbcredentials or .cifs under some directory lets /root/.smbcredentials.

One note here is (many people prefer to store the password under /root) for security reasons as root directory is usually readable only by administrator and would prevent a non-privileged user to read the user / pass which are stored in plain text.

.smbcredentials is described in mount.cifs man page, here is what it says about credentials variable understood by mount / mount.cifs command  file syntax:
 

 

credentials=filename
    specifies a file that contains a username and/or password. The format of the file is:

         username=value
         password=value


For a CIFS (Common Internet File System) which is a new implementation of old Windows Share (SMB protocol) avaiable in newer Windows XP / 7 / 10 machines, to do the cifs mount manually:
 

mount -v -t cifs //WINSHARESERVER/topsecretfiles /mnt/network/ -o credentials=/mnt/creds-file

or use 

 

mount.cifs //WINSSHARE/topsecretfiles /mnt/network/ -o credentials=/root/.creds-file

 

For old smbfs protocol for backward compatibility so older Win 2000 or Winblows server XP PCs configured to also access the Linux samba mount.

mount -t smbfs //WINHARESERVER/topsecretfiles /mnt/network/ -o credentials=/mnt/.smbcredentials


Once you have the defined .smbcredentials file name, be sure to also protect it with properly set permissions like 0600 (rw) readable only for root user. 

chmod 0600 /root/.smbcredentials

Note that in that example .smbcredentials is set to be a hidden file on purpose as this is a hidden file it will make it slightly less seenable if introduder breaks on the server (an example of security through obscurity)

 

Next lets see how to mount the Windows Samba Share permanently with predefined user / pass server login

For many non secured Windows shares one can use /etc/fstab line definition as simple as:
 

//server-share-name/sharename  /mnt/shares/sharename  cifs  guest,uid=1000,iocharset=utf8  0


For password protected Win Share mounts however, the simplest way to do is via /etc/fstab line add like so:

 

 

 

//servername/sharename  /mnt/shares/sharename  cifs  username=msusername,password=mspassword,iocharset=utf8,sec=ntlm  0  0


Note that the sec=ntlm is optional and remote samba server or Windows Share server version has to support this kind of authentication and in some cases you could safely reove sec=ntlm, just use it, when you know what you're doing. iocharset is good to have as for Russian / Bulgarian e.g.  Cyrillic, Chineese, Indian and other exotic languages and other strange language encoding to be supported and properly shown on the mounted share it should be properly defined …, 

A good permissions would be:

chmod 600 ~/.smbcredentials

To use the external /root/.smbcredentials password it shold be like so:

 

 

 

 

 

 

 

# cat /root/.smbcredentials

username=msusername
password=mssecretpassword
56#

 

 

Finally /root/.smbcredentials record should be as so:
 

//share-server-name/sharename /mnt/shares/windowsshare cifs credentials=/home/ubuntuusername/.smbcredentials,iocharset=utf8,sec=ntlm 0 0


Note You should already have

/mnt/shares/windowshare created on server B (the ount client) with:

mkdir -p  /mnt/shares/windowshare


To mount /etc/fstab defined filesystem to mount on next server boot then do

mount /mnt/shares/windowshare


or completely mount / remount all present /etc/fstab filesystems with the common

mount -a


(but here be careful as this might cause you troubles already other NFS or whatever FS is mounted and being read by clients) :

And you the remote Samba Share (mount location) – should be reachable with ping command and traceroute and remote server ports 139, 445 etc. should be up running opened and connectable from server B share-server-name/sharename

If you face some issues when trying to mount remote share with mount -t smbfs / mount.cifs then you can use smbclient with debug option to find out some more on the connectivity / authentication issue by using the smb share server IP address instead of hostnae and lets say a debug level of 3 like so:

 

 

 

 

smbclient -d3 -L //10.5.8.118/Files -A /root/.smbcredentials

[0] smbclient -d3 -L //10.2.3.111/Files -A /home/acteam/.smbcredentials     lp_load_ex: refreshing parameters
Initialising global parameters
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[global]"
WARNING: The "syslog" option is deprecated
added interface eth0 ip=10.2.3.127 bcast=10.2.3.255 netmask=255.255.255.0
Client started (version 4.3.11-Ubuntu).
Connecting to 10.2.3.111 at port 445
Doing spnego session setup (blob length=120)
got OID=1.3.6.1.4.1.311.2.2.30
got OID=1.2.840.48018.1.2.2
got OID=1.2.840.113554.1.2.2
got OID=1.2.840.113554.1.2.2.3
got OID=1.3.6.1.4.1.311.2.2.10
got principal=not_defined_in_RFC4178@please_ignore
GENSEC backend 'gssapi_spnego' registered
GENSEC backend 'gssapi_krb5' registered
GENSEC backend 'gssapi_krb5_sasl' registered
GENSEC backend 'spnego' registered
GENSEC backend 'schannel' registered
GENSEC backend 'naclrpc_as_system' registered
GENSEC backend 'sasl-EXTERNAL' registered
GENSEC backend 'ntlmssp' registered
GENSEC backend 'ntlmssp_resume_ccache' registered
GENSEC backend 'http_basic' registered
GENSEC backend 'http_ntlm' registered
GENSEC backend 'krb5' registered
GENSEC backend 'fake_gssapi_krb5' registered
Got challenge flags:
Got NTLMSSP neg_flags=0x62898215
NTLMSSP: Set final flags:
Got NTLMSSP neg_flags=0x62088215
NTLMSSP Sign/Seal – Initialising with flags:
Got NTLMSSP neg_flags=0x62088215
NTLMSSP Sign/Seal – Initialising with flags:
Got NTLMSSP neg_flags=0x62088215
Domain=[TMGRID] OS=[Windows Server 2012 R2 Standard 9600] Server=[Windows Server 2012 R2 Standard 6.3]

 

        Sharename       Type      Comment
        ———       —-      ——-
        ADMIN$          Disk      Remote Admin
        C$              Disk      Default share
        Files           Disk
        IPC$            IPC       Remote IPC
        MappedDrive     Disk
Connecting to 10.2.3.111 at port 139
Connecting to 10.2.3.111 at port 139
Connection to 10.2.3.111 failed (Error NT_STATUS_RESOURCE_NAME_NOT_FOUND)
NetBIOS over TCP disabled — no workgroup available

 

Sum it up

Lets Summarize a bit, here I described how to mount smbfs and cifs mount shares with mount command, how to define the auto mount on server boot via /etc/fstab, how to mount manually /etc/fstab defined mount and what should be the syntax of .smbcredentials user / pass file and also pointed how to debug problems on samba / windows server location share mounts with smbclient command.
 

How to remove ‘active contents’ from PDF file on Linux / Strip Active Contents from PDF


July 18th, 2019

how-to-remove-active-content-from-pdf-with-ghoscript-on-gnu-linux.svg

I'm updating my Autiobography (CV) with my latest job eployeers, technology expertise and certifications and usually use the EuroPassCV standard web service to update already generated PDF files.The service as web based application service allows easy edit from the web as most web services which is quite handy and then allows Export to DOCX or PDF file format. So far so good but today I faced a really weird problem after, I've used successfully EuroPassCV service  and downloaded the PDF to my computer and tried to submit my Curriculum Vitae application to SAP's Successfactor newly created account for the purpose I faced a weird I error saying

"The system does not allow files with Active contents. Please …"

the-system-does-not-allow-files-with-active-contents-pdf-error-successfactors-errors

Of course if this error message was received on a Start-up application on Application upload that would be fine, but come on this is SAP's Successfactors, it cannot accept a standard generated PDF from EuroPass which nowadays is a standard for CV here in Europe and hosted on of official European Union website europa.eu

To me this is a clear signal SAP needs an experienced ICT specialists and Quality Assurance testers like me to fix their mess and I will be willing to help them if they contact me until its too late for them, but let me go back to the topic of this article which was how to remove active contents from a PDF file 🙂

So first lets make clear what is Active content in a file ?

Active contents is content that includes programs like Internet polls, JavaScript applications, stock tickers, animated images, ActiveX applications, action items, streaming video and audio, weather maps, embedded objects, and much more. Active content contains programs that trigger automatic actions on a Web page without the user's knowledge or consent.
Active contents (Macros) could exist in many file formats that are used daily in most companies / organizations daily, active content can be contained in documents such as MS Excel,  Word, PDF, PowerPoint and so on.

So why does some applications disable document support for Active contents?

Well just for the reason of security, Active contents could often be some kind of malware or crapware and they can mess up with the web application (in case of bugs) or even mess up with server software if it is a complex warm like behavior exploiting some kind of vulnerability.
One thing to say about active contents removal on file upload by applications is that this practice could only be tolerated if the organization had already adapted a security through obscurity which most likely is the case with SAP's Successfactors and many other applications out there.

So next question is how to  Panicea (Resolution) Active Contents existing in a PDF file

Assuming you have a GNU / Linux Desktop or server with ghostscript package installed (which is the case by default with virtually any modern Linux distribution), removing Active Contents from PDF to make possible file to be submitted to the picky Security Conscious application with a single command:
 

gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=CV-Georgi_Dimitrov_Georgiev-Europass-20190718-EN-noact-content.pdf -dBATCH CV-Georgi_Dimitrov_Georgiev-Europass-20190718-EN.pdf


After that the stripped active contents PDF file would succeed in uploading to web app.
 

 

 

Howto create Linux Music Audio CD from MP3 files / Create playable WAV format Audio CD Albums from MP3s


July 16th, 2019

cdburning-audio-music-cd-from-mp3-on-linuxcomapct-disc-tux-linux-logo

Recently my Mother asked me to prepare a Music Audio CD for her from a popular musician accordionist Stefan Georgiev from Dobrudja who has a unique folklore Bulgarian music.

As some of older people who still remember the age of the CD and who had most likely been into the CD burning Copy / Piracy business so popular in the countries of the ex-USSR so popular in the years 1995-2000 audio ,  Old CD Player Devices were not able to play the MP3 file format due to missing codecs (as MP3 was a proprietary compression that can't be installed on every device without paying the patent to the MP3 compression rights holder.

The revolutionary MP3 compression used to be booming standard for transferring Music data due to its high compression which made an ordinary MP3 of 5 minutes of 5MB (10+ times more compression than an ordinary classic WAV Audio the CPU intensiveness of MP3 files that puts on the reading device, requiring the CD Player to have a more powerful CPU.

Hence  due to high licensing cost and requirement for more powerful CPU enabled Audio Player many procuders of Audio Players never introduced MP3 to their devices and MP3 Neve become a standard for the Audio CD that was the standard for music listening inside almost every car out there.

Nowdays it is very rare need to create a Audio CD as audio CDs seems to be almost dead (As I heard from a Richard Stallman lecture In USA nowadays there is only 1 shop in the country where you can still buy CD or DVD drives) and only in third world as Africa Audio CDs perhaps are still in circulation.

Nomatter that as we have an old Stereo CD player on my village and perhaps many others, still have some old retired CD reading devices being able to burn out a CD is a useful thing.

Thus to make mother happy and as a learning excercise, I decided to prepare the CD for her on my Linux notebook.
Here I'll shortly describe the takes I took to make it happen which hopefully will be useful for other people that need to Convert and burn Audio CD from MP3 Album.

 

1. First I downloaded the Album in Mp3 format from Torrent tracker

My homeland Bulgaria and specific birth place place the city of Dobrich has been famous its folklore:  Galina Durmushlijska and Stefan Georgiev are just 2 of the many names along with Оркестър Кристал (Orchestra Crystal) and the multitude of gifted singers. My mother has a santiment for Stefan Georgiev, as she listened to this gifted accordinist on her Uncle's marriage.

Thus In my case this was (Стефан Георгиев Хора и ръченици от Добруджа) the album full song list here If you're interested to listen the Album and Enjoy unique Folklore from Dobrudja (Dobrich) my home city, Stefan Georgiev's album Hora and Rachenica Dances is available here

 


Stefan_Georgiev-old-audio-Music-CD-Hora-i-Rychenici-ot-Dobrudja-Horos-and-Ruchenitsas-from-Dobrudja-CD_Cover
I've downloaded them from Bulgarian famous torrent tracker zamunda.net in MP3 format.
Of course you need to have a CD / DVD readed and write device on the PC which nowdays is not present on most modern notebooks and PCs but as a last resort you can buy some cheap External Optical CD / DVD drive for 25 to 30$ from Amazon / Ebay etc.

 

2. You will need to install a couple of programs on Linux host (if you don't have it already)


To be able to convert from command line from MP3 to WAV you will need as minimum ffmpeg and normalize-audio packages as well as some kind of command line burning tool like cdrskin  wodim which is
the fork of old good known cdrecord, so in case if you you're wondering what happened with it just
use instead wodim.

Below is a good list of tools (assuming you have enough HDD space) to install:

 

root@jeremiah:/ # apt-get install –yes dvd+rw-tools cdw cdrdao audiotools growisofs cdlabelgen dvd+rw-tools k3b brasero wodim ffmpeg lame normalize-audio libavcodec58

 

Note that some of above packages I've installed just for other Write / Read operations for DVD drives and you might not need that but it is good to have it as some day in future you will perhaps need to write out a DVD or something.
Also the k3b here is specific to KDE and if you're a GNOME user you could use Native GNOME Desktop app such brasero or if you're in a more minimalistic Linux desktop due to hardware contrains use XFCE's native xfburn program.

If you're a console / terminal geek like me you will definitely enjoy to use cdw
 

root@jeremiah:/ # apt-cache show cdw|grep -i description -A 1
Description-en: Tool for burning CD's – console version
 Ncurses-based frontend for wodim and genisoimage. It can handle audio and

Description-md5: 77dacb1e6c00dada63762b78b9a605d5
Homepage: http://cdw.sourceforge.net/

 

3. Selecting preferred CD / DVD / BD program to use to write out the CD from Linux console


cdw uses wodim (which is a successor of good old known console cdrecord command most of use used on Linux in the past to burn out new Redhat / Debian / different Linux OS distro versions for upgrade purposes on Desktop and Server machines.

To check whether your CD / DVD drive is detected and ready to burn on your old PC issue:

 

root@jeremiah:/# wodim -checkdrive
Device was not specified. Trying to find an appropriate drive…
Detected CD-R drive: /dev/cdrw
Using /dev/cdrom of unknown capabilities
Device type    : Removable CD-ROM
Version        : 5
Response Format: 2
Capabilities   :
Vendor_info    : 'HL-DT-ST'
Identification : 'DVDRAM GT50N    '
Revision       : 'LT20'
Device seems to be: Generic mmc2 DVD-R/DVD-RW.
Using generic SCSI-3/mmc   CD-R/CD-RW driver (mmc_cdr).
Driver flags   : MMC-3 SWABAUDIO BURNFREE
Supported modes: TAO PACKET SAO SAO/R96P SAO/R96R RAW/R16 RAW/R96P RAW/R96R

You can also use xorriso (whose added value compared to other console burn cd tools is is not using external program for ISO9660 formatting neither it use an external or an external burn program for CD, DVD or BD (Blue Ray) drive but it has its own libraries incorporated from libburnia-project.org libs.

Below output is from my Thinkpad T420 notebook. If the old computer CD drive is there and still functional in most cases you should not get issues to detect it.

cdw ncurses text based CD burner tool's interface is super intuitive as you can see from below screenshot:

cdw-burn-cds-from-console-terminal-on-GNU-Linux-and-FreeBSD-old-PC-computer

CDW has many advanced abilities such as “blanking” a disk or ripping an audio CD on a selected folder. To overcome the possible problem of CDW not automatically detecting the disk you have inserted you can go to the “Configuration” menu, press F5 to enter the Hardware options and then on the first entry press enter and choose your device (by pressing enter again). Save the setting with F9.
 

4. Convert MP3 / MP4 Files or whatever format to .WAV to be ready to burn to CD


Collect all the files you want to have collected from the CD album in .MP3 a certain directory and use a small one liner loop to convert files to WAV with ffmpeg:
 

cd /disk/Music/Mp3s/Singer-Album-directory-with-MP3/

for i in $( ls *.mp3); do ffmpeg -i $i $i.wav; done


If you don't have ffmpeg installed and have mpg123 you can also do the Mp3 to WAV conversion with mpg123 cmd like so:

 

for i in $( ls ); do mpg123 -w $i.wav $i.mp3; done


Another alternative for conversion is to use good old lame (used to create Mp3 audio files but abling to also) decode
mp3 to wav.

 

lame –decode somefile.mp3 somefile.wav


In the past there was a burn command tool that was able to easily convert MP3s to WAV but in up2date Linux modern releases it is no longer available most likely due to licensing issues, for those on older Debian Linux 7 / 8 / 9 / Ubuntu 8 to 12.XX / old Fedoras etc. if you have the command you can install burn and use it (and not bother with shell loops):

apt-get install burn

or

yum install burn


Once you have it to convert

 

$ burn -A -a *.mp3
 

 

5. Fix file naming to remove empty spaces such as " " and substitute to underscores as some Old CD Players are
unable to understand spaces in file naming with another short loop.

 

for f in *; do mv "$f" `echo $f | tr ' ' '_'`; done

 

6. Normalize audio produced .WAV files (set the music volume to a certain level)


In case if wondering why normalize audio is needed here is short extract from normalize-audio man page command description to shed some light.

"normalize-audio  is  used  to  adjust  the volume of WAV or MP3 audio files to a standard volume level.  This is useful for things like creating mp3 mixes, where different recording levels on different albums can cause the volume to  vary  greatly from song to song."
 

cd /disk/Music/Mp3s/Singer-Album-directory-with-MP3/

normalize-audio -m *.wav

 

7. Burn the produced normalized Audio WAV files to the the CD

 

wodim -v -fix -eject dev='/dev/sr0' -audio -pad *.wav


Alternatively you can conver all your MP3 files to .WAV with anything be it audacity
or another program or even use 
GNOME's CDBurn tool brasero (if gnome user) or KDE's CDBurn which in my opinion is
the best CD / DVD burning application for Linux K3B.

Burning Audio CD with K3b is up to few clicks and super easy and even k3b is going to handle the MP3 to WAV file Conversion itself. To burn audio with K3B just run it and click over 'New Audio CD Project'.

k3b-on-debian-gnu-linux-burn-audio-cd-screenshot

For those who want to learn a bit more on CD / DVD / Blue-Ray burning on GNU / Linux good readings are:
Linux CD Burning Mini Howto, is Linux's CD Writing Howto on ibiblio (though a bit obsolete) or Debian's official documentation on BurnCD.
 

8. What we learned here


Though the accent of this tutorial was how to Create Audio Music CD from MP3 on GNU / Linux, the same commands are available in most FreeBSD / NetBSD / OpenBSD ports tree so you can use the same method to build prepare Audio Music CD on *BSDs.

In this article, we went through few basic ways on how to prepare WAV files from MP3 normalize the new created WAV files on Linux, to prepare files for creation of Audio Music CD for the old mom or grandma's player or even just for fun to rewind some memories. For GUI users this is easily done with  k3b,  brasero or xfburn.

I've pointed you to cdw a super useful text ncurses tool that makes CD Burninng from plain text console (on servers) without a Xorg / WayLand  GUI installed super easy. It was shortly reviewed what has changed over the last few years and why and why cdrecord was substituted for wodim. A few examples were given on how to handle conversion through bash shell loops and you were pointed to some extra reading resources to learn a bit more on the topic.
There are plenty of custom scripts around for doing the same CD Burn / Covnersion tasks, so pointing me to any external / Shell / Perl scripts is mostly welcome.

Hope this learned you something new, Enjoy ! 🙂

Upgrade Debian Linux 9 to 10 Stretch to Buster and Disable graphical service load boot on Debian 10 Linux / Debian Buster is out


July 9th, 2019

howto-upgrade-debian-linux-debian-stretch-to-buster-debian-10-buster

I've just took a time to upgrade my Debian 9 Stretch Linux to Debian Buster on my old school Laptop (that turned 11 years old) Lenovo Thinkpad R61 . The upgrade went more or less without severe issues except few things.

The overall procedure followed is described n a few websites out there already and comes up to;

 

0. Set the proper repository location in /etc/apt/sources.list


Before update the sources.list used are:
 

deb [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster main contrib non-free
deb-src [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster main contrib non-free

 

deb [arch=amd64,i386] http://security.debian.org/ buster/updates main contrib non-free
deb-src [arch=amd64,i386] http://security.debian.org/ buster/updates main contrib non-free

deb [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster-updates main contrib non-free
deb-src [arch=amd64,i386] http://ftp.bg.debian.org/debian/ buster-updates main contrib non-free

deb http://ftp.debian.org/debian buster-backports main


For people that had stretch defined in /etc/apt/sources.list you should change them to buster or stable, easiest and quickest way to omit editting with vim / nano etc. is run as root or via sudo:
 

sed -i 's/stretch/buster/g' /etc/apt/sources.list
sed -i 's/stretch/buster/g' /etc/apt/sources.list.d/*.list

The minimum of config in sources.list after the modification should be
 

deb http://deb.debian.org/debian buster main
deb http://deb.debian.org/debian buster-updates main
deb http://security.debian.org/debian-security buster/updates main

Or if you want to always be with latest stable packages (which is my practice for notebooks):

deb http://deb.debian.org/debian stable main
deb http://deb.debian.org/debian stable-updates main
deb http://security.debian.org/debian-security stable/updates main

 

1. Getting list of hold packages if such exist and unholding them, e.g.

 

apt-mark showhold


Same could also be done via dpkg

dpkg –get-selections | grep hold


To unhold a package if such is found:

echo "package_name install"|sudo dpkg –set-selections

For those who don't know what hold package is this is usually package you want to keep at certain version all the time even though after running apt-get upgrade to get the latest package versions.
 

2. Use df -h and assure you have at least 5 – 10 GB free space on root directory / before proceed

df -h /

3. Update packages list to set new set repos as default

apt update

 

4. apt upgrade
 

apt upgrade

Here some 10 – 15 times you have to confirm what you want to do with configuration that has changed if you're unsure about the config (and it is not critical service) you're aware as such as Apache / MySQL / SMTP etc. it is best to install the latest maintainer version.

Hopefully here you will not get fatal errors that will interrupt it.

P.S. It is best to run apt-update either in VTTY (Virtual console session) with screen or tmux or via a physical tty (if this is not a remote server) as during the updates your GUI access to the gnome-terminal or konsole / xterm whatever console used might get cut. Thus it is best to do it with command:
 

screen apt upgrade

 

5. Run dist-upgrade to finalize the upgrade from Stertch to Buster

 

Once all is completed of the new installed packages, you will need to finally do, once again it is best to run via screen, if you don't have installed screen install it:

 

if [ $(which screen) ]; then echo 'Installed'; else apt-get install –yes screen ; fi

screen apt dist-upgrade


Here once again you should set whether old configuration to some e services has to stay or the new Debian maintainer package shipped one will overwrite the old and locally modified (due to some reason), here do wisely whatever you will otherwise some configured services might not boot as expected on next boot.

 

6. What if you get packages failed on update


If you get a certain package failed to configure after installed due to some reason, if it is a systemd service use:

 

journalctl -xe |head -n 50


or fully observer output of journalctl -xe and decide on yourself.

In most cases

dpkg-reconfigure failed-package-name


should do the trick or at least give you more hints on how to solve it.

 

Also if a package seems to be in inconsistent or broken state after upgrade  and simple dpkg-reconfigure doesn't help, a good command
that can help you is

 

dpkg-reconfigure -f package_name

 

or you can try to workaround a failed package setup with:
 

dpkg –configure -a

 
If dpkg-reconfigure doesn't help either as I experienced in prior of Debian from Debian 6 -> 7 an Debian 7 ->8 updates on some Computers, then a very useful thing to try is:
 

apt-get update –fix-missing 

apt-get install -f


At certain cases the only work around to be able to complete the package upgrade is to to remove the package with apt remove but due to config errors even that is not possible to work around this as final resort run:
 

dpkg –remove –force-remove-reinstreq

 

7. Clean up ununeeded packages

 

Some packages are left over due to package dependencies from Stretch and not needed in buster anymore to remove them.
 

apt autoremove

 

8. Reboot system once all upgrade is over

 

/sbin/reboot

 

9. Verify your just upgraded Debian is in a good state

 

root@noah:~# uname -a;
Linux noah 4.19.0-5-rt-amd64 #1 SMP PREEMPT RT Debian 4.19.37-5 (2019-06-19) x86_64 GNU/Linux

 

root@noah:~# cat /etc/issue.net
Debian GNU/Linux 10
 

 

root@noah:~# lsb_release -a
No LSB modules are available.
Distributor ID:    Debian
Description:    Debian GNU/Linux 10 (buster)
Release:    10
Codename:    buster

 

root@noah:~# hostnamectl
   Static hostname: noah
         Icon name: computer-laptop
           Chassis: laptop
        Machine ID: 4759d9c2f20265938692146351a07929
           Boot ID: 256eb64ffa5e413b8f959f7ef43d919f
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.19.0-5-rt-amd64
      Architecture: x86-64

 

10. Remove annoying picture short animation with debian logo looping

 

plymouth-debian-graphical-boot-services

By default Debian 10 boots up with annoying screen hiding all the status of loaded services state .e.g. you cannot see the services that shows in [ FAILED ] state and  which do show as [ OK ] to revert back the old behavior I'm used to for historical reasons and as it shows a lot of good Boot time debugging info, in previous Debian distributions this was possible  by setting the right configuration options in /etc/default/grub

which so far in my config was like so

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash scsi_mod.use_blk_mq=y dm_mod.use_blk_mq=y zswap.enabled=1 text"


Note that zswap.enabled=1 passed option is because my notebook is pretty old machine from 2008 with 4GB of memory and zswap does accelerate performance when working with swap – especially helpful on Older PCs for more you can read more about zswap on ArchLinux wiki
After modifying this configuration to load the new config into grub the cmd is:
 

/usr/sbin/update-grub

 
As this was not working and tried number of reboots finally I found that annoying animated gif like picture shown up is caused by plymouth below is excerpts from Plymouth's manual page:


       "The plymouth sends commands to a running plymouthd. This is used during the boot process to control the display of the graphical boot splash."

Plymouth has a set of themes one can set:

 

# plymouth-set-default-theme -l
futureprototype
details
futureprototype
joy
lines
moonlight
softwaves
spacefun
text
tribar

 

I tried to change that theme to make the boot process as text boot as I'm used to historically with cmd:
 

update-alternatives –config text.plymouth

 
As after reboot I hoped the PC will start booting in text but this does not happened so the final fix to turn back to textmode service boot was to completely remove plymouth
 

apt-get remove –yes plymouth

How to start / Stop and Analyze system services and improve Linux system boot time performance


July 5th, 2019

systemd-components-systemd-utilities-targets-cores-libraries
This post is going to be a very short one and to walk through shortly to System V basic start / stop remove service old way and the new ways introduced over the last 10 years or so with the introduction of systemd on mass base across Linux distributions.
Finally I'll give you few hints on how to check (analyze) the boot time performance on a modern GNU / Linux system that is using systemd enabled services.
 

1. System V and the old days few classic used ways to stop / start / restart services (runlevels and common wrapper scripts)

 

The old fashioned days when Linux was using SystemV / e.g. no SystemD used way was to just go through all the running services with following the run script logic inside the runlevel the system was booting, e.g. to check runlevel and then potimize each and every run script via the respective location of the bash service init scripts:

 

root@noah:/home/hipo# /sbin/runlevel 
N 5

 

Or on some RPM based distros like Fedora / RHEL / SUSE Enterprise Linux to use chkconfig command, e.g. list services:

~]# chkconfig –list

etworkManager  0:off   1:off   2:on    3:on    4:on    5:on    6:off
abrtd           0:off   1:off   2:off   3:on    4:off   5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
anamon          0:off   1:off   2:off   3:off   4:off   5:off   6:off
atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
auditd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
avahi-daemon    0:off   1:off   2:off   3:on    4:on    5:on    6:off

And to start stop the service into (default runlevel) or respective runlevel:

 

~]#  chkconfig httpd on

~]# chkconfig –list httpd
httpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

 

 

~]# chkconfig service_name on –level runlevels

 


Debian / Ubuntu and other .deb based distributions with System V (which executes scripts without single order but one by one) are not having natively chkconfig but instead are famous for update-rc.d init script wrapper, here is few basic use  of it:

update-rc.d <service> defaults
update-rc.d <service> start 20 3 4 5
update-rc.d -f <service>  remove

Here defaults means default set boot runtime for system and numbers are just whether service is started or stopped for respective runlevels. To check what is your default one simply run /sbin/runlevel

Other useful tool to stop / start services and analyze what service is running and which not in real time (but without modifying boot time set for a service) – more universal nowadays is to use the service command.

root@noah:/home/hipo# service –status-all
 [ + ]  acpid
 [ – ]  alsa-utils
 [ – ]  anacron
 [ + ]  apache-htcacheclean
 [ – ]  apache2
 [ + ]  atd
 [ + ]  aumix

root@noah:/home/hipo# service cron restart/usr/sbin/service command is just a simple wrapper bash shell script that takes care about start / stop etc. operations of scripts found under /etc/init.d

For those who don't want to tamper with too much typing and manual configuration there is an all distribution system V compatible ncurses interface text itnerface sysv-rc-conf which could make your life easier on configuring services on non-systemd (old) Linux-es.

To install on Debian distros:

debian:~# apt-get install sysv-rc-conf

debian:~# sysv-rc-conf


SysV RC Conf desktop on GNU Linux using sysv-rc-conf systemV and systemd
 

2. SystemD basic use Start / stop check service and a little bit of information
for the novice

As most Linux kernel based distributions except some like Slackware and few others see the full list of Linux distributions without systemd (and aha yes slackw. users loves rc.local so much – we all do 🙂  migrated and are nowadays using actively SystemD, to start / stop analyze running system runnig services / processes

systemctl – Control the systemd system and service manager

To check whether a service is enabled

systemctl is-active application.service

To check whether a unit is in a failed state

systemctl is-failed application.service

To get a status of running application via systemctl messaging

# systemctl status sshd
● ssh.service – OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2019-07-06 20:01:02 EEST; 2h 3min ago Main PID: 1335 (sshd) Tasks: 1 (limit: 4915) CGroup: /system.slice/ssh.service └─1335 /usr/sbin/sshd -D юли 06 20:01:00 noah systemd[1]: Starting OpenBSD Secure Shell server… юли 06 20:01:02 noah sshd[1335]: Server listening on 0.0.0.0 port 22. юли 06 20:01:02 noah sshd[1335]: Server listening on :: port 22. юли 06 20:01:02 noah systemd[1]: Started OpenBSD Secure Shell server.

To enable / disable application with systemctl systemctl enable application.service

systemctl disable application.service

To stop / start given application systemcl stop sshd

systemctl stop tor

To reload running application

systemctl reload sshd

Some applications does not have the right functionality in systemd script to reload configuration without fully restarting the app if this is the case use systemctl reload-or-restart application.service

systemctl list-unit-files

Then to view the content of a single service unit file:

:~# systemctl cat apache2.service
# /lib/systemd/system/apache2.service
[Unit]
Description=The Apache HTTP Server
After=network.target remote-fs.target nss-lookup.target

[Service]
Type=forking
Environment=APACHE_STARTED_BY_SYSTEMD=true
ExecStart=/usr/sbin/apachectl start
ExecStop=/usr/sbin/apachectl stop
ExecReload=/usr/sbin/apachectl graceful
PrivateTmp=true
Restart=on-abort

[Install]
WantedBy=multi-user.target


converting-traditional-init-scripts-to-systemd-graphical-diagram

systemd's advancement over normal SystemV services it is able to track and show dependencies
of a single run service for proper operation on other services

:~# systemctl list-dependencies sshd.service

 


● ├─system.slice
● └─sysinit.target
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ├─keyboard-setup.service
●   ├─kmod-static-nodes.service
●   ├─proc-sys-fs-binfmt_misc.automount
●   ├─sys-fs-fuse-connections.mount
●   ├─sys-kernel-config.mount
●   ├─sys-kernel-debug.mount
●   ├─systemd-ask-password-console.path
●   ├─systemd-binfmt.service
….

.

 

You can also mask / unmask service e.g. make it temporary unavailable via systemd with

sudo systemctl mask nginx.service

it will then appear as masked if you do list-unit-files

If you want to change something on a systemd unit file this is done with

systemctl edit –full nginx.service

In case if some modificatgion was done to systemd service files e.g. lets say to
/etc/systemd/system/apache2.service or even you've made a Linux system Upgrade recently
that added extra systemd service config files it will be necessery to reload all files
present in /etc/systemd/system/* with:

systemctl daemon-reload


Systemd has a target states which are pretty similar to the runlevel concept (e.g. runlevel 5 means graphical etc.), for example to check the default target for a system:

One very helpful feature is to restart systemd but it seems this is not well documented as of now and though this might work after some system package upgrade roll-outs it is always better to reboot the system, but you can give it a try if restart can't be done due to application criticallity.

To restart systemd and its spawned subprocesses do:
 

systemctl daemon-reexec

 

root@noah:/home/hipo# systemctl get-default
graphical.target


 to check all targets possible targets

root@noah:/home/hipo# systemctl list-unit-files –type=target
UNIT FILE                 STATE   
basic.target              static  
bluetooth.target          static  
busnames.target           static  
cryptsetup-pre.target     static  
cryptsetup.target         static  
ctrl-alt-del.target       disabled
default.target            static  
emergency.target          static  
exit.target               disabled
final.target              static  
getty.target              static  
graphical.target          static  

you can put the system in Single user mode if you like without running the good old well known command:

/sbin/init 1 

command with

systemctl rescue

You can even shutdown / poweroff / reboot system via systemctl (though I never did that and I don't recommend) 🙂
To do so use:

systemctl halt
systemctl poweroff
systemctl reboot


For the lazy ones that don't want to type all the time like crazy to configure and manage simple systemctl set services take a look at chkservice – an ncurses text based menu systemctl management interface

As chkservice is relatively new it is still not present in stable Stretch Debian repositories but it is in current testing Debian unstable Buster / Sid – Testing / Unstable distribution and has installable package for Ubuntu / Arch Linux and Fedora

chkservice-Linux-systemctl-ncurses-text-menu-service-management-interface-start-chkservice
Picture Source Tecmint.com

chkservice linux help screen


3. Analyzing and fix performance boot slowness issues due to a service taking long to boot


The first very useful thing is to know how long exactly all daemons / services got booted
on your GNU / Linux OS.

linux-server:~# systemd-analyze 
Startup finished in 4.135s (kernel) + 3min 47.863s (userspace) = 3min 51.998s

As you can see it reports both the kernel boot time and userspace (surrounding services
that had to boot for the system to be considered fully booted).


Once you have the system properly booted you have a console or / ssh access

root@pcfreak:/home/hipo# systemd-analyze blame
    2min 14.172s tor@default.service
    1min 40.455s docker.service
     1min 3.649s fail2ban.service
         58.806s nmbd.service
         53.992s rc-local.service
         51.458s systemd-tmpfiles-setup.service
         50.495s mariadb.service
         46.348s snort.service
         34.910s ModemManager.service
         33.748s squid.service
         32.226s ejabberd.service
         28.207s certbot.service
         28.104s networking.service
         23.639s munin-node.service
         20.917s smbd.service
         20.261s tinyproxy.service
         19.981s accounts-daemon.service
         18.501s loadcpufreq.service
         16.756s stunnel4.service
         15.575s oidentd.service
         15.376s dev-sda1.device
         15.368s courier-authdaemon.service
         15.301s sysstat.service
         15.154s gpm.service
         13.276s systemd-logind.service
         13.251s rsyslog.service
         13.240s lpd.service
         13.237s pppd-dns.service
         12.904s NetworkManager-wait-online.service
         12.540s lm-sensors.service
         12.525s watchdog.service
         12.515s inetd.service


As you can see you get a list of services time took to boot in secs and you can
further debug each of it to find out why it boots so slow (netwok / DNS / configuration isssue whatever).

On a servers it is useful to look up for some processes slowing it down like gdm.service etc.

 

Close up words rant on SystemD vs SysemV

init-and-systemd-comparison-commands-linux-booting-1

A lot could be ranted on what is better systemd or systemV. I personally hated systemd since day since I saw it being introduced first in Fedora / CentOS linuxes and a bit later in my beloved desktop used Debian Linux.
I still remember the bugs and headaches with systemd's intruduction as it is with all new the early adoption of technology makes a lot of pain in the ass.
Eventually systemd has become a standard and with my employment as a contractor through Itelligence GmBH for SAP AG I now am forced to work with systemd daily on SLES 12 based Linuces and I was forced to get used to it. 
But still there is my personal preference to SystemV even though the critics of slow boot etc.but for managing a multitude of Linux preinstalled servers like Virtual Machines and trying to standardize a Data Center with Tens of Thousands of Linuxes running on different Hypervisors VMWare / OpenXen + physical hosts etc. systemd brings a bit of more standardization that makes it a winner.