Posts Tagged ‘amount’

How to split large files in Windows via split command line and File Archive GUI tool easily

Tuesday, October 22nd, 2024

Moving around a very large files especially Virtualbox Virtual Machines or other VM formats between Windows host and OneDrive might be a problem due to either Azure Cloud configured limitations, or other reasons that your company Domain Administrator has configured, thus if you have to migrate your old Hardware Laptop PC Windows 10 to a newer faster better Harware / Better Performance Notebook Computer with Windows 11 and you still want to keep and move your old large files in this short and trivial article, will explain how.

The topic is easily and most of novice sysadmins should have already be faced to bump into something like this but anyways i found useful to mention about Git for Windows, as it is really useful too thus wrote this small article.

The moved huge files, in my case an experimental Virtual Machines Images which I needed to somehow migrate on the new Freshly installed Windows laptop, the Large files were 40 / 80 etc. Gigabytes or whatever large amount of files from your PC to the Cloud Onedrive and of course the most straight forward thing i tried was to simply add the file for inclusion into the Onedrive storage (via OneDrive tool setup interface), however this file, failed due to OneDrive Cloud file format security limitations or Antivirus solutions configured to filter out the large file copying or even a prohibition to be able to include any kind of Virtual Machines ISOs straight into the cloud.

With this big files comes the question:

How to copy the Virtual Machines from your Old Hardware Laptop to the Cloud (without being able to use an external SSD Hard Drive or a USB SSD Flash drive, due to Domain policy configured for your windows to be unable to copy to externally connected Drive but only to read from such.) ?
 

Here are few sample approaches to do it both from command line (useful if you have to repeat the process or script it and deploy to multiple hosts) or for single hosts via an Archiver tool:

 

1. Using split command Git for Windows (Bash) MINGW64 shell 

Download Git for Windows – https://git-scm.com/download install it and you will get the MINGW64 bash for Windows executable.

Run it either invoke bash command from command line or trigger Windows Run command prompt (Windows button + R) and type full path to executable
 

C:\Program Files\Git\git-bash.exe


Git-for-Windows-bash-for-windows-MINGW64-windows-11-screenshot

Use the integrated program split and to cut it into pieces use:

 

# split MyVeryLargeFileVM.vdi -b 800m


To split the .VDI virtualbox file to lets say 5 Gigabite pieces:

# split MyVeryLargeFile.vdi -b 5g

The output files will be named pieces will be named as in a normal UNIX / GNU split command in the format and each piece of 5GB will be named like:

xaa
xab
xac
xad

If you want to get a more meaningful name for the spilitted files you can set a generated split file prefix with suffixes to be 5 digits long:

# split MyVeryLargeFile.vdi MyVeryLargeFileVM-parts_ -b 5g -d -a 5

  • the -d flag for using numerical suffixes (instead of default aa, ab, ac, etc…),
  • and the option -a 5 to tell it I want the suffixes to be 5 digits long:

2. Split large files by Archiving them with Winrar (ShareWare) tool

If you have already Winrar installed and you don't want to bother with too much typing from the command line, You can use good old WinRAR as a file splitter/joiner as well.

To split a file into smaller files, select "Store" as the compression method and enter the desired value (bytes) into "Split to volumes" box.
This way you can have split files named as filename.part1.rar, filename.part2.rar, etc.

WinRAR_cut-split-large-files-into-pieces-screenshot-Windows

3. Split files with 7-Zip (FreeWare)

Assuming you have the 7-Zip installed on the PC, you can do the archiving of the Big file to a smaller pieces one, you can create the splitted file from 7Zip interfaces menus:

7zip-file-split-files-into-multiple-pieces-windows-screenshot

Or directly cut the single file into multiple volumes, directly from Windows Explorer by Selecting the file and using fall down menus :
7zip-creation-of-multiple-parts-file-from-single-one-screenshot-Windows

7-zip-split-huge-files-to-lower-parts-set-volume-size

Sum it up what learned ? 

What we learned is how to cut large files into multiple single consequential ones for easy copy between Network sides, via both Git 4 Windows and manual copy paste of parted multiple files to OneDrive / DropBox / pCloud or Google Drive.
There is a plenty of other approaches to take as there is also file GUI tools, besides using GNU Win / Gnu Tools for Windows or Cygwin / Gsplit GUI tool  or some kind of the many Archiver toolsavailable for Windows, another option to split the large files is to use a bunch of PowerShell and Batch scripts written that can help you do the file split for both binaries files or Text files. but i'll stop here as I believe that is pretty much enough for most basic needs.

 

Fix staled NFS on server with dmesg error log nfs: server nfs-server not responding, still trying

Saturday, March 16th, 2019

NFS_Filesystem-fix-staled-NFS-System-dmesg-error-nfs-server-not-responding-still-trying

On a server today I've found to have found a number of NFS mounts mounted through /etc/fstab file definitions that were hanging;
 

nfs-server:~# df -hT


 command kept hanging as well as any attempt to access the mounted NFS directory was not possible.
The server with the hanged Network File System is running SLES (SuSE Enterprise Linux 12 SP3) a short investigation in the kernel logs (dmesg) as well as /var/log/messages reveales following errors:

 

nfs-server:~# dmesg
[3117414.856995] nfs: server nfs-server OK
[3117595.104058] nfs: server nfs-server not responding, still trying
[3117625.032864] nfs: server nfs-server OK
[3117805.280036] nfs: server nfs-server not responding, still trying
[3117835.209110] nfs: server nfs-server OK
[3118015.456045] nfs: server nfs-server not responding, still trying
[3118045.384930] nfs: server nfs-server OK
[3118225.568029] nfs: server nfs-server not responding, still trying
[3118255.560536] nfs: server nfs-server OK
[3118435.808035] nfs: server nfs-server not responding, still trying
[3118465.736463] nfs: server nfs-server OK
[3118645.984057] nfs: server nfs-server not responding, still trying
[3118675.912595] nfs: server nfs-server OK
[3118886.098614] nfs: server nfs-server OK
[3119066.336035] nfs: server nfs-server not responding, still trying
[3119096.274493] nfs: server nfs-server OK
[3119276.512033] nfs: server nfs-server not responding, still trying
[3119306.440455] nfs: server nfs-server OK
[3119486.688029] nfs: server nfs-server not responding, still trying
[3119516.616622] nfs: server nfs-server OK
[3119696.864032] nfs: server nfs-server not responding, still trying
[3119726.792650] nfs: server nfs-server OK
[3119907.040037] nfs: server nfs-server not responding, still trying
[3119936.968691] nfs: server nfs-server OK
[3120117.216053] nfs: server nfs-server not responding, still trying
[3120147.144476] nfs: server nfs-server OK
[3120328.352037] nfs: server nfs-server not responding, still trying
[3120567.496808] nfs: server nfs-server OK
[3121370.592040] nfs: server nfs-server not responding, still trying
[3121400.520779] nfs: server nfs-server OK
[3121400.520866] nfs: server nfs-server OK


It took me a short while to investigate and check the NetApp remote NFS storage filesystem and investigate the Virtual Machine that is running on top of OpenXen Hypervisor system.
The NFS storage permissions of the exported file permissions were checked and they were in a good shape, also a reexport of the NFS mount share was re-exported and on the Linux
mount host the following commands ran to remount the hanged Filesystems:

 

nfs-server:~# umount -f /mnt/nfs_share
nfs-server:~# umount -l /mnt/nfs_share
nfs-server:~# umount -lf /mnt/nfs_share1
nfs-server:~# umount -lf /mnt/nfs_share2
nfs-server:~# mount -t nfs -o remount /mnt/nfs_share


that fixed one of the hanged mount, but as I didn't wanted to manually remount each of the NFS FS-es, I've remounted them all with:

nfs-server:~# mount -a -t nfs


This solved it but, the fix seemed unpermanent as in a time while the issue started reoccuring and I've spend some time
in further investigation on the weird NFS hanging problem has led me to the following blog post where the same problem was described and it was pointed the root cause of it lays
in parameter for MTU which seems to be quite high MTU 9000 and this over the years has prooven to cause problems with NFS especially due to network router (switches) configurations
which seem to have a filters for MTU and are passing only packets with low MTU levels and using rsize / wzise custom mount NFS values in /etc/fstab could lead to this strange NFS hangs.

Below is a list of Maximum Transmission  Unit (MTU) for Media Transport excerpt taken from wikipedia as of time of writting this article.

https://www.pc-freak.net/images/Maximum-Transmission-Unit-for-Media-Transport-diagram-3.png

In my further research on the issue I've come across this very interesting article which explains a lot on "Large Internet" and Internet Performance

I've used tracepath command which is doing basicly the same as traceroute but could be run without root user and discovers hops (network routers) and shows MTU between path -> destionation.

Below is a sample example

nfs-server:~# tracepath bergon.net
 1?: [LOCALHOST]                      pmtu 1500
 1:  192.168.6.1                                           0.909ms
 1:  192.168.6.1                                           0.966ms
 2:  192.168.222.1                                         0.859ms
 3:  6.192.104.109.bergon.net                              1.138ms reached
     Resume: pmtu 1500 hops 3 back 3

 

Optiomal pmtu for this connection is to be 1500 .traceroute in some cases might return hops with 'no reply' if there is a router UDP  packet filtering implemented on it.

The high MTU value for the Storage network connection interface on eth1 was evident with a simple:

 

 nfs-server:~# /sbin/ifconfig |grep -i eth -A 2
eth0      Link encap:Ethernet  HWaddr 00:16:3E:5C:65:74
          inet addr:100.127.108.56  Bcast:100.127.109.255  Mask:255.255.254.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth1      Link encap:Ethernet  HWaddr 00:16:3E:5C:65:76
          inet addr:100.96.80.94  Bcast:100.96.83.255  Mask:255.255.252.0
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1


The fix was as simple to lower MTU value for eth1 Ethernet interface to 1500 which is the value which most network routers are configured too.

To apply the new MTU to the eth1 interface without restarting the SuSE SLES networking , I first used ifconfig one time with:

 

 nfs-server:~# /sbin/ifconfig eth1 mtu 1500
 nfs-server:~# ip addr show
 …


To make the setting permanent on next  SuSE boot:

I had to set the MTU=1500 value in

 

nfs-server:~#/etc/sysconfig/network/ifcfg-eth1
nfs-server:~#  ip address show eth1
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 8c:89:a5:f2:e8:d8 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/24 brd 192.168.0.255 scope global eth1
       valid_lft forever preferred_lft forever

 


Then to remount the NFS mounted hanged filesystems once again ran:
 

nfs-server:~# mount -a -t nfs


Many network routers keeps the MTU to low as 1500 also because a higher values causes IP packet fragmentation when using NFS over UDP where IP packet fragmentation and packet
reassembly requires significant amount of CPU at both ends of the network connection.
Packet fragmentation also exposes network traffic to greater unreliability, since a complete RPC request must be retransmitted if a UDP packet fragment is dropped for any reason.
Any increase of RPC retransmissions, along with the possibility of increased timeouts, are the single worst impediment to performance for NFS over UDP.
This and many more is very well explained in Optimizing NFS Performance page (which is a must reading) for any sys admin that plans to use NFS frequently.

Even though lowering MTU (Maximum Transmission Union) value does solved my problem at some cases especially in a modern local LANs with Jumbo Frames, allowing and increasing the MTU to 9000 bytes
might be a good idea as this will increase the amount of packet size.and will raise network performance, however as always on distant networks with many router hops keeping MTU value as low as 1492 / 5000 is always a good idea.

 

Baby boomers and Generation X, Y, Z – Generational Marketing and 4 Common personality stereotype traits of people born over the last 60 years

Saturday, August 18th, 2018

baby-bommers-and-x-y-z-generations

Those who are employed in the realm of Social or Internet Marketing definitely have to know the existence of at least 4 different conditional stereotypes, these are Baby Boomers and Generation X, Generation Y and Generation Z (Millenials).

According to Socielogist Karl Mannheim (who is among the founding fathers of classical socielogy) – "All members of a generation share a similar collective experience" or in other words people are categorized in generations depending on when they were born.

As stereotypes they're generalization of people born in different periods of time and sharing same or similar traits.
Because of the age and the conditions they grew up and as they share those general spirit of time and age, they tend to be more or less behaving in a similar ways in how they think save / spend money or share some common approach to life choices and attitude towards life and worldview.

But before proceeding to the 4 main cohert provisional stereotypes, its worthy to mention how these four common trait generations came to existence with a little bit of pre-history.

The pre WW I and WW II world situation and the First and Second World War played a pivotal role in forming the social conditions necessery for the development of the baby boomers.

* The depression Era people

Born in period: 1912 – 1921 who came at full maturity around 1930-1939 right in the beginning of WW I (all of whom are already deceased) as of 2018 as a cause of the war uncertainty and the havoc and the war conditions were very conservative, compulsive savers, tried their best to maintain a low debt. They had the mindset (responsibility) to leave some kind of legacy to their children. They were very patriotic, oriented towards work before pleasure, had a great respect for authority and had a strong sense of moral obligation. For all this character traits of this people undoubtfully a key role played the strong belief in God mostly all people had at the time.

The next in line conditional stereotype of people that came to earth are the:

* The World War II Generation
 

Born in year period: 1922 to 1927 who came to a mature age exactly at the terrible years of Second World War.

People of that time were either fighters for or against the Axis Powers or the Central Powers with the common shared goal to fight against the enemy (of course there are multiple of people who were just trying to survive and not taking a side in this meaningless war).

The current amount of people living are estimated to few million of deathbed elders  worldwide.

As above conditional generations types mentioned are of importance for historical reasons and most of the people belonging to those depression pre WWI and WW II era are dead or just a few millions an overall in un less-consuming age (excluding the medicine consumption which is higher compared to youngsters).

I'll further proceed further with the Baby Boomers, GEN X, Y, Zs who are de-facto the still active members participating to society and economy more or less.

baby-boomers-generation-x-y-z-chart-table-by-year-of-birth

So what are these 4 Stereotypes of Generations that and why are so important for the modern marketers or business manager?

 

1. BABY BOOMERS also called for a short (Boomers)
 

we-are-who-are-baby-boomers

These are people who have been defined by a birth year range (period) from early to mid 1940s  until 1960 and 1964.

 In Europe and North America, boomers are widely identified with privilege, as many grew up in a time of widespread government subsidies in post-war housing and education, and increasing affluence.

As a group, baby boomers were considered the wealthiest, most active, and most physically fit generation up to the era in which they arrived, and were amongst the first to grow up genuinely expecting the world to improve with time. They were also the generation that received peak levels of income; they could therefore reap the benefits of abundant levels of food, apparel, retirement programs, and sometimes even "midlife crisis" products. The increased consumerism for this generation has been regularly criticized as excessive (and that's for a good reason).

One feature of the boomers was that they have tended to think of themselves as a special generation, very different from those that had come before or that has come afterward. In the 1960s, as the relatively large numbers of young people became teenagers and young adults, they, and those around them, created a very specific rhetoric around their cohort, and the changes they were bringing about. This rhetoric had an important impact in the self perceptions of the boomers, as well as their tendency to define the world in terms of generations, which was a relatively new phenomenon. The baby boom has been described variously as a "shockwave" and with a methapors such as as "the pig in the python".

 

2. Generation X / GEN X

generation-x-who-are-they-gen-x-explained-picture

Generation X is considered the people born in the following birth year period 1960 forward in time until 1980s. A specific feature in the 60s-80s period was the shifting societal values, perhaps the spring of this generation was also connected to the increasing role and spread of communism in the world.
Sometimes this generation was referred as the "latchkey generation".
The term generation X itself was popularized largely by Douglas Coupland in his novel 1991 novel Generation X Tales for an Accelerated Culture

A very common trait for Generation X was the reduced adult supervision over kids when compared to previous generations a result of increasing divorce rates and the increased role of one parent children upbringing (in most cases that was the mother) which had to be actively involved as a workforce and lacked physically the time to spend enough time with its children and the increased use of childcare options in one parent families.

They were dubbed the "MTV" (Music Television) generation – that was a hit and most popular music TV in the early 1990s.
The kids representing generation X were described as slackers, cynical and disaffected.

The cultural influences dominating the tastes and feelings of the teen masses of that generation was musical genres such as punk music, heavy metal music, grunge and hip-hop and indie films (independent films)  produced outside of the major film studio system.

According to many researches in midtime those generation are described as active, happy and achieving a work-life balance kind of lifestyle.

People belonging to Generation X are described as people with Enterpreneural tendencies.

Just to name a few of the celebrities and successful people who belong to this generation, that's Google's founder Sergey Brinn & Larry Page (born in 1973), Richard Stallman (founder of Free Software movement) as well movie and film producer celebrities such as Georgi Clooney, Lenny Kravitz, Quantin Tarantino, Kevin Smith, David Fincher etc.

According to United Kingdom survey study of 2500+ workers conducted by Workfront, GEN X are found to be among the hardest working employees in today's workforce. They are also ranked high by fellow workers for having a strong work ethics (about 59.5%), being helpful (55.4%) and very skilled (54.5%) of respondents as well marked as the best troubleshooters / problem solvers (41.6%) claimed so.
According to research conducted by Viacom, gen x they have a high desire for flexibility and fulfillment at work.

3. Generation Y (Millenials) – GEN Y
 

who-are-generation-y-millenials-explained

Following Generation X came on earth Genreation Y the birth period dated for this kids were years are stretchy year period that this generation is described are years 1980s – 1990s to yearly 2000s where birth period range of those ppl ends.
This kids are descendants of the GEN X and second wave Baby Boomers.
In the public this generation is referred as "echo boomers".

The Millenials characteristics are different based on the region of birth, they're famous for the increased familiarity with communication, media and digital technologies.

millenials-focus-on-technology-innovation-and-their-technological-preferences

There upbringing was marked by increase in liberal approach to politics.
The Great recession crisis of the 2000s played a major impact on this generation because it has caused historical high levels of un-employment among youngsters and led to a possible long term economic and social damage to this generation.

millennials-are-heavily-influenced-by-their-peers
Gen Y according are less brand loyal and the speed of the Internet has led the cohort to be similarly flexible and changing in its fashion, style consciousness and where
and how it is communicated with.

As I am born in 1983 me and my generation belongs to Generation Y and even though Bulgaria before 1991 was a Communist regime country, I should agree that I and many of my friends share a very similar behavior and way of thinking to the GEN Y stereotype described, but as I was born in a times of transition and Bulgaria as a Soviet Union Satellite at the time has been lacking behind in fashion and international culture due to the communist regime, me and my generation seem to be sharing a lot of common stereotype characteristics with Generation X such as the punk-rock, metal, hip-hop culture MTV culture and partly because of the GEN X like overall view on life.

Among most famous representative successful people of the Millenials generation are Mark Zuckerberg (Facebook founder), Prince William (the second in line to the British throne), Kim Jong Un (the leader dictator of North Korea) etc.
 

 

4. Generation Z ( GEN Z) / iGeneration / Generation Sensible (Post Millenials)
 

who-are-generation-y-millenials-explained

Following Millenials generation is GEN Z, demographers and researchers typically set as a starting birth date period of those generation 1990s and mid 2000s. As of time of writting there is still no clear consensus regarding ending birth years.

This is the so called Internet Generation because this generation used the internet and Smart Mobile Phone technology since a very young age, they are very confortable with technology (kinda of wired) and addicted to social media such as Facebook / Twitter / Instagram etc. Because of the level of digital communication, many people of this generation are more introvert oriented and often have problems expressing themselves freely in groups. Also they tend to lack the physical communication and more digitally community oriented, even though this depends much also on the specific personality and in some cases it is exactly the opposite.

 

 * Summary
 

As a Marketer, Human Resources hiring personal specialist, a CEO or some kind of project / business manager it is a good idea to be aware of these 4 common stereotypes. However as this are stereotypes (and a theory) as everything theoritized the data is slighly biased and untrue. The marketer practice shows that whoever conducts a marketing and bases his sales on this theoritizing should consider this to be just one aspect of the marketing campaign those who are trying to sell, stuff ideas or ideology to any of those generation should be careful not to count 100% on the common traits found among the above 4 major groups and consider the individuality of person everyone has and just experiment a little bit to see what works and what doesn't.

Also it should be mentioned these diversification of stereotypes are mostly valid for the US citizens and Westerners but doesn't fully fit to ex-communist countries or countries of the Soviet union, those countries have a slightly different personality traits of person born in any of the year periods defined, same is more or less true for the poor parts of Africa and India, Vietnam, China and mostly all of the coomunist countries ex and current. It should be said that countries who belonged to the Soviet Union many of which are current Russian Federation Republics have a personality traits that are often mixture of the 4 stereotypes and even have a lot of the traits that were typical for the WW I and WW II generations, which makes dealing with this people a very weird experience.

Nomatter the standard error that should always considered when basing a marketing research hypothesis on Generational Marketing (using generational segmentation in marketing best potential customer targets), having a general insight and taking in consideration those stereotypes could seriously help in both marketing as well as HR specific fields like Change Management.

generation-x-y-z-characteristics

If you're a marketer, I recommend you take a quick look also on following very educative article Generational Makarketing and how to target each of the GEN X, Y, Z and Baby Boomers and what works best for each of them.

Nomatter what just like all Theories, the theory of Boomers and the Generation segmantation is not completely true, but it gives a good soil for reasoning as well definitely helps for people involved in sociology and business.

Comments and feedback on the article are mostly welcome as the topic is very broad and there is much more to be said …

Hope the article was interesting to you ….

What was your Generation like?

Optimizing Linux TCP/IP Networking to increase Linux Servers Performance

Tuesday, April 8th, 2008

optimize-linux-servers-for-network-performance-to-increase-speed-and-decrease-hardware-costs-_tyan-exhibits-hpc-optimized-server-platforms-featuring-intel-xeon-processor-e7-4800-v3-e5-2600-supercomputing-15_full

Some time ago I thought of ways to optimize my Linux Servers network performance.

Even though there are plenty of nice articles on the topic on how to better optimize Linux server performance by tunning up the kernel sysctl (variables).

Many of the articles I found was not structed in enough understandable way so I decided togoogle around and  found few interesting websites which gives a good overview on how one can speed up a bit and decrease overall server loads by simply tuning few basic kernel sysctl variables.

Below article is a product of my research on the topic on how to increase my GNU / Linux servers performance which are mostly running LAMP (Linux / Apache / MySQL / PHP) together with Qmail mail servers.

The article is focusing on Networking as networking is usual bottleneck for performance.
Below are the variables I found useful for optimizing the Linux kernel Network stack.

Implementing the variables might reduce your server load or if not decrease server load times and CPU utilization, they would at lease increase thoroughput so more users will be able to access your servers with (hopefully) less interruptions.
That of course would save you some Hardware costs and raise up your Servers efficiency.

Here are the variables themselves and some good example:
 

# values.net.ipv4.ip_forward = 0 ( Turn off IP Forwarding )

net.ipv4.conf.default.rp_filter = 1

# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0

# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0 ( same as above )
net.ipv4.conf.default.accept_source_route = 0

# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
( - || - )net.ipv4.tcp_fin_timeout = 40

# ( Decrease FIN timeout ) - Useful on busy/high load
serversnet.ipv4.tcp_keepalive_time = 4000 ( keepalive tcp timeout )
net.core.rmem_default = 786426 - Receive memory stack size ( a good idea to increase it if your server receives big files )
net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608 ( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
( maximum amount of option memory buffers )

# like a homework investigate by yourself what the variables below stand for :)
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024

 

# Below are newly added experimental
#net.core.rmem_max = 16777216
#net.core.wmem_max = 16777216
##kernel.msgmni = 1024
##kernel.sem = 250 256000 32 1024
##vm.swappiness=0
kernel.sched_migration_cost=5000000

 

Also a good sysctl.conf file which one might want to substitite or use as a skele for some productive server is ready for download here


Even if you can't reap out great CPU reduction benefits from integrating above values or similar ones, your overall LAMP performance to end customers should increase – at some occasions dramatically, at others little bit but still noticable.

If you're unsure on exact kernel variable values to use check yourself what should be the best values that fits you according to your server Hardware – usually this is done by experimenting and reading the kernel documentation as provided for each one of uplisted variables.

Above sysctl.conf is natively created to run on Debian and on other distributions like CentOS, Fedora Slackware some values might either require slight modifications.

Hope this helps and gives you some idea of how network optimization in Linux is usually done. Happy (hacking) tweakening !

The Treasures of Hermitage, Saint Peterburg Russia and its imitation Hermitage in Amsterdam

Thursday, January 17th, 2013

Sankt-Peterburg-Hermitage

Since some time, I'm very interested in Russia culture as it is so inter-related to our culture in Bulgaria. .It is sad that nowadays Bulgaria is not with Russia, we share much more cultural and heritage roots with Russia than any Western European county or even generally Europe. Thus I believe for every Bulgarian and generally for anyone interested in art, maybe the most remarkable place to visit if you're in Russia is the city of Saint Peterburg. There is found the Russian National State museum. The Hermitage is a museum which historically used to be Winster Palace (Residency) for the Russian Tsar / King. What is unique about this museum is it is one of the oldest and largest museums in the world. The musem was founded in 1764 and since 1852 is in open for public visits. The museum consists of 3 000 000 unique items. Only a small number of those 3 million are exposed for public visitors. The museum contains artifacts from all around the world including very rare things like Mummy sarcophagus. The amount of gold in mostly everywhere is remarkable the ornaments and everything is amazing. For people who value art it is a must see. I still did not have the opportunity to see it with my own eyes. But even seeing it on a video is very  worthy.


 

The Treasures of Hermitage, Saint Petersburg, Russia (State Hermitage Musem of Russia)

As I'm currently in Holland, it was very interesting to find out one of the worthy things to see in Holland is an imitation of st. Peterburg's Hermitage in Amsterdam. There is plenty of beautiful and unique works in it too, as you can check in below video.


 

Hermitage Museum Amsterdam Tour an imitation of Hermitage State Museum of Russia

Enjoy ! 🙂

Speeding up Apache through apache2-mpm-worker and php5-cgi on Debian / How to improve Apache performance and decrease server memory consumption

Friday, March 18th, 2011

speeding up apache through apache2-mpm-worker and php5-cgi on Debian Linux / how to improve apache performance and decrease server responce time
By default most Apache running Linux servers on the Internet are configured to use with the mpm prefork apache module
Historically prefork apache module is the predecessor of the worker module therefore it's believed to be a way more tested and reliable, if you need a critical reliable webserver configuration.

However from my experience by so far with the Apache MPM Worker I can boldly say that many of the rumors concerning the unreliabity of apache2-mpm-worker are just myths.

The old way Apache handles connections e.g. the mod prefork is the well known way that high amount of the daemons on Linux and BSD are still realying on.
When prefork is a used by Apache, every new TCP/IP connection arriving at your Linux server on the Apache configured port let's say on port 80 is being served by Apache in a way that the Apache process (mother process) parent does fork a new Apache parent copy in order to serve the new request.
Thus by using the prefork Apache needs to fork new process (if it doesn't have already an empty forked one waiting for connections) and serve the HTTP request of the new client, after the request of the client is completed the newly forked Apache usually dies (even though it again depends on the way the Apache server is configured via the Apache configuration – apache2.conf / httpd.conf etc.).

Now you can imagine how slow and memory consuming it is that all the time the parent Apache process spawns new processes, kills old ones etc. in order to fulfill the client requests.

Now just to compare the Apace mpm prefork does not use the old forking way, but relies on a few Apache processes which handles all the requests without constantly being destroyed and recreated like with the prefork module.
This saves operations and system resources, threaded programming has already been proven to be more efficient way to handle tasks and is heavily adopted in GUI programming for instance in Microsoft Windows, Mac OS X, Linux Gnome, KDE etc.

There is plenty of information and statistical data which compares Apache running with prefork and respectively worker modules online.
As the goal of this article is not to went in depths with this kind of information I would not say more on it but let you explore online a bit more about them in case if you're interested.

The purpose of this article is to explain in short how to substitute the Apache2-MPM-Prefork and how your server performance could benefit out of the use of Apache2-MPM-Worker.
On Debian the default Apache process serving module in Apache 1.3x,Apache 2.0x and 2.2x is prefork thus the installation of apache2-mpm-worker is not "a standard way" to install Apache

Deciding to swith from the default Debian apache-mpm-prefork to apache-mpm-worker is quite a serious and responsible decision and in some cases might cause troubles, if you have decided to follow my article be sure to consider all the possible negative consequences of switching to the apache worker !

Now after having said a bunch of info which might be not necessary with the experienced system admin I'll continue on with the steps to install the apache2-mpm-worker.

1. Install the apache2-mpm-worker

debian:~# apt-get install apache2-mpm-worker php5-cgi
Reading state information... Done
The following packages were automatically installed and are no longer required:
The following packages will be REMOVED apache2-mpm-prefork libapache2-mod-php5
The following NEW packages will be installed apache2-mpm-worker
0 upgraded, 1 newly installed, 2 to remove and 46 not upgraded.
Need to get 0B/259kB of archives.After this operation, 6193kB disk space will be freed.

As you can notice in below's text confirmation which will appear you will have to remove the apache2-mpm-prefork and the apache2-mpm-worker modules before you can proceed to install the apache2-mpm-prefork.

You might ask yourself if I remove my installed libphp how would I be able to use my Apache with my PHP based websites? And why does the apt package manager requires the libapache2-mod-php5 to get removed.
The explanation is simple apache2-mpm-worker is not thread safe, in other words scripts which does use the php fork(); function would not work correctly with the Apache worker module and will probably be leading to PHP and Apache crashes.
Therefore in order to install the apache mod worker it's necessary that no libapache2-mod-php5 is existent on the system.
In order to have a PHP installed on the server again you will have to use the php5-cgi deb package, this is the reason in the above apt-get command I'm also requesting apt to install the php5-cgi package next to apache2-mpm-worker.

2. Enable the cgi and cgid apache modules

debian:~# a2enmod cgi
debian:~# a2enmod cgid

3. Activate the mod_actions apache modules

debian:~# cd /etc/apache2/mods-enabled
debian:~# ln -sf ../mods-available/actions.load
debian:~# ln -sf ../mods-available/actions.conf

4. Add configuration options in order to enable mod worker to use the newly installed php5-cgi

Edit /etc/apache2/mods-available/actions.conf vim, mcedit or nano (e.g. your editor of choice and add inside:

&ltIfModule mod_actions.c>
Action application/x-httpd-php /cgi-bin/php5
</IfModule>

After completing all the above instructions, you might also need to edit your /etc/apache2/apache2.conf to tune up, how your Apache mpm worker will serve client requests.
Configuring the <IfModule mpm_worker_module> in apache2.conf is necessary to optimize your newly installed mpm_worker module for performance.

5. Configure the mod_worker_module in apache2.conf One example configuration for the mod worker is:

<IfModule mpm_worker_module>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>

Consider the fact that this configuration is just a sample and it's in no means configured for serving Apache requests for high load Apache servers and you need to further play with the values to have a good results on your server.

6. Check that all is fine with your Apache configurations and no syntax errors are encountered

debian:~# /usr/sbin/apache2ctl -t
Syntax OK

If you get something different from Syntax OK track the error and fix it before you're ready to restart the Apache server.

7. Now restart the Apache server

debian:~# /etc/init.d/apache2 restart

All should run fine and hopefully your PHP scripts should be interpreted just fine through the php5-cgi instead of the libapache2-mod-php5.
Using the /usr/bin/php5-cgi will increase with some percentage your server CPU load but on other hand will drasticly decrease the Webserver memory consumption.
That's quite logical because the libapache2-mod-hp5 is loaded once during apache server whether a new instance of /usr/bin/php5-cgi is invoked during each of Apache requests via the mod worker.

There is one serious security flow coming with php5-cgi, DoS against a server processing scripts through php5-cgi is much easier to be achieved.
An example for a denial attack which could affect a website running with mod worker and php5-cgi, could be simulated from a simple user with a web browser which holds up the f5 or ctrl + r browser page refresh buttons.
In that case whenever php5-cgi is used the CPU load would rise drastic, one possible solution to this denial of service issues is by installing and using libapache2-mod-evasive like so:

8. Install libapache2-mod-evasive

debian:~# apt-get install libapache2-mod-evasive
The Apache mod evasive module is a nice apache module to minimize HTTP DoS and brute force attacks.
Now with mod worker through the php5-cgi, your apache should start serving requests more efficiently than before.
For some performance reasons some might even want to try out the fastcgi with the worker to boost the Apache performance but as I have never tried that I can't say how reliable a a mod worker with a fastcgi would be.

N.B. ! If you have some specific php configurations within /etc/php5/apache2/php.ini you will have to set them also in /etc/php5/cgi/php.ini before you proceed with the above instructions to install Apache otherwise your PHP scripts might not work as expected.

Mod worker is also capable to work with the standard mod php5 Apache module, but if you decide to go this route you will have to recompile your PHP lib manually from source as in Debian this option is not possible with the default php library.
This installation worked fine on Debian Lenny but suppose the same installation should work fine on Debian Squeeze as well as Debian testing/unstable.
Feedback on the afore-described mod worker installation is very welcome!

How to easy add Joomla 1.5 donate Paypal capabilities with Joomla PAYPAL DONATION MODULE

Wednesday, June 15th, 2011

PayPal donation Module Joomla Screenshot

Many joomla CMS installations are for Non-profit organizations or Non Government organizations. These are organizations which are not officially making profit and therefore this instituations are interested into donations to support their activities.

In this occasions adding Joomla paypal capabilities is very essential. There are plenty of modules which enables Joomla to support paypal monetary payments, however many of them are either paid or requires registration and thus it’s quite time consuming to set up a decent PayPal supporting module for Joomla.
After a bit of investigation thanks God, I’ve come across a module that is free of charge, easily downloadable (wihtout registration) and is also relatively easy to configure, these module is called PAYPAL DONATION MODULE
I’ve mirored the module to my server, just in case if the module disappears in the future.

Here are a very brief explanation on how the module can be downloaded installed and configured:

First Download (mod_ojdonation_pp) Paypal Donation Module here

Install it as joomla module via:

Extensions -> Install/Uninstall
menu

Afterwards, go to:

Extensions -> Module Manager

In the list of modules you will notice the Donate module which will be disabled. Use the Enable button to enable it.

Next by clicking on the Donate Module Name, one can configure the module, where the most essential configuration values that needs to be filled in are:

1. Title: – The title of the donation form:
2. Donation Title: – Title of donation picture to show in the webpage
3. Donation Amount: – Default donation amount user will donate with paypal by clicking on Donate button
4. Currency – Default currency the donators will use to donate to configured paypal account
5. Paypal ID: – The email address of paypal account your donators will donate to (This was a bit hard to understand since Paypal ID is not a number ID but the email address configured as an username in PayPal).
6. Donation Description: – Description text to appear before the Donate button
7. Donation Footer: – Text to appear after the Donate button

There are two ways one could add the donation module to show the donation form, on the joomla website:
a. One is to enable the donation button on every joomla webpage (I don’t like this kind of behaviour).

To use this kind of donate button display approach, you will have to select from the Donation module, conf options:
– Show on FrontPage: and Show Title:

Also make sure the Enabled: option is set to Yes

b. Second approach is to set the PayPal Donation form only to appear on a single menu, to do so:

While in Paypal Donation Module configuration in Menu Assignment section, select:

Select Menu Item(s) from the List
instead of the default All value set for Menus.

The last setting to be choosen is the paypal donation form page location (where exactly on the selected pages the form will appear).

The form location is set from the Position: dropdown menu, the option which I found to be the best one for me was the bottom option. However just play with the Position setting and choose the one that will be best for you.

Then scroll on in the Menu Selection: and choose only the menus where you want a paypal donation form to appear.

Finally to save all the recent made settings, click on Apply and refreshing in a new page should show you paypal’s money donation form in joomla

If all is configured fine with Joomla’s – Paypal Donation Module you should get on your webpage:

PayPal donation Module in Joomla Screenshot
 

How to harden Linux Security and imprpove network efficiency on Kernel sysctl Level to Stop SYN flood

Friday, July 8th, 2011

Power up Linux and protect against DDoS with sysctl var optimization

Some long time ago I’ve written an article Optimizing Linux tcp/ip networking

In the article I’ve examined a number of Linux kernel sysctl variables, which significantly improve the way TCP/IP networking is handled by a non router Linux based servers.

As the time progresses I’ve been continuing to read materials on blogs and internet sites on various tips and anti Denial of Service rules which one could apply on newly installed hosting (Apache/MySql/Qmail/Proxy) server to improve webserver responce times and tighten the overall security level.

In my quest for sysctl 😉 I found a few more handy sysctl variables apart from the old ones I incorporate on every Linux server I adminstrate.
The sysctl variables improves the overall network handling efficiency and protects about common SYN/ACK Denial of service attacks.

Here are the extra sysctl variables I started incorporating just recently:

############ IPv4 Sysctl Settings ################
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'
net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
##################################################
#
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routers
net.ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignments
net.ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interface
net.ipv6.conf.default.max_addresses = 1
#####################################################

 

To use this settings paste the above sysctl variables in /etc/sysctl.conf and ask sysctl command to read and apply the newly added conf settings:

server:~# sysctl -p
...

Hopefully you should not get errors while applying the sysctl settings, if you get some errors, it’s possible some of the variable is differently named (depending on the Linux kernel version) or the Linux distribution on which sysctl’s are implemented.

For some convenience I’ve created unified sysctl variables /etc/sysct.conf containing the newly variables I started implementing to servers with the ones I already exlpained in my previous post Optimizing Linux TCP/IP Networking

Here is the optimized / hardened sysctl.conf file for download

I use this exact sysctl.conf these days on both Linux hosting / VPS / Mail servers etc. as well as on my personal notebook 😉

Here is also the the complete content of above’s sysctl.conf file, just in case if somebody wants to directly copy/paste it in his /etc/sysctl.conf

# Sysctl kernel variables to improve network performance and protect against common Denial of Service attacks
# It's possible that not all of the variables are working on all Linux distributions, test to make sure
# Some of the variables might need a slight modification to match server hardware, however in most cases it should be fine
# variables list compiled by hip0
### https://www.pc-freak.net
#### date 08.07.2011
############ IPv4 Sysctl Kernel Settings ################
net.ipv4.ip_forward = 0
# ( Turn off IP Forwarding )
net.ipv4.conf.default.rp_filter = 1
# ( Control Source route verification )
net.ipv4.conf.default.accept_redirects = 0
# ( Disable ICMP redirects )
net.ipv4.conf.all.accept_redirects = 0
# ( same as above )
net.ipv4.conf.default.accept_source_route = 0
# ( Disable IP source routing )
net.ipv4.conf.all.accept_source_route = 0
# ( - || - )net.ipv4.tcp_fin_timeout = 40
# ( Decrease FIN timeout ) - Useful on busy/high load server
net.ipv4.tcp_keepalive_time = 4000
# ( keepalive tcp timeout )
net.core.rmem_default = 786426
# Receive memory stack size ( a good idea to increase it if your server receives big files )
##net.ipv4.tcp_rmem = "4096 87380 4194304"
net.core.wmem_default = 8388608
#( Reserved Memory per connection )
net.core.wmem_max = 8388608
net.core.optmem_max = 40960
# ( maximum amount of option memory buffers )
# tcp reordering, increase max buckets, increase the amount of backlost
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.tcp_reordering = 5
##net.core.hot_list_length = 256
net.core.netdev_max_backlog = 1024
#Enable ExecShield protection (randomize virtual assigned space to protect against many exploits)
kernel.randomize_va_space = 1
#Increase the number of PIDs processes could assign this is very needed especially on more powerful servers
kernel.pid_max = 65536
# Prevent against the common 'syn flood attack'net.ipv4.tcp_syncookies = 1
# Controls the use of TCP syncookies two is generally a better idea, though you might experiment
#net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
###################################################
############## IPv6 Sysctl Settings ################
# Number of Router Solicitations to send until assuming no routers are present.
net.ipv6.conf.default.router_solicitations = 0
# Accept Router Preference in RA? Again not necessery if the server is not a router
net.ipv6.conf.default.accept_ra_rtr_pref = 0
# Learn Prefix Information in Router Advertisement (Unnecessery) for non-routersnet.
ipv6.conf.default.accept_ra_pinfo = 0
# disable accept of hop limit settings from other routers (could be used for DoS)
net.ipv6.conf.default.accept_ra_defrtr = 0
# disable ipv6 global unicasts server assignmentsnet.
ipv6.conf.default.autoconf = 0
# neighbor solicitations to send out per address (better if disabled)
net.ipv6.conf.default.dad_transmits = 0
# disable assigning more than 1 address per network interfacenet.
ipv6.conf.default.max_addresses = 1
#####################################################
# Reboot if kernel panic
kernel.panic = 20

These sysctl settings will tweaken the Linux kernel default network settings performance and you will notice the improvements in website responsiveness immediately in some cases implementing this kernel level goodies will make the server perform better and the system load might decrease even 😉

This optimizations on a kernel level are not only handy for servers, their implementation on Linux Desktop should also have a positive influence on the way the network behaves and could improve significantly the responce times of opening pages in Firefox/Opera/Epiphany Torrent downloads etc.

Hope this kernel tweakenings are helpful to someone.
Cheers 😉