Posts Tagged ‘strange’

Fixing strange Debian Linux Squeeze system overloads and Apache Webserver crashes

Friday, October 19th, 2012

For quite some time, my home run server pre-installed with Debian Squeeze Linux has been crashing in a very strange circumstances inside dmesg kernel log and in /var/log/messages on times, when this crashes occur I see errors / warnings spitting not very helpful kernel debug messages like this:


Oct 16 11:32:28 pcfreak kernel: [66657.797930] Pid: 0, comm: swapper Not tainted 2.6.32-5-amd64 #1
Oct 16 11:32:28 pcfreak kernel: [66657.797931] Call Trace:
Oct 16 11:32:28 pcfreak kernel: [66657.797933] [] ? select_nohz_load_balancer+0x94/0x163
Oct 16 11:32:28 pcfreak kernel: [66657.797943] [] ? __report_bad_irq+0x30/0x7d
Oct 16 11:32:28 pcfreak kernel: [66657.797945] [] ? note_interrupt+0x105/0x16e
Oct 16 11:32:28 pcfreak kernel: [66657.797948] [] ? handle_fasteoi_irq+0x93/0xb5
Oct 16 11:32:28 pcfreak kernel: [66657.797952] [] ? handle_irq+0x17/0x1d
Oct 16 11:32:28 pcfreak kernel: [66657.797954] [] ? do_IRQ+0x57/0xb6
Oct 16 11:32:28 pcfreak kernel: [66657.797956] [] ? ret_from_intr+0x0/0x11
Oct 16 11:32:28 pcfreak kernel: [66657.797957] [] ? poll_idle+0x28/0x5b
Oct 16 11:32:28 pcfreak kernel: [66657.797963] [] ? poll_idle+0xa/0x5b
Oct 16 11:32:28 pcfreak kernel: [66657.797965] [] ? cpuidle_idle_call+0x94/0xee
Oct 16 11:32:28 pcfreak kernel: [66657.797968] [] ? cpu_idle+0xa2/0xda
Oct 16 11:32:28 pcfreak kernel: [66657.797971] [] ? early_idt_handler+0x0/0x71
Oct 16 11:32:28 pcfreak kernel: [66657.797974] [] ? start_kernel+0x3dc/0x3e8
Oct 16 11:32:28 pcfreak kernel: [66657.797976] [] ?x86_64_start_kernel+0xf9/0x106

and this:


Oct 16 15:53:14 pcfreak kernel: [82297.972509] apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Oct 16 15:53:30 pcfreak kernel: [82297.972513] apache2 cpuset=/ mems_allowed=0
Oct 16 15:53:30 pcfreak kernel: [82297.972515] Pid: 8943, comm: apache2 Not tainted 2.6.32-5-amd64 #1
Oct 16 15:53:30 pcfreak kernel: [82297.972517] Call Trace:
Oct 16 15:53:30 pcfreak kernel: [82297.972523] [] ? oom_kill_process+0x7f/0x23f
Oct 16 15:53:30 pcfreak kernel: [82297.972527] [] ? timekeeping_get_ns+0xe/0x2e
Oct 16 15:53:30 pcfreak kernel: [82297.972529] [] ? __out_of_memory+0x12a/0x141
Oct 16 15:53:30 pcfreak kernel: [82297.972531] [] ? out_of_memory+0x140/0x172
Oct 16 15:53:30 pcfreak kernel: [82297.972534] [] ? __alloc_pages_nodemask+0x4ec/0x5fb
Oct 16 15:53:30 pcfreak kernel: [82297.972538] [] ? do_wp_page+0x386/0x707
Oct 16 15:53:30 pcfreak kernel: [82297.972541] [] ? autoremove_wake_function+0x9/0x2e
Oct 16 15:53:30 pcfreak kernel: [82297.972544] [] ? __wake_up_common+0x44/0x72
Oct 16 15:53:30 pcfreak kernel: [82297.972547] [] ? __wake_up+0x30/0x44
Oct 16 15:53:30 pcfreak kernel: [82297.972549] [] ? handle_mm_fault+0x704/0x80f
Oct 16 15:53:30 pcfreak kernel: [82297.972553] [] ? do_page_fault+0x2e0/0x2fc
Oct 16 15:53:30 pcfreak kernel: [82297.972556] [] ? page_fault+0x25/0x30


Oct 16 18:41:55 pcfreak kernel: [ 6582.554746] Mem-Info:
Oct 16 18:41:55 pcfreak kernel: [ 6582.554747] Node 0 DMA per-cpu:
Oct 16 18:41:55 pcfreak kernel: [ 6582.554751] CPU 0: hi: 0, btch: 1 usd: 0
Oct 16 18:41:55 pcfreak kernel: [ 6582.554753] CPU 1: hi: 0, btch: 1 usd: 0
Oct 16 18:41:55 pcfreak kernel: [ 6582.554755] Node 0 DMA32 per-cpu:
Oct 16 18:41:55 pcfreak kernel: [ 6582.554758] CPU 0: hi: 186, btch: 31 usd: 0
Oct 16 18:41:55 pcfreak kernel: [ 6582.554760] CPU 1: hi: 186, btch: 31 usd: 0
Oct 16 18:41:55 pcfreak kernel: [ 6582.554762] Node 0 Normal per-cpu:
Oct 16 18:41:55 pcfreak kernel: [ 6582.554765] CPU 0: hi: 186, btch: 31 usd: 5
Oct 16 18:41:55 pcfreak kernel: [ 6582.554767] CPU 1: hi: 186, btch: 31 usd: 0
Oct 16 18:41:55 pcfreak kernel: [ 6582.554773] active_anon:1580557 inactive_anon:308231 isolated_anon:9504
Oct 16 18:41:55 pcfreak kernel: [ 6582.554775] active_file:148 inactive_file:220 isolated_file:32
Oct 16 18:41:55 pcfreak kernel: [ 6582.554776] unevictable:0 dirty:5 writeback:494 unstable:0
Oct 16 18:41:55 pcfreak kernel: [ 6582.554777] free:12063 slab_reclaimable:4262 slab_unreclaimable:17553
Oct 16 18:41:55 pcfreak kernel: [ 6582.554778] mapped:148 shmem:43 pagetables:89423 bounce:0
Oct 16 18:41:55 pcfreak kernel: [ 6582.554781] Node 0 DMA free:15880kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive
_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15328kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB
slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct 16 18:41:55 pcfreak kernel: [ 6582.554794] lowmem_reserve[]: 0 2947 7995 7995
Oct 16 18:41:55 pcfreak kernel: [ 6582.554798] Node 0 DMA32 free:24672kB min:4212kB low:5264kB high:6316kB active_anon:2153732kB inactive_anon:538456kB activ
e_file:32kB inactive_file:56kB unevictable:0kB isolated(anon):6912kB isolated(file):0kB present:3017744kB mlocked:0kB dirty:16kB writeback:336kB mapped:184kB
shmem:168kB slab_reclaimable:4400kB slab_unreclaimable:21908kB kernel_stack:1816kB pagetables:131140kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scann
ed:42 all_unreclaimable? no
Oct 16 18:41:55 pcfreak kernel: [ 6582.554812] lowmem_reserve[]: 0 0 5048 5048
Oct 16 18:41:55 pcfreak kernel: [ 6582.554815] Node 0 Normal free:7700kB min:7216kB low:9020kB high:10824kB active_anon:4168496kB inactive_anon:694468kB acti
ve_file:560kB inactive_file:824kB unevictable:0kB isolated(anon):31104kB isolated(file):128kB present:5169180kB mlocked:0kB dirty:4kB writeback:1640kB mapped
:408kB shmem:4kB slab_reclaimable:12648kB slab_unreclaimable:48296kB kernel_stack:3488kB
pagetables:226552kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_
scanned:224 all_unreclaimable? no
Oct 16 18:41:55 pcfreak kernel: [ 6582.554829] lowmem_reserve[]: 0 0 0 0
Oct 16 18:41:55 pcfreak kernel: [ 6582.554832] Node 0 DMA: 2*4kB 2*8kB 3*16kB 4*32kB 3*64kB 3*128kB 1*256kB 1*512kB 2*1024kB 2*2048kB 2*4096kB = 15880kB
Oct 16 18:41:55 pcfreak kernel: [ 6582.554842] Node 0 DMA32: 114*4kB 73*8kB 273*16kB 256*32kB 85*64kB 10*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 2
4672kB
Oct 16 18:41:55 pcfreak kernel: [ 6582.554852] Node 0 Normal: 867*4kB 103*8kB 7*16kB 13*32kB 7*64kB 1*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 0*4096kB = 7700
kB
Oct 16 18:41:55 pcfreak kernel: [ 6582.554862] 89702 total pagecache pages
Oct 16 18:41:55 pcfreak kernel: [ 6582.554864] 89257 pages in swap cache
Oct 16 18:41:55 pcfreak kernel: [ 6582.554866] Swap cache stats: add 4155760, delete 4066503, find 618278/801429
Oct 16 18:41:55 pcfreak kernel: [ 6582.554868] Free swap = 0kB
Oct 16 18:41:55 pcfreak kernel: [ 6582.554870] Total swap = 5787636kB
Oct 16 18:41:55 pcfreak kernel: [ 6582.581389] 2096640 pages RAM
Oct 16 18:41:55 pcfreak kernel: [ 6582.581392] 60657 pages reserved
Oct 16 18:41:55 pcfreak kernel: [ 6582.581394] 330845 pages shared
Oct 16 18:41:55 pcfreak kernel: [ 6582.581397] 2012293 pages non-shared

It took, me long time of thinking and pondering what is causing this errors ….
I thought it is due to some failing RAM bank or some kind of conflict URL hardware inconpitability, I had some thoghts that it is possible Hard Disk is failing or have some bad blocks; However as I bought the machine brand new and besides that it is not assembled one PC but brand one Lenovo ThinkEdge, I’ve further thought and investigated if Apache is failing due to some problem with Apache modules. After reviewing all system installed modules, I’ve found php5-suhosin, was installed on the system (probably as a dependency package) to something else I previously installed ??
I don’t have a very positive feedback on some other servers I configured with Apache, whether php5-suhosin was installed so decided to try removing it ….:


# dpkg -r php5-suhosin
.....
# dpkg --purge php5-suhosin

I’ve also lowered down a bit the StartServers and MaxSpareServers (mod_prefork_modules) section in /etc/apache2/apache2.conf:

StartServers there was set to: 700 and MaxSpareServers to 150, I’ve changed the values to read as so:



StartServers 500
MinSpareServers 100
MaxSpareServers 120
MaxClients 1000
MaxRequestsPerChild 10000

Just for info machine is with 8 Gigabytes of Memory and has 1x 2 cores CPU:


# free -m |grep -i 'mem:'
Mem: 7953 7871 81 0 91 753


# cat /proc/cpuinfo |grep -i proces -A 5
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Pentium(R) CPU G630 @ 2.70GHz
stepping : 7
--
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel(R) Pentium(R) CPU G630 @ 2.70GHz
stepping : 7

After that to make module already loaded in system memory by Apache main (parent) process, I restarted Apache as well:


# apache2ctl -k restart

Following that changes – Thanks God! ; I no longer experience the weird errors and Server overloads 😉