Freezing kernel 2.6.8.1 on HP DL380 G3 (high load kswapd)

[Krigler Pavol]

I have HP DL380 G3 P4 with HT with 1GB of RAM, 3x72GB SCSI with RAID 5 on Smart Array 5i. Running vanilla kernel 2.6.8.1 with compiled SMP, HT, IO-APIC, ACPI, NOHIGHMEM, IRQBALLANCE. Server freezes 2x per week, no keyboard and ping responses. After three days I have high iowait load (50-70%) and small system idle (50-20%). Always when I run top the kswapd0 is on the first place with big TIME+. I try also put boot options acpi=noidle but with no effect. Here are few lines from top.

top - 18:14:33 up 3 days, 5:18, 1 user, load average: 1.35, 0.74, 0.64
Tasks: 88 total, 1 running, 87 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.2% us, 16.0% sy, 0.0% ni, 22.3% id, 56.5% wa, 0.0% hi, 0.2% si
Mem: 905912k total, 902148k used, 3764k free, 4764k buffers
Swap: 1954296k total, 27112k used, 1927184k free, 26064k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
47 root 15 0 0 0 0 S 21.4 0.0 137:38.66 kswapd0
11720 postgres 15 0 16216 9988 14m S 7.1 1.1 0:59.01 postmaster
21727 postgres 18 0 18328 11m 14m D 2.9 1.3 0:00.09 postmaster
11429 nobody 15 0 79456 2264 73m S 1.0 0.2 0:11.28 httpd

My .config file is in attachment

Krigler Pavol

[dirk dierickx · Accepted solution]

Are you running out of memory? How much memory is the largest process taking.
In the 2.6.x kernels you can adjust the swapping with the vm.swappiness settings.

sysctl vm.swappiness

if you set this to 0 the system will try to swap as less as possible and all processes will try to stay in memory, if set to 100 the most active process will get all the ram and all others are swapped out.

[Don_89]

If your running Insight Manager agetns, disable them and see what happens.

[Krigler Pavol]

I am running on Slackware 10 and no Insight Manager agetns are installed. I think that there is no problem with swapping. Problem is in huge iowait load when disks are used. Read or write operations are very slow directly after boot. I use cciss driver for smart array 5i. My vm.swappiness is from default set to 60.
When I run disk performace test iowait peak is 90% and some errors shown.


hdparm -tT /dev/cciss/c0d0

/dev/cciss/c0d0:
Timing buffer-cache reads: 2716 MB in 2.00 seconds = 1357.53 MB/sec
BLKFLSBUF failed: Invalid request code
Timing buffered disk reads: 296 MB in 3.01 seconds = 98.22 MB/sec
BLKFLSBUF failed: Invalid request code

[Piotr Dziubanii]

what is your bios revision? if it's not currrent, try upgrade

p.

[Krigler Pavol]

After upgrade of System BIOS (version 2004.06.23) and flashing Smart Array 5i controller (version 2.58 B) there are no freezes, but the a huge iowait is still there. I think it has something to do with performance of SA 5i controller. I have found some forum tips (, but it requires to buy a new array controller or "+" to SA 5i):

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=206252
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=664924

[Piotr Dziubanii]

we've got similar problems, strange system load and instability

hardware: prolian ml350g3 and raid1 on compaq smart array

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=703662

[HGN]

Hi

We have quite a few of these servers(ML350 with smary array raid controller with raid 1) in our environment and have experienced many freeze like this. The server has very high swap usage also, from you case dont see much swap usage, for us when we call redhat they suggest us to upgrade to latest kernel which our application people are not comfortable with since the same environment works fine with similar other servers.

Rgds

Gopi