high load when copying files on DL 585

[DavidDD_1]

Hello,

We noticed strange behavior running our new servers ProLiant DL585 4 of them. When copying files this is what happens example:

if we want to copy 2Gb file on the same File system the Load of the server goes up way to much example:

#we create 2Gb big file:
dd if=/dev/zero of=test.file bs=1024 count=2048000

#we look at the load (the server is not doing anything)
cat /proc/loadavg
0.24 1.21 1.22 1/1423 6644

#then we copy the file a few times over and over and check load between:
cp -a test.file test.me
cat /proc/loadavg
3.93 1.92 1.45 1/1756 7117
cp -a test.file test.me
cat /proc/loadavg
9.50 4.06 2.23 1/3159 8845
cp -a test.file test.me
cat /proc/loadavg
16.05 7.92 3.81 2/5269 11406
cp -a test.file test.me
cat /proc/loadavg
14.16 9.32 4.64 1/5293 11768
cp -a test.file test.me
cat /proc/loadavg
14.58 11.03 5.72 1/5268 12187

As you can see load is going up way too much! We tried the same test with HP DL 360 with 1Gb Memory and Single processor 32Bit and the result is:

cat /proc/loadavg
2.19 1.78 1.26 2/109 24104
cp -a test.file test.me
cat /proc/loadavg
2.74 2.06 1.42 2/108 24343
cp -a test.file test.me
cat /proc/loadavg
3.21 2.44 1.62 1/115 24834
cp -a test.file test.me
cat /proc/loadavg
3.66 2.92 1.89 1/109 26202
cp -a test.file test.me
cat /proc/loadavg
3.13 2.92 2.01 1/115 26472

As you can see the load is much lower then the 64Bit server with 4 Processors and 16Gb Memory is this normal? When the load is more then 10 on the DL 585. The responces are slow for what ever you want to do on the server. Does anyone else have the same problem on a DL 585 server?

Please help us find the solution.

Data about servers:

ProLiant DL585 G1
4x processors 2.8Ghz
16Gb Memory
3x disk SCSI 320 300Gb / RAID 5

SuSE Enterprise Server 9
Kernel: 2.6.5-7.252-smp
Partition 550Gb EXT3 + 1Gb swap

[Stuart Browne]

What's the RAID controller in the DL585? how much cache does it have on it. What is the cache mode set to?

Is all of the memory recognized by the system (/proc/meminfo) ? What are the buffers and cache falues (from /proc/meminfo again) between the copies?

[DavidDD_1]

Hello,

thank you for your help... here are the requested infos...

The Contoller is Smar Array 5i (i think plus) and it has 48Mb Cache onboard.

The memory is all there 16Gb of it... below are some output info for you:

Controller Smart Array 5i in Slot 0
Bus Interface: pci
slot: 0
Cache serialnumber: xxxxxxxx
RAID 6 (ADG) status: False
Controller Status: OK
Chassis Slot: 1
Hardware Revision: Rev B
Firmware Version: 2.62
Rebuild Priority: Low
Expand Priority: Low
Surface Scan Delay: 15 sec
Cache Board Present: True
Cache Status: OK
Accelerator Ratio: 50/50 (read/write)
Read Cache Size: 24 MB
Write Cache Size: 24 MB
Total Cache Size: 48 MB
Battery Backed Cache Size: 48 MB
Non Battery Backed Cache Size: 0 MB
Battery Pack Count: 1
Battery Status: OK

this meminfo was done when the load is 1:

cat /proc/loadavg ; cat /proc/meminfo
1.02 0.89 0.78 1/378 32344
MemTotal: 16249084 kB
MemFree: 122984 kB
Buffers: 641344 kB
Cached: 13553280 kB
SwapCached: 0 kB
Active: 9699884 kB
Inactive: 5897792 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 16249084 kB
LowFree: 122984 kB
SwapTotal: 1052216 kB
SwapFree: 1051924 kB
Dirty: 7700 kB
Writeback: 0 kB
Mapped: 987092 kB
Slab: 443868 kB
Committed_AS: 2494880 kB
PageTables: 52772 kB
VmallocTotal: 536870911 kB
VmallocUsed: 10276 kB
VmallocChunk: 536860575 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB


this meminfo was done when the copying is in progress (one file 2Gb) and load raises up to 14:

cat /proc/loadavg ; cat /proc/meminfo
14.34 5.84 2.63 1/4576 9033
MemTotal: 16249084 kB
MemFree: 1788864 kB
Buffers: 497672 kB
Cached: 9170668 kB
SwapCached: 0 kB
Active: 5457200 kB
Inactive: 7622724 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 16249084 kB
LowFree: 1788864 kB
SwapTotal: 1052216 kB
SwapFree: 1050680 kB
Dirty: 190184 kB
Writeback: 0 kB
Mapped: 3353196 kB
Slab: 585784 kB
Committed_AS: 15354988 kB
PageTables: 690824 kB
VmallocTotal: 536870911 kB
VmallocUsed: 10276 kB
VmallocChunk: 536860575 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

at this load if for instance i do "df -h" it takes about 3.5 seconds for the command to output:

time df -h
Filesystem Size Used Avail Use% Mounted on
/dev/cciss/c0d0p2 549G 221G 301G 43% /
tmpfs 7.8G 12K 7.8G 1% /dev/shm

real 0m3.569s
user 0m0.001s
sys 0m0.002s

[Stuart Browne]

What I find interesting here is that before doing the copy, the Cached value is about 13GB. After the copy, it's only 9GB. You can see similar moves in the Active values. Where did 4GB of cache-data go? Odd.

The only place they can go is back into active memory.

What else are these servers doing at the same time?

The more I look at it, the RAID controller is highly inadequate for the job you're putting it through.

Use either top or vmstat, show how many processes are waiting. Grab sar (iostat package I believe) and see what the IO wait times are like.

[DavidDD_1]

l the server is running apache2 and mysql it serves only one php script which writes stats to the mysql about client status (browser, system, ip,....) there are not so many hits on this php script. usually around 200 concurrent apache processes running.

# ps aux |grep http |wc -l
171

Before this server we had an 1 processor server of another brand and it did its job good (based on the load which was always around 1 and up to 3 when copying large files without large server performance issues)

The same problem is on all 4 server (other one is serving only mysql and the other two are serving only web page)

here are the tests about vmstat and sar (iowait)

attached is the output of two programs:

vmstat 1
and
sar -P ALL 1 0

they are both in one txt file...

the logging of both programs was done on start with load of 0.8 and then i stared to copy file by file (one at a time) of (2Gb) the load raised up to 23 and when i canceled copy the load went back to 2 and then I ended the capture of vmstat and sar (when load was at 23 I did (df) and it took 5seconds to complete)

[Stuart Browne]

Those are some scary numbers. Those IO waits are just scary.

This could be caused either by an old driver revision, or the write-cache mode of the container being set up on an odd manner.

What kernel revision, and cciss driver version are you using? What is the write-cache mode? (WriteBack, WriteThrough.. ?)

[DavidDD_1]

The kernel is a 2.6.5-7.252-smp (Sles 9 - SP3)

cciss driver is: HP CISS Driver (v 2.6.8) (defualt SLES 9 SP3)
I have found a new driver on http://sourceforge.net/projects/cciss (cciss-2.6.10-1) but i didnt install it yet. I've searched on HP site but did not find an binary one for 252 kernel. Latest was for 191.

What is the write-cache mode? (WriteBack, WriteThrough.. ?) where do i look for this? I didnt find it? But here are some infos about the controler setup:

Controller Smart Array 5i in Slot 0
Bus Interface: pci
slot: 0
Cache serialnumber: XXXXXX
RAID 6 (ADG) status: False
Controller Status: OK
Chassis Slot: 1
Hardware Revision: Rev B
Firmware Version: 2.62
Rebuild Priority: Low
Expand Priority: Low
Surface Scan Delay: 15 sec
Cache Board Present: True
Cache Status: OK
Accelerator Ratio: 50/50 (read/write)
Read Cache Size: 24 MB
Write Cache Size: 24 MB
Total Cache Size: 48 MB
Battery Backed Cache Size: 48 MB
Non Battery Backed Cache Size: 0 MB
Battery Pack Count: 1
Battery Status: OK

Smart Array 5i in Slot 0
array A
Interface Type: Parallel SCSI
Unused Space: 0 MB
Status: OK

Smart Array 5i in Slot 0
logicaldrive 1
Size: 558 GB
Fault Tolerance: 5
Heads: 255
Sectors per Track: 63
Cylinders: 65535
Stripe Size: 16 KB
Status: Ok
Array Accelerator: Enabled
Has Data On Drive: True
Preferred Controller Chassis Slot: 1

physicaldrive 1:0
SCSI Bus: 1
SCSI ID: 0
Status: OK
Drive Type: Data Drive
Interface Type: Parallel SCSI
Size: 300 GB
Transfer Mode: Ultra 3 Wide
Transfer Speed: 160 MB/Sec
Rotational Speed: 10000
Firmware Revision: HPB1
serialnumber: XXXXXXXXXXX

[Stuart Browne]

The write cache mode is set in the controller BIOS. I'm not sure if you can do it through software whilst online.

If memory serves, write-back will get you the best performance and reliability (given you have battery-backup for the controller).

But yea. Update those cciss drivers. I'm pretty sure the 'binary' packages just re-compile the driver for the kerenel you're running anyway, so you need the dev-environment set up enough on one of these servers to get the driver for you.

[DavidDD_1]

I have rebooted the server and looked for an option write-back in the bios and in the controller 5i setup at boot but there was nothing to configure (you could just create an Logical Drive)

The only options I have found were in the Array Configuration in Linux and they are shown abow.

I updated the cciss driver from the official web page (used the version 2.6.10)

But it did not help much.

Any other ideas?