ProLiant Servers (ML,DL,SL)
1748124 Members
3246 Online
108758 Solutions
New Discussion

very very slow reads (50-250kB/s) with DL380G5 P400 512+battery, 4x600GB TOSHIBA SAS in RAID 5

 
LinusM
Collector

very very slow reads (50-250kB/s) with DL380G5 P400 512+battery, 4x600GB TOSHIBA SAS in RAID 5

Six weeks ago I started to experience a huge performance drop with a DL380G5 server running ESXi 4.1. No disks are reported failed by hpacucli nor with the vSphere client (see attached files).

 

Here is a test done on a VM on that host:

$  hdparm -t /dev/mapper/VolGroup00-LogVol00

/dev/mapper/VolGroup00-LogVol00:
 Timing buffered disk reads:    2 MB in 15.32 seconds = 133.70 kB/sec

I started by switching the battery, then the whole server chassi including the raid controller (= inserting the disks in another dl380g5 system), then updating the firmwares and running a memory check. I also updated the host to ESXi 5.1.

 

An automatically initiated parity optimization scan took over one month, I had hopes that after it the speeds would be back to normal, but it didn't help.

 

I have not been able to extract smartctl info from the physical drives, but could one of the disks might be behaving badly even it is reported as OK?

 

I have run out of options and I hope someone is a lot wiser and wants to help!

 

 

5 REPLIES 5
LinusM
Collector

Re: very very slow reads (50-250kB/s) with DL380G5 P400 512+battery, 4x600GB TOSHIBA SAS in RAID 5

The disk leds blink all four about every second, all green.

 

Do anyone have a clue on where to start?

Marko71
Frequent Advisor

Re: very very slow reads (50-250kB/s) with DL380G5 P400 512+battery, 4x600GB TOSHIBA SAS in RAID 5

Check to make sure you have got your 512MB cache set to 100% writes, if it is then as the controller turns off the drive on caches by default then you will have no read ahead caching enabled at all.

 

Make sure it's set to at least 25% read caching.

 

Other than this all I can think of is a fault controller or bad cabling.

 

Hope this helps.

Marko71
Frequent Advisor

Re: very very slow reads (50-250kB/s) with DL380G5 P400 512+battery, 4x600GB TOSHIBA SAS in RAID 5

Actually, I just read your diags output & see your caching is ok. 

 

I notice it can't work out what transfer rates those drives support.  Although they almost certainly support both 3Gb/s &  6Gb/s it's odd that it doesnt know.  This might indicate an incompatability between the drives and the controller, or cables or SAS backplane.

 

 

LinusM
Collector

Re: very very slow reads (50-250kB/s) with DL380G5 P400 512+battery, 4x600GB TOSHIBA SAS in RAID 5

Thank you for your time and suggestions. As it didn't help to place the disks a new chassi with tested cables I think it's not the cables. The same disks have been working ok for two years and suddenly a drop.

 

Do anyone else have any hints?

Jan Soska
Honored Contributor

Re: very very slow reads (50-250kB/s) with DL380G5 P400 512+battery, 4x600GB TOSHIBA SAS in RAID 5

Hello,

we've had something similar in the same controller and 300GB 10k sas drives. Raid group we created as backup partition, worked well for almost 5 years. Suddenly, server backup started to be really poor, causing were slow responses during backup without any visible reason. It was very strange. After 5 weeks of such behavior, one of disk in this raid5 diskgroup raised predictive failure. After replacing the drive, everything works well now...

 

So - maybe try to but your drives under heavy workload - if there isany hw problem should be visible sooner.

Or, if you have time, try to replace your drives one by one and rebuild raid - it is highly likely one drive causes problem.

 

Jan