DL 380 G5 RAID 5 Slow

[Chris Camm]

I have a DL 380 G5 with P400 controller.

Disks 1 and 2 are in RAID0+1
Disks 2-8 are in RAID 5

The disks are all 68GB SAS 15Krpm

Reading and writing to the RAID 0+1 array is very quick.

Reading from the RAID 5 is very slow, writing even slower.

This problem was not there on the same setup on a DL 380 G4 with four disks in the RAID 5 array. It is also not there on any of my other DL380 G5 Server with a P400 Controller and only four disks in the RAID5 array.

The array cache is set to 100% read and 0% write.

I have the latest firmware patches on the server, raid controller and disks. The latest SSD's are installed too.

Any ideas greatly appreciated.

[juan quesada]

RAID 5
Advantages:
â ¢ Has high read performance.
â ¢ Data is not lost if one physical drive fails.
â ¢ More drive capacity is usable than with RAID 1+0â parity information requires only the storage
space equivalent to one physical drive.
Disadvantages:
â ¢ Has relatively low write performance.
â ¢ Data is lost if a second drive fails before data from the first failed drive is rebuilt.

http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c00729544/c00729544.pdf

The behavior you have is odd, can you run and ADU (array diagnostic utility) report and attach it?

Regards

[Chris Camm]

Please find attached the ARU report

Thanks

[juan quesada]

Hi,

the Image is broke.

*.zip file must have an *.txt file inside.

regards

[Chris Camm]

Very sorry.

Here is the txt file

Regards

Chris

[juan quesada]

ADU is OK. storage subsystem is fine.

If you have w2k3 updating the Smart Array P400 driver to revision 6.6.2.32 improves performance to expected levels.

If w2k there is no solution as a new driver to resolve the issue.

As a workaround:
The write caching can be disabled by setting the Read/Write ratio to 100/0 on the logical drive being utilized

[Chris Camm]

I am currently running with 6.8.0.32 I will downgrade and see what happens.

Thanks

[Mi6t0]

Try with the newest driver for p400 (6.10.x.x).

About the read/write priority - you don't have a cache module,so it is just an automatic setting. With cache module you may feel better performance and you will be able to edit the read/write prio.

Kind regards,

[KarloChacon]

fi the recommendations above dont work:
latest driver / disable cache

getting an BBWC update would be the only
solution in case you have 256 MB BBWC right now

regards

[juan quesada]

customer have 512

Total Memory Size 512 MBytes

[Chris Camm]

Downgraded to 6.6.2.32 and rebooted.

Same problem

Upgraded to 6.10.0.32 and rebooted. Also upgraded the BIOS to 2008.1.24 and the disk firmware to HPD7.

Running some tests now

[KarloChacon]

hi again

and what about juan quesada suggestion

" As a workaround:
The write caching can be disabled by setting the Read/Write ratio to 100/0 on the logical drive being utilized"

did you give it a try?

regards

[Chris Camm]

This has always been disabled.

Many thanks

The server has just rebooted though unexpectedly so I am now thinking of rolling back to the original version of the driver if its a driver issue.

I will keep you informed.

Thanks again

[Chris Camm]

I have been doing more testing and it seems to be that writing to the server is a lot quicker than reading from the server.

More investigation seems to point to the TCP Offload Engine. I have now done the following:

Run:

netsh int ip set chimney disabled

I have Disabled Receive-Side Scaling, TCP Offload Engine all under the teaming on the server. I have installed the latest drivers for the NIC's and the teaming software. Deleted the TEAM and recreated it. Rebooted after all these changes. I have made sure that the Switch is set to AUTO/AUTO and that the NIC's are also set to AUTO/AUTO.

I am running out of ideas. Searching the forum as found some help as I have gone through everything in the post:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1158952&admit=109447626+1204648547079+28353475

But I am still having the same errors.

Anymore suggestions greatly appreciated.

[juan quesada · Accepted solution]

BTW this procedure is use without the need of disabling anything, no TOE or RSS so the NIC will have all functions available.

When you try to update the driver it will not overwrite the existing one, so, what we do is actually delete the driver from windows registry and we installed manually. This way it will work

Solution.
Download the driver for your OS version on a USB Key, save the driver into a temporary folder in the affected server, make sure that the folder and its contents are not marked as read only.
1) Go to the device manager and right click on the NIC device and select Uninstall
2) Go to Start > Programs > Accesories > System Tools > System Information.
3) On the left side click on the + sign to expand the menu > Click on the + sign to expand Network menu > click on adapter.
4) On the right page look for the HP NC31xx Fast Ethernet NIC Driver location. Copy the driver location and then go to the folder where the driver resides and manually delete it.
5) Reboot the server and stop windows driver installation. If it prompts for you to choose the new driver cancel all request and get into windows, once there install the driver from the .exe file that you downloaded.
6) Reboot the server. By now you should be able to use the NIC card.

Regards

[Chris Camm]

I have managed to get hold of a spare G5 server so I am going to test your suggestion on this so that I can keep rebooting the servers without affecting the users. I will let you know if it resolves the issue.

Many thanks for your help

[Chris Camm]

I have managed to update the NIC card.

Driver Name BXND52X.SYS
Driver Version 4.1.3.0
VDB Driver Version 4.1.5.0
Bootcode Version 1.9.6

Teaming is set to 9.0.0.0

There are now no packets being dropped. However to speed is still really slow:

When saving a word file to a DL380 G4 it takes 1 second. Saving the same file to a DL 380 G5 takes 15 seconds.

Any suggestions are greatly appreciated.

Thanks

Chris

[KarloChacon]

hi

what if you try the drivers from Broadcom those are linked here

http://forums12.itrc.hp.com/service/forums/questionanswer.do?threadId=1211628

try those and let us know

regards

[Chris Camm]

I have tried to install the broadcom drivers but it wont let me.

I have changed the settings on the pc's NIC to none for the Checksum offload and disable for the Large Checksum offload

With this done we get a throughput of on the Team Utilisation:

575Mbps on the RX
261Mbps on the TX

This is a vast improvement.

However, on the production server if I run this test I can get the speed up to 520Mbps on the RX and 227Mbps on the TX at its peek but the average is 163Mbps on the RX and 112Mbps on the TX.

I have updated to the latest SSD's too version 8.0

Thanks again for any suggestions.

[Phillip Thayer]

Have you tried doenloading the latest version of the smart start CD, boot from it and then run the firmware update check on the servers. Updat whatever firmware it flags as needing to be updated and reboot the server into windows?

Sometimes I have found that simply having the firmware up-to-date takes care of a lot of the problems that happen like this.

Phil

[Patrick Zhang]

I met with the same problem with Chris.
The server is HP DL380G5, with 8*146GB 10Krpm SAS HDs (Raid5 using P400-256MB).
And the average write speed is 30MB/s as I tested. I've used a P400-512MB to have a try, almost the same write speed.
Any manners to improve the performance will be appreciated.

[Chris Camm]

I have put the latest firmware CD in and it found no updates needed.

Still running tests using pc's and servers on the same network switch.

Any other suggestions welcomed.

[Patrick Zhang]

The problem was solved!
We upgraded the Raid contoller from 256MB cache to 512MB with the battery. And adjust the cache percent, 50% for read and 50% for write. The current writing speed is about 95 MBps and the former is 30 MBps.
Maybe you could have a try like this.

[ViceSquad]

This is interesting because I have the same configuration (P400 256mb) and found the RAID5 painfully slow. In fact, when I did performance tests of RAID5 vs RAID10 the same operation that took 11 minutes on the RAID10 took more than 40 minutes on the RAID5.

Will installing the battery cache help here or do I need to upgrade the memory to 512mb as suggested by the recent poster?

[Chris Camm]

Update - I have booted the production server with the latest firmware CD and it found no updates where needed. Since the reboot:

Writing files to the Server 600Mbs
Reading files from the server 212Mbs.

Now I must be really missing something here because RAID 5 is supposed to be quick on Reading files from the server and not writing to it.

I changed the cache ratio several times and went through all the settings to see if it made a difference and it has made no significant improvement. There is no one in the office other than me writing to the server so this is about as quick as its going to get. I even enabled the Physical Drive Write Cache State.

I am know not sure if the problem is with the NIC or the Array now.

Has anyone managed to get good read speed on the servers? Writing even to the RAID 5 is fine its the same as writing to the RAID 0+1.

Thanks for any help