ProLiant Servers (ML,DL,SL)
cancel
Showing results for 
Search instead for 
Did you mean: 

DL360 G5 P400i Raid 5 SLOW

SOLVED
Go to solution
hongman
Advisor

DL360 G5 P400i Raid 5 SLOW

Hi

I have 2 DL360 G5's which were running 2x146GB SAS Drives each in Raid 1 array. All hardware identical in both servers.

I recently added an extra disk to each server and did an online migration to Raid 5.

One server completed the task in ~8hrs, the other took over around 48hrs!

The server that took the longest is having some dreadful performance issues. All drivers and firmware is up to date, including firmware for the drives themselves.

I called HP Support but unfortunately after a week of calls the answer I'm getting from this "tech" is that my Windows install is broken and I need to reinstall the OS. I mean, really.

I dont have BBWC as far as I know, but beofre people start screaming that I need this, please bear in mind that the other server is running fine without as well.

I have no idea what else to do. I used an disk speed test utility and on a 128MB file, I get 25.4Mbps on one server and 900Kbps on the bad one.

Backups are taking over 12 hrs from the 1hr 40 mins it used to, with rates on that gone from 900MB/sec to 125MB/sec (according to BEWS)

Even opening general apps (for example, ADU or ACU or anything else) takes probably 4-5 times longer on this server now than the other.

All of this happened since the Raid 5 Migration. Help please! I can provide any logs or configurations you want.
35 REPLIES
TTr
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

> I dont have BBWC as far as I know...

This tells me that you are not sure that you don't have BBWC. You should verify this with ACU/ADU.

> All of this happened since the Raid 5 Migration

Were both servers performing the same before adding the disks? Maybe both had the BBWC memory and the memory and/or battery failed on the slow server.
hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

ADU/ACU do not mention anything about BBWC so Im pretty sure I dont have it!

Both servers were performing the same before the migration.
hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

Actually...

Array Accelerator Present - True
Cache Status - OK
Array Accelerator (Cache) Ratio - 50%/50%
Total Cache Size - 256MB
Cache Backup Power Source - Batteries
Battery/Capacitor Pack Count - 1
Battery/Capacitor Status - OK

Does this indicate a BBWC?
TTr
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

Yes it does. Which server is this from? Probably the one that run fine.

Check the other server. It probably has the BBWC option too so check the status. If the battery fails to charge, the raid controller will disable the write cache and the result is a slow server.
hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

This part is the same on both servers :(

Any other ideas?
TTr
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

> Array Accelerator (Cache) Ratio - 50%/50%

Not much comes to mind. Double check the 50/50 ratio. Also check the status of all the disks and logical drives and compare all the settings from one server to the other.

Ensure that you are accessing the right server / right ACU. Sometimes the ACU client will connect to the ACU software on another server.
hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

The Cache ratio is just set there because I dont know what it should be at, in truth. I have had it on pretty much every setting and it doesnt make an awful lot of difference, so I thought halfway would be best.

Everything else has been checked through, run out of ideas.
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

hi hongman
as advised previously


This is how we thank in the forum

http://forums11.itrc.hp.com/service/forums/helptips.do?#33

hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

Can you please stop harassing me for points. I issued points for your help in the other thread.

As this thread is still ongoing I think it is more appropriate to hand them out at the end, not after every post.
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

wow you are twitchy NVM
Hongman here you go

hi get disk model nr

The model number you can get remotely by going in to Hp tools and the array diagnostic utility , and on to the array controller , and onto the physical disc a model number it has this type of format BF14684970 ,when here you can press control f to get a search box open , just type the word Model in there to help with the search

now go to Hp.com put model nr in and get latest firmware,,

there are advisories on this at the moment
dont know the nr at the moment
enjoy

like to know how you get on :)
hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

I have the lastest firmware for the disks, and after it was installed the server has been rebooted.

No difference :(
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

there are many similar to this


http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=125&prodSeriesId=1127655&prodTypeId=329290&objectID=c01296286

just put your model nr in google and type advisory should bring up something nut not all sas drives are listed yet
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

ok from this page

http://h20000.www2.hp.com/bizsupport/TechSupport/DriverDownload.jsp?lang=en&cc=us&prodNameId=3288144&taskId=135&prodTypeId=15351&prodSeriesId=1121486〈=en&cc=us&submit=Go%20%BB

get latest firmware maint c it is vers 8.6 install all latest firmwares via this

the softtware cd rom it contains drivers and utils ,
you can use this in windows
update all drivers after firmware, is done reboot and
see what happens if all okay fine if not
boot again from the software psp cd and run offline diagnostics save file


the System Survey report with â View Levelâ = Advanced and Category = ALL.



also run the adu utility and post that file here too


only do these diagnostics after firmwares are all updated and drivers too..

and i will look at them
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

best to use this page for firmware cd and software cd

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareIndex.jsp?lang=en&cc=us&prodNameId=3288144&prodTypeId=15351&prodSeriesId=1121486&swLang=8&taskId=135&swEnvOID=1005


this could be important new firware for you ontroller
Online ROM Flash Component for Windows - Smart Array P400 and P400i (multi-part download) 6.86
16 Jul 2009

hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

The firmware versions of the disks are the very latest and are not affected by the advisories, according to HP website.

I am waiting to do the offline diagnostics, it will be a few days at least becuase I am not onsite everyday + i have to arrange downtime.

Attached is the ADU report.
hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

Already running 6.86 for Array Controller - please trust me, everything is as up to date as it can be!
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

ok you want to post survey and adu report
by the way the is no tape device on the same controller is there..?
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

ok Hongman
from what i read in report i reckon
take out battery restart server,it may moan its loosing a bit of data caught in the battery but proced

is all okay now if so shutdown reinsert battery all should be ok as the rebuild is not quite finished thats why it mentions parity qued

Consolidated Error Report:
Controller: Smart Array P400i in slot 0
Device: Logical Drive 1
....................
Logical Drive 1 Warning Background parity initialization is currently queued or in progress on this logical drive. If background parity initialization is queued, it will start when I/O is performed on the drive. When background parity initialization completes, the performance of the logical drive will improve.

Cache Failure Failed From Cache Data Loss Status
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

also expect it to be slow while rebuilding , circa 15 mins per gig if it is not being use d, much slower if it is being used ,


best to put rebuild priority high in acu when not in production hours..
hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

Like I said, I'll have to arrange for downtime to do this as Im not on site everyday.

However, the server that is running fine also has that error message about parity queue and its running fine?

The array's were migrated nearly 2 weeks ago now. that is plenty of time for it to rebuild completely right?

Anyways, I guess I need to get some more downtime in to do some offline tests.
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

go in to acu and click on status messages
and all should be clear,,

goodluck :)
TTr
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

Have you rebooted the servers since you added the drives / did the raid migration? Try rebooting them first without messing around with the battery. Maybe when they are offline or start up without any i/o load the rebuild will complete and the inconsistencies will clear up. If not then you consider doing the battery removal process as suggested above.
hongman
Advisor

Re: DL360 G5 P400i Raid 5 SLOW

The servers have both been rebooted 2-3 times each since the migration completed.

Further frustration, now the backups are taking 14 hours from the usual 1hr 45 mins :( According to the logs as soon as it hits the bad server the rate drops to 63.5MB/s from 8-900 MB/s. Sigh.

I'll try get round to ripping the battery out this weekend!
marcus1234
Honored Contributor

Re: DL360 G5 P400i Raid 5 SLOW

looks like the only viable option atm goodluck