ProLiant Servers (ML,DL,SL)
cancel
Showing results for 
Search instead for 
Did you mean: 

Smart Array P420i extremely slow writes

SOLVED
Go to solution
kthusid
Occasional Contributor

Smart Array P420i extremely slow writes

Hi,

 

We have a couple of new Hp Proliant DL380p running Xenserver 6.2 SP1. On one server, the write performance is normal for 1-2 days, and then it drops to 1/50 of normal performance. The read perfomance is not affected.

 

We have upgraded firmware to all the newest versions, upgraded to Xenserver 6.2 SP1, installed newest HPSA driver from Citrix (ver 3.4.4-125), reformatted the RAID, tried RAID 6/10, removed the server from pool and added it again.

 

We have also installed a SATA disk on the onboard sata controller, and it is not affected by this problem.

 

Nothing seems to make any difference. There is no error in the ILO from the controller or disks. This is driving me mad, as everything is running, just extremely slow, and after each firmware/driver/version update, it is running fine for a couple of days, then it slows down. A reboot fixes the problem for a while again.

 

I have attached two pictures, the first one is an Atto test from when the system is working, and the second, when it is slowed down.

 

A bit info:

Xenserver build 70446c

 

Proliant dl380p

Dual Intel Xeon CPU E5-2620 0 @ 2.00GHz

System ROM: P70 12/20/2013

 

HP Smart Array P420i Controller
Firmware Version 5.22
 

Any info suggestions are welcome, as I am running out of options.

4 REPLIES
waaronb
Respected Contributor

Re: Smart Array P420i extremely slow writes

How many and what type of drives are in the array? What are the write caching settings for the P420? Any errors about the supercap not being charged or anything?

There was an update to the P420 firmware you should check out. Version 5.42 is the latest.

They fixed a problem we had on one of ours, where it kept reporting the supercap was charging and write cache was being temporarily disabled. It would say this despite having several days to charge the thing up.

That firmware version had a note that it fixed an issue with some controllers not properly reporting the cap being charged back up, and since applying it, that issue went away for us on that system.

It typically showed up as the supercap reporting a full charge for the first 24 hours or so after the system started up. During that time, everything was great and write-caching worked as expected, but then it reported the low charge state and disabled the write cache.

You should check in the SSA utility and look specifically at the reported state of the battery/supercap on the cache module... does it say disabled or charging?
kthusid
Occasional Contributor

Re: Smart Array P420i extremely slow writes

There are 8 600GB 10K sas drives in the array. The write cache is enabled, but I agree that this is probably the issue.

I have temporarely enabled nobatterywritechache and so far the problems are gone.

 

The firmware was upgraded about a week ago to version 5.22 but if the 5.42 specifcally adresses this issue, it seems it would solve my problems.

 

But first I would like to find the root cause. How do I check if the battery/capacitor is ok? It shows ok in ILO, so where else to check?

waaronb
Respected Contributor
Solution

Re: Smart Array P420i extremely slow writes

I'm not familiar with the management software HP has for Citrix Xenserver ... I just looked, and there's not a lot of stuff in it's drivers/downloads page.

ILO 4 has the "System Information" -> "Storage" tab. There's a spot there for the controller where it lists "Cache Module Status". I assume if the supercap was discharged/charging, it might say something there, but maybe not.

The "Integrated Management Log" may show something.

Here's that part of the firmware 5.42 revision history:
"On some power cycles, the supercap would always report that it was charging, even though charging had completed."

If you're getting consistently better write performance by telling it to enable write caching even if the battery isn't present or charged, then hopefully that's what caused the bad performance.

If your workload is pretty write-intensive, consider setting more of the cache memory to write instead of the default 75% or whatever. Also, RAID 6 is kind of slow for writing anyway. If it's an option, you might consider RAID 50...that would work pretty well on 8 drives using 2 parity groups. Spacewise that gives you the same usable space as 8 drives in a RAID 6, but the performance should be better for writes.

RAID 50 just might meet your redundancy needs as well as ADG, but you may have a business need for RAID 6.
kthusid
Occasional Contributor

Re: Smart Array P420i extremely slow writes

Waroonb was right, there was a problem with the firmware/write cache. After upgrading til controller firmware 5.42, it seems like the problem is fixed, and we have been running now for a week with this new firmware, and so far, it is running as expected.