MSA30-SA6402 slow I/O on DL380 G4's

[Mike Texter]

Hi,

I'm using 2 - DL380 G4 systems for data destruction, each contains an SA 6402 raid controller (with a 256MB BBWC) attached to a single-bus MSA30 enclosure.  Controller firmware is up to date at v. 2.84.  All batteries are in good working condition.  I build raid-0 arrays of the disks i am shredding, and then run a DoD 4-pass shred on them. 

The issue is, they each should run at U320 speeds (320mbps), but when running the shred command from centOS, i get, on average, 20mbps of bandwidth.  I first noticed this isssue when my versions of CentOS were different, one system was running at 5.4, the other at 5.7.  The one at 5.7 was running at roughly 20mbps, while the 5.4 machine was running much much faster, around 160mbps across all shred processes combined.  I then (foolishly) updated the 5.4 machine to 5.7 to see if it would duplicate the issue, and sure enough, it did.  I then reloaded both systems with centOS 6.2, which seemed to have solved the issue temporarily.  The shreds were running at a shared capacity of ~160mpbs again.  However, about an hour or two into each shred, the shred speeds drop back to the 20mpbs range - permanently it seems. 

Edit: The shred speed appears to be around 20mbps for most (~90% or so) of the time, but does occasionally jump back up to what i would consider normal.

Initially, i thought firmware/drivers were to blame, so i updated everything using the HP ProLiant Service Package (PSP) for the systems in question - nothing.  I also updated CentOS's packages to the latest, and have been browsing their forums for more information on this topic - no luck.

Another interesting tidbit - The shelves shred at normal speeds while attached in a JBOD configuration to another system (I basically plug into an lvd-scsi card in another intel-based system (a PE2850 with similar specs) and they shred just fine, albeit as individual disks

TL/DR:

Long story short, I've narrowed the issue down to the DL380G4 systems themselves, the SA6402 controllers, or the cciss driver on CentOS 5.7 and later.  Since speeds were normal on an older version of CentOS, it would lead me to believe that it is not hardware related.  

I really do apologize for the length of this post, and thanks a lot for your time to anyone who actually reads this.

[Mike Texter]

Possible symptom: the drives are downshifting their speeds.  I didn't know this type of thing existed, but apparently there was a known issue (solved with a firmware upgrade) with HP SCSI disks where they would downshift from U320 to U160 speeds.  I'll doublecheck my drives' firmware levels.  They do shred at normal speeds in a JBOD configuration, attached to a raw scsi card on another system, so maybe there is a firmware mismatch between my HP products.