After upgrading VO015300JWCNL SSD firmware from HPD0 to HPD8 - write performance is dead slow

[laimis17]

I've successfully upgraded ssd from  from HPD0 to HPD8, but afterwards  write performance is dead slow ~0.64MB/s.

Read -  all good.

Any ideas how firmware could've caused this and what are the solutions ?

We've made similar upgrade for another model VO015300JWSSV from HPD1 to HPD8. Everything is fine there.

[AmRa]

This need deep log analysis, hence please raise support ticket with HPE tech support team for further assistance.

[laimis17]

Sadly, i don't have a support.

But this issue could affect serious Enterprise Customers and i could be of help being test subject. 

[AmRa]

Login to HPE Smart Storage Administrator and verify Physical and logical drive status.

Have you checked by updating smart array controller firmware and driver to latest.

[laimis17]

I don't believe it's about controller. I've tested it with different RAID controllers and SAN storage devices.

Write/read performance results are the same.

I believe it's about SSD firmware that caused it.

[AmRa]

With reference to CUSTOMER BULLETIN: HPE SAS Solid State Drives - Critical Firmware Upgrade Required for Certain HPE SAS Solid State Drive Models to Prevent Drive Failure at 32,768 Hours of Operation.

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00092491en_us

And as resolution suggested to immediately upgrade the drive firmware to version HPD8, which HPE has released to prevent the issue described above.

Which means HPE SAS Solid State Drives firmware HPD8 is more stable.

[laimis17]

This is exaclty what i've done.

And what i'm trying to say that upgrading VO015300JWCNL to HPD8 causes this strange behaviour with write operations which makes it unusable in production.

I hope this post will make enterprise customers think twice before upgrading this model. This HPE 15.36TB SSD is not cheap as you know.  As I've upgraded another 8x HPE 15.36TB SSD VO015300JWSSV(of newer generation) drives to new firmware without a problem, i'm sure about consistency of my results.

A serious bug from my perspective if You ask.
Prior to upgrading VO015300JWCNL to HPD8, this SSD with HPD0 firmware operated normally. So i am afraid of trying this with other drives of this generation.

[SanjeevGoyal]

Hello,

I can understand your issue, but do not think the performance Issue is only due to hard disk firmware updates.
So many other factors are also possible.

Please check/update as below.

System bios 
SA controller firmware (SA controller should be updated with the latest If you are updating hard disk firmware)

If the issue persist after updating bios and SA controller firmware.
I would suggest, you have to log a proper case with HPE, and share the appropriate logs for further analysis.

If you feel this was helpful please click the KUDOS! thumb below!   
Regards,

[laimis17]

I follow.

It's known practice when storage manufacturer does not allow 3rd party drives.

But this SSD is recognized with any vendor. Besides HPE DL380 Gen10 I tried Dell R640 and storage device Promise E5320sD, they both produce same performance results. So how this can be about BIOS/controller/drivers ?

Or with this SSD firmware HPE introduces something we don't know about, that was not mentioned in the release notes ? And i'd like to know about it.

Ok, let's think outside the box. Can I reflash the firmware with some other controller by using image ? In this case i believe it's G1633HD8.rel from RPM ?

And as a fallback option, can we downgrade it and where to find HPD0 firmware ?

[laimis17]

I did one more test. I've put it into another storage and checked it's SMART results:

Here's how affected drive looks like:

-------------------------------------------------------------------------------
PdId: 24
Model Number: HP VO015300JWCNL
Drive Type: SAS SSD
SMART Status: Enabled
SMART Health Status: FAILURE PREDICTI< THRESHO
EXC2016D37ascq=0x1d [asc=5d, ascq=1d]
Current Drive Temperature: 24 C
Drive Trip Temperature: 60 C
Power On Hours: 1129858336 Hours
Manufactured in week 37 of year 2016
Recommended maximum start stop count: 20296 times
Current start stop count: 288 times

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
R: 0 0 0 0 0 0.000 0
W: 0 0 0 0 0 0.000 0
V: 0 0 0 0 0 0.000 0

SMART Self-test log: N/A

Here's how good drive looks like:

-------------------------------------------------------------------------------
PdId: 2
Model Number: HP VO015300JWSSV
Drive Type: SAS SSD
SMART Status: Enabled
SMART Health Status: OK
Current Drive Temperature: 28 C
Drive Trip Temperature: 60 C
Power On Hours: 15523 Hours
Manufactured in week 46 of year 2018
Recommended maximum start stop count: 0 times
Current start stop count: 39 times

Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
R: 0 0 0 0 0 2.375 0
W: 0 0 0 0 0 45.532 0
V: 0 0 0 0 0 0.000 0

Take a look how firmware messes Total power on hours parameter on affected drive:

1129858336 hours... which is 128 891 years!

And by the way i left it for a week. Total power on hours parameters did not change.

So now that I showed how firmware messes Total power on hours parameter(and maybe others), what HPE is going to do about it  ?

[Torsten.]

You probably already hit the bug:

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-a00092491en_us

[laimis17]

I thought i was clear that drive worked flawlesly before this firmware upgrade it had only about 10k hours.

And this bulletin you mentioned of 32k hours was the reason for this upgrade and it messed things up.

Can someone explain how 10k hours after firmware became  128k years ?

Did anyone successfully upgraded VO015300JWCNL or has one in posession ?