Seeking Advice on the correct way to update drive firmware on DL360 Gen 9 Server

[adamdb_uk]

Hi,

    I've got a number of DL360 Gen 9 Servers running Windows Server 2012 R2 and acting as appliances for a larger infrastructure service. They have a P440Ar Array Controller to which  is attached the following.

2x200gb RAID 1 (OS) - Logical Drive 1

2x1.6tb RAID 0 (DATA) - Logical Drive 2

2x1.6tb RAID 0 (DATA) - Logical Drive 3

Ouput showing firmware revs etc. at the end of the post.

According to the config on the HPSSA

=> ctrl slot=0 ld 1 show

Smart Array P440ar in Slot 0 (Embedded)

   Array A

      Logical Drive: 1
         Size: 186.28 GB
         Fault Tolerance: 1
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 47874
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         Unrecoverable Media Errors: None
         MultiDomain Status: OK
         Caching:  Disabled
         Unique Identifier: 600508B1001CBDA64FC1D17AFB949DAE
         Disk Name: \\.\PhysicalDrive0 (Disk 0) (Bus: 0,Target: 1,Lun: 0)
         Mount Points: Offline 350 MB Partition Number 1, Offline 100 MB Partiti
on Number 2, C:\ 185.8 GB Partition Number 4
         Logical Drive Label: 02CEABC6PDNLH0BRH84CZHBEBD
         Mirror Group 1:
            physicaldrive 2I:1:5 (port 2I:box 1:bay 5, SATA SSD, 200 GB, OK)
         Mirror Group 2:
            physicaldrive 2I:1:6 (port 2I:box 1:bay 6, SATA SSD, 200 GB, OK)
         Drive Type: Data
         LD Acceleration Method: Smart Path

So caching is disabled and the LD is using SmartPath.  This is the same config (bar raid level) for the other LD's as well.

My understanding is when making new LD's inside  HPSS this is the default mode it's created in.

So my question is what is the correct approach to update the drive firmware in this scenario?

Should it be

1. Windows Smart update tool - says it's ok to use this unless you use this unless you are running in 'Zero Memory Mode' but for all my searching I can't confirm what this actually means! Does this mean drives running with SmartPath and caching disabled (as below below in my configuration).

Online firmware flashing of drives attached to an HPE Smart Array controller running in Zero Memory (ZM) mode or an HPE ProLiant host bus adapter (HBA) is NOT supported. Only offline firmware flashing of drives is supported for these configurations.

• Online drive firmware update available for Smart Array Controllers configured in systems running supported Linux, Microsoft Windows, and VMware environments. All other OSes would require an offline update using the SPP and HP SUM.

2. If it IS the case I can't use the Smart update tool (kind of crazy for the HPSSA to create arrays like this a default if you can't then use the smart tools to update the firmware later!) and I must use the offline SPP method I have a couple of further issues.

I can't locate EITHER of the required firmwares when using the SPP custom build tool

200gb drive firmware

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_9ab28dbb320d40dbab08c67e35#tab4

1.6tb drive firmware

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_642c151cab514527a689a5f4ef#tab2

which means I have to bake my own ISO. I got a base bootable ISO from the custom builder and downloaded it.

Unpacked it and added the exe's from the above links to the packages folder.

Problem is now finding a tool which will bake me a new ISO image. I've tried a few and so far no luck. There are not that many free tools to create bootable ISOs (although plenty of you want a USB image). So any advice on how to bake my own ISO which will boot via virtual CD attached to ILO would be welcomed as yes these servers sit in a remote DC I've no desire to spend a day sitting in.

Anyway, hoping an expert will tell me it's all fine and I can use the windows update tool! If not I welcome any feedback on the 2nd option.

many thanks.

Ad

--------------- Array Config and firmware info---------------------

Model Smart Array P440ar Controller
Firmware Version 6.88
Controller Type HPE Smart Array
Encryption Status Not Enabled
Encryption ASIC Status  OK
Encryption Critical Security Parameter NVRAM Status  OK
-Drive Enclosure Port 1I Box 1
Status  OK
Drive Bays 4
-Drive Enclosure Port 2I Box 1
Status  OK
Drive Bays 4
-Logical Drive 01
Status  OK
Capacity 186 GiB
Fault Tolerance RAID 1/RAID 1+0
Logical Drive Type Data LUN
Encryption Status Not Encrypted
-Physical Drive in Port 2I Box 1 Bay 5
Status  OK
Serial Number xxxxxxxxxxxxx
Model MO0200EBTJU
Media Type SSD
Capacity 200 GB
Location Port 2I Box 1 Bay 5
Firmware Version HPG2
Drive Configuration Configured
Encryption Status Not Encrypted
-Physical Drive in Port 2I Box 1 Bay 6
Status  OK
Serial Number xxxxxxxxxxxxxx
Model MK0200GEYKC
Media Type SSD
Capacity 200 GB
Location Port 2I Box 1 Bay 6
Firmware Version 4IWTHPG1
Drive Configuration Configured
Encryption Status Not Encrypted
-Logical Drive 02
Status  OK
Capacity 2980 GiB
Fault Tolerance RAID 0
Logical Drive Type Data LUN
Encryption Status Not Encrypted
-Physical Drive in Port 1I Box 1 Bay 1
Status  OK
Serial Number xxxxxxxxxxxxxxxxx
Model MO001600JWFWQ
Media Type SSD
Capacity 1600 GB
Location Port 1I Box 1 Bay 1
Firmware Version HPD2
Drive Configuration Configured
Encryption Status Not Encrypted
-Physical Drive in Port 1I Box 1 Bay 2
Status  OK
Serial Number xxxxxxxxxxxxxxxxxxx
Model MO001600JWFWQ
Media Type SSD
Capacity 1600 GB
Location Port 1I Box 1 Bay 2
Firmware Version HPD2
Drive Configuration Configured
Encryption Status Not Encrypted
-Logical Drive 03
Status  OK
Capacity 2980 GiB
Fault Tolerance RAID 0
Logical Drive Type Data LUN
Encryption Status Not Encrypted
-Physical Drive in Port 1I Box 1 Bay 3
Status  OK
Serial Number xxxxxxxxxxxxxxxxxx
Model MO001600JWFWQ
Media Type SSD
Capacity 1600 GB
Location Port 1I Box 1 Bay 3
Firmware Version HPD2
Drive Configuration Configured
Encryption Status Not Encrypted
-Physical Drive in Port 1I Box 1 Bay 4
Status  OK
Serial Number xxxxxxxxxxxxxxxx
Model MO001600JWFWQ
Media Type SSD
Capacity 1600 GB
Location Port 1I Box 1 Bay 4
Firmware Version HPD2
Drive Configuration Configured
Encryption Status Not Encrypted

< update>

So I've had nearly 300 reads but as yet no responses. I guess this is one area people don't go over very often!

I did have another thought about a suitable approach over the weekend. Assuming that I am not able to use the Smart update tool when using Smart Path would it be permissable to change the config on the controller to using traditional caching and then run the smart update tool? This would be a lot easier that going the SPP route.

I'm not 100% certain but I guess it would be something akin to

ctrl slot=0 array a modify ssdsmarthpath=disable   <turn off smartpath>

ctrl slot=0 logicaldrive 1 modify aa=enable     <enable 'array accelerator' aka cache for logical drive 1>

ctrl slot=0 logicaldrive 2 modify aa=enable    <enable 'array accelerator' aka cache for logical drive 2>

ctrl slot=0 logicaldrive 3 modify aa=enable    <enable 'array accelerator' aka cache for logical drive 3>

I believe that would enable caching on all 3 drives as per the default (10% read / 90 % write)

I would hope then that I could use the windows smart update tool to perform the updates.

Once completed I could then turn smartpath back on

ctrl slot=0 array a modify ssdsmarthpath=enable

which would disable to above configuration.

Could any validate if my approach is correct?

thanks.

[SanjeevGoyal]

Hello,

I hope all is well.

I would suggest you, update the SA P440ar controller first before the hard disk firmware update.

You should have the complete data backup before the hard disk firmware update.

You can use the latest SPP 2020.03.0 for firmware update.

https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_8ccab9e4b99047138c2e608f97#tab-history

as well as you can create ISO customer image with the new firmware.

Please let me know for further assistance.

If you feel this was helpful please click the KUDOS! thumb below!   

Regards,

[adamdb_uk]

Thanks for the response. Sadly if you read my post fully you'd see a couple of things.

1. Neither of the firmwares I need are available via custom SPP image builder.

2. I am remote to the servers so if i MUST use SPP rather than the windows smart update (preferred method) then I need to be able to bake my own bootable ISO image containing the missing smart update .exe files from the download links posted in the original question (which can be attached to the server via ILO). I will not be able to use USB boot or any other method if SPP is the only way to perform this work. So far I've created my custom SPP (containing minimal drivers and content to keep the size down), extracted it locally and added the missing smart update .exe files for the required firmwares into the 'packages' folder. Sadly I cannot find a tool to turn this back into a bootable SPP (and i've tried quite a few).

So whilst I appreciate your advice, it's not really helpful in the current context. I did have a call open with support as well but I was getting the same 'stock responses' from them. Use SPP, here's the USB creation tool... none of which are any use to me.

I would much prefer it if someone could just technically vet the steps I'd outlined for changing the caching mode and if this allow the use the Smart update tool or for that matter if the smart update tool can be used under windows when the arrays are configured to use HPE SmartPath, again a question to which nobody ever seems to know the answer and yet this is the default configuration when you deploy a new array via HP SSA.

We are only doing ALL of this because I originally raised a call for excessive wear rate on a couple of SSD's which were in RAID1 and had a much higher wear rate than other systems configured in the same way.

The stock response is of course, please upgrade the firmware and I've been trying to get straight answers ever since to no avail. I reached out on here in the hope someone could simply confirm the criteria for doing the update via the Smart update tool. But it seems nobody seems to know!

Thanks for you input.

[BPSingh]

@adamdb_uk 

Greetings!

I understand that that you haven't attempted to flash the firmware online, a little skeptical because of the following statement on the firmware download page.

Online firmware flashing of drives attached to an HPE Smart Array controller running in Zero Memory (ZM) mode or an HPE ProLiant host bus adapter (HBA) is NOT supported. Only offline firmware flashing of drives is supported for these configurations. Also all Gen9 servers are running Windows Server 2012 R2.

If I understand correctly, "HPE Smart Array controller running in Zero Memory (ZM) mode" refers to a controller without a cache. Cache goes default with P440.

It is tested in the lab that to enable Smart Path user need to have a cache module installed even if user is using a RAID 0 , RAID 1 and RAID 1 + 0. And from the output you have shared caching is disabled as Smartpath is enabled.

If we read the following paragraph, we understand that both SmartPath and normal IO path are active on the controller. Also array expansion and RAID-level migrations aren't possible without a cache module. So in my opinion online firmware upgrade should go through. May I know if you have tried flashing HDD drive firmware online (from windows OS) on any of the servers? I would advise you to take a valid data backup before you plan to try or do that on a test server first.

HP SSD Smart Path and the normal I/O path are both active simultaneously.The Smart Array device driver coordinates with theSmart Array firmware for keeping an up-to-date map of the disk and volume layout. With this map, requests are checked and qualified to bypass the Smart Array firmware and use the Smart Path. HPE SSD Smart Path qualified requests are generally small read requests for volumes of any RAID level and small write requests to RAID 0. The normal I/O path handles large requests and most writes. If a request goes through HPE SSD Smart Path and results in an error, the request is retried through the normal I/Opath. If volume expansions, rebuilds, re-configurations, or RAID-level migrations are in progress, all requests use the normal I/Opath.

Coming to your query on creating custom SPP and make it bootable. 

Please be informed that .rpm packages are actually only used while flashing firmware offline irrespective of the OS installed on the server. So you will need to add .rpm files of the drive firmware to the custom SPP if you plan to update offline.

Could you share an ftp so that I can upload a document for making custom SPP for your reference?

Regards,

[adamdb_uk]

Hi,

   first off thanks for providiing some clarification on the function of smartpath and it's relation to array cache. I think I understand a little more how it functions now.

With regard to testing the windows update tool sadly I do not have a test server I can run it against so I guess I will have to take the plunge and attempt the update on one of nodes to see what happens. Assume a reboot will be needed after the update is applied and smartpath can be left enabled.

With regard to constructing the ISO feel free to upload any files here

https://drive.google.com/drive/folders/1Fz_OIqjEBI83bMA9fFsn9eQg3XeZANnU?usp=sharing

Sadly I don't have an ftp available.

With regard to what goes into the ISO's I was told by support to place the exe's in the package folder. It did strike me as odd when you can clearly see the bootable ISO uses a linux bootloader when initialising. I take it if I adopt this approach I'd need to obtain the equivalent linux update package and pull the rpms from there?

Anyway hopefully using the smart windows tool will work. Not sure how these appliances are backed up so I'd better get that change paperwork done!

And a share on how to make the bootable ISO will be gratefully received. I'm not sure why this information isn't just shared as a KB article as I'm sure I'm not the first person who's asked for this.

Oh well.

thanks again

Adam.

[BPSingh]

Hello Again,

Thank you for the response.

Assume a reboot will be needed after the update is applied and smartpath can be left enabled.--- Yes you are right, reboot is required for the changes to take place.

You can start with downloading the latest compatible SPP for gen9 and then add the components (RPM) manually as mentioned in the doc file which I uploaded. Please do not miss to tick the "Make Bootable ISO File" option. Before you add the files manually please check the content report for that particular SPP as the desired firmware file may be already present in it.

https://techlibrary.hpe.com/us/en/enterprise/servers/products/service_pack/spp/index.aspx

Document has been uploaded, hope it helps.

Cheers!