MSL5026 serious problems

[Neven Kaic]

Hi,

Recently I downloaded HP's LT&T 3.3 in order to upgrade firmware on my MSL5026 library (it has two SDLT drives). It is connected to Compaq Proliant 1850R server with Windows NT 4.0 server (+ SP6a). Previously, MSL was running firmware version 4.14.

When I started LT&T software, it offered to download new version of firmware for both, library (version 4.21) and drives (version V075). Afterwards, I started the upgrade process and everything went fine with the upgrade of library. However, almost immediatelly all LED lights were on (on both SDLT drives) and from that moment library can not initialize these drives.

Of course, I tried to turn off the library and wait and then turn it on again. Didn't help. I reseted all settings to the factory defaults. Nothing. Then I downgraded firmware to 3.11. Nothing. Then upgraded to 4.14. Nothing again.

I'm out of any ideas.

Please help ....

[Eugeny Brychkov]

Go to LTT's firmware directory and compare firmware files (sizes nd core with 'fc' command) with located at ftp://ftp.hp.com/pub/information_storage/software/ltt/firmware/
Are they identical?
Eugeny

[Neven Kaic]

Eugeny,

I checked it and they are the same ones.

Neven

[David Ruska]

Are there any error messages on the MSL front panel?

Do the drives still appear on the host?

If you can send us the eventlog files in the logs folder of LTT, we can see if the drive firmware updates were completed successfully. Send them to ltt_team@hp.com

[David Ruska]

We updated firmware on an MSL/SDLT to watch the behavior:

---
LTT updates the library first and waits for it to completely power up - drives power LED normal green
(does not appear that library resets the drives at power-up)

LTT downloads drive 1 first - drive power LED green until download transfer complete
- drive 1 activity/power LEDs both flash, while drive powers up with new fw
- then drive 1 power-up cycles through all 3 LED's quickly
- then drive 1 completes with power LED green.

LTT downloads drive 2 - drive power LED green until download transfer complete
- same LED sequence as drive 1
---

Can you tell us where the behavior differed for your library?

[Neven Kaic]

Well, during my update process the first phase (update of the library) went OK, but then the problems started. Although I was not constantly looking at drives' LEDs, it seems to me that LTT downloaded firmware simultaneously to both drives (maybe it was sequentially, but quite quick) and then all 3 LEDs turned on and stayed that way. I was waiting for the LTT software to finish its waiting period (I think it is 740s timeout), but nothing happened. Then another pop-up window came up saying something like "LTT could not finish I/O process, but when it is finished this window will close ...". Of course, nothing happened and all 3 LEDs were still on. So, I let the server and library in that condition for couple of hours, but to no avail. Finally, I turned off everything in order to connect my reserve backup device and run my daily backup. Eventually, I turned on the library next day, but it was behaving in the same way. Then I started with the downgrading of firmware on library to see if this would change the situation, but nothing happened.

Regarding error messages, there are no error messages on the LCD screen. I think this is because SCSI adapter of my server does not "see" these drives any more (it sees only the library).

Somehow I have a feeling that there is something wrong with either LTT software or with firmware release.

[David Ruska]

The drive firmware updates can be done in parallel on some OSes (on win2k we force sequential updates). So in your case, it may have done the updates in parallel.

The firmware update process in the drive is done by recieving the new image, and then copying that image into flash. The LTT software checks the firmware image with a CRC before sending it, and the distributed images with LTT are always tested before being released. Therefore, I do not beleive there should be any problem with the firmware file.

There are two things that may have gone wrong. First, it may be possible that the transfer was corrupted while sending the image to the drive (e.g. cabling problems). I'm not sure if the SDLT drive implements it's own CRC check on the image before starting an update. That may be a weak link.

Second, the process to flash cannot be interrupted, or the drive may be left with incomplete firmware. A power glitch is one possible interruption.

One other possible interaction may have been another host app attempting to communicate to the drives during the firmware update process. This could be a backup app or monitoring software (e.g. Compaq Insight Manager). (note: we haven't been able to cause a failure by doing this purposely.) Any chance that there was another app that may have attempted to access the drives?

If you can send us the event logs (from the LTT logs folder) we can look for any clues to something going wrong. Send them to ltt_team@hp.com.

Based on your observations of the drive LEDs and not seeing the drives from the host, that seems to indicate the drive firmware flash failed and the drives are dead. I suggest you call HP to have the drives replaced.

[Neven Kaic]

Hi,

I stopped all services on the Windows NT 4.0 server (e.g. backup and anti-virus related, SNMP, Compaq Management Agents etc.) before I started the upgrade process. Additionally, all my servers are connected to very advanced UPS which is supposed to remove all voltage fluctuations. Furthermore, all equipment is properly grounded.

Anyway, I have to admit that I can not recall stopping Compaq Insight Manager 7.0 ...

However, in the mean time I called HP service team and they replaced the drives.

Finally, I'd like to ask you a simple question: library is currently running on firmware version 4.14 - is it safe to perform an upgrade of library firmware to 4.21, with the SDLT drives physically removed (both drives are on firmware version V052/CPQ2) and after the finnished upgrade installed again?

[David Ruska]

Neven,

We did verify that SDLT drives do a CRC check before starting the firmware flash, so the problem somehow happened during the flash update itself. Loss of power is the most obvious risk, but it sounds like that should not have been the case for you. It may be possible that some host commands interrupted the update.

One way to prevent any host interactions would be to do the firmware update from a firmware update tape, with the drives disconnected from the host. LTT can create SDLT firmware update tapes (one of the selectable tests).

> Finally, I'd like to ask you a simple question: library is currently running on firmware version 4.14 - is it safe to perform an upgrade of library firmware to 4.21, with the SDLT drives physically removed (both drives are on firmware version V052/CPQ2) and after the finnished upgrade installed again?

There shouldn't be any issue with updating the library firmware with the drives installed, but you can remove them (or just disconnect them from the bus) if you wish.

FYI, the library controller has separate boot code, so it should not be possible to create a brain-dead controller if something were to go wrong in the flash process.

* Could you send the event logs from the LTT logs folder to ltt_team@hp.com? We would like to see if we can understand what went wrong with your SDLT firmware udpates. Thanks.