topic Re: Invisible replacement disk in Operating System - HP-UX

Invisible replacement disk

Peter Bosch — Wed, 28 Jul 2010 10:51:04 GMT

In a blade 870c I had 4 disks: 2 x 74 GB, 15 KRPM HDD in bay1/2 and 2 x 146 GB, 10 KRPM in bay3/4. They were configured in two RAID-1 sets and subsequently into two volume groups, vg00 and vg01.
Volume group vg01 became too small so I bought replacement disks: 2 x 300 GB, 10 KRPM. I removed the existing volume group and deleted the corresponding RAID-1 set. After that I physically replaced the disks.
Since the new disks got new hardware paths I have used the "replace target" option of the sasmgr command to "move" the disks to the original device files. After that I removed the device files that were no longer applicable.
When I tried to build a new RAID-1 set on the new disk I got an error message (truth be told, I don't recall the exact message). First I thought I made a mistake in the command, but when I tried the same command on the original (146 GB) disks everything worked fine. Back to the new disks. After some try-and-error I noticed that one of the new disks (the one in bay 3) had disappeared from "sasmgr get_info" and reported "NO_HW" in ioscan. Replacing that disk with another disk (same characteristics: 300 GB 10 KRPM) did not resolve the problem.
When I swap the disks in bay 3 and bay 4 the problem remains the same: the disk in bay 3 does not work while the one in bay 4 does. Multiple rmsf / ioscan / insf commands didn't solve my problem.
Normally I would suspect a problem with bay 3. But in that case, why do the original 146 GB disks both work with no problems?

Re: Invisible replacement disk

Deeos — Wed, 28 Jul 2010 11:27:37 GMT

hi,

Are you using same make and model of disks

#insf -eC disk

to recreate device file system on disk

Re: Invisible replacement disk

Peter Bosch — Wed, 28 Jul 2010 11:44:21 GMT

Yes, both disks are the same make and model. And as said, I've tried multiple rmsf / ioscan / insf commands. But since ioscan doesn't even see the disk in bay 3, insf is basically useless at this point.
What I forgot to mention before: I did try a powerdown and cold reboot so the controller has been reset. It made no difference, however.

Re: Invisible replacement disk

Torsten. — Wed, 28 Jul 2010 12:08:11 GMT

I just did such a disk replacement and noticed that the disk only appears in any "sasmgr ... -q raid" or "... -q lun=all" etc. command if the device files are created.

The only command that always show installed disks is the "sasmgr ... -q phy=all".

Does it show the disks?

Re: Invisible replacement disk

Deeos — Wed, 28 Jul 2010 12:09:25 GMT

in bay 1 74Gb Disk
In bay 2 74Gb disk

In bay 3 146Gb disk
In bay 4 146Gb disk

I think, scenerio like that

which disks are in vg01?

Have you checked inside connection of bay3?

Re: Invisible replacement disk

Peter Bosch — Wed, 28 Jul 2010 12:24:42 GMT

@Torsten: it does not. For what I've seen so far, "sasmgr -q phy=all" only appears to show disks that have device files (but I might be mistaken).

@deeos: your thoughts are correct. vg01 was on the 146 GB disks (those in bay 3/4) which have been replaced by 300 GB disks. At the moment I don't have physical access to the server, but when I put back the 146 GB disks last time, everything worked just fine. So it seems to me the connector should be okay. However, as soon as I can I will doublecheck.

Re: Invisible replacement disk

Peter Bosch — Wed, 28 Jul 2010 12:28:29 GMT

@Thorsten... I made a mistake in my last reply. "sasmgr -q phy=all" *does* in fact show the disks (all four of them). Sorry about the inconvenience.

Re: Invisible replacement disk

Deeos — Wed, 28 Jul 2010 12:29:52 GMT

hi,

can you check?

#sasmgr get_info -D /dev/ -q lun=all

#sasmgr get_info -D /dev/ -q raid

Re: Invisible replacement disk

Torsten. — Wed, 28 Jul 2010 12:31:32 GMT

If you see 4 disks "up" there, how about ioscan? Are the disks there? I'm not exactly sure about the relation between the other sasmgr commands and the device files, but maybe you need to remove and re-create the device files?

Re: Invisible replacement disk

Deeos — Wed, 28 Jul 2010 12:40:02 GMT

Please post output !

#sasmgr get_info -D /dev/ -q lun=all

#sasmgr get_info -D /dev/ -q raid

what does output of

#ioscan -funC disk

Have you done?

#rmsf -H

#insf -eC disk

Regards
Deeos

Re: Invisible replacement disk

Peter Bosch — Wed, 28 Jul 2010 12:47:16 GMT

sasmgr -q phy=all shows 4 disks "up"
sasmgr -q raid shows 1 logical drive (RAID-1) and 1 physical drive
ioscan shows 2 disks: 1 IR volume (the logical drive) and 1 "ordinary" disk.

I don't have exact command output readily available and to my regret I cannot logon to the system at this time. Will try to include output at the soonest available opportunity.

Re: Invisible replacement disk

N,Vipin — Wed, 28 Jul 2010 12:47:41 GMT

I would check the firmware of SAS controller and will update if it is not latest. I have seen some versions of firmware will not detect the 300GB disks.

Re: Invisible replacement disk

Torsten. — Wed, 28 Jul 2010 13:20:14 GMT

This is the latest firmware I can find:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=3257036&prodNameId=3257037&swEnvOID=54&swLang=13&mode=2&taskId=135&swItem=ux-70290-2

FW version: 1.23.42.00, EFI driver: 3.05.01.00

Make sure to have the latest driver:

https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=SerialSCSI-00

Re: Invisible replacement disk

Peter Bosch — Wed, 28 Jul 2010 13:30:24 GMT

My colleagues and I have thought about firmware as well but we can't explain why one disk would be visible (and available) and another identical disk would not. We couldn't explain why swapping the disks made no difference (bay 4 remains okay, no matter which disk is in there, while bay 3 won't work) either. Nevertheless, we will try to do a firmware update.

As for the requested command output, I have attached a file to this reply.

Re: Invisible replacement disk

Deeos — Wed, 28 Jul 2010 14:03:35 GMT

hi,

Could you just let me know which command you have used to add new disks?

I have seen only two disk in lun output, I hasn't show bay 3 disk.

Regards
Deeos

Re: Invisible replacement disk

Torsten. — Wed, 28 Jul 2010 14:25:56 GMT

The disk is obviously alive:

Info for PHY ID : 2
PHY Health : UP
Port SAS Address : 0x500605b000000002
Attached SAS Address : 0x5000c5001dd17d09
Current Link Rate : 3 Gbps
Max Link Rate : 3 Gbps

I cannot find any document about FW changes anywhere, but maybe it helps.
Did you already reset the controller?

Re: Invisible replacement disk

Peter Bosch — Thu, 29 Jul 2010 06:29:48 GMT

It has even been powered off so I would say so. I'm planning to do the firmware upgrade tomorrow (Friday).

Re: Invisible replacement disk

Peter Bosch — Mon, 02 Aug 2010 06:27:53 GMT

Well... the driver update was succesful but it didn't solve the problem. The firmware upgrade failed to install because of an assertion error. See attachment for details. I have logged a hardware call with HP now.

Re: Invisible replacement disk

Peter Bosch — Thu, 05 Aug 2010 05:40:11 GMT

The firware upgrade has been implemented sucessfully - I had to use fs0: and used blk1: during the previous attempt. The problem remains, however. Will add more info as it becomes available.

Re: Invisible replacement disk

Peter Bosch — Mon, 16 Aug 2010 07:10:48 GMT

We were finally able to create a 300 GB raid volume. To get there, we had to erase the headers on the disks and (under power-off conditions) physically move one of the disks of the existing 72 GB disks to bay 3. Upon starting the machine, the existing array was recognized on bay1+bay3 and we could add a new array on bay2+bay4.