Discussion: Device special file changed for unknown reason.

[stephen peng]

Guys,

There was a situation in my rx2660 box. The IR volume dsf used to be c1t0d0, and under some circumstances which I was not aware of, the dsf changed to c9t1d0; in fact, all disk dsfs changed, like a HSV200 volume changed from c2t0d1 to c1t0d0, and all HW path remained unchanged!

1, what would cause these changes? You could offer every possibility.

2, would OS boot under such situation? Why? and if it could, what problem would it cause?

Regards

Stephen

[Ken Grabowski]

The CTD numbers are directly related to the controller (FCA) and SCIS addresses.  If the addressing at the storage array changed, then so will the CTD.  Have you been doing a storage migration?   Have any of your VG/LV's had an issue? What operating system are you using?

[stephen peng]

Hi Ken,
I don't think there was a migration, maybe just replacing a fault local hard drive or so. Of course all VGs had issue cause all the dsf changed. It was an HP-UX 11.23 box.

regards
Stephen

[Ken Grabowski]

Have you checked with your storage team? Have you recovered your volume groups?

[Ken Grabowski]

Steven,

HSV volumes are SAN attached storage.  Normally you would have multiple paths to each disk.  Anywhere from two to four in most environments.  Were your volume groups built with alternate paths, or only the primary?  Did a fiber path go away? 

What does your vgdisplay -v show?

Run  "ioscan -funC fc" to get your fiber controller card address and then check their status with "fcmsutil /dev/fcd#"

A properly configured volume group with SAN attached drives should have a primary disk path and an alternate path for each fiber path.  The CTD address would normally change for each path based on the FC address.

[stephen peng]

Hi Ken,

1,Just check attached file you will see the difference,

2,I have ajusted VGs' configuration and they are alright now.

3, It was interesting that it seemed that HSV volume c2t0d0 took over dsf c1t0d0 and IR volume had to use c9t0d0

4, why could OS boot with c9t0d0?

Regards

Stephen 

[Ken Grabowski]

All the SCSI addresses remained the same for the SAN disk. I would check with the SAN storage team and see if the WWN's changed or anything else that could affect the controller address.

I've never seen the internal disk controller address change, but I could guess that if a change in the SAN caused a conflict with the internal controller the OS might cause the change!  But if you have a support contract I would suggest you open a ticket and find out from HP how it could happen.

The boot is a simple answer. The setboot is configured with the hardware path not the CTD address. So the boot goes to 0/1/1/0.0.0.0.0, not c1t0d0 or c9t0d0.

[Torsten.]

Really strange.

All the instance numbers have changed.

Did somebody manipulate the ioinit database?

[stephen peng]

Hi Torsten,
I don't think no one would know how to do this manipuation, most possibly, hard drive replaced; maybe one of the two san switch was replaced, it is a cluster enviroment, and the other box remained nothing changed.

regards
Stephen

[Ken Grabowski]

If the World Wide Name WWN changes on the SAN disks it will change the C value of the device path. Replacing a local disk will affect the IR Volume at the SAS controller level but will not change the device path at all.  If changes occur at the SAN array or fabric router level, they need to be planned in advance.  Volume group map files should be created. Volume groups removed (exported) and systems shutdown until the change is completed. After the change, then volume groups can be recreated and imported from the map files. 

These kinds of changes should never occur unannounced or unplanned. The affect of changing without the proper planning  are system crashes and missing disks, like you just experienced.

[Torsten.]

I still think there was something wrong with ioconfig file(s), because all disk devices got new instance numbers and for that reason new device files.

[Ken Grabowski]

It's a binary file that is modified by insf, rmsf, and ioscan.  Not easily played with and pretty much only used to keep data between boots.  While running the data is kept in the kernel data structure. I rarely rule out anything as a possibility, but somebody modifying this file manually is pretty hard to do.

At boot the system does an ioscan!  I've done a lot of SAN migrations and when disks are moved between arrays they wind up with new WWN's and that means their hardware path has changed.  The C value will change, and if the SCSI was modified too, then the T and D values could change too!  This could also happen within the same SAN if the storage team decided to move disks to different control's (FA ports) to balance or "Fan Out", disk load in something like and EMC DMX or vMax style array.

I'm still surprised by the change to the IR Volume, but I think somebody doing something unannounced on the SAN or fabric is more likely than somebody playing with or stepping on the ioconfig file. 

[Torsten.]

There are procedures to modify ioconfig to change exactly these values. (e.g. to make them the same among several servers).

I don't think there where changes in the SAN, because all the h/w paths are still the same.

I really wonder why the IR Volume is no longer instance 1 ...

[stephen peng]

Guys,

It is a real discussion!:)

Maybe I should post full output of ioscan -fnCdisk, and you can notice that instance number of DVD changed also. It looked like IR volume disappeared for a while.

By the way, there was a MCA before I aware this weird thing. I run errdump mca at EFI Shell> prompt, and paste the output to a file as an input to Machine Check Analyzer, but the tools panicked after I start analyze. What wrong did I do in producing MCA file?

Regards

Stephen

[stephen peng]

Guys,

Liked I said before, I was curious when I found that the OS could still boot up under such circumstance, and unliked usual, vgdisplay -v vg00 showed in attachment, and the OS seemed ok.

c1t0d0 was not there, but vg00 could still activated, it really conflicted my understanding of LVM; it is not so hard to understand why os could boot(EFI located /stand, right?), but it is really hard to explain, for me, whether there was REAL relationship between vg00 and the PV that consist it.

regards

Stephen.

[Ken Grabowski]

EFI is not in /stand, /stand is in vg00 which is in an EFI partition.  You give setboot the hardware address of the EFI disk(s) , for example "Primary bootpath : 0/0/12/1/0.10.15.0.0.0.1".

Exactly how the boot translates that into the disk that's in vg00 in spite of lvmtab having a different device path, is beyound me.  But I assume since it already has the hardware address it doesn't have to worry about it.

[Torsten.]

This is not OK:

Cur PV                      1     
Act PV                      0     

I'm sure the system has booted with original configuration, then during boot the change happened.

Let's see

# vgdisplay -v vg00

# strings /etc/lvmtab

Check the syslog file for related information.

[stephen peng]

Torsten,

Check syslog.log and event.log as attachment.

There was an MCA indeed, but nothing wrong with vg00 and filesystem, in fact, I could rlogin the box, so I thought network service was OK. since I could read syslog, so the filesystems should be OK; there were some FIBRE syslog which I could not interpret.

strings /etc/lvmtab showed:

/dev/vg00
/dev/dsk/c1t0d0s2

It was really beyond my understanding.

regards

Stephen

[Torsten.]

execute:

/opt/resmon/bin/resdata -R 175636482 -r /system/events/ipmi_fpl/ipmi_fpl -n 175636481 -a

/opt/resmon/bin/resdata -R 175636482 -r /system/events/ipmi_fpl/ipmi_fpl -n 175636482 -a

vg00 lists still the old device file - you should correct this asap.

[stephen peng]

Torsten,

I am not on site now, and I captured those syslog and event.log or other information before I noticed the stranges. I corrected the mismatch thing right after I found it, and all I want to know is WHY.

For resdata output, you could check out event.log, and I am pretty sure that were some MCA notices. I am not that agreed with you about the "OK" thing cause I don't see any "NOT OK" thing except that vg00's mismatch, and you could also check the rc.log before I did the correction.

Regards

Stephen

[Torsten.]

Look at this:

your new ioscan:

disk         9  0/1/1/0.0.0.0.0      sdisk          CLAIMED     DEVICE       HP      IR Volume
                               /dev/dsk/c9t0d0     /dev/dsk/c9t0d0s2   /dev/rdsk/c9t0d0    /dev/rdsk/c9t0d0s2
                               /dev/dsk/c9t0d0s1   /dev/dsk/c9t0d0s3   /dev/rdsk/c9t0d0s1  /dev/rdsk/c9t0d0s3

but your lvmtab:

/dev/vg00
/dev/dsk/c1t0d0s2

and because of this mismatch vgdisplay shows

 VG Name                     /dev/vg00
VG Write Access             read/write    
VG Status                   available                
Max LV                      255   
Cur LV                      8     
Open LV                     8     
Max PV                      16    
Cur PV                      1     
Act PV                      0     

But without current command outputs and logs it is impossible to help.

[stephen peng]

Torsten,

From my aspect, the os show no difference whether with right dsf to form vg00 or not, only output of vgdisplay -v vg00 look terribly wrong. Of course I could not activate other vgs in the system. My opinion was that vmunix ignored mismatch of real dsf c9t0d1 and c1t0d0 in /etc/lvmtab, and just recognize hardware path set by setboot.

To hard for me to express myself  in English:)

regards

Stephen

[Ken Grabowski]

Stephen,

Torsten is correct. Even though the OS was able to boot and activate vg00, it was operating in a state that would not be considered stable.  If you did rebuild lvmtab and now have the correct volume in vg00 the vgdisplay should now show the active volumes as 1 rather than 0.  If you haven't rebuilt it, then your probably still unstable.

[stephen peng]

Ken,
I did not realize that there was a problem with this box, until I checked vgdisplay -v vg00, before that checking, everybody assumed it alright. And I did correct it, simple vgexport and vgimport, as soon as I noticed that problem.

regards
Stephen

[Ken Grabowski]

So then "strings /etc/lvmtab" shows the correct information now?  The procedure to rebuild lvmtab is to copy it to the side and run vgscan.