Problems booting since re-installing online JFS

[EMCD]

Hi, could you help me with a problem I'm having?
We have an RP7410 with IBM storage attached.
I reinstalled OnlineJFS on our server to resolve a license expired problem. The kernel was rebuilt, so the install required a reboot. When bcheckrc was issued it failed to fix the
SAN vxfs filesystems and dropped into a shell (bcheckrc)#.
I had to comment-out the SAN file systems in /etc/fstab and issue a reboot. Once
the server was up I uncommented the entries in /etc/fstab and issued a mount -a. The SAN file systems all mounted successfully. I then shutdown the server with a reboot to check the SAN file systems would mount but once again bcheckrc 'couldn't fix vxfs filesystems' and dropped into a shell. I have appended the output from the reboot.

EXAMPLE:
/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist) Volume
group "/dev/vg00" has been successfully changed.
vgchange: Warning: Couldn't attach to the volume group physical volume
"/dev/dsk/vpath1":
A component of the path of the physical volume does not exist.
vgchange: Warning: Couldn't attach to the volume group physical volume
"/dev/dsk/vpath2":
A component of the path of the physical volume does not exist.

etc

vgchange: Warning: couldn't query physical volume "/dev/dsk/vpath1":
The specified path does not correspond to physical volume attached to
this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/vpath2":
The specified path does not correspond to physical volume attached to
this volume group

etc

vgchange: Warning: couldn't query all of the physical volumes.
vgchange: Couldn't activate volume group "/dev/vg02":
Quorum not present, or some physical volume(s) are missing.

vgdisplay: Volume group not activated.
vgdisplay: Cannot display volume group "/dev/vg02".
Resynchronized volume group /dev/vg00
vxfs fsck: sanity check: root file system OK (mounted read/write)
Checking hfs file systems
/sbin/fsclean: /dev/vg00/lvol1 (mounted) ok HFS file systems are OK, not
running fsck Checking vxfs file systems Recreating and recovering qlog
devices vxfs fsck: /dev/vg00/rlvol3:/dev/vg00/rlvol3 is currently
mounted Running parallel replay fsck ...
vxfs fsck: /dev/vg02/rlvol06:Cannot open /dev/vg02/rlvol06: No such
device or address /dev/vg02/rlvol06:file system check failure, aborting
...
vxfs fsck: /dev/vg02/rlvol11:Cannot open /dev/vg02/rlvol11: No such
device or address /dev/vg02/rlvol11:file system check failure, aborting
...

etc

The following filesystems could not be repaired :
/dev/vg02/rlvol01
/dev/vg02/rlvol02

etc

couldn't fix vxfs filesystems, dropping into a shell ...
(bcheckrc)#

[Rasheed Tamton]

Hi,

Did you make sure that all your disks are claimed:

ioscan -fCdisk|grep -iv claim

check all disks are available in the VGs; expecially vg02
vgdisplay -v|grep "PV Status"

Do a pvdisplay on the disks in question

check lvmtab
strings /etc/lvmtab

[Frank_W]

Hi

I don´t think that you have a problem with online JFS.
Your server just don´t see all the disks.

- What´s the output of ioscan -fnkCdisk ?

- Are you using some spezial IBM Software to connect to the disks?

[EMCD]

Hi, the ioscan shows all the IBM storage. This is my customers server, who I've relayed these questions back to. I will reply with the full answers to your questions, but in the meantime I've attached a script I asked him to run, to gather info such as ioscan.

Cheers

[Frank_W]

Hi Ewan,

the output of ioscan looks ok. Every device-file ( /dev/dsk/cXtYdZ ) seems to be ok and status "claimed"

But there should be a software that "translates" these device-files into the new device-files like "/dev/dsk/vpath1". This Software should be from IBM and doesnt work now.

( for example EMC uses powerpath software and HP uses securepath )

maybe some other threads with the topic vpath and IBM can help :

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=711403

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=65122

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=625235

[TTr · Accepted solution]

I suspect that this problem probably existed for the begining when vg01 and vg02 were setup. The server was probably never rebooted to realize the problem. So it was the reboot that made the problem show up not OnlineJFS. OnlineJFS is a filesystem software and has nothing to do with volumes. I agree with Frank that there is a multipath software that is running, probably from IBM, that converts the UNIX device files for the LUN devices /dev/dsk/cXtYdZ to virtual disk devices /dev/dsk/vpathX. These devices are probably NOT visible early in the boot process when all the volume group activation takes place and that's why you get the errors.

You should not need to comment out the /etc/fstab entries and rebott. All you should have to do is run "vgchange -a y /dev/vg01" (and for vg02) and issue a mount command for the missing mountpoints.

Look in /etc/rc.log in case there is a startup script of the multipath software or look in "swlist -fileset" to find the same.

ioscan is not going to show the vpath? devices because they do not exist at the hardware level, they are virtual paths that are created by the multipath software.

By the way, IBM has discontinued the use of their SDD software on the DS-series disk arrays. So the add-on software is either from the shark days or from an SVC appliance.

[EMCD]

Closed