Reboot and volume group not active

[Coolmar]

I have an hp-ux system attached to a Hitachi SAN. I have to move the data off the Hitachi and over to an IBM8300 Shark. The shark disk was assigned, and the HP-UX system found it. I added it to vg01 and then did a lvextend to mirror the data. Everything is fine and "current". We had to reboot the server on the weekend and the volume group with the shark disk attached didn't come up (see errors below). All I had to do was a "vgchange -a y /dev/vg01" and then mountall and everything was fine again. Why would I have to activate the volume group after a reboot?

vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c38t0d0":
The path of the physical volume refers to a device that does not
exist, or is not configured into the kernel.
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c35t0d0":
The path of the physical volume refers to a device that does not
exist, or is not configured into the kernel.
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c38t0d1":
The path of the physical volume refers to a device that does not
exist, or is not configured into the kernel.
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c35t0d1":

[Patrick Wallek]

It sounds as if the system could not see the disks until it was completely booted.

Do you have any special drivers / daemons that load that are required to access the data on the IBM shark array?

[Coolmar]

No, I have no drivers loaded and would really like to avoid that if at all possible. We figured we didn't need them because the disks came right up in the ioscan and then we were able to write to them without a problem.

[Mustafa Gulercan]

hi sally;
is there a serviceguard manager at system?
is there a cluster?

[Coolmar]

No...no serviceguard and no cluster.

[Mustafa Gulercan]

check the automatic volume group activation at /etc/lvmrc.
AUTO_VG_ACTIVATE= 0 or 1 ?

[Patrick Wallek]

I don't think this issue has anything to do with the "AUTO_VG_ACTIVATE" setting.

I think the real issue is the "Couldn't attach to the volume group physical volume" messages posted.

Are the PV names mentioned (c35t0d0, c38t0d0, c35t0d1, c38t0d1) mentioned the ones that are part of your VG01?

If so, I think you need to figure out when / why those messages are occurring. Do they just occurr when the system is rebooted? If that is fixed, then you may also fix your VG activation issue.

[Coolmar]

Yeah, all the devices mentioned are the Shark LUNs. We get the errors after every reboot.

[Coolmar]

When I do a strings /etc/lvmtab, I get the following:

/dev/vg00
/dev/dsk/c1t2d0
/dev/dsk/c2t2d0

/dev/vg01
/dev/dsk/c6t12d6
/dev/dsk/c4t12d6
/dev/dsk/c6t12d5
/dev/dsk/c4t12d5
/dev/dsk/c38t0d0
/dev/dsk/c35t0d0
/dev/dsk/c38t0d1
/dev/dsk/c35t0d1

Then when I do a vgscan -p -v, I get the following:
/dev/vg00
/dev/dsk/c1t2d0
/dev/dsk/c2t2d0

/dev/vg01
/dev/dsk/c38t0d0
/dev/dsk/c38t0d1
/dev/dsk/c6t12d5
/dev/dsk/c6t12d6
/dev/dsk/c35t0d0
/dev/dsk/c35t0d1
/dev/dsk/c4t12d5
/dev/dsk/c4t12d6


Following Physical Volumes belong to one Volume Group.
Unable to match these Physical Volumes to a Volume Group.
Use the vgimport command to complete the process.
/dev/dsk/c8t0d4
/dev/dsk/c10t0d4

Could /dev/dsk/c8t0d4 and c10t0d4 be the problem? It is a LUN from another SAN, that is still visible to the system, but not part of the volume group.

[Coolmar]

I did a pvremove of the two devices above, rebooted and no luck. Again, had to wait for the system to come back up and then activate the VG and then mountall.

[Sandman!]

Sally,

It's possible one of the disks on the Shark array is bad. Do an ioscan to see if any show status other than "CLAIMED".

On the other hand it could very well be a quorum problem since the VG is mirrored, try activating the VG without quorum checking...

# vgchange -a y -q n /dev/vg01

~cheers

[Coolmar]

The devices are all claimed and I can write to them. So you think i should run the command disabling Quorum and then reboot?

[Sandman!]

Yes Sally.

[Scott Riley]

Sally, I'm not Shark expert,but are devices c38t0d0 and c35t0d0 two paths to the same lun? And likewise c38t0d1 and c35t0d1?

If you have multiple paths to the same lun on some kinds of controller-based arrays, you can have access problems down the alternate path without using the manufacturer's path failover software.

[Coolmar]

Sandman - same error. Thanks for trying though.

[Coolmar]

Yes, Scott....the C35's are the failover paths.

[DCE]

Sally,

Here is the link to the IBM drivers - it looks like you may need to use it in order to fully utilize the shark.

http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000053&rs=555


Download Description
The Subsystem Device Driver is a pseudo device driver designed to support the multipath configuration environments in the IBM TotalStorage Enterprise Storage Server, the IBM TotalStorage DS family, and the IBM System Storage SAN Volume Controller. It resides in a host system with the native disk device driver and provides the following functions:

- Enhanced data availability
- Dynamic I/O load-balancing across multiple paths
- Automatic path failover protection
- Concurrent download of licensed internal code
- Path-selection policies for the host system

[Scott Riley]

Sally, in that case it sounds like you either need to install the IBM multipath softwre, or possible make some configuration adjustments to the array.

The array most likely has settings for HPUX support for your LUNS - has someone checked those to make sure they are set correctly?

[Coolmar]

Hi Scott....The SAN people look after that and we are not allowed anywhere near it, not that I would be much good anyway ;0) Do you have anything specific in mind that I could ask them in regards to the HP setup on the SAN?

Thanks again,
Sally

[Scott Riley]

It would be related to how the array handles SCSI_trespass commands. With alternate paths, HPUX will switch I/O to the alternate path by issuing a SCSI_trespass command down that path, and the array is expected to respond with I/O down that path.

I know on other arrays there are specific bit settings/flags that you need to set to tell the array it is dealing with HPUX. I would ask the storage folks to double check those bit settings/flags on those luns to be sure they are set for HPUX support. Arrays (inluding your Hitachi) supporting multi-oses usually treat Windows, Solaris and Linux the same, but AIX and HPUX are configured differently.

Sorry I can't be more specific, like I said I'm not a Shark guy. Your storage team should have explicit configuration information for HPUX though...

[Sandman!]

Are there any errors logged in syslog or dmesg? Could you post the output of the command below:

# vgdisplay -v vg01

If you have multiple paths to each of the vg01 PVs, I would suggest you vgreduce the alternate paths from vg01. After that reboot the server and see if this goes okay i.e. you don't have to manually activate vg01 after the reboot. If all is well then procedd with adding the alternate PV links to vg01 with vgextend.

before rebooting...
# vgreduce /dev/vg01 /dev/dsk/c#t#d#

If okay add PV links back in...
# vgextend /dev/vg01 /dev/dsk/c#t#d#

~hope it helps

[sathish kannan]

Hi Sally, are you seeing the same error message on c4 and c6 controllers. Are they belongs to IBM Shark? Have you checked rc.boot log file and /var/adm/syslog/syslog.log boot messages.

If you are receiving only on IBM Shark devices, I woud recommed to check the settings with IBM Shark for your host connectivity. Also remove PV link and check the reboot.

Regards
Sathish

[Coolmar]

This is what our SAN guy has told me...ESS rings a bell, should the hp-ux luns be ESS?

I can look at creating different LUN characteristics. Yours are open systems DS size, there are also Opens systems ESS size, Open systems Block and iseries - protected and iseries unprotected.

[Coolmar]

To answer the above...there are no errors except for the initial post upon booting. I have checked the rc.log and the only problems are where vg01 is needed because the volume group won't come up in the boot process which happens before it gets to the rc.log startup scripts.

No, I am not getting the same errors on c6 and c4 and those devices are attached to the Hitachi SAN.

[Coolmar]

I removed the failover paths (c35t0d0) and rebooted and I still get the same problem.