topic Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy in Operating System - HP-UX

vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Ramaprasad K — Wed, 29 Apr 2009 08:29:25 GMT

Hi experts,

Server: rp5470
OS: HP-UX 11.11

The above error appears for VGs which contain 4 LUNs presented from EVA 3000 as shown below:

#vgchange -a y vg02
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c4t0d1":
Device busy
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c4t0d2":
Device busy
vgchange: Warning: couldn't query physical volume "/dev/dsk/c4t0d1":
The specified path does not correspond to physical volume attached to this volume group
vgchange: Warning: couldn't query physical volume "/dev/dsk/c4t0d2":
The specified path does not correspond to physical volume attached to this volume group
vgchange: Warning: couldn't query all of the physical volumes.
vgchange: Couldn't activate volume group "vg02":
Quorum not present, or some physical volume(s) are missing.

Above error is reported for each of vg03 and vg04. Following is the /etc/lvmtab:

#strings /etc/lvmtab
/dev/vg01
/dev/dsk/c2t2d0
/dev/vg02
/dev/dsk/c4t0d1
/dev/dsk/c4t0d2
/dev/vg03
/dev/dsk/c4t0d3
/dev/vg04
/dev/dsk/c4t0d4

***************

Diskinfo for above disks show 0 bytes but in the ioscan, the disks show to be claimed.
--------------------------------
SCSI describe of /dev/rdsk/c4t0d1:
vendor: HP
product id: HSV100
type: direct access
size: 0 Kbytes
bytes per sector: 0
rev level: 3020
---------------------------------
above o/p is similar for other 3 disks with diskinfo command

***********

#ioscan -funC disk
Class I H/W Path Driver S/W State H/W Type Description
=================================================================================
disk 0 0/0/1/1.2.0 sdisk CLAIMED DEVICE HP 146 GST3146807LC
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 1 0/0/2/0.2.0 sdisk CLAIMED DEVICE HP 146 GST3146807LC
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 2 0/0/2/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 305
/dev/dsk/c3t2d0 /dev/rdsk/c3t2d0
disk 3 0/10/0/0.1.0.0.0.0.1 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d1 /dev/rdsk/c4t0d1
disk 4 0/10/0/0.1.0.0.0.0.2 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d2 /dev/rdsk/c4t0d2
disk 5 0/10/0/0.1.0.0.0.0.3 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d3 /dev/rdsk/c4t0d3
disk 6 0/10/0/0.1.0.0.0.0.4 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d4 /dev/rdsk/c4t0d4
disk 7 0/10/0/0.1.0.0.0.0.5 sdisk CLAIMED DEVICE HP HSV100
/dev/dsk/c4t0d5 /dev/rdsk/c4t0d5

********************

Is this HW issue? Should I log call with HP?

Regards,
Ram.

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Torsten. — Wed, 29 Apr 2009 08:46:54 GMT

>> size: 0 Kbytes

You should check the EVA side - how about the status of the vdisks?

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Ramaprasad K — Wed, 29 Apr 2009 09:26:36 GMT

Torsten,

Thanks for your reply.

Forgot to update that I checked the Status of Vdisk in CV-EVA. Irony is, all Vdisks presented to the server show operational state to be Good and in the navigation pane, none of Vdisks show warning.

What could be other reason?

Regards,
Ram.

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Henry Fauni — Wed, 29 Apr 2009 22:18:46 GMT

Try doing "ioscan -fnC disk" to actually probe hardware. It may come back with "NO_HW" this time. If so, check EVA3000 presentation to verify correct host. On the EVA, also check the host ports that they're matching up with the host port WWN. Are the virtual disks connected to multiple hosts? If so, are they activated on the other node? Are you using "Secure Path" for multi-pathing?

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Basheer_2 — Thu, 30 Apr 2009 05:13:08 GMT

try,

ioscan
pvdisplay /dev/dsk/c4t0d3
pvdisplay /dev/dsk/c4t0d4
(these will tell you if they are assigned to any volume group)

mv /etc/lvmtab /etc/lvmtab /etc/lvmtab.bak1

vgscan
(recreate /etc/lvmtab)

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Ramaprasad K — Tue, 05 May 2009 04:06:33 GMT

Thanks Henry and Basheer for your replies.

@ Henry
No chance, even after rebooting server once again. Presentation is all fine. No changes done. Also, Vdisks are not connected to multiple hosts.

@ Basheer
tried mv /etc/lvmtab /etc/lvmtab.old and then vgscan. No luck. Still same error as earlier.

*********************************************

FYI, On 26th the server hanged and was not responding. Even was not able to telnet. Console was also hanging. So hardbooted server and it came up. While booting, it threw messages saying unable to activate VGs. So checked the Oldsyslog and at the point of time of the issue, found following logs:

--------------------------------------------

Apr 26 20:04:05 disney vmunix: DIAGNOSTIC SYSTEM WARNING:
Apr 26 20:04:05 disney vmunix: The diagnostic logging facility has started receiving excessive
Apr 26 20:04:05 disney vmunix: errors from the I/O subsystem. I/O error entries will be lost
Apr 26 20:04:05 disney vmunix: until the cause of the excessive I/O logging is corrected.
Apr 26 20:04:05 disney vmunix: If the diaglogd daemon is not active, use the Daemon Startup command
Apr 26 20:04:05 disney vmunix: in stm to start it.
Apr 26 20:04:05 disney vmunix: If the diaglogd daemon is active, use the logtool utility in stm
Apr 26 20:04:05 disney vmunix: to determine which I/O subsystem is logging excessive errors.
--------------------------------------------

Any idea why even after booting the PVs are not shown up. In ioscan, everthing seems to be fine, all coressponding PVs are CLAIMED, same is case with link to EVA.

Any idea on what could be the issue?

Regards,
Ram.

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Torsten. — Tue, 05 May 2009 04:43:44 GMT

The firmware of your (active/passive) EVA looks very old and unsupported. Since it is a/p, you must have secure path. What does secure path report? ("spmgr display").

Because the vdisks are visible but zero in size, the problem could be related either to secure path anyhow or it is likely on the array side.

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Ramaprasad K — Tue, 05 May 2009 04:59:53 GMT

Sorry missed posting complete error from Oldsyslog file. here it is. I feel there was a temporary glitch or reset in PV link, don't know for what reason.
---------------------------------------------------
Apr 25 23:00:52 disney ftpd[7863]: FTP session closed
Apr 26 20:03:40 disney vmunix: LVM: Path (device 0x1f040100) to PV 0 in VG 2 Failed!
Apr 26 20:03:40 disney vmunix: LVM: vg[2]: pvnum=0 (dev_t=0x1f040100) is POWERFAILED
Apr 26 20:03:40 disney vmunix: LVM: vg[2]: pvnum=1 (dev_t=0x1f040200) is POWERFAILED
Apr 26 20:03:40 disney vmunix: LVM: Path (device 0x1f040300) to PV 0 in VG 3 Failed!
Apr 26 20:03:40 disney vmunix: LVM: vg[3]: pvnum=0 (dev_t=0x1f040300) is POWERFAILED
Apr 26 20:03:40 disney vmunix: LVM: Path (device 0x1f040400) to PV 0 in VG 4 Failed!
Apr 26 20:03:40 disney vmunix: LVM: vg[4]: pvnum=0 (dev_t=0x1f040400) is POWERFAILED
Apr 26 20:04:05 disney vmunix: DIAGNOSTIC SYSTEM WARNING:
Apr 26 20:04:05 disney vmunix: The diagnostic logging facility has started receiving excessive
Apr 26 20:04:05 disney vmunix: errors from the I/O subsystem. I/O error entries will be lost
Apr 26 20:04:05 disney vmunix: until the cause of the excessive I/O logging is corrected.
Apr 26 20:04:05 disney vmunix: If the diaglogd daemon is not active, use the Daemon Startup command
Apr 26 20:04:05 disney vmunix: in stm to start it.
Apr 26 20:04:05 disney vmunix: If the diaglogd daemon is active, use the logtool utility in stm
Apr 26 20:04:05 disney vmunix: to determine which I/O subsystem is logging excessive errors.
-----------------------------------------------

Torsten- There is no secure path installed. Also, in CV-EVA, the Vdisk size is not zero. Even the Vdisk status shows to be in good condition.

Any idea?

Ram.

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Torsten. — Tue, 05 May 2009 06:05:43 GMT

From my point of view your configuration is the problem.

It looks like your server has only 1 HBA, connected to a single controller of the EVA.

If the controller failes or gives the ownership of the LUN to the other controller for any reason, your server is disconnected.

Looks like this is what happened here.

Re: vgchange: Warning: Couldn't attach to volume group physical volume "/dev/dsk/c4t0d1": Device busy

Ramaprasad K — Wed, 06 May 2009 04:12:43 GMT

Torsten,

You are right. The Server has only 1 HBA and connected to SAN Switch. Probably need to check zoning if both controllers are in same zone as the port connected from server. If not- seems to be so, can I add other controller into the zone?

But this seems to be illogical on controller part of taking ownership of VDISK. What I understand is one among two controllers will be master which takes ownership of entire EVA rather than VDISKs, which would mean any server zoned only with slave controller at that point in time wouldn't see its LUNs, which probably wouldn't be the case in my setup. Anyhow, shall check and get back.

Regards,
Ram.