topic Re: iSCSI LVM - VGRA appears corrupted in Operating System - HP-UX

iSCSI LVM - VGRA appears corrupted

Bill Costigan — Fri, 15 Feb 2008 20:55:32 GMT

Looking for hints:

I created an iSCSI disk on an HP storage blade and presented it to a bl860c running HPUX 11iv2

I can create a VG on it but every LVM command returns an error saying:

VGRA appears corrupted

I also see the following in syslog. I don't know if it is related:
Feb 15 14:18:49 bl860c krsd[3814]: Delay time is 300 seconds
Feb 15 14:21:52 bl860c islpd[2774]: SLPFindSrvs failed: Connection timed out

except for the messages and a longer than expected delay in executing the commands, everything seems to be working.

Here is the output of the commands:

bl860c (11.23) #vgextend vg01 /dev/dsk/c2t0d0
Volume group "vg01" has been successfully extended.
VGRA appears corrupted
bl860c (11.23) #vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 32
Cur LV 1
Open LV 1
Max PV 32
Cur PV 1
Act PV 1
Max PE per PV 3839
VGDA 2
PE Size (Mbytes) 16
Total PE 3839
Alloc PE 3125
Free PE 714
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/lvol1
LV Status available/syncd
LV Size (Mbytes) 50000
Current LE 3125
Allocated PE 3125
Used PV 1

--- Physical volumes ---
PV Name /dev/dsk/c1t0d0
PV Name /dev/dsk/c2t0d0 Alternate Link
PV Status available
Total PE 3839
Free PE 714
Autoswitch On
Proactive Polling On

Re: iSCSI LVM - VGRA appears corrupted

Marco A. — Fri, 15 Feb 2008 22:31:15 GMT

Hello,

Seems like you have to apply the lastest LVM commulative patch.

These are the patches :

Patch Name: PHKL_36745
Patch Description: s700_800 11.23 LVM Cumulative Patch

Patch Name: PHCO_36744
Patch Description: s700_800 11.23 LVM commands patch

The first one requires reboot.

Regards,

Marco

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Fri, 15 Feb 2008 22:53:15 GMT

Thanks.

That's what I'll do.

Re: iSCSI LVM - VGRA appears corrupted

George Genov — Wed, 12 Mar 2008 15:11:05 GMT

Hello,

Have you managed to mount the iscsi disk?
Because I can make VG and then LV but I cannot mount it.

BR

George

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Sat, 15 Mar 2008 15:41:14 GMT

I have not been back in the office since the middle of Feb, so I haven't had a chance to test it with the patches.

You may want to open a new question and list the error you are seeing with the mount.

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Wed, 19 Mar 2008 17:37:12 GMT

These patches were already installed. This is waht I get when I try to create the volume group on using the iSCSI disk.

bl860c (11.23) #mkdir /dev/vg01/
bl860c (11.23) #mknod /dev/vg01/group c 64 0x010000
bl860c (11.23) #pvcreate -f /dev/rdsk/c1t0d0
Physical volume "/dev/rdsk/c1t0d0" has been successfully created.
bl860c (11.23) #vgcreate -l 32 -p 32 -s 16 vg01 /dev/dsk/c1t0d0
Increased the number of physical extents per physical volume to 3839.
Volume group "/dev/vg01" has been successfully created.
VGRA appears corrupted
bl860c (11.23) #

Re: iSCSI LVM - VGRA appears corrupted

Siju Vadakkan — Wed, 19 Mar 2008 19:29:31 GMT

please check whether the following patch is installed or not :

PHCO_33244

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Wed, 19 Mar 2008 19:38:03 GMT

This patch seems to be for 11.0. Is there one for 11.23

Patch Name: PHCO_33244

Patch Description: s700_800 11.00 LVM commands cumulative patch

Re: iSCSI LVM - VGRA appears corrupted

Michael Steele_2 — Wed, 19 Mar 2008 20:00:06 GMT

There solutions to this 11.23 problem with patches.

http://www11.itrc.hp.com/service/patch/patchDetail.do?patchid=PHKL_36745&caller=james#Patch%20Dependencies:

Dependencies:

PHCO_36744
PHKL_31500

YES or NO they are or are not installed?

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Thu, 20 Mar 2008 13:13:49 GMT

Yes they are installed:

bl860c (11.23) #swlist -l product |grep PHKL_31500
PHKL_31500 1.0 Sept04 base patch
bl860c (11.23) #swlist -l product |grep PHCO_36744
PHCO_36744 1.0 LVM commands patch

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Thu, 20 Mar 2008 13:15:58 GMT

And so is:

bl860c (11.23) #swlist -l product |grep PHKL_36745
PHKL_36745 1.0 LVM Cumulative Patch

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Thu, 20 Mar 2008 16:15:18 GMT

I'm thinking this is more a iSCSI issue than an LVM issue. I was able to create the VG and LVOL and use it despite the VGRA messages.

But I'm getting the following errors about the iSSCI disk that contains the VG.

Mar 20 13:08:28 bl860c vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Mar 20 13:08:28 bl860c vmunix: LVM: VG 64 0x010000: PVLink 31 0x010000 Failed! The PV is not accessible.
Mar 20 13:08:33 bl860c vmunix: LVM: VG 64 0x010000: PVLink 31 0x010000 Recovered.
Mar 20 13:10:06 bl860c vmunix:
Mar 20 13:10:06 bl860c vmunix: SCSI: Read error -- dev: b 31 0x010000, errno: 126, resid: 1024,
Mar 20 13:10:06 bl860c vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Mar 20 13:10:06 bl860c vmunix: LVM: VG 64 0x010000: PVLink 31 0x010000 Failed! The PV is not accessible.
Mar 20 13:10:11 bl860c vmunix: LVM: VG 64 0x010000: PVLink 31 0x010000 Recovered.

The HPUX server is a blade [BL860c] and the iSCSI storage is another blade SB600c in the same enclosure.

It might be a settings issue.

The iSCSI config on the SB600C has
Max Data Segment Length - 65536 Bytes
First Burst - 65536 Bytes
Max Burst - 262144 Bytes
Number of Receive Buffers - 10

Re: iSCSI LVM - VGRA appears corrupted

Michael Steele_2 — Thu, 20 Mar 2008 17:40:50 GMT

0x010000 This looks like a controller.

Paste in this report:

/usr/sbin/cstm<<-EOF
runutil logtool
rs
EOF

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Thu, 20 Mar 2008 20:11:03 GMT

Here is the output.

We are seeing IO errors but the path is to the iSCSI disks. This is the SB600c (which is running the Microsoft iSCSI target)

Logtool Utility>
Logtool Utility>rs
Select Raw processing file /var/stm/logs/os/log227.raw.cur
Number of entries analyzed is 1.
Number of entries analyzed is 51.
Number of entries analyzed is 101.
Number of entries analyzed is 151.
Number of entries analyzed is 201.
Number of entries analyzed is 251.
Number of entries analyzed is 301.
Number of entries analyzed is 351.
Number of entries analyzed is 401.
Number of entries analyzed is 451.
Number of entries analyzed is 501.
Number of entries analyzed is 551.
Number of entries analyzed is 596.
The Select Raw operation completed.

The Display Raw Summary operation is currently in progress.

-- Converting a (860) byte raw log file to text. --
Preparing the Logtool Utility: View Raw Summary File ...

.... bl860c.advizex.com : 10.250.1.33 ....

-- Logtool Utility: View Raw Summary --

Summary of: /var/stm/logs/os/log227.raw.cur

Date/time of first entry: Thu Mar 20 16:27:56 2008

Date/time of last entry: Thu Mar 20 17:05:44 2008

Number of LPMC entries: 0
Number of System Overtemp entries: 0
Number of LVM entries: 0
Number of Logger Event entries: 0

Number of I/O Error entries: 596

Device paths for which entries exist:

(162) 255/0/0.0.0.0
(434) 255/0/1.0.0.1

Products for which entries exist:

(596)

Product Qualifiers for which entries exist:

(596)

Logger Events for which entries exist:

(596) sdisk
-- Logtool Utility: View Raw Summary --
View - To View the file.
Print - To Print the file.
SaveAs - To Save the file.
Enter Done, Help, Print, SaveAs, or View: [Done]
Display of the raw log file summary was successful.

Logtool Utility>#.
bl860c (11.23) #

Re: iSCSI LVM - VGRA appears corrupted

Michael Steele_2 — Thu, 20 Mar 2008 20:24:30 GMT

OK. This is a Hardware problem. Call HP and refer to this report. When you see the start and stop date of Mar 20 and 596 errors collected then you've got a device on the brink of failure. Anything over 100 or so is enough. In your case you've got:

(162) 255/0/0.0.0.0
(434) 255/0/1.0.0.1

...both hw paths to devices with over a hundred errors collected in a 24 hour period.

Cross refercence these HW paths with ioscan to determine the devices and call HP.

Good job.

#################
Date/time of first entry: Thu Mar 20 16:27:56 2008

Date/time of last entry: Thu Mar 20 17:05:44 2008

Number of LPMC entries: 0
Number of System Overtemp entries: 0
Number of LVM entries: 0
Number of Logger Event entries: 0

Number of I/O Error entries: 596

Device paths for which entries exist:

(162) 255/0/0.0.0.0
(434) 255/0/1.0.0.1

Re: iSCSI LVM - VGRA appears corrupted

Bill Costigan — Thu, 20 Mar 2008 20:28:34 GMT

These are not real real devices. They are iSCSI accessed through a network interface.

We have VMware accessing the same physical array and disk drives without errors.

Re: iSCSI LVM - VGRA appears corrupted

Michael Steele_2 — Thu, 20 Mar 2008 22:56:42 GMT

Even if its SAN over LAN its generating 600 SCSI errors in 24 hours and someone from HP will want to know.

Can you get to the MP and review the error logs?

You should have an ip address for a telnet connection into this bl860c that will allow you to review the MP logs.

Re: iSCSI LVM - VGRA appears corrupted

Michael Steele_2 — Fri, 21 Mar 2008 01:47:32 GMT

Here's what I have from a ioscan using iSCSI.

255/0 = your virtual node
255/0/0.0 = your initiator session id (* ISID *)
255/0/0.0.0 = your virtual bus
255/0/0.0.0.0 = 8 disk array luns / 1 virtual lun
255/0/1.0.0.1 = 8 disk array luns / 1 virtual lun

So your getting errors from
(162) 255/0/0.0.0.0 = 8 disk array luns
(434) 255/0/1.0.0.1 = 8 disk array luns

Try the 'iscsiutil' command. Do you have this installed? If so paste in these reports:

iscsiutil -sG

The man page on iscsiutil states the following:

Issue the iscsiutil -sG command and look at the following statistics:

Number of Discovery session open failures

If there is a non-zero value for this statistic, determine the cause of failure by either looking at the message logged in the /var/adm/syslog/syslog.log file or by monitoring the EMS/STM log files. The most common cause for failure would be an incorrect configuration of the components of the iSCSI target address:

'iscsiutil' is based upon stm / logtool log files.

Re: iSCSI LVM - VGRA appears corrupted

Michael Steele_2 — Fri, 21 Mar 2008 01:48:43 GMT

Also check your /etc/opt/resmon/*.logs for related events. Paste in any event numbers.

Re: iSCSI LVM - VGRA appears corrupted

George Genov — Fri, 21 Mar 2008 05:48:06 GMT

Hello,

As you can see int the link:

http://docs.hp.com/en/iSCSI_SM/iSCSI_SM.htm

the MS Windows is not supported as iSCSI target for HP-UX. So that is why you and I see all of these strange messages.

P.S. I think that there is nothing wrong with your hardware.

BR

George