iSCSI LVM - VGRA appears corrupted

[Bill Costigan]

Looking for hints:

I created an iSCSI disk on an HP storage blade and presented it to a bl860c running HPUX 11iv2

I can create a VG on it but every LVM command returns an error saying:

VGRA appears corrupted

I also see the following in syslog. I don't know if it is related:
Feb 15 14:18:49 bl860c krsd[3814]: Delay time is 300 seconds
Feb 15 14:21:52 bl860c islpd[2774]: SLPFindSrvs failed: Connection timed out

except for the messages and a longer than expected delay in executing the commands, everything seems to be working.

Here is the output of the commands:

bl860c (11.23) #vgextend vg01 /dev/dsk/c2t0d0
Volume group "vg01" has been successfully extended.
VGRA appears corrupted
bl860c (11.23) #vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 32
Cur LV 1
Open LV 1
Max PV 32
Cur PV 1
Act PV 1
Max PE per PV 3839
VGDA 2
PE Size (Mbytes) 16
Total PE 3839
Alloc PE 3125
Free PE 714
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/lvol1
LV Status available/syncd
LV Size (Mbytes) 50000
Current LE 3125
Allocated PE 3125
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c1t0d0
PV Name /dev/dsk/c2t0d0 Alternate Link
PV Status available
Total PE 3839
Free PE 714
Autoswitch On
Proactive Polling On

[Marco A. · Accepted solution]

Hello,

Seems like you have to apply the lastest LVM commulative patch.


These are the patches :

Patch Name: PHKL_36745
Patch Description: s700_800 11.23 LVM Cumulative Patch

Patch Name: PHCO_36744
Patch Description: s700_800 11.23 LVM commands patch

The first one requires reboot.

Regards,


Marco

[Bill Costigan]

Thanks.

That's what I'll do.

[George Genov]

Hello,

Have you managed to mount the iscsi disk?
Because I can make VG and then LV but I cannot mount it.

BR

George

[Bill Costigan]

I have not been back in the office since the middle of Feb, so I haven't had a chance to test it with the patches.

You may want to open a new question and list the error you are seeing with the mount.

[Bill Costigan]

These patches were already installed. This is waht I get when I try to create the volume group on using the iSCSI disk.

bl860c (11.23) #mkdir /dev/vg01/
bl860c (11.23) #mknod /dev/vg01/group c 64 0x010000
bl860c (11.23) #pvcreate -f /dev/rdsk/c1t0d0
Physical volume "/dev/rdsk/c1t0d0" has been successfully created.
bl860c (11.23) #vgcreate -l 32 -p 32 -s 16 vg01 /dev/dsk/c1t0d0
Increased the number of physical extents per physical volume to 3839.
Volume group "/dev/vg01" has been successfully created.
VGRA appears corrupted
bl860c (11.23) #

[Siju Vadakkan]

please check whether the following patch is installed or not :

PHCO_33244

[Bill Costigan]

This patch seems to be for 11.0. Is there one for 11.23

Patch Name: PHCO_33244

Patch Description: s700_800 11.00 LVM commands cumulative patch

[Michael Steele_2]

There solutions to this 11.23 problem with patches.

http://www11.itrc.hp.com/service/patch/patchDetail.do?patchid=PHKL_36745&caller=james#Patch%20Dependencies:

Dependencies:

PHCO_36744
PHKL_31500

YES or NO they are or are not installed?

[Bill Costigan]

Yes they are installed:

bl860c (11.23) #swlist -l product |grep PHKL_31500
PHKL_31500 1.0 Sept04 base patch
bl860c (11.23) #swlist -l product |grep PHCO_36744
PHCO_36744 1.0 LVM commands patch

[Bill Costigan]

And so is:

bl860c (11.23) #swlist -l product |grep PHKL_36745
PHKL_36745 1.0 LVM Cumulative Patch

[Bill Costigan]

I'm thinking this is more a iSCSI issue than an LVM issue. I was able to create the VG and LVOL and use it despite the VGRA messages.

But I'm getting the following errors about the iSSCI disk that contains the VG.

Mar 20 13:08:28 bl860c vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Mar 20 13:08:28 bl860c vmunix: LVM: VG 64 0x010000: PVLink 31 0x010000 Failed! The PV is not accessible.
Mar 20 13:08:33 bl860c vmunix: LVM: VG 64 0x010000: PVLink 31 0x010000 Recovered.
Mar 20 13:10:06 bl860c vmunix:
Mar 20 13:10:06 bl860c vmunix: SCSI: Read error -- dev: b 31 0x010000, errno: 126, resid: 1024,
Mar 20 13:10:06 bl860c vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 1024.
Mar 20 13:10:06 bl860c vmunix: LVM: VG 64 0x010000: PVLink 31 0x010000 Failed! The PV is not accessible.
Mar 20 13:10:11 bl860c vmunix: LVM: VG 64 0x010000: PVLink 31 0x010000 Recovered.

The HPUX server is a blade [BL860c] and the iSCSI storage is another blade SB600c in the same enclosure.

It might be a settings issue.

The iSCSI config on the SB600C has
Max Data Segment Length - 65536 Bytes
First Burst - 65536 Bytes
Max Burst - 262144 Bytes
Number of Receive Buffers - 10

[Michael Steele_2]

0x010000 This looks like a controller.

Paste in this report:

/usr/sbin/cstm<<-EOF
runutil logtool
rs
EOF

[Bill Costigan]

Here is the output.

We are seeing IO errors but the path is to the iSCSI disks. This is the SB600c (which is running the Microsoft iSCSI target)


Logtool Utility>
Logtool Utility>rs
Select Raw processing file /var/stm/logs/os/log227.raw.cur
Number of entries analyzed is 1.
Number of entries analyzed is 51.
Number of entries analyzed is 101.
Number of entries analyzed is 151.
Number of entries analyzed is 201.
Number of entries analyzed is 251.
Number of entries analyzed is 301.
Number of entries analyzed is 351.
Number of entries analyzed is 401.
Number of entries analyzed is 451.
Number of entries analyzed is 501.
Number of entries analyzed is 551.
Number of entries analyzed is 596.
The Select Raw operation completed.

The Display Raw Summary operation is currently in progress.

-- Converting a (860) byte raw log file to text. --
Preparing the Logtool Utility: View Raw Summary File ...

.... bl860c.advizex.com : 10.250.1.33 ....

-- Logtool Utility: View Raw Summary --

Summary of: /var/stm/logs/os/log227.raw.cur

Date/time of first entry: Thu Mar 20 16:27:56 2008

Date/time of last entry: Thu Mar 20 17:05:44 2008



Number of LPMC entries: 0
Number of System Overtemp entries: 0
Number of LVM entries: 0
Number of Logger Event entries: 0

Number of I/O Error entries: 596


Device paths for which entries exist:

(162) 255/0/0.0.0.0
(434) 255/0/1.0.0.1

Products for which entries exist:

(596)

Product Qualifiers for which entries exist:

(596)

Logger Events for which entries exist:

(596) sdisk
-- Logtool Utility: View Raw Summary --
View - To View the file.
Print - To Print the file.
SaveAs - To Save the file.
Enter Done, Help, Print, SaveAs, or View: [Done]
Display of the raw log file summary was successful.

Logtool Utility>#.
bl860c (11.23) #

[Michael Steele_2]

OK. This is a Hardware problem. Call HP and refer to this report. When you see the start and stop date of Mar 20 and 596 errors collected then you've got a device on the brink of failure. Anything over 100 or so is enough. In your case you've got:

(162) 255/0/0.0.0.0
(434) 255/0/1.0.0.1

...both hw paths to devices with over a hundred errors collected in a 24 hour period.

Cross refercence these HW paths with ioscan to determine the devices and call HP.

Good job.

#################
Date/time of first entry: Thu Mar 20 16:27:56 2008

Date/time of last entry: Thu Mar 20 17:05:44 2008


Number of LPMC entries: 0
Number of System Overtemp entries: 0
Number of LVM entries: 0
Number of Logger Event entries: 0

Number of I/O Error entries: 596


Device paths for which entries exist:

(162) 255/0/0.0.0.0
(434) 255/0/1.0.0.1

[Bill Costigan]

These are not real real devices. They are iSCSI accessed through a network interface.

We have VMware accessing the same physical array and disk drives without errors.

[Michael Steele_2]

Even if its SAN over LAN its generating 600 SCSI errors in 24 hours and someone from HP will want to know.

Can you get to the MP and review the error logs?

You should have an ip address for a telnet connection into this bl860c that will allow you to review the MP logs.

[Michael Steele_2]

Here's what I have from a ioscan using iSCSI.

255/0 = your virtual node
255/0/0.0 = your initiator session id (* ISID *)
255/0/0.0.0 = your virtual bus
255/0/0.0.0.0 = 8 disk array luns / 1 virtual lun
255/0/1.0.0.1 = 8 disk array luns / 1 virtual lun

So your getting errors from
(162) 255/0/0.0.0.0 = 8 disk array luns
(434) 255/0/1.0.0.1 = 8 disk array luns

Try the 'iscsiutil' command. Do you have this installed? If so paste in these reports:

iscsiutil -sG

The man page on iscsiutil states the following:

Issue the iscsiutil -sG command and look at the following statistics:

Number of Discovery session open failures

If there is a non-zero value for this statistic, determine the cause of failure by either looking at the message logged in the /var/adm/syslog/syslog.log file or by monitoring the EMS/STM log files. The most common cause for failure would be an incorrect configuration of the components of the iSCSI target address:

'iscsiutil' is based upon stm / logtool log files.

[Michael Steele_2]

Also check your /etc/opt/resmon/*.logs for related events. Paste in any event numbers.

[George Genov]

Hello,

As you can see int the link:

http://docs.hp.com/en/iSCSI_SM/iSCSI_SM.htm

the MS Windows is not supported as iSCSI target for HP-UX. So that is why you and I see all of these strange messages.

P.S. I think that there is nothing wrong with your hardware.

BR

George

[Bill Costigan]

Not Supported. Thanks.

I was hoping to used the storage blade in the blade server to provide storage for the HPUX blade. Guess not.

Thank you everyone for your help and suggestions.