Persistent Reservation issues

Jian-ping Hui — Mon, 02 Mar 2009 07:26:58 GMT

On HP-UX 11.31, we have two nodes which share two LUNs:
[vega8087{root}:/home/jphui/temp]>ioscan -m lun /dev/rdisk/disk32
Class I Lun H/W Path Driver S/W State H/W Type Health Description
======================================================================
disk 32 64000/0xfa00/0xc esdisk CLAIMED DEVICE online HP HSV210
0/4/1/1.0x50001fe150102e68.0x4001000000000000
0/4/1/1.0x50001fe150102e69.0x4001000000000000
0/4/1/1.0x50001fe150102e6a.0x4001000000000000
0/4/1/1.0x50001fe150102e6b.0x4001000000000000
0/4/1/1.0x50001fe150102e6e.0x4001000000000000
0/4/1/1.0x50001fe150102e6c.0x4001000000000000
0/4/1/1.0x50001fe150102e6f.0x4001000000000000
0/4/1/1.0x50001fe150102e6d.0x4001000000000000
/dev/disk/disk32 /dev/rdisk/disk32
[vega8087{root}:/home/jphui/temp]>ioscan -m lun /dev/rdisk/disk42
Class I Lun H/W Path Driver S/W State H/W Type Health Description
======================================================================
disk 42 64000/0xfa00/0x13 esdisk CLAIMED DEVICE online COMPAQ MSA1000 VOLUME
0/4/1/0.0x500805f300120051.0x1000000000000
/dev/disk/disk42 /dev/rdisk/disk42

We wrote a C program to issue "PERSISTENT IN" and "PERSISTENT OUT" by IOCTL to reserve the LUN.

Below are some behavior in the test:
1. If we reserved the shared LUNs on 1 node with the reservation type 5(Write Exclusive - Registrants
Only), the command "diskinfo" on the other node will not get the correct information:
[vega8097{root}:/home/jphui/temp]>diskinfo /dev/rdisk/disk42
SCSI describe of /dev/rdisk/disk42:
vendor: COMPAQ
product id: MSA1000 VOLUME
type: direct access
size: 0 Kbytes <=== This value is not correct
bytes per sector: 0 <=== This value is not correct

The command will get successful if this node also reserved the shared LUNs. Looks this is a little
different with the behavior on HP-UX 11.23. On HP-UX 11.23, we can get the information correctly
although the other node has reserved the LUN.

I'm wondering if this is the expected behavior on HP-UX 11.31. If so, how can we make the 2nd node
to show the correct information while running "diskinfo"? Is there any special system command to run to
register the 2nd node?

2. If we run our C program to reserve the LUN /dev/rdisk/disk42, it will always succeed. But if we run
the program to reserve the LUN /dev/rdisk/disk32, it will fail with "Reservation Conflict". If we
try to run the program continuously, it will succeed after couples of runs. Looks this is strange.
I checked the information of two LUNs:
[vega8087{root}:/home/jphui/temp]>scsimgr get_info -D /dev/rdisk/disk32

STATUS INFORMATION FOR LUN : /dev/rdisk/disk32

Generic Status Information

SCSI services internal state = UNOPEN
Device type = Direct_Access
EVPD page 0x83 description code = 1
EVPD page 0x83 description association = 0
EVPD page 0x83 description type = 3
World Wide Identifier (WWID) = 0x600508b40006a8ff0001200000080000
Serial number = "PBA23D99SV309F"
Vendor id = "HP "
Product id = "HSV210 "
Product revision = "6110"
Other properties = ""
SPC protocol revision = 2
Open count (includes chr/blk/pass-thru/class) = 0
Raw open count (includes class/pass-thru) = 0
Pass-thru opens = 0
LUN path count = 8
Active LUN paths = 4
Standby LUN paths = 4
Failed LUN paths = 0
Maximum I/O size allowed = 2097152
Preferred I/O size = 2097152
Outstanding I/Os = 0
I/O load balance policy = round_robin
Path fail threshold time period = 0
Transient time period = 120
Tracing buffer size = 1024
LUN Path used when policy is path_lockdown = NA
LUN access type = T10 Asymmetric Active-Active
Asymmetric logical unit access supported = Both implicit and explicit
Asymmetric states supported = ao_sup, an_sup
Preferred paths reported by device = Yes
Preferred LUN paths = 0

Driver esdisk Status Information :

Capacity in number of blocks = 209715200
Block size in bytes = 512
Number of active IOs = 0
Special properties =
Maximum number of IO retries = 45
IO transfer timeout in secs = 30
FORMAT command timeout in secs = 86400
START UNIT command timeout in secs = 60
Timeout in secs before starting failing IO = 120
IO infinite retries = false
[vega8087{root}:/home/jphui/temp]>scsimgr get_info -D /dev/rdisk/disk42

STATUS INFORMATION FOR LUN : /dev/rdisk/disk42

Generic Status Information

SCSI services internal state = UNOPEN
Device type = Direct_Access
EVPD page 0x83 description code = 1
EVPD page 0x83 description association = 0
EVPD page 0x83 description type = 3
World Wide Identifier (WWID) = 0x600805f300120050ac370fb176b40009
Serial number = "P56350GX3QZ01J"
Vendor id = "COMPAQ "
Product id = "MSA1000 VOLUME "
Product revision = "4.32"
Other properties = ""
SPC protocol revision = 4
Open count (includes chr/blk/pass-thru/class) = 0
Raw open count (includes class/pass-thru) = 0
Pass-thru opens = 0
LUN path count = 1
Active LUN paths = 1
Standby LUN paths = 0
Failed LUN paths = 0
Maximum I/O size allowed = 2097152
Preferred I/O size = 2097152
Outstanding I/Os = 0
I/O load balance policy = round_robin
Path fail threshold time period = 0
Transient time period = 120
Tracing buffer size = 1024
LUN Path used when policy is path_lockdown = NA
LUN access type = NA
Asymmetric logical unit access supported = No
Asymmetric states supported = NA
Preferred paths reported by device = No
Preferred LUN paths = 0

Driver esdisk Status Information :

Capacity in number of blocks = 204799680
Block size in bytes = 512
Number of active IOs = 0
Special properties =
Maximum number of IO retries = 45
IO transfer timeout in secs = 30
FORMAT command timeout in secs = 86400
START UNIT command timeout in secs = 60
Timeout in secs before starting failing IO = 120
IO infinite retries = false

Looks there are some differences except for the disk vendor related information:
(1) SPC protocol version
The value of /dev/rdisk/disk32 is:
SPC protocol revision = 2

The value of /dev/rdisk/disk42 is:
SPC protocol revision = 4
(2) LUN path
The values of /dev/rdisk/disk32 is:
LUN path count = 8
Active LUN paths = 4
Standby LUN paths = 4
The values of /dev/rdisk/disk42 is:
LUN path count = 1
Active LUN paths = 1
Standby LUN paths = 0

I'm not sure which difference affected our reservation process. Any comments here?

Would you please help to give some suggestions on the above two questions? Appreciate for
your helps in advance.

Re: Persistent Reservation issues

Duncan Edmonstone — Mon, 02 Mar 2009 08:16:28 GMT

I don't pretend to completelyt understand this stuff but, here goes...

Care to share what your C code is actually doing? From the notes you provided I'm *assuming* it is issuing an ioctl to the disk with the SIOC_SET_BUS_EXCL usage.

Did you read the notes in /usr/include/sys/scsi.h relating to the use of these SIOC_*_EXCL ioctls in 11.31?

Specifically that for target and bus exclusive requests on new agile DSFs will have changed behaviour, and will eventually be deprecated in favour of lun exclusive modes only.

Incidentally what happens if you issue the reservation request to an old legacy DSF instead of the new agile DSF? Do you get consistent behaviour then?

HTH

Duncan

Re: Persistent Reservation issues

Jian-ping Hui — Mon, 02 Mar 2009 08:49:17 GMT

Really thank you for the quick response, Duncan.

Yes, I've tried on the old legacy dsf with the same command. I can see the same error. Their behaviors are consistent.

In my C program, I do:
1. Open the device like /dev/rdisk/disk32.
2. Set the flag SIOC_SET_BUS_EXCL by the call
ioctl(fd, SIOC_EXCLUSIVE, &flag);
3. Register the node to the LUN by issuing "Persistent OUT" command with "REGISTER" by iotcl to the device.
...
ioctl(fd, SIOC_IO, &io)
4. Reserve the LUN by issuing "Persistent OUT" command with "RESERVE" by ioctl to the device.
...
ioctl(fd, SIOC_IO, &io)
5. Set the flag SIOC_REL_BUG_EXCL by the call
ioctl(fd, SIOC_EXCLUSIVE, &flag);
6. Close the device.

Thanks again.

Re: Persistent Reservation issues

Duncan Edmonstone — Mon, 02 Mar 2009 11:48:06 GMT

So as I said, did you see the following in scsi.h:

/*
* Definition of SIOC_EXCLUSIVE requests
* In 11.31 all these requests will be supported when issued on a legacy
* device file.
* In 11.31 and onwards all below exclusive requests will be supported
* but there will be change in behaviour for target and bus exclusive
* requests on new style dev_t. The will result in setting/unsetting
* of lun exclusive mode.
* Post 11.31 only setting/unsetting of lun exclusive mode will be supported */

Also remember that issuing the ioctl to the agile or legacy DSF could result in the ioctl going down any of the LUN paths. Associated with the device. To send the IO to a specific path you'd have to set leg_mpath_enable to 0 for that LUN. This is different to 11iv2, where unless you have some sort of MPIO like HP Secure Path or EMC PowerPath installed, the ioctl will always be issue down the path specified by the DSF. I think you need to carefully read the man page for sioc_io(7)

HTH

Duncan

Re: Persistent Reservation issues

Pradeep Dubey — Mon, 13 Apr 2009 07:52:42 GMT

On similar topic, I want to know, whether
HPUX uses the following obsolete Persistent Reservation types:

READ SHARED(0h), READ EXCLUSIVE(2h), SHARED ACCESS (4h)

topic Re: Persistent Reservation issues in General

Persistent Reservation issues

Re: Persistent Reservation issues

Re: Persistent Reservation issues

Re: Persistent Reservation issues

Re: Persistent Reservation issues