iscsi initator repeatedly tries to login to target

[Jason Long_2]

I've deleted a LUN on our iSCSI storage array, but I've got two HP-UX boxes that keep trying to access it, generating many warning notifications (one every 6 seconds) from our storage array. The warning notifications are like this:

iSCSI login to target 'iqn.2001-05.com.equallogic:blahblahblah' from initiator 'iqn.1986-03.com.hp:blahblahblah' failed with status 0x1203:
requested target was not found


The targets are still showing up on both boxes in `iscsiutil -p -O' and `ioscan -H 255'.

Is there any way to tell the iSCSI initiator to stop trying to login to that target, other than a reboot of the server? (which I think worked when this happened before, but a reboot isn't so convenient this time)

Jason Long
Messiah College

--

HP-UX 11.11 with iSCSI initiator 03d
HP-UX 11.23 with iSCSI initiator 03e
EqualLogic iSCSI storage array

[marie-noelle jeanson]

Jason,

iSCSI initiator may be trying to login again and again as there may be an active open from upper layer modules to the deleted targets.

Can you try to close all the lun opens to these deleted targets? Rebooting solves this problem because it closes all the active opens to these deleted targets. Hopefully the above would solve it too.

Marie.

[Jason Long_2]

Marie,

I don't understand "close all the lun opens to these targets"... How would I do that? Is there a way to check for an "active open"?

There was an LVM volume group on the LUN. Before deleting it on the array side, I umounted, and "exported" the volume group, so I don't think HP-UX has any reason to still be accessing it.

Jason

[marie-noelle jeanson]

Jason,

The statistics for the particular target can be obtained using the below:

iscsiutil /dev/iscsi -s -S -C

For example the below indicates that the session is open (as number of closes is less than number of opens so far taken place).

Number of session opens from upper layers 5
Number of session closes from upper layers 4

You may check what you see on your system and if you find any session open.

Marie.

[Jason Long_2]

Seems like you're on the right track. As you guessed, the number of upper layer Opens is greater than the number of upper layer Closes. (see below)

I see when I deactivate the LUN on the target the session state on the initiator changed to "Transient". I didn't really see any other changes, so I guess in some way the iSCSI layer is acting as if this LUN is still on the system, just giving I/O errors.

Is the "upper layer" here the SCSI layer? Is there a way to tell the SCSI layer the device has been removed?


Jason


----
Session # 7
------------
Session Instance No. : 12
ISID : 0x80000000000c
Session State : Transient
Target Name : iqn.2001-05.com.equallogic:6-8a0900-b3af20701-ad3ff0b614c45a3a-banlvm-g2
Target Alias :
No. of Target Addresses : 1

Address # 1
------------
IP Address : 153.42.101.6
iSCSI TCP Port : 3260
iSCSI Portal Group Tag : 1

User Configured:
----------------

Authenticaton Method :
CHAP Method : CHAP_UNI
Initiator CHAP Name :
CHAP Secret :
Header Digest : None,CRC32C (default)
Data Digest : None,CRC32C (default)


Session statistics
-------------------

Session instance # 12
---------------------
Number of session opens from upper layers 66
Number of session closes from upper layers 58
Number of session reopens 2
Number of session opens in the reopen context 52
Number of session opens aborted in the reopen context 0
Number of session closes aborted in the reopen context 0
Number of connection opens that failed 0
Number of successful login redirections 2
Number of async events received for dead connections 0
Number of destination hostname resolution failures 0
Number of destination address routing failures 0
Number of session logouts 2
Number of session state machine transitions to online state 2
Number of session state machine transitions to transient state 2
Number of session state machine transitions to offline state 2
Number of requests to enable session flow control 0
Number of requests to disable session flow control 9
Number of async events
"target will drop all connections" received 0
Number of SendTargets command send failures 0
Number of unexpected events received in session online state 0
Number of unexpected events received in session transient state 0
Number of unexpected events received in session offline state 0
Number of Task Management commands sent 0
Number of Task Management commands failed 0
Number of login failures due to not configuring CHAP secret 0
Number of times there was no connection associated with an
authentication response 0
Number of times the connection signature stamp did not match
with the authentication response 0
Number of times the target sent a "bad next stage" during
login negotiation 0
Number of login context allocation failures 0
Number of unsolicited NOP-INs sent by the target 111337

Connection # 0
---------------
Number of times the login failed 0
Number of exception status class values returned by target 55
Number of PDU headers with Protocol errors received by intiator 0
Number of times iswd daemon failed to open a connection 0
Number of failures to send a login command
due to kernel memory allocation failure 0
Number of asynchronous failures waiting for a login response 0
Number of asynchronous failures waiting for a logout response 0
Number of unexpected TCP closes in the active state 0
Number of timeouts on FIN after sending a logout command 0
Number of TCP connection open timeouts 0
Number of unexpected connection closes after a login command 0
Number of unexpected connection closes after a logout command 0
Number of target authentication timeouts 0
Number of target authentication failures 0
Number of temporary redirection requests 3
Number of permanent redirection requests 0

Protocol Data Unit (PDU) State Machine Statistics
--------------------------------------------------
Number of kernel memory allocation failures 0
Number of streams message allocation failures 0
Number of PDU transmission failures due to
an offline connection 0
Number of streams message duplication failures 0
Number of PDU exchange timeouts 0
Number of PDU exchanges aborted 0
Number of PDU exchanges abandoned 0

SCSI I/O State Machine Statistics
----------------------------------
Number of I/Os issued on this connection 4162
Number of I/O timeouts 0
Number of kernel memory allocation failures 0
Number of streams message allocation failures 0
Number of I/Os delayed while waiting for resources 0
Number of I/Os failed due to an offline connection 0
Number of I/Os failed due to memory resource constraints 0
Number of invalid Data-In PDUs received 0
Number of unexpected R2T PDUs received during a Read I/O 0
Number of unexpected Data-In PDUs received during a Write I/O 0
Number of Data-In PDUs with incorrect residual count 0
Number of SCSI Response PDUs with incorrect residual count 0
Number of I/O underruns 0
Number of I/O underflows 3169
Number of I/O overflows 0
Number of I/O failures due to response code errors 0
Number of Data-In PDUs received without data 0
Number of invalid R2T PDUs received 0
Number of I/Os that failed to respond to an R2T
due to kernel memory allocation failures 0
Number of I/O failures due to streams message
concatenation memory allocation failures 0
Number of holes seen in the status sequencing 0

Async event statistics
-----------------------
Number of SCSI Async events received 0
Number of "target requests logout" Async events received 2
Number of "target will drop connection" Async events received 0
Number of "target requests parameter negotiation" Async
events received 0
Number of "vendor specific" Async events received 0

Reject statistics
-----------------
Number of Reject PDUs due to Data Digest Errors 0
Number of Reject PDUs due to SNACK rejects 0
Number of Reject PDUs due to Protocol Errors 0
Number of Reject PDUs due to excessive Immediate Commands 0
Number of Reject PDUs due to "Task in Progress" 0
Number of Reject PDUs due to Invalid Data ACK 0
Number of Reject PDUs due to Invalid PDU field 0
Number of Reject PDUs due to Lack of Target Resources 0
Number of Reject PDUs due to Negotiation Resets 0
Number of Reject PDUs due to target Waiting for Logout 0
Number of Reject PDUs due to Miscellaneous reasons 0

[marie-noelle jeanson]

Jason,

The lun may have been opened by file system or by any other userspace application. Scsi subsystem does not open the lun by itself.

All lun opens are user generated ones. You need to unmount the filesystem if any is mounted on the affected lun. Or, need to close any application/script if any has opened the lun.

Looks like it is not easy to figure out who has opened it though...

Marie

[Jason Long_2]

As I said, I had umounted the filesystem, and exported the LVM volume group, prior to deactiving the LUN.

If there was an application still using the disk, I would not have been able to umount it.

I came across rmsf(1) which looks like it could remove the disks from the hardware tree, but since "most drivers do not support the ability to be removed from the system" and the iSCSI documentation did not mention rmsf (that I can remember), I am hesitant to try it on a live system.