topic Re: Node cannot reach lock device /dev/dsk/c2t0d1 in Operating System - HP-UX

Node cannot reach lock device /dev/dsk/c2t0d1

Amir Haboosheh — Thu, 11 Sep 2008 21:38:36 GMT

i am getting this warning every few minutes or so on the smh gui in the serviceguard manager from node2.

the cluster works fine and moving the package to node2 that sends the alert works good.
so i am not sure what's the cause or the fix for this. any ideas?

i have hpux 11.31 with sep 2008 patch bundle
sg v.18

2x rx2660 connected to an msa30 using an msa1500 with msa1000 controllers (direct connection, no fc switch).

the device is the shared msa30 and node1 h/w path is c4t0d1 and node2 is c2t0d1. everything is the same so i don't know why they came out different (the first time i installed hpux 11.23 they were the same c2t0d1) but from what i read it doesn't matter as long as my cluster configuration is correct (which i think it is).

any help on this matter would be appreciated.
these are brand new equipment about 5 months old (took us a while to install ;)

thanks,
amir

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Patrick Wallek — Thu, 11 Sep 2008 21:56:36 GMT

The first thing I would try is an ioscan to make sure that the disk is actually still available to this node.

You could also try 'diskinfo /dev/rdsk/c2t0d1' and see what it returns.

The cluster lock disk will not come into play until one node fails. At that point, each node tries to access the lock disk. The one that succeeds becomes the new cluster.

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Amir Haboosheh — Thu, 11 Sep 2008 23:41:56 GMT

everything looks good with diskinfo and ioscan. and it's not that the error is constant it's just there for one min and gone the next. on/off/on/off...

i know the arbitration moment is the only time the lockdisk will matter but the last thing i need is for it not do find the lockdisk when the failure happens.

amir

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Patrick Wallek — Fri, 12 Sep 2008 01:23:30 GMT

If the error is intermittent, it sounds like you have an intermittent problem accessing the disk.

Is there any log on the MSA? Check /var/adm/syslog/syslog.log and see if there are any disk errors there.

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Amir Haboosheh — Fri, 12 Sep 2008 03:13:36 GMT

this is interesting this is what is at the end of the syslog and keeps repeating on and off:

Sep 11 20:25:02 opiunix2 cmdisklockd[3503]: Cluster lock disk /dev/dsk/c2t0d1 is
inaccessible

Sep 11 20:28:43 opiunix2 above message repeats 4 times

Sep 11 20:26:22 opiunix2 cmdisklockd[3503]: Cluster lock disk /dev/dsk/c2t0d1 is
accessible

Sep 11 20:28:43 opiunix2 above message repeats 4 times

Sep 11 20:29:17 opiunix2 cmdisklockd[3503]: Cluster lock disk /dev/dsk/c2t0d1 is
inaccessible

Sep 11 20:30:37 opiunix2 cmdisklockd[3503]: Cluster lock disk /dev/dsk/c2t0d1 is
accessible

Sep 11 20:47:42 opiunix2 cmdisklockd[3503]: Cluster lock disk /dev/dsk/c2t0d1 is
inaccessible

Sep 11 20:48:43 opiunix2 above message repeats 4 times

Sep 11 20:44:47 opiunix2 cmdisklockd[3503]: Cluster lock disk /dev/dsk/c2t0d1 is
accessible

Sep 11 20:48:43 opiunix2 above message repeats 3 times

Sep 11 20:50:27 opiunix2 cmdisklockd[3503]: Cluster lock disk /dev/dsk/c2t0d1 is
accessible

Sep 11 20:53:22 opiunix2 cmdisklockd[3503]: WARNING: Cluster lock disk /dev/dsk/
c2t0d1 has failed: I/O error. Until it is fixed, a single failure could cause al
l nodes in the cluster to crash.

Sep 11 20:53:22 opiunix2 cmdisklockd[3503]: Cluster lock disk /dev/dsk/c2t0d1 is
inaccessible

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Amir Haboosheh — Fri, 12 Sep 2008 03:23:48 GMT

when i search the error messages on the forums it implies a hardware issue. that doesn't make sense. everything has been working perfectly on 11.23 on fri i installed 11.31 using new drives. and since setting up sg i have been seeing the errors. but node fail over works fine.

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Torsten. — Fri, 12 Sep 2008 04:08:51 GMT

"2x rx2660 connected to an msa30 using an msa1500 with msa1000 controllers (direct connection, no fc switch)."

How did you manage this?

If you have 2 FC ports on the MSA1500, did you connect 1 FC port of the MSA to 1 Server?

BTW, make sure the configured profile for the LUNs is set to HP(-UX).

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Sandeep_Chaudhary — Fri, 12 Sep 2008 05:10:09 GMT

check for fiber connectivity.
If possibly change the lock disk.

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Amir Haboosheh — Fri, 12 Sep 2008 12:38:41 GMT

msa30 is scsi so 2 fc from each server and scsi port to msa30 (don't ask it's what i got to work with and that cannot be changed).
i confimed with hp and it is a supported configuratino. the one thing i had to do originally with 11.23 to make it work is update the firmware to active/active on the msa1500.

the msa1500 profile is set to hpux

connectivity it good.
how can i change th lock disk? doen's it have to be the msa30 which is the only shared drive between the the server?

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Torsten. — Fri, 12 Sep 2008 12:45:28 GMT

I think I understand now ...

You have (apart from the MSA1500) a MSA30MI shared directly between the servers, right?

Did you configure one of the SCSI controllers in 1 server to ID6?

If not, you have both controllers at ID7 - this is a conflict and would explain what you see.

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Torsten. — Fri, 12 Sep 2008 12:52:59 GMT

If your config is like this

Server_A <==> MSA30 <==> Server_B
Server_A <==> MSA30 <==> Server_B

you need to configure the HBA SCSI address of either Server_A **or** Server_B from 7 to 6.

Can you post an "ioscan -fn" of both servers?

You will see devices like

/dev/mpt_

You need to do

# mptconfig -i 6 /dev/mpt11

Confirm: Do you want to change scsi parameters (y/[n]): y

Scan For Devices to avoid id conflict ... [Passed]
Status .................................. [Successful]

for example.

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Amir Haboosheh — Fri, 12 Sep 2008 14:16:11 GMT

it more like this:

ServerA> MSA1500+1000> MSA30
the two msa1000 controllers and inside the msa1500 unit. the msa1500 has a scsi i/o module that connects to the msa30. the msa 1500 also has 2x fc modules that each server is connected to.

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Torsten. — Fri, 12 Sep 2008 14:41:04 GMT

So the MSA30 is only the extension to the MSA1500.

Do you have only a single FC cable between the MSA1500 and each server?

What firmware is running on the MSA?

When you was using 11.23 - was secure path installed?

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Amir Haboosheh — Fri, 12 Sep 2008 15:07:42 GMT

>Do you have only a single FC cable between the MSA1500 and each server?
yes

>What firmware is running on the MSA?
v.7 active/active

>When you was using 11.23 - was secure path installed?
no

Re: Node cannot reach lock device /dev/dsk/c2t0d1

Amir Haboosheh — Fri, 12 Sep 2008 15:14:45 GMT

i attached the ioscan -fn output from each server.