disk array, lvm, POWERFAILED

[Marcel Preda]

Hi,

I have a `little' problem.
More than a week ago (2005.01.15), we have change the location of a feew HP servers + a CLARIION disk array.

The disk array is mounted on a L1000 class machine (hostname=l1000db2) on two LV (vg02 && vg03).



Yesterday, a proplem has ocurred, see below in syslog file.

I have no idea if it is just a coincidence,
but a feew minutes before the firts error message someone has run a script from HP , check.dat (hardware inspection).
and there was a command "lvlnboot -v",
you can see it in the syslog file
" Jan 24 16:32:48 l1000db2 LVM[8338]: lvlnboot -v"


Next, the syslog:
################




Jan 19 16:46:40 l1000db2 pwgrd: Invalid entry found in passwd file /etc/passwd
Jan 19 16:49:59 l1000db2 : su : + ta root-vscsftp1
Jan 19 16:54:46 l1000db2 : su : + ta root-oracle
Jan 19 16:57:49 l1000db2 : su : + ta root-oracle
Jan 24 16:28:12 l1000db2 ftpd[7725]: FTP LOGIN FROM 172.16.10.31 [172.16.10.31], root
Jan 24 16:32:48 l1000db2 LVM[8338]: lvlnboot -v
Jan 24 16:37:15 l1000db2 vmunix: LVM: Recovered Path (device 0x1f060700) to PV 3 in VG 2.
Jan 24 16:37:15 l1000db2 vmunix: LVM: Restored PV 3 to VG 2.
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f071400) to PV 3 in VG 3 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070500) to PV 2 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070300) to PV 1 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070700) to PV 3 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: vg[2]: pvnum=1 (dev_t=0x1f060300) is POWERFAILED
Jan 24 16:37:44 l1000db2 vmunix: LVM: vg[2]: pvnum=2 (dev_t=0x1f060500) is POWERFAILED
Jan 24 16:37:44 l1000db2 vmunix: LVM: vg[2]: pvnum=3 (dev_t=0x1f060700) is POWERFAILED
Jan 24 16:37:47 l1000db2 vmunix: LVM: vg[3]: pvnum=3 (dev_t=0x1f061400) is POWERFAILED

############

I have no idea how to solve this problem now.
Should I try a restart of the server ?!


Thanks,

Marcel

[Ravi_8]

Hi,

lvlnboot -v just shows the boot disk, no harm in using this command.

LOg shows one of the disk in VG2 is failed,
#ioscan -fnC disk
check all the disks are claimed or not.
propably you may have to replace the failed disk

[baiju_3]

Hi Marcel,

If you have powerpath installed then do,

Powermt config
Powermt check

again verify dead devices with powermt display.

You may consider a reboot , if you are not able to clear dead paths.

EMC has recommendations for settings on HP Unix and Clariion Arrays .Refer EMC Web .

You have to change the PV time out to 120 or more ,if PV time out is default.

Check with "pvdisplay -v /dev/rmt/c#t#d#"

Set PV time out value by,

pvchange -t 120 /dev/rmt/c#t#d#

Regards,
Baiju.

[Sundar_7]

IF you have alternate paths to the LUNS then you dont have to restart the server.

# vgdisplay -v /dev/vg02

# vgdisplay -v /dev/vg03

Do you see "Alternate Link" for the PVs ?

[Florian Heigl (new acc) · Accepted solution]

The HP script either triggered the error or made the os aware of it, You have to find out which is the case.

Check the Clariions logs for 'array shutdown for trespass' messages.

/opt/Navisphere/bin/navicli -h getlog -2000

If there are more than 10-20 of these errors, this is suspicious.
Are You running in a quick loop configuration?

If You don't know, check it using /opt/fcms/bin/fcmsutil /dev/td | grep Topology

If it is a quickloop (plain FC-AL):
There are conditions where HP-UX overloads the Clariions Storage Processor to a point I better don't talk about.

Check Your kernel parameter max_fcp_reqs and compare it to emc's documetation.
a rule of thumb: max_fcp_reqs=(512/2/systems_in_loop)
Also, check the scsi queue depth against what emc wants.

It seems like You don't have Navisphere Agent installed - You should.

(We had a lot of problems in our SAN until I stopped begging for permission and simply set that parameter to what it needed to be :)

[Marcel Preda]

Hi,
sorry for my late response, other problems come too :-(


About "ioscan -fnC disk"
all disks are CLAIMED, like:
....
disk 28 0/7/0/0.8.0.1.0.0.1 sdisk CLAIMED DEVICE DGC C4400WDR1
/dev/dsk/c7t0d1 /dev/rdsk/c7t0d1
....




About powermt command , no such command
About "pvdisplay -v /dev/rmt/c#t#d#".

Probable I dont have such devices.
ls -l /dev/rmt/c*
crw-rw-rw- 2 bin bin 205 0x030000 Mar 16 2001 /dev/rmt/c3t0d0BEST
crw-rw-rw- 2 bin bin 205 0x030080 Mar 16 2001 /dev/rmt/c3t0d0BESTb
crw-rw-rw- 2 bin bin 205 0x030040 Apr 13 2001 /dev/rmt/c3t0d0BESTn
crw-rw-rw- 2 bin bin 205 0x0300c0 Mar 16 2001 /dev/rmt/c3t0d0BESTnb
crw-rw-rw- 1 bin bin 205 0x030001 Mar 16 2001 /dev/rmt/c3t0d0DDS
crw-rw-rw- 1 bin bin 205 0x030081 Mar 16 2001 /dev/rmt/c3t0d0DDSb
crw-rw-rw- 1 bin bin 205 0x030041 Mar 16 2001 /dev/rmt/c3t0d0DDSn
crw-rw-rw- 1 bin bin 205 0x0300c1 Mar 16 2001 /dev/rmt/c3t0d0DDSnb

Looks to be just magnetic tapes.



About vgdisplay:

All disks have "Alternate Link"

vgdisplay -v /dev/vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 13915
VGDA 8
PE Size (Mbytes) 4
Total PE 55652
Alloc PE 55652
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol1
LV Status available/syncd
LV Size (Mbytes) 222608
Current LE 55652
Allocated PE 55652
Used PV 4


--- Physical volumes ---
PV Name /dev/dsk/c6t0d1
PV Name /dev/dsk/c7t0d1 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d3
PV Name /dev/dsk/c7t0d3 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d5
PV Name /dev/dsk/c7t0d5 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d7
PV Name /dev/dsk/c7t0d7 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On


On #vgdisplay -v /dev/vg03
is a similar situation, alal disks have "Alternate Link"



About the topology:
l1000db2 # /opt/fcms/bin/fcmsutil /dev/td0 | grep Topology
Topology = PRIVATE_LOOP
l1000db2 # /opt/fcms/bin/fcmsutil /dev/td1 | grep Topology
Topology = PRIVATE_LOOP

I was traing:

l1000db2 # /opt/Navisphere/bin/navicli getlog
ABORT instruction (core dumped)

The clariion is connected to the L1000 server via optical fibre, so I dindn't know what IP to user as "-h host" option :-(


BTW: I'm not an expert in HP-UX, in any case not disks-array and Navisphere.
Also I'm about 2000 kms distance by the servers.


Thanks for your time,
I'll notice you about what will happen.


Marcel

[Florian Heigl (new acc)]

if everything is CLAIMED and PV Status is "available" too, then I'd say there is no need to reboot.

The Clariion has two service processors which are usually connected to the site lan, but if noone can tell You that won't do any good.

Either have someone get You that ip address or hostname, or search through the disks - there is another parameter (documented in a file called agent.config that is *somewhere*) to use a scsi-device for communication.
this definitely works if You see any LUNs on 0/7/0 with a capacity of 0 or 8 MB.

just use /opt/Navisphere/bin/navicli and hit enter for a list of commands.

I kind of think You're having the same problem we had, but I can't tell for sure.

First find a way to reproduce the problem at a low-risk time, then - if You wish - try changing the kernel parameter (ask either HP or EMC^2 on it, they both have the relevant EMC^2 document on what to change to make HP-UX a 'supported' client in loop configurations) and see if the problem persists. (I'd bet it doesn't)

[Florian Heigl (new acc)]

The Navisphere cli Manual is called 069001038-1 at EMC^2, and I'll copy a bit but You might want to search for a pdf...:

navicli [-d device ] [-h hostname] {-np} [-m] [-p] [-t timeout] -[v|q] command

(-m suppress output, -np don't poll the box, which sometimes may help performance of command, -p appears to be preview mode, -q suppresses *error* messages -t sets the timeout for the command, from my experience this is passed to the box, not to the cli program -v extensive error descriptions)

so, this could be
navicli -d /dev/dsk/c7t0d0 getlog

I have never tried using a *data* lun for this, and I'm not sure if it'll work. If this goes somewhere through the scsi mode pages it might, but maybe it just breaks everything :))

So better find someone that knows the ip, afterwards search for the EMC^2 config file agent.config and edit it to allow access for root@localhost and on how to access the clariion.

afterwards /sbin/init.d/agent stop and /sbin/init.d/agent start and from that moment on You'll have a lot of output on every single soft scsi error and whatever the clariion notices right into Your syslog.

:)

[Nguyen Anh Tien]

Hi Marcel.
>>>Jan 24 16:32:48 l1000db2 LVM[8338]: lvlnboot is OK. it is no harm for your server.
But
=========
Jan 24 16:37:15 l1000db2 vmunix: LVM: Recovered Path (device 0x1f060700) to PV 3 in VG 2.
Jan 24 16:37:15 l1000db2 vmunix: LVM: Restored PV 3 to VG 2.
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f071400) to PV 3 in VG 3 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070500) to PV 2 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070300) to PV 1 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070700) to PV 3 in VG 2 Failed!
===
are problem.
Now issue these command to diagnostic
#vgdisplay -v vg02
#vgdisplay -v vg02
Pls post result for me/
tienna

[Marcel Preda]

Hi Florian,

I think that it's a problem with Navisphere too.

l1000db2 # /opt/Navisphere/bin/navicli
ABORT instruction (core dumped)


l1000db2 # /sbin/init.d/agent stop
Navisphere Agent is not running
l1000db2 # /sbin/init.d/agent start
Starting Navisphere Agent
/sbin/init.d/agent[37]: 26412 Abort(coredump)


Marcel

[Marcel Preda]

l1000db2 # vgdisplay -v vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 13915
VGDA 8
PE Size (Mbytes) 4
Total PE 55652
Alloc PE 55652
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
Hi Nguyen,


the oputput of the commands:
l1000db2 # vgdisplay -v vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 13915
VGDA 8
PE Size (Mbytes) 4
Total PE 55652
Alloc PE 55652
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0


--- Logical volumes ---
LV Name /dev/vg02/lvol1
LV Status available/syncd
LV Size (Mbytes) 222608
Current LE 55652
Allocated PE 55652
Used PV 4


--- Physical volumes ---
PV Name /dev/dsk/c6t0d1
PV Name /dev/dsk/c7t0d1 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d3
PV Name /dev/dsk/c7t0d3 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d5
PV Name /dev/dsk/c7t0d5 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d7
PV Name /dev/dsk/c7t0d7 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On



____________________________________


l1000db2 # vgdisplay -v vg03
--- Volume groups ---
VG Name /dev/vg03
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 12
Act PV 12
Max PE per PV 26592
VGDA 24
PE Size (Mbytes) 32
Total PE 33964
Alloc PE 33964
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg03/lvol1
LV Status available/syncd
LV Size (Mbytes) 1086848
Current LE 33964
Allocated PE 33964
Used PV 12


--- Physical volumes ---
PV Name /dev/dsk/c6t1d1
PV Name /dev/dsk/c7t1d1 Alternate Link
PV Status available
Total PE 2399
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d2
PV Name /dev/dsk/c7t1d2 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d3
PV Name /dev/dsk/c7t1d3 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d4
PV Name /dev/dsk/c7t1d4 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d5
PV Name /dev/dsk/c7t1d5 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d6
PV Name /dev/dsk/c7t1d6 Alternate Link
PV Status available
Total PE 3323
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d7
PV Name /dev/dsk/c7t1d7 Alternate Link
PV Status available
Total PE 2399
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d0
PV Name /dev/dsk/c7t2d0 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d1
PV Name /dev/dsk/c7t2d1 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d2
PV Name /dev/dsk/c7t2d2 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d3
PV Name /dev/dsk/c7t2d3 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d4
PV Name /dev/dsk/c7t2d4 Alternate Link
PV Status available
Total PE 3323
Free PE 0
Autoswitch On

[Roxana_4]

OFF TOPIC.
Buna ziua stimate domn Marcel.
Ma numesc Roxana Colta si sunt din Brasov.
Cu scuzele de rigoare, imi permit sa va abordez cu un subiect diferit de problema dumneavoastra.
Deoarece sunteti, deocamdata, singurul roman cu experienta in servere HP-UX de pe acest forum, imi permit sa va solicit in mod expres parerea in urmatoarea problema: am niste DTC-uri mai vechi (modelele HP 2345A si HP 2340A) pe care le folosesc pentru a conecta niste terminale la un server HP e3000 cu MPE/iX 7.0 ; am achizitionat un server Inegrity rx2600 cu HP-UX 11iv2 pe care urmeaza sa il instalez si configurez ; pot sa folosesc DTC-urile vechi pe acest server, HP-UX 11i le recunoaste la configurare ?

Putem gasi o alta cale de a comunica in mod direct ? Adresa mea este roxana@engineer.com . Va multumesc anticipat si imi cer scuze pentru indrazneala de a va aborda pe acest forum.

Roxana

[Mohanasundaram_1]

Hi Marcel,

I don't know what Roxana said before this. But I can see it is not a single disk (LUN)failure. If you go through the syslog.log message, you can see that PV1, PV2 and PV3 of VG2 have issued this message.

If more than one LUN is causing such messages I would suspect something is wrong with the controller on the Clarion or HBA on the system or anything between them.

If you are experiencing performance problem due to this message, then it may be an intermittent problem. That is, that path may become available for sometime and then go off-line and again become available. This may go on until the problem is fixed. There may be a considerable performance degradation due to this.

Let us know if this is the case. You can also provide the syslog.log in its entirety.

With regards,
Mohan.

[Florian Heigl (new acc)]

marcel - if the navicli is broken, then find the person responsible or the service contract, have an EMC technician come to the site and check the event logs.

[Marcel Preda]

Hi again,

I'll try to contact the person who can contact the tech support, the maintenance contract is no more available.


In a feew minutes I'll put the syslog file.


Thanks for your time.

Marcel

[Marcel Preda]

The entirely syslog file.

If some of you have some time to read it...

Thanks,

Marcel

[Mohanasundaram_1]

Hi Marcel,

I can see the powerfailed message and recovered message for PVs in VG2 and VG3. So the problem must be as I was guessing. one of the path is going on and off frequently. Trace the entire path and isolate the problem.

Contrary to what you have indicated in the first post, vg3 seems to be the most battered one due to this problem. Your problem lies in one of the components - HBA, cable, switch, Clariion controller.

With regards,
Mohan.

[Marcel Preda]

Hi again,

thanks for the reply Mohan, .

A good news,
on other server(which was down till today) I have found a Navispher which seems works (no core dump).
But I still cant find the IP of the clariion array, if it exist.

Maybe some of you could help me to find it in a config file.
I cant figure which of the file could be:



l1000db1 # pwd
/etc/Navisphere
l1000db1 # ls -ltr
total 4
drwxr-xr-x 2 root sys 96 Mar 27 2001 logs
drwxr-xr-x 2 root sys 96 Mar 27 2001 backup
lrwxr-xr-x 1 root sys 32 Mar 27 2001 agent.config -> /opt/Navisphere/bin/agent.config
lrwxr-xr-x 1 root sys 30 Mar 27 2001 clsendtrap -> /opt/Navisphere/bin/clsendtrap
drwxrwxrwx 2 root sys 96 Mar 27 2001 log
lrwxr-xr-x 1 root sys 31 Mar 30 2001 Navimon.cfg -> /opt/Navisphere/bin/Navimon.cfg
drwxr-xr-x 2 root sys 2048 Jan 26 15:47 messages

Thanks,

Marcel

[Marcel Preda]

Hi,

looking in /etc/Navisphere/agent.config
I have found 2 devices: c6t0d0 & c7t0d2

ussing those:

l1000db1 # navicli -d c6t0d0 getlog -2000 | grep -i trespass
01/26/2005 16:42:04 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba9220 14318b0
01/26/2005 16:42:06 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba89cc 14318b0
01/26/2005 16:42:08 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba89cc 14318b0
01/26/2005 16:42:11 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba89cc 14318b0
01/26/2005 16:42:13 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba89cc 14318b0
01/26/2005 16:42:15 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba8408 14318b0
01/26/2005 16:42:17 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba8408 14318b0
l1000db1 # navicli -d c7t0d2 getlog -2000 | grep -i trespass
01/26/2005 16:42:25 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0b] baa4b4 14318b0
01/26/2005 16:42:26 Bus 0 Enclosure 0 Disk 0(606) Unit Shutdown for Trespass [0x01] ba82c0 14318b0
01/26/2005 16:42:28 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x11] 184db0 14318a8
01/26/2005 16:42:28 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x09] 184db0 14318a8
01/26/2005 16:42:29 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] 184db0 14318a8
01/26/2005 16:42:30 Bus 0 Enclosure 0 Disk 2(606) Unit Shutdown for Trespass [0x03] 184db0 14318a8
01/26/2005 16:42:30 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0a] 184db0 14318a8
01/26/2005 16:42:30 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x10] 184db0 14318a8
01/26/2005 16:42:32 Bus 0 Enclosure 0 Disk 4(606) Unit Shutdown for Trespass [0x05] 184db0 14318a8
01/26/2005 16:42:34 Bus 0 Enclosure 0 Disk 6(606) Unit Shutdown for Trespass [0x07] ba97e4 14318b0
01/26/2005 16:42:34 Bus 0 Enclosure 0 Disk 4(606) Unit Shutdown for Trespass [0x05] 184db0 14318a8
01/26/2005 16:42:34 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0c] ba84ac 14318b0
01/26/2005 16:42:35 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0e] ba9d04 14318b0
01/26/2005 16:42:35 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x0f] 184db0 14318a8
01/26/2005 16:42:35 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x12] baa038 14318b0
01/26/2005 16:42:37 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x13] baa224 14318b0
01/26/2005 16:42:37 Bus 0 Enclosure 0 Disk 0(606) Unit Shutdown for Trespass [0x01] 184db0 14318a8
01/26/2005 16:42:37 Bus 0 Enclosure 0 Disk 2(606) Unit Shutdown for Trespass [0x03] 184db0 14318a8
01/26/2005 16:42:37 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x14] 826d30 ba8884
01/26/2005 16:42:37 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x09] 826d30 ba8884
01/26/2005 16:42:38 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x10] 826d30 ba8884


what the hell could be the problem?
unfortunately I can not verify the cables, hardvare in general, I'm out of site.


Thanks, Marcel

[Florian Heigl (new acc)]

our agent config reads like this:
# SAMPLES:
# device c0t0d0 NAVISPHERE-1 "NAVISPHERE-1"

# automatically detect manageable devices
device auto auto

maybe that's enough, but I'm not very convinced:

root@xxxxx:/opt/Navisphere/bin>./navicli getlog -100
Error: getlog command failed
Cannot access device



In the meantime You should use fcmsutil to check for adapter/cabling errors...

/opt/fcms/bin/fcmsutil /dev/tdN stat

root@xxxxx:/opt/Navisphere/bin>/opt/fcms/bin/fcmsutil /dev/td2 stat
Wed Jan 26 16:53:47 2005
Channel Statistics

Statistics From Link Status Registers ...
Loss of signal 22 Bad Rx Char 2379
Loss of Sync 46 Link Fail 41
Received EOFa 0 Discarded Frame 0
Bad CRC 0 Protocol Error 0

Do a reset on the counters and see, if Loss of signal, Bad RX or similar things increase, if yes there might be a hardware issue

These are also interesting:
Storm Statistics ...
Elastic Store Error Storm 0
Link Fail storm . 0
LIP(f8, xx) storm . 0
Loss Of Signal Storm 0
Out Of Sync Storm 0
Link Fault Storm 0
NOS_OLS Storm 1

[Marcel Preda]

Thanks for your reply,

I got this, I have to reset the counters , after.

l1000db1 # /opt/fcms/bin/fcmsutil /dev/td0 stat
Wed Jan 26 16:57:13 2005
Channel Statistics

Statistics From Link Status Registers ...
Loss of signal 0 Bad Rx Char 5
Loss of Sync 0 Link Fail 0
Received EOFa 0 Discarded Frame 0
Bad CRC 0 Protocol Error 0

Channel State M/C Statistics ...
TL Errata D40 0
Unexpected OCM for FCP_DATA phase 0
LSM Link Fail State 0
No. of times LUP timer popped 0
HBA Changing AL_PA Across LIP 0
Channel Reset Request 0
Channel Offline Request 0
CE_FCP_FREEZE Request 4
CE_FCP_UNFREEZE Request 4
CE_FCP_FREEZE Cancel 0
CE_LDN Event 16
CE_LUP Event 17
ERQ frzn CMs, after TL is resumed 0
FCP frzn CMs, after TL is resumed 0
LDN EICM, when ERQ/FCP not frzn 0
FTO EICM, when ERQ/FCP not frzn 0
CE_FCP_FROZEN Event 0
Number Of Delayed Freeze Operations 0
CE_FCP_UNFREEZE recvd in CS_FCP_FROZEN 545
Error Idle Interrupts 558
ERQ/FCP Assists Resumed 561
Loop State T/O, when LSM is not ONLINE 0
Loop State T/O, when LSM is ONLINE 0
Loop State T/O, when LSM is in ARB 0
Credit Errors 0
LIPs due to OLS 0

FLOGI Statistics ...
FLOGIs sent 20
FLOGI Resps recvd 0
LS_RJTs for FLOGI 0
Short FLOGI Resps 0
Version_Hi Error in FLOGI Resps 0
Version_Lo Error in FLOGI Resps 0
No class 3 support in FLOGI Resps 0
Invalid datasize/resvd2/fport_nport in FLOGI Resps 0
Invalid RA_TOV in FLOGI Resps 0
Invalid ED_TOV in FLOGI Resps 0
FLOGI Retries skipped due to Parse Errors 16
FTO OCMs for FLOGIs sent 20
FLOGIs timedout 0
FLOGI retries > High Water Mark 17

Fabric Statistics ...
FANs received 0
FANs timedout 0
FANs recvd in CS_FLOGI_SENT 0
Unexpected FANs received 0
ns_open failures 0
Changes detected in own nport_id 0
Changes detected in Fabric PWWN 0
Changes detected in Fabric NWWN 0
Channel in TD_FABRIC topology 0
Channel in TD_PUBLIC_LOOP topology 0
Channel in TD_PRIVATE_LOOP topology 17
Topology mode toggles during initialization 0
Topology changes across LDN/LUP 0
Successful initializations in P2P mode 0
Successful initializations in Loop mode 17

Storm Statistics ...
Elastic Store Error Storm 0
Link Fail storm . 0
LIP(f8, xx) storm . 0
Loss Of Signal Storm 0
Out Of Sync Storm 0
Link Fault Storm 0
NOS_OLS Storm 0

PCI Error Interrupt Statistics ...
MPE Interrupts 0
CRS Interrupts 0
DER Interrupts 0
PER Interrupts 0

Target State M/C Statistics ...
Target Thread Alloc. Fail 0
Frames with length < FC-PH Hdr length 0
Frames with incorrect ROUTING FC-PH Hdr 0
Frames with invalid CS_CTL FC-PH Hdr 0
Frames with invalid TYPE FC-PH Hdr 0
Frames with invalid DF_CTL FC-PH Hdr 0
Invalid SOF/EOF Received 0
Multi-Frame Sequences Received 0
SFS Received With S_ID/D_ID Mismatch 0
SFS Received exceeding TD_SFTBFSZ 0
Inbound non-Class3 Frames 0
Inbound Frame large than TD_SFTBFSZ 0
Kmalloc Failures 0
Out of order Multi-Frame sequences 0

I/O Error Statistics ...
FCP Frames Recvd As Responder 0
Bad Unassisted FCP frame 0
Bad S_ID in Unassisted FCP frame 0
No cdb for I/O 0
IRB entries on ERQ 0
SCSI Linked list entries 0
Late ABTS Responses Received 0
Too many mapping entries for a single I/O 0

SF state m/c statistics...
Rcvd ABTS dropped due to lack of SFT 0
No reply before timeout 3
SF failed due to LDN 0
SF failed due to ASN, LKF or FTO 542
SF thread Allocation failures 0
Unable to map SF payload 0
SFT in ERQ during Chip Reset 0
Late ICM from Non-compliant target 0



l1000db1 # /opt/fcms/bin/fcmsutil /dev/td1 stat
Wed Jan 26 16:59:08 2005
Channel Statistics

Statistics From Link Status Registers ...
Loss of signal 0 Bad Rx Char 5
Loss of Sync 0 Link Fail 0
Received EOFa 0 Discarded Frame 0
Bad CRC 0 Protocol Error 0

Channel State M/C Statistics ...
TL Errata D40 0
Unexpected OCM for FCP_DATA phase 0
LSM Link Fail State 0
No. of times LUP timer popped 0
HBA Changing AL_PA Across LIP 0
Channel Reset Request 0
Channel Offline Request 0
CE_FCP_FREEZE Request 0
CE_FCP_UNFREEZE Request 0
CE_FCP_FREEZE Cancel 0
CE_LDN Event 1
CE_LUP Event 2
ERQ frzn CMs, after TL is resumed 0
FCP frzn CMs, after TL is resumed 0
LDN EICM, when ERQ/FCP not frzn 0
FTO EICM, when ERQ/FCP not frzn 0
CE_FCP_FROZEN Event 0
Number Of Delayed Freeze Operations 0
CE_FCP_UNFREEZE recvd in CS_FCP_FROZEN 527
Error Idle Interrupts 528
ERQ/FCP Assists Resumed 528
Loop State T/O, when LSM is not ONLINE 0
Loop State T/O, when LSM is ONLINE 0
Loop State T/O, when LSM is in ARB 0
Credit Errors 0
LIPs due to OLS 0

FLOGI Statistics ...
FLOGIs sent 5
FLOGI Resps recvd 0
LS_RJTs for FLOGI 0
Short FLOGI Resps 0
Version_Hi Error in FLOGI Resps 0
Version_Lo Error in FLOGI Resps 0
No class 3 support in FLOGI Resps 0
Invalid datasize/resvd2/fport_nport in FLOGI Resps 0
Invalid RA_TOV in FLOGI Resps 0
Invalid ED_TOV in FLOGI Resps 0
FLOGI Retries skipped due to Parse Errors 1
FTO OCMs for FLOGIs sent 5
FLOGIs timedout 0
FLOGI retries > High Water Mark 2

Fabric Statistics ...
FANs received 0
FANs timedout 0
FANs recvd in CS_FLOGI_SENT 0
Unexpected FANs received 0
ns_open failures 0
Changes detected in own nport_id 0
Changes detected in Fabric PWWN 0
Changes detected in Fabric NWWN 0
Channel in TD_FABRIC topology 0
Channel in TD_PUBLIC_LOOP topology 0
Channel in TD_PRIVATE_LOOP topology 2
Topology mode toggles during initialization 0
Topology changes across LDN/LUP 0
Successful initializations in P2P mode 0
Successful initializations in Loop mode 2

Storm Statistics ...
Elastic Store Error Storm 0
Link Fail storm . 0
LIP(f8, xx) storm . 0
Loss Of Signal Storm 0
Out Of Sync Storm 0
Link Fault Storm 0
NOS_OLS Storm 0

PCI Error Interrupt Statistics ...
MPE Interrupts 0
CRS Interrupts 0
DER Interrupts 0
PER Interrupts 0

Target State M/C Statistics ...
Target Thread Alloc. Fail 0
Frames with length < FC-PH Hdr length 0
Frames with incorrect ROUTING FC-PH Hdr 0
Frames with invalid CS_CTL FC-PH Hdr 0
Frames with invalid TYPE FC-PH Hdr 0
Frames with invalid DF_CTL FC-PH Hdr 0
Invalid SOF/EOF Received 0
Multi-Frame Sequences Received 0
SFS Received With S_ID/D_ID Mismatch 0
SFS Received exceeding TD_SFTBFSZ 0
Inbound non-Class3 Frames 0
Inbound Frame large than TD_SFTBFSZ 0
Kmalloc Failures 0
Out of order Multi-Frame sequences 0

I/O Error Statistics ...
FCP Frames Recvd As Responder 0
Bad Unassisted FCP frame 0
Bad S_ID in Unassisted FCP frame 0
No cdb for I/O 0
IRB entries on ERQ 0
SCSI Linked list entries 0
Late ABTS Responses Received 0
Too many mapping entries for a single I/O 0

SF state m/c statistics...
Rcvd ABTS dropped due to lack of SFT 0
No reply before timeout 0
SF failed due to LDN 0
SF failed due to ASN, LKF or FTO 527
SF thread Allocation failures 0
Unable to map SF payload 0
SFT in ERQ during Chip Reset 0
Late ICM from Non-compliant target 0

PM

[Florian Heigl (new acc)]

Hah!

Seems I'm guessing right :)

If there are no hardware issues to be seen
with fcmsutil follow the following.
Otherwise get rid of the hardware problems and consider following it afterwards.

In short:
change the kernel paramter
â â max_fcp_reqs 512 512 Static N/A M â â
to something according the formula I wrote above.

To explain the issue:

when hp-ux hit's a timeout for a single device for any reason, it will issue a full ioscan and check availability of all devices the kernel knows.
Now, the Clariion is kind of an active-passive array, even if a single device has an alternate path only one of these paths is active at a time.
now when the alternate path is checked by hp-ux the Clariion needs to switch the lun over to the other SP. This triggers the message 'CRU / UNIT shutdown for trespass'.
In a quickloop this is a bad situation as the trespass will also change all other hosts to switch until (ad nauseam) there's so much switching that simple commands hitting many LUNs (lvlnboot, vgcfgbackup) will overload the storage processor.

Specific to hp-ux is that a HP box does allow a maximum of 512 requests / s / path whitout asking the Clariion how much it can take.
FC4500: 256reqs / SP
FC4700: 512reqs / SP
This number must (I think) even be divided by two as soon as the trespass occurs.
I even took it in half as it count's per path.
So, if You have three hp-ux hosts connected to it:
512/2/3/2=42
42 is a good number, isn't it?

In the end just set it lower than
There is no noticeable performance degradation and You follow emc^2 r

[Florian Heigl (new acc)]

5 RX Chars is fine, I'd say, and I don't see anything too troublesome elsewhere.

[Marcel Preda]

Thanks for you time and for reply,

one more thing, those statistics ar from other server (I have said it a feew posts above) conected to the same clariion .

To explain: first(2 weeks ago) were two HP servers in a cluster ( I have relized that is a cluster whet I shutdown the servers :-) ).

We have changed the location, 25 kilometers.
After that the main node, didn't want to restart - complains in GSP console about a defective DIMM (memeory).
The secondary node didn't want to start as cluster.
SO I have startet it as standalone.

Today has come a person from HP, he cleans (with a vacuum cleaner) the broken server, and now it is on line.
So the statistics are from this one.

In any case, tomorrow at first hour I have to remount the clarrion to main node, and rebuild the cluster.
I cant do it now, because those servers are on production.

Thanks again for all your help.
If the problem persist I'll try to change the kernel params.

What I want to say to you is that the clariion is online for almost four years .
The same HP servers.


Marcel

[Marcel Preda]

Sorry for the first stats posted, as I told you was other server.

Now the fcmsutil statistics from the server which `owns' now the clariion:

l1000db2 # /opt/fcms/bin/fcmsutil /dev/td0 stat
Wed Jan 26 18:25:53 2005
Channel Statistics

Statistics From Link Status Registers ...
Loss of signal 0 Bad Rx Char 33
Loss of Sync 0 Link Fail 0
Received EOFa 0 Discarded Frame 0
Bad CRC 0 Protocol Error 0

Channel State M/C Statistics ...
TL Errata D40 0
Unexpected OCM for FCP_DATA phase 1
LSM Link Fail State 0
No. of times LUP timer popped 0
HBA Changing AL_PA Across LIP 0
Channel Reset Request 0
Channel Offline Request 0
CE_FCP_FREEZE Request 1020
CE_FCP_UNFREEZE Request 1020
CE_FCP_FREEZE Cancel 0
CE_LDN Event 1261
CE_LUP Event 1262
ERQ frzn CMs, after TL is resumed 0
FCP frzn CMs, after TL is resumed 0
LDN EICM, when ERQ/FCP not frzn 0
FTO EICM, when ERQ/FCP not frzn 0
CE_FCP_FROZEN Event 0
Number Of Delayed Freeze Operations 0
CE_FCP_UNFREEZE recvd in CS_FCP_FROZEN 3784
Error Idle Interrupts 4025
ERQ/FCP Assists Resumed 5045
Loop State T/O, when LSM is not ONLINE 0
Loop State T/O, when LSM is ONLINE 0
Loop State T/O, when LSM is in ARB 0
Credit Errors 0
LIPs due to OLS 1

FLOGI Statistics ...
FLOGIs sent 1265
FLOGI Resps recvd 0
LS_RJTs for FLOGI 0
Short FLOGI Resps 0
Version_Hi Error in FLOGI Resps 0
Version_Lo Error in FLOGI Resps 0
No class 3 support in FLOGI Resps 0
Invalid datasize/resvd2/fport_nport in FLOGI Resps 0
Invalid RA_TOV in FLOGI Resps 0
Invalid ED_TOV in FLOGI Resps 0
FLOGI Retries skipped due to Parse Errors 1261
FTO OCMs for FLOGIs sent 1265
FLOGIs timedout 0
FLOGI retries > High Water Mark 1262

Fabric Statistics ...
FANs received 0
FANs timedout 0
FANs recvd in CS_FLOGI_SENT 0
Unexpected FANs received 0
ns_open failures 0
Changes detected in own nport_id 0
Changes detected in Fabric PWWN 0
Changes detected in Fabric NWWN 0
Channel in TD_FABRIC topology 0
Channel in TD_PUBLIC_LOOP topology 0
Channel in TD_PRIVATE_LOOP topology 1262
Topology mode toggles during initialization 0
Topology changes across LDN/LUP 0
Successful initializations in P2P mode 0
Successful initializations in Loop mode 1262

Storm Statistics ...
Elastic Store Error Storm 0
Link Fail storm . 0
LIP(f8, xx) storm . 0
Loss Of Signal Storm 0
Out Of Sync Storm 0
Link Fault Storm 0
NOS_OLS Storm 0

PCI Error Interrupt Statistics ...
MPE Interrupts 0
CRS Interrupts 0
DER Interrupts 0
PER Interrupts 0

Target State M/C Statistics ...
Target Thread Alloc. Fail 0
Frames with length < FC-PH Hdr length 0
Frames with incorrect ROUTING FC-PH Hdr 0
Frames with invalid CS_CTL FC-PH Hdr 0
Frames with invalid TYPE FC-PH Hdr 0
Frames with invalid DF_CTL FC-PH Hdr 0
Invalid SOF/EOF Received 0
Multi-Frame Sequences Received 0
SFS Received With S_ID/D_ID Mismatch 0
SFS Received exceeding TD_SFTBFSZ 0
Inbound non-Class3 Frames 0
Inbound Frame large than TD_SFTBFSZ 0
Kmalloc Failures 0
Out of order Multi-Frame sequences 0

I/O Error Statistics ...
FCP Frames Recvd As Responder 0
Bad Unassisted FCP frame 0
Bad S_ID in Unassisted FCP frame 0
No cdb for I/O 0
IRB entries on ERQ 0
SCSI Linked list entries 0
Late ABTS Responses Received 0
Too many mapping entries for a single I/O 0

SF state m/c statistics...
Rcvd ABTS dropped due to lack of SFT 0
No reply before timeout 212
SF failed due to LDN 0
SF failed due to ASN, LKF or FTO 2764
SF thread Allocation failures 0
Unable to map SF payload 0
SFT in ERQ during Chip Reset 0
Late ICM from Non-compliant target 0


l1000db2 # /opt/fcms/bin/fcmsutil /dev/td1 stat
Wed Jan 26 18:25:57 2005
Channel Statistics

Statistics From Link Status Registers ...
Loss of signal 0 Bad Rx Char 22
Loss of Sync 0 Link Fail 0
Received EOFa 0 Discarded Frame 0
Bad CRC 0 Protocol Error 0

Channel State M/C Statistics ...
TL Errata D40 0
Unexpected OCM for FCP_DATA phase 0
LSM Link Fail State 0
No. of times LUP timer popped 0
HBA Changing AL_PA Across LIP 0
Channel Reset Request 0
Channel Offline Request 0
CE_FCP_FREEZE Request 61
CE_FCP_UNFREEZE Request 61
CE_FCP_FREEZE Cancel 0
CE_LDN Event 252
CE_LUP Event 253
ERQ frzn CMs, after TL is resumed 0
FCP frzn CMs, after TL is resumed 0
LDN EICM, when ERQ/FCP not frzn 0
FTO EICM, when ERQ/FCP not frzn 0
CE_FCP_FROZEN Event 0
Number Of Delayed Freeze Operations 0
CE_FCP_UNFREEZE recvd in CS_FCP_FROZEN 1824
Error Idle Interrupts 2016
ERQ/FCP Assists Resumed 2076
Loop State T/O, when LSM is not ONLINE 0
Loop State T/O, when LSM is ONLINE 0
Loop State T/O, when LSM is in ARB 0
Credit Errors 0
LIPs due to OLS 1

FLOGI Statistics ...
FLOGIs sent 256
FLOGI Resps recvd 0
LS_RJTs for FLOGI 0
Short FLOGI Resps 0
Version_Hi Error in FLOGI Resps 0
Version_Lo Error in FLOGI Resps 0
No class 3 support in FLOGI Resps 0
Invalid datasize/resvd2/fport_nport in FLOGI Resps 0
Invalid RA_TOV in FLOGI Resps 0
Invalid ED_TOV in FLOGI Resps 0
FLOGI Retries skipped due to Parse Errors 252
FTO OCMs for FLOGIs sent 256
FLOGIs timedout 0
FLOGI retries > High Water Mark 253

Fabric Statistics ...
FANs received 0
FANs timedout 0
FANs recvd in CS_FLOGI_SENT 0
Unexpected FANs received 0
ns_open failures 0
Changes detected in own nport_id 0
Changes detected in Fabric PWWN 0
Changes detected in Fabric NWWN 0
Channel in TD_FABRIC topology 0
Channel in TD_PUBLIC_LOOP topology 0
Channel in TD_PRIVATE_LOOP topology 253
Topology mode toggles during initialization 0
Topology changes across LDN/LUP 0
Successful initializations in P2P mode 0
Successful initializations in Loop mode 253

Storm Statistics ...
Elastic Store Error Storm 0
Link Fail storm . 0
LIP(f8, xx) storm . 0
Loss Of Signal Storm 0
Out Of Sync Storm 0
Link Fault Storm 0
NOS_OLS Storm 0

PCI Error Interrupt Statistics ...
MPE Interrupts 0
CRS Interrupts 0
DER Interrupts 0
PER Interrupts 0

Target State M/C Statistics ...
Target Thread Alloc. Fail 0
Frames with length < FC-PH Hdr length 0
Frames with incorrect ROUTING FC-PH Hdr 0
Frames with invalid CS_CTL FC-PH Hdr 0
Frames with invalid TYPE FC-PH Hdr 0
Frames with invalid DF_CTL FC-PH Hdr 0
Invalid SOF/EOF Received 0
Multi-Frame Sequences Received 0
SFS Received With S_ID/D_ID Mismatch 0
SFS Received exceeding TD_SFTBFSZ 0
Inbound non-Class3 Frames 0
Inbound Frame large than TD_SFTBFSZ 0
Kmalloc Failures 0
Out of order Multi-Frame sequences 0

I/O Error Statistics ...
FCP Frames Recvd As Responder 0
Bad Unassisted FCP frame 0
Bad S_ID in Unassisted FCP frame 0
No cdb for I/O 0
IRB entries on ERQ 0
SCSI Linked list entries 0
Late ABTS Responses Received 0
Too many mapping entries for a single I/O 0

SF state m/c statistics...
Rcvd ABTS dropped due to lack of SFT 0
No reply before timeout 16
SF failed due to LDN 0
SF failed due to ASN, LKF or FTO 1764
SF thread Allocation failures 0
Unable to map SF payload 0
SFT in ERQ during Chip Reset 0
Late ICM from Non-compliant target 0