topic Re: disk array, lvm, POWERFAILED in Operating System - HP-UX

disk array, lvm, POWERFAILED

Marcel Preda — Tue, 25 Jan 2005 06:28:11 GMT

Hi,

I have a `little' problem.
More than a week ago (2005.01.15), we have change the location of a feew HP servers + a CLARIION disk array.

The disk array is mounted on a L1000 class machine (hostname=l1000db2) on two LV (vg02 && vg03).

Yesterday, a proplem has ocurred, see below in syslog file.

I have no idea if it is just a coincidence,
but a feew minutes before the firts error message someone has run a script from HP , check.dat (hardware inspection).
and there was a command "lvlnboot -v",
you can see it in the syslog file
" Jan 24 16:32:48 l1000db2 LVM[8338]: lvlnboot -v"

Next, the syslog:
################

Jan 19 16:46:40 l1000db2 pwgrd: Invalid entry found in passwd file /etc/passwd
Jan 19 16:49:59 l1000db2 : su : + ta root-vscsftp1
Jan 19 16:54:46 l1000db2 : su : + ta root-oracle
Jan 19 16:57:49 l1000db2 : su : + ta root-oracle
Jan 24 16:28:12 l1000db2 ftpd[7725]: FTP LOGIN FROM 172.16.10.31 [172.16.10.31], root
Jan 24 16:32:48 l1000db2 LVM[8338]: lvlnboot -v
Jan 24 16:37:15 l1000db2 vmunix: LVM: Recovered Path (device 0x1f060700) to PV 3 in VG 2.
Jan 24 16:37:15 l1000db2 vmunix: LVM: Restored PV 3 to VG 2.
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f071400) to PV 3 in VG 3 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070500) to PV 2 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070300) to PV 1 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070700) to PV 3 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: vg[2]: pvnum=1 (dev_t=0x1f060300) is POWERFAILED
Jan 24 16:37:44 l1000db2 vmunix: LVM: vg[2]: pvnum=2 (dev_t=0x1f060500) is POWERFAILED
Jan 24 16:37:44 l1000db2 vmunix: LVM: vg[2]: pvnum=3 (dev_t=0x1f060700) is POWERFAILED
Jan 24 16:37:47 l1000db2 vmunix: LVM: vg[3]: pvnum=3 (dev_t=0x1f061400) is POWERFAILED

############

I have no idea how to solve this problem now.
Should I try a restart of the server ?!

Thanks,

Marcel

Re: disk array, lvm, POWERFAILED

Ravi_8 — Tue, 25 Jan 2005 06:39:06 GMT

Hi,

lvlnboot -v just shows the boot disk, no harm in using this command.

LOg shows one of the disk in VG2 is failed,
#ioscan -fnC disk
check all the disks are claimed or not.
propably you may have to replace the failed disk

Re: disk array, lvm, POWERFAILED

baiju_3 — Tue, 25 Jan 2005 11:29:05 GMT

Hi Marcel,

If you have powerpath installed then do,

Powermt config
Powermt check

again verify dead devices with powermt display.

You may consider a reboot , if you are not able to clear dead paths.

EMC has recommendations for settings on HP Unix and Clariion Arrays .Refer EMC Web .

You have to change the PV time out to 120 or more ,if PV time out is default.

Check with "pvdisplay -v /dev/rmt/c#t#d#"

Set PV time out value by,

pvchange -t 120 /dev/rmt/c#t#d#

Regards,
Baiju.

Re: disk array, lvm, POWERFAILED

Sundar_7 — Tue, 25 Jan 2005 11:43:30 GMT

IF you have alternate paths to the LUNS then you dont have to restart the server.

# vgdisplay -v /dev/vg02

# vgdisplay -v /dev/vg03

Do you see "Alternate Link" for the PVs ?

Re: disk array, lvm, POWERFAILED

Florian Heigl (new acc) — Tue, 25 Jan 2005 12:25:55 GMT

The HP script either triggered the error or made the os aware of it, You have to find out which is the case.

Check the Clariions logs for 'array shutdown for trespass' messages.

/opt/Navisphere/bin/navicli -h getlog -2000

If there are more than 10-20 of these errors, this is suspicious.
Are You running in a quick loop configuration?

If You don't know, check it using /opt/fcms/bin/fcmsutil /dev/td | grep Topology

If it is a quickloop (plain FC-AL):
There are conditions where HP-UX overloads the Clariions Storage Processor to a point I better don't talk about.

Check Your kernel parameter max_fcp_reqs and compare it to emc's documetation.
a rule of thumb: max_fcp_reqs=(512/2/systems_in_loop)
Also, check the scsi queue depth against what emc wants.

It seems like You don't have Navisphere Agent installed - You should.

(We had a lot of problems in our SAN until I stopped begging for permission and simply set that parameter to what it needed to be :)

Re: disk array, lvm, POWERFAILED

Marcel Preda — Tue, 25 Jan 2005 13:55:01 GMT

Hi,
sorry for my late response, other problems come too :-(

About "ioscan -fnC disk"
all disks are CLAIMED, like:
....
disk 28 0/7/0/0.8.0.1.0.0.1 sdisk CLAIMED DEVICE DGC C4400WDR1
/dev/dsk/c7t0d1 /dev/rdsk/c7t0d1
....

About powermt command , no such command
About "pvdisplay -v /dev/rmt/c#t#d#".

Probable I dont have such devices.
ls -l /dev/rmt/c*
crw-rw-rw- 2 bin bin 205 0x030000 Mar 16 2001 /dev/rmt/c3t0d0BEST
crw-rw-rw- 2 bin bin 205 0x030080 Mar 16 2001 /dev/rmt/c3t0d0BESTb
crw-rw-rw- 2 bin bin 205 0x030040 Apr 13 2001 /dev/rmt/c3t0d0BESTn
crw-rw-rw- 2 bin bin 205 0x0300c0 Mar 16 2001 /dev/rmt/c3t0d0BESTnb
crw-rw-rw- 1 bin bin 205 0x030001 Mar 16 2001 /dev/rmt/c3t0d0DDS
crw-rw-rw- 1 bin bin 205 0x030081 Mar 16 2001 /dev/rmt/c3t0d0DDSb
crw-rw-rw- 1 bin bin 205 0x030041 Mar 16 2001 /dev/rmt/c3t0d0DDSn
crw-rw-rw- 1 bin bin 205 0x0300c1 Mar 16 2001 /dev/rmt/c3t0d0DDSnb

Looks to be just magnetic tapes.

About vgdisplay:

All disks have "Alternate Link"

vgdisplay -v /dev/vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 13915
VGDA 8
PE Size (Mbytes) 4
Total PE 55652
Alloc PE 55652
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol1
LV Status available/syncd
LV Size (Mbytes) 222608
Current LE 55652
Allocated PE 55652
Used PV 4

--- Physical volumes ---
PV Name /dev/dsk/c6t0d1
PV Name /dev/dsk/c7t0d1 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d3
PV Name /dev/dsk/c7t0d3 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d5
PV Name /dev/dsk/c7t0d5 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d7
PV Name /dev/dsk/c7t0d7 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

On #vgdisplay -v /dev/vg03
is a similar situation, alal disks have "Alternate Link"

About the topology:
l1000db2 # /opt/fcms/bin/fcmsutil /dev/td0 | grep Topology
Topology = PRIVATE_LOOP
l1000db2 # /opt/fcms/bin/fcmsutil /dev/td1 | grep Topology
Topology = PRIVATE_LOOP

I was traing:

l1000db2 # /opt/Navisphere/bin/navicli getlog
ABORT instruction (core dumped)

The clariion is connected to the L1000 server via optical fibre, so I dindn't know what IP to user as "-h host" option :-(

BTW: I'm not an expert in HP-UX, in any case not disks-array and Navisphere.
Also I'm about 2000 kms distance by the servers.

Thanks for your time,
I'll notice you about what will happen.

Marcel

Re: disk array, lvm, POWERFAILED

Florian Heigl (new acc) — Tue, 25 Jan 2005 18:52:21 GMT

if everything is CLAIMED and PV Status is "available" too, then I'd say there is no need to reboot.

The Clariion has two service processors which are usually connected to the site lan, but if noone can tell You that won't do any good.

Either have someone get You that ip address or hostname, or search through the disks - there is another parameter (documented in a file called agent.config that is *somewhere*) to use a scsi-device for communication.
this definitely works if You see any LUNs on 0/7/0 with a capacity of 0 or 8 MB.

just use /opt/Navisphere/bin/navicli and hit enter for a list of commands.

I kind of think You're having the same problem we had, but I can't tell for sure.

First find a way to reproduce the problem at a low-risk time, then - if You wish - try changing the kernel parameter (ask either HP or EMC^2 on it, they both have the relevant EMC^2 document on what to change to make HP-UX a 'supported' client in loop configurations) and see if the problem persists. (I'd bet it doesn't)

Re: disk array, lvm, POWERFAILED

Florian Heigl (new acc) — Tue, 25 Jan 2005 19:08:22 GMT

The Navisphere cli Manual is called 069001038-1 at EMC^2, and I'll copy a bit but You might want to search for a pdf...:

navicli [-d device ] [-h hostname] {-np} [-m] [-p] [-t timeout] -[v|q] command

(-m suppress output, -np don't poll the box, which sometimes may help performance of command, -p appears to be preview mode, -q suppresses *error* messages -t sets the timeout for the command, from my experience this is passed to the box, not to the cli program -v extensive error descriptions)

so, this could be
navicli -d /dev/dsk/c7t0d0 getlog

I have never tried using a *data* lun for this, and I'm not sure if it'll work. If this goes somewhere through the scsi mode pages it might, but maybe it just breaks everything :))

So better find someone that knows the ip, afterwards search for the EMC^2 config file agent.config and edit it to allow access for root@localhost and on how to access the clariion.

afterwards /sbin/init.d/agent stop and /sbin/init.d/agent start and from that moment on You'll have a lot of output on every single soft scsi error and whatever the clariion notices right into Your syslog.

:)

Re: disk array, lvm, POWERFAILED

Nguyen Anh Tien — Tue, 25 Jan 2005 20:48:13 GMT

Hi Marcel.
>>>Jan 24 16:32:48 l1000db2 LVM[8338]: lvlnboot is OK. it is no harm for your server.
But
=========
Jan 24 16:37:15 l1000db2 vmunix: LVM: Recovered Path (device 0x1f060700) to PV 3 in VG 2.
Jan 24 16:37:15 l1000db2 vmunix: LVM: Restored PV 3 to VG 2.
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f071400) to PV 3 in VG 3 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070500) to PV 2 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070300) to PV 1 in VG 2 Failed!
Jan 24 16:37:44 l1000db2 vmunix: LVM: Path (device 0x1f070700) to PV 3 in VG 2 Failed!
===
are problem.
Now issue these command to diagnostic
#vgdisplay -v vg02
#vgdisplay -v vg02
Pls post result for me/
tienna

Re: disk array, lvm, POWERFAILED

Marcel Preda — Wed, 26 Jan 2005 02:44:22 GMT

Hi Florian,

I think that it's a problem with Navisphere too.

l1000db2 # /opt/Navisphere/bin/navicli
ABORT instruction (core dumped)

l1000db2 # /sbin/init.d/agent stop
Navisphere Agent is not running
l1000db2 # /sbin/init.d/agent start
Starting Navisphere Agent
/sbin/init.d/agent[37]: 26412 Abort(coredump)

Marcel

Re: disk array, lvm, POWERFAILED

Marcel Preda — Wed, 26 Jan 2005 02:47:56 GMT

l1000db2 # vgdisplay -v vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 13915
VGDA 8
PE Size (Mbytes) 4
Total PE 55652
Alloc PE 55652
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
Hi Nguyen,

the oputput of the commands:
l1000db2 # vgdisplay -v vg02
--- Volume groups ---
VG Name /dev/vg02
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 13915
VGDA 8
PE Size (Mbytes) 4
Total PE 55652
Alloc PE 55652
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg02/lvol1
LV Status available/syncd
LV Size (Mbytes) 222608
Current LE 55652
Allocated PE 55652
Used PV 4

--- Physical volumes ---
PV Name /dev/dsk/c6t0d1
PV Name /dev/dsk/c7t0d1 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d3
PV Name /dev/dsk/c7t0d3 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d5
PV Name /dev/dsk/c7t0d5 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t0d7
PV Name /dev/dsk/c7t0d7 Alternate Link
PV Status available
Total PE 13913
Free PE 0
Autoswitch On

____________________________________

l1000db2 # vgdisplay -v vg03
--- Volume groups ---
VG Name /dev/vg03
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 1
Open LV 1
Max PV 16
Cur PV 12
Act PV 12
Max PE per PV 26592
VGDA 24
PE Size (Mbytes) 32
Total PE 33964
Alloc PE 33964
Free PE 0
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg03/lvol1
LV Status available/syncd
LV Size (Mbytes) 1086848
Current LE 33964
Allocated PE 33964
Used PV 12

--- Physical volumes ---
PV Name /dev/dsk/c6t1d1
PV Name /dev/dsk/c7t1d1 Alternate Link
PV Status available
Total PE 2399
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d2
PV Name /dev/dsk/c7t1d2 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d3
PV Name /dev/dsk/c7t1d3 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d4
PV Name /dev/dsk/c7t1d4 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d5
PV Name /dev/dsk/c7t1d5 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d6
PV Name /dev/dsk/c7t1d6 Alternate Link
PV Status available
Total PE 3323
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t1d7
PV Name /dev/dsk/c7t1d7 Alternate Link
PV Status available
Total PE 2399
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d0
PV Name /dev/dsk/c7t2d0 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d1
PV Name /dev/dsk/c7t2d1 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d2
PV Name /dev/dsk/c7t2d2 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d3
PV Name /dev/dsk/c7t2d3 Alternate Link
PV Status available
Total PE 2815
Free PE 0
Autoswitch On

PV Name /dev/dsk/c6t2d4
PV Name /dev/dsk/c7t2d4 Alternate Link
PV Status available
Total PE 3323
Free PE 0
Autoswitch On

Re: disk array, lvm, POWERFAILED

Roxana_4 — Wed, 26 Jan 2005 03:57:41 GMT

OFF TOPIC.
Buna ziua stimate domn Marcel.
Ma numesc Roxana Colta si sunt din Brasov.
Cu scuzele de rigoare, imi permit sa va abordez cu un subiect diferit de problema dumneavoastra.
Deoarece sunteti, deocamdata, singurul roman cu experienta in servere HP-UX de pe acest forum, imi permit sa va solicit in mod expres parerea in urmatoarea problema: am niste DTC-uri mai vechi (modelele HP 2345A si HP 2340A) pe care le folosesc pentru a conecta niste terminale la un server HP e3000 cu MPE/iX 7.0 ; am achizitionat un server Inegrity rx2600 cu HP-UX 11iv2 pe care urmeaza sa il instalez si configurez ; pot sa folosesc DTC-urile vechi pe acest server, HP-UX 11i le recunoaste la configurare ?

Putem gasi o alta cale de a comunica in mod direct ? Adresa mea este roxana@engineer.com . Va multumesc anticipat si imi cer scuze pentru indrazneala de a va aborda pe acest forum.

Roxana

Re: disk array, lvm, POWERFAILED

Mohanasundaram_1 — Wed, 26 Jan 2005 06:52:03 GMT

Hi Marcel,

I don't know what Roxana said before this. But I can see it is not a single disk (LUN)failure. If you go through the syslog.log message, you can see that PV1, PV2 and PV3 of VG2 have issued this message.

If more than one LUN is causing such messages I would suspect something is wrong with the controller on the Clarion or HBA on the system or anything between them.

If you are experiencing performance problem due to this message, then it may be an intermittent problem. That is, that path may become available for sometime and then go off-line and again become available. This may go on until the problem is fixed. There may be a considerable performance degradation due to this.

Let us know if this is the case. You can also provide the syslog.log in its entirety.

With regards,
Mohan.

Re: disk array, lvm, POWERFAILED

Florian Heigl (new acc) — Wed, 26 Jan 2005 07:26:05 GMT

marcel - if the navicli is broken, then find the person responsible or the service contract, have an EMC technician come to the site and check the event logs.

Re: disk array, lvm, POWERFAILED

Marcel Preda — Wed, 26 Jan 2005 07:44:05 GMT

Hi again,

I'll try to contact the person who can contact the tech support, the maintenance contract is no more available.

In a feew minutes I'll put the syslog file.

Thanks for your time.

Marcel

Re: disk array, lvm, POWERFAILED

Marcel Preda — Wed, 26 Jan 2005 08:10:15 GMT

The entirely syslog file.

If some of you have some time to read it...

Thanks,

Marcel

Re: disk array, lvm, POWERFAILED

Mohanasundaram_1 — Wed, 26 Jan 2005 08:26:44 GMT

Hi Marcel,

I can see the powerfailed message and recovered message for PVs in VG2 and VG3. So the problem must be as I was guessing. one of the path is going on and off frequently. Trace the entire path and isolate the problem.

Contrary to what you have indicated in the first post, vg3 seems to be the most battered one due to this problem. Your problem lies in one of the components - HBA, cable, switch, Clariion controller.

With regards,
Mohan.

Re: disk array, lvm, POWERFAILED

Marcel Preda — Wed, 26 Jan 2005 10:40:18 GMT

Hi again,

thanks for the reply Mohan, .

A good news,
on other server(which was down till today) I have found a Navispher which seems works (no core dump).
But I still cant find the IP of the clariion array, if it exist.

Maybe some of you could help me to find it in a config file.
I cant figure which of the file could be:

l1000db1 # pwd
/etc/Navisphere
l1000db1 # ls -ltr
total 4
drwxr-xr-x 2 root sys 96 Mar 27 2001 logs
drwxr-xr-x 2 root sys 96 Mar 27 2001 backup
lrwxr-xr-x 1 root sys 32 Mar 27 2001 agent.config -> /opt/Navisphere/bin/agent.config
lrwxr-xr-x 1 root sys 30 Mar 27 2001 clsendtrap -> /opt/Navisphere/bin/clsendtrap
drwxrwxrwx 2 root sys 96 Mar 27 2001 log
lrwxr-xr-x 1 root sys 31 Mar 30 2001 Navimon.cfg -> /opt/Navisphere/bin/Navimon.cfg
drwxr-xr-x 2 root sys 2048 Jan 26 15:47 messages

Thanks,

Marcel

Re: disk array, lvm, POWERFAILED

Marcel Preda — Wed, 26 Jan 2005 10:53:05 GMT

Hi,

looking in /etc/Navisphere/agent.config
I have found 2 devices: c6t0d0 & c7t0d2

ussing those:

l1000db1 # navicli -d c6t0d0 getlog -2000 | grep -i trespass
01/26/2005 16:42:04 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba9220 14318b0
01/26/2005 16:42:06 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba89cc 14318b0
01/26/2005 16:42:08 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba89cc 14318b0
01/26/2005 16:42:11 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba89cc 14318b0
01/26/2005 16:42:13 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba89cc 14318b0
01/26/2005 16:42:15 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba8408 14318b0
01/26/2005 16:42:17 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] ba8408 14318b0
l1000db1 # navicli -d c7t0d2 getlog -2000 | grep -i trespass
01/26/2005 16:42:25 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0b] baa4b4 14318b0
01/26/2005 16:42:26 Bus 0 Enclosure 0 Disk 0(606) Unit Shutdown for Trespass [0x01] ba82c0 14318b0
01/26/2005 16:42:28 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x11] 184db0 14318a8
01/26/2005 16:42:28 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x09] 184db0 14318a8
01/26/2005 16:42:29 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0d] 184db0 14318a8
01/26/2005 16:42:30 Bus 0 Enclosure 0 Disk 2(606) Unit Shutdown for Trespass [0x03] 184db0 14318a8
01/26/2005 16:42:30 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0a] 184db0 14318a8
01/26/2005 16:42:30 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x10] 184db0 14318a8
01/26/2005 16:42:32 Bus 0 Enclosure 0 Disk 4(606) Unit Shutdown for Trespass [0x05] 184db0 14318a8
01/26/2005 16:42:34 Bus 0 Enclosure 0 Disk 6(606) Unit Shutdown for Trespass [0x07] ba97e4 14318b0
01/26/2005 16:42:34 Bus 0 Enclosure 0 Disk 4(606) Unit Shutdown for Trespass [0x05] 184db0 14318a8
01/26/2005 16:42:34 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0c] ba84ac 14318b0
01/26/2005 16:42:35 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x0e] ba9d04 14318b0
01/26/2005 16:42:35 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x0f] 184db0 14318a8
01/26/2005 16:42:35 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x12] baa038 14318b0
01/26/2005 16:42:37 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x13] baa224 14318b0
01/26/2005 16:42:37 Bus 0 Enclosure 0 Disk 0(606) Unit Shutdown for Trespass [0x01] 184db0 14318a8
01/26/2005 16:42:37 Bus 0 Enclosure 0 Disk 2(606) Unit Shutdown for Trespass [0x03] 184db0 14318a8
01/26/2005 16:42:37 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x14] 826d30 ba8884
01/26/2005 16:42:37 Bus 0 Enclosure 1 Disk 0(606) Unit Shutdown for Trespass [0x09] 826d30 ba8884
01/26/2005 16:42:38 Bus 0 Enclosure 2 Disk 0(606) Unit Shutdown for Trespass [0x10] 826d30 ba8884

what the hell could be the problem?
unfortunately I can not verify the cables, hardvare in general, I'm out of site.

Thanks, Marcel

Re: disk array, lvm, POWERFAILED

Florian Heigl (new acc) — Wed, 26 Jan 2005 10:57:55 GMT

our agent config reads like this:
# SAMPLES:
# device c0t0d0 NAVISPHERE-1 "NAVISPHERE-1"

# automatically detect manageable devices
device auto auto

maybe that's enough, but I'm not very convinced:

root@xxxxx:/opt/Navisphere/bin>./navicli getlog -100
Error: getlog command failed
Cannot access device

In the meantime You should use fcmsutil to check for adapter/cabling errors...

/opt/fcms/bin/fcmsutil /dev/tdN stat

root@xxxxx:/opt/Navisphere/bin>/opt/fcms/bin/fcmsutil /dev/td2 stat
Wed Jan 26 16:53:47 2005
Channel Statistics

Statistics From Link Status Registers ...
Loss of signal 22 Bad Rx Char 2379
Loss of Sync 46 Link Fail 41
Received EOFa 0 Discarded Frame 0
Bad CRC 0 Protocol Error 0

Do a reset on the counters and see, if Loss of signal, Bad RX or similar things increase, if yes there might be a hardware issue

These are also interesting:
Storm Statistics ...
Elastic Store Error Storm 0
Link Fail storm . 0
LIP(f8, xx) storm . 0
Loss Of Signal Storm 0
Out Of Sync Storm 0
Link Fault Storm 0
NOS_OLS Storm 1