LVM powerfailed SCSI: Abort abandoned -- lbolt , disk failures

[Franky Leeuwerck_2]

Dear forum group,

One of our HP-UX machines regurarly has troubles with one or more of its disks.

In the syslog, we see messages like this :
scb->cdb: 2a 00 00 08 0a a0 00 00 10 00
SCSI: Abort abandoned -- lbolt: 451001, dev: 1f045000, io_id: 402368d, status: 200
LVM: vg[1]: pvnum=3 (dev_t=0x1f045000) is POWERFAILED
scb->cdb: 28 00 00 00 00 10 00 00 04 00
SCSI: Abort abandoned -- lbolt: 30690427, dev: 1f015000, io_id: 1025e79, status: 200
LVM: vg[1]: pvnum=2 (dev_t=0x1f015000) is POWERFAILED
DIAGNOSTIC SYSTEM WARNING:
The diagnostic logging facility has started receiving excessive
errors from the I/O subsystem

Sometimes, but not always, we have to stop the diagnostics in order to prevent the /var directory from filling up completely.



This has happened already a few times during the last 2 months.
Each time, a reboot of the HP-UX solved the problem.

Any idea what can cause this ?

Thanks in advance,
Franky Leeuwerck

[Sunil Sharma_1]

HI,

One of your disk has problem. Check the scsi connection and power connection to disk.

Disk device is c1t5d0

for more information have a lookm on

http://forums1.itrc.hp.com/service/forums/bizsupport/questionanswer.do?threadId=245085

Sunil

[Bharat Katkar]

Hi franky,
This error we recevied was due to duplicate SCSI ID on a SCSI bus. This happens if you have two SCSI adapters on a single SCSI bus. Suggest to check one.
Also check the SCSI termination if it is done properly and look out for any faulty SCSI cable or any loose connection.

Hope this helps.
Regards,

[Franky Leeuwerck_2]

Hi,
How can I find out if there are two SCSI adapters on a single SCSI bus on this remote system ?

Franky

[Franky Leeuwerck_2]

To complete the information : each time we also see NO_HW for the corresponding disks in the 'ioscan -fnC disk' output.

The disks are actually not responding ( 'stale' in pvdisplay outputs ).

Franky

[Bharat Katkar]

Franky ,
Use the ioscan command and try looking at H/w paths displayed for SCSI cntrl's.
In my case we had connected DLT library to two different server and it was a single SCSI bus two adapters. That may or may not be your problem.
Regards,

[Franky Leeuwerck_2]

Hello Mr. Bharat Katkar,

This is the ioscan output :
Class I H/W Path Driver S/W State H/W Type Description
=======================================================================
bc 0 root CLAIMED BUS_NEXUS
bc 1 8 ccio CLAIMED BUS_NEXUS I/O Adapter
bc 2 8/0 bc CLAIMED BUS_NEXUS Bus Converter
tty 0 8/0/0 mux2 CLAIMED INTERFACE MUX
ext_bus 0 8/4 c720 CLAIMED INTERFACE GSC add-on Fast/Wide
SCSI Interface
target 0 8/4.3 tgt CLAIMED DEVICE
tape 0 8/4.3.0 stape CLAIMED DEVICE Quantum DLT4000
target 1 8/4.7 tgt CLAIMED DEVICE
ctl 0 8/4.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 1 8/12 c720 CLAIMED INTERFACE GSC add-on Fast/Wide
SCSI Interface
target 2 8/12.5 tgt CLAIMED DEVICE
disk 0 8/12.5.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC
target 3 8/12.6 tgt CLAIMED DEVICE
disk 1 8/12.6.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
target 4 8/12.7 tgt CLAIMED DEVICE
ctl 1 8/12.7.0 sctl CLAIMED DEVICE Initiator
ba 0 8/16 bus_adapter CLAIMED BUS_NEXUS Core I/O Adapter
ext_bus 3 8/16/0 CentIf CLAIMED INTERFACE Built-in Parallel Int
erface
ext_bus 2 8/16/5 c720 CLAIMED INTERFACE Built-in SCSI
target 5 8/16/5.2 tgt CLAIMED DEVICE
disk 2 8/16/5.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 6x/32
x
target 6 8/16/5.3 tgt CLAIMED DEVICE
tape 1 8/16/5.3.0 stape CLAIMED DEVICE HP C1533A
target 7 8/16/5.5 tgt CLAIMED DEVICE
disk 3 8/16/5.5.0 sdisk CLAIMED DEVICE SEAGATE ST39173N
target 8 8/16/5.6 tgt CLAIMED DEVICE
disk 4 8/16/5.6.0 sdisk CLAIMED DEVICE SEAGATE ST39175LW
target 9 8/16/5.7 tgt CLAIMED DEVICE
ctl 2 8/16/5.7.0 sctl CLAIMED DEVICE Initiator
lan 0 8/16/6 lan2 CLAIMED INTERFACE Built-in LAN
ps2 0 8/16/7 ps2 CLAIMED INTERFACE Built-in Keyboard/Mou
se
ba 1 8/20 bus_adapter CLAIMED BUS_NEXUS Core I/O Adapter
tty 1 8/20/2 asio0 CLAIMED INTERFACE Built-in RS-232C
bc 3 10 ccio CLAIMED BUS_NEXUS I/O Adapter
ext_bus 4 10/12 c720 CLAIMED INTERFACE GSC add-on Fast/Wide
SCSI Interface
target 10 10/12.5 tgt CLAIMED DEVICE
disk 5 10/12.5.0 sdisk CLAIMED DEVICE SEAGATE ST39236LC
target 11 10/12.6 tgt CLAIMED DEVICE
disk 6 10/12.6.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
target 12 10/12.7 tgt CLAIMED DEVICE
ctl 3 10/12.7.0 sctl CLAIMED DEVICE Initiator
processor 0 32 processor CLAIMED PROCESSOR Processor
memory 0 49 memory CLAIMED MEMORY Memory

Franky

[Bharat Katkar]

Franky one more thing,

# vgdisplay -v vg01
See which pv's it is using e.g. c0t1d0,.. and let me know.

# ioscan -fnC disk
This output too.
Regards,

[Franky Leeuwerck_2]

These are the outputs you asked (after the last reboot).
To me, this looks fine.

vgdisplay -v vg01
--- Volume groups ---
VG Name /dev/vg01
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 3
Open LV 3
Max PV 16
Cur PV 4
Act PV 4
Max PE per PV 2171
VGDA 8
PE Size (Mbytes) 4
Total PE 8680
Alloc PE 8000
Free PE 680
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

--- Logical volumes ---
LV Name /dev/vg01/lvol1
LV Status available/syncd
LV Size (Mbytes) 6000
Current LE 1500
Allocated PE 3000
Used PV 2

LV Name /dev/vg01/lvol2
LV Status available/syncd
LV Size (Mbytes) 6000
Current LE 1500
Allocated PE 3000
Used PV 2

LV Name /dev/vg01/lvol3
LV Status available/syncd
LV Size (Mbytes) 4000
Current LE 1000
Allocated PE 2000
Used PV 4


--- Physical volumes ---
PV Name /dev/dsk/c1t6d0
PV Status available
Total PE 2170
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t6d0
PV Status available
Total PE 2170
Free PE 340
Autoswitch On

PV Name /dev/dsk/c1t5d0
PV Status available
Total PE 2170
Free PE 0
Autoswitch On

PV Name /dev/dsk/c4t5d0
PV Status available
Total PE 2170
Free PE 340
Autoswitch On


ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 0 8/12.5.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC
/dev/dsk/c1t5d0 /dev/rdsk/c1t5d0
disk 1 8/12.6.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c1t6d0 /dev/rdsk/c1t6d0
disk 2 8/16/5.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 6x/32x
/dev/dsk/c2t2d0 /dev/rdsk/c2t2d0
disk 3 8/16/5.5.0 sdisk CLAIMED DEVICE SEAGATE ST39173N
/dev/dsk/c2t5d0 /dev/rdsk/c2t5d0
disk 4 8/16/5.6.0 sdisk CLAIMED DEVICE SEAGATE ST39175LW
/dev/dsk/c2t6d0 /dev/rdsk/c2t6d0
disk 5 10/12.5.0 sdisk CLAIMED DEVICE SEAGATE ST39236LC
/dev/dsk/c4t5d0 /dev/rdsk/c4t5d0
disk 6 10/12.6.0 sdisk CLAIMED DEVICE SEAGATE ST39173WC
/dev/dsk/c4t6d0 /dev/rdsk/c4t6d0

[Bharat Katkar]

Hi franky,
The problem is related to the following SCSI controller i.e.
target 4 8/12.7 tgt CLAIMED DEVICE
ctl 1 8/12.7.0 sctl CLAIMED DEVICE Initiator

and the disk is:
disk 0 8/12.5.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC
/dev/dsk/c1t5d0 /dev/rdsk/c1t5d0

Atleast from the output you posted it doesn't look like SCSI ID duplication, so if possible reboot the server, go to PDC and check all the SCSI controller's , see if 8/12.7 is repeated anywhere.

Before that i would suggest you check the cabling and SCSI termination physically and this is not possible remotely. You need to take help of any local engineer there.

Hope that helps.
Regards,

[Franky Leeuwerck_2]

So it looks like there is really a problem with the SCSI controller and/or disk power connectors.

We've been checking this several times by our local IT responsable but without result.

We'd better contact an HP Support Engineer, I guess.


Thanks all for your help.
Franky Leeuwerck

[Mohanasundaram_1]

Hi Franky,

It may not be one SCSI bus but two, which is giving you this problem.

From your first message, 2 disks have given the powerfailed message,

c4t5d0 and c1t5d0
------------------
ext_bus 4 10/12 c720 CLAIMED INTERFACE GSC add-on Fast/Wide SCSI Interface
target 10 10/12.5 tgt CLAIMED DEVICE
disk 5 10/12.5.0 sdisk CLAIMED DEVICE SEAGATE ST39236LC

ext_bus 1 8/12 c720 CLAIMED INTERFACE GSC add-on Fast/Wide SCSI Interface
target 2 8/12.5 tgt CLAIMED DEVICE
disk 0 8/12.5.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC


and these two must belong to the same volume group, whose group file should have the minor number 0x010000.

Is there an external Storage unit? if so, was it powered-off during the time this error came? that would explain why 2 disks from 2 different SCSI bus would give this error.

If that was not the case, you need to have a thorough investigation of the SCSI bus, including the cables and terminators. Check for any kink in the cable connectors. Ensure you have put the correct type of terminators.

Provide more details about the server and storage.

Cheers,
Mohan.

[Franky Leeuwerck_2]

Dear Mohan,

Thanks for your reply.
In the mean time we had again such powerfailure message on two disks.
We contacted an HP engineer to come over and to have everything checked thoroughly.

Regards,
Franky

[Bharat Katkar]

Hi franky,
lets us know what was the problem. That will help us all.
Thanks,
bharat.

[Franky Leeuwerck_2]

Dear Bharat,

If possible, I'll let you know what the problem was. The local IT responsable, a few thousands miles from my place, must contact an HP technician. I don't know when the technician will be able to do his intervention. So, it may take a few days before I can post the outcome.

Franky

[Franky Leeuwerck_2]

Hi,

The SE replaced both of the drives in question. They have been remirrored and all data seems to be intact.

The SE could not find any other problems. Hopefully the system will be stable for a while.

Thanks for your help.
Franky