topic Re: errors in syslog in Operating System - HP-UX

errors in syslog

Brian Bientz — Wed, 14 Jan 2004 08:55:30 GMT

We are getting the following errors in syslog:

Jan 14 08:50:01 prodcl1b vmunix: SCSI: Request Timeout; Abort Tag -- lbolt: 829523321, dev: 1f036000, io_id: 389644d
Jan 14 08:50:02 prodcl1b vmunix: LVM: Recovered Path (device 0x1f036000) to PV 1 in VG 0.
Jan 14 08:50:02 prodcl1b vmunix: LVM: Restored PV 1 to VG 0.
Jan 14 08:50:04 prodcl1b vmunix: LVM: Failed to automatically resync PV 1f036000 error: 5

Can anyone provide some insight? At the time the messages were appearing in the syslog, performance on the node was pathetic. A simple ls command was taking many seconds to return. A number of processes were timing out.

Any insight would be appreciated. Thanks!

Re: errors in syslog

Sunil Sharma_1 — Wed, 14 Jan 2004 09:03:31 GMT

Hi,

These are SCSI errors. it seems your one disk has problem.

0x1f036000 suppose to be minor number of disk. but it suppose to be only 8 digit hear it's 10 digit.

once again check the syslog and find out the disk whihc has problem.

Sunil

Re: errors in syslog

Jakes Louw — Wed, 14 Jan 2004 09:03:57 GMT

Hi
You are experiencing hardware errors on a disk.
Do the following:

ll /dev/dsk | grep 1f036000

This will give the CTD file that is giving the errors.
Do a pvdisplay on this disk to see which VG, but judging from the output you sent, I reckon this is VG00.

I hope you have root mirrored!

The performance drop is caused by the system processing I/O resubmits due to excessive error recovery.

Re: errors in syslog

Jakes Louw — Wed, 14 Jan 2004 09:07:25 GMT

Just a correction to myself: you do have mirroring, as the message "Restored PV 1 to VG 0." indicates that a mirrored pair has had one of the disks come back from an error.
The failure to resync, however, indicates a terminal failure.

Report to HP immediately if you have a contract, or else tell your boss you need money for a Time and Material call!

Re: errors in syslog

Dietmar Konermann — Wed, 14 Jan 2004 09:08:53 GMT

The dev_t 0x1f036000 is major# 0x1f (31), minor# 0x036000.

# lsdev -b 31
Character Block Driver Class
188 31 sdisk disk

So the disk at c3t6d0 looks suspicious.

Best regards...
Dietmar.

Re: errors in syslog

Cheryl Griffin — Wed, 14 Jan 2004 09:12:44 GMT

# ioscan -fnC disk
Make sure the disk shows "claimed" and has a valid description

Test the disk with dd.
# dd if=/dev/dsk/c3t6d0 of=/dev/null bs=64 count=1000
If successful, the results of this command will look similar to:
100+0 records in
100+0 records out

Look for any of the following:
*IO errors
*the number of blocks read in do not match was is read out
*the command hangs or immediately produces an error

# diskinfo /dev/rdsk/c3t6d0
Look for any of the following which indicate a HW problem:
*the command produces an error
*disk size reports zero or is incorrect

Re: errors in syslog

Stuart Abramson_2 — Wed, 14 Jan 2004 09:14:35 GMT

1. Get the "dev:" entry from the lbolt:

# dmesg | grep lbolt | grep dev:

SCSI: Abort -- lbolt: 18346341, dev: e7015000, io_id: 122e9a3
SCSI: Request Timeout -- lbolt: 18351441, dev: e7015000
SCSI: Abort -- lbolt: 18351441, dev: e7015000, io_id: 122e9be
SCSI: Request Timeout -- lbolt: 18356641, dev: e7015000
SCSI: Abort -- lbolt: 18356641, dev: e7015000, io_id: 122e9cf
SCSI: Request Timeout -- lbolt: 18362141, dev: e7015000
SCSI: Abort -- lbolt: 18362141, dev: e7015000, io_id: 122e9e0
SCSI: Request Timeout -- lbolt: 74105435, dev: 1f000000
SCSI: Abort Tag -- lbolt: 74105435, dev: 1f000000, io_id: 4ead34

Here we have two:

1f
e7

2. This is the major number of the device in question. Convert the first
two digits of the device from hex to decimal:

# printf "%#d\n" 0x1f
31

3. find out what driver this major number is. It tells us the type of
device:

# lsdev 31

Character Block Driver Class
188 31 sdisk disk

So, this is probably a disk !

4. Find the device file entry from the remainder of the lbolt error:

SCSI: Abort Tag -- lbolt: 74105435, dev: 1f000000, io_id: 4ead34

This is the minor number for the device that is failing.

a. Block device:

# ll -R /dev/ | grep 31 | grep 0x000000

brw-r----- 1 bin sys 31 0x000000 Jul 15 16:25 c0t0d0

Or:

b. Character Device:

# ll -R /dev/ | grep 188 | grep 0x000000
crw-r----- 1 bin sys 188 0x000000 Oct 11 07:15 c0t0d0

5. Find the Hardware Address:

# lssf /dev/dsk/c0t0d0
sdisk card instance 0 SCSI target 0 SCSI LUN 0 section 0
at address 0/0/0.0.0 /dev/dsk/c0t0d0

6. Find the type of device:

# diskinfo /dev/rdsk/c0t0d0# diskinfo /dev/rdsk/c0t0d0
SCSI describe of /dev/rdsk/c0t0d0:
vendor: DGC
product id: C2300WDR1
type: direct access
size: 4102875 Kbytes
bytes per sector: 512

So, we have a Nike disk at hardware address 0/0/0.0.0, device file
/dev/dsk/c0t0d0

Re: errors in syslog

David_246 — Wed, 14 Jan 2004 09:16:53 GMT

Hi,

Looks like /stand is having problems, at least your underlying disk has.
As your root is mirrored there is no real problem, your system keeps running as it should. However, you lost your backup disk now. Ones your other disk will be failing too, you have a problem.

For now try to find out which disks are used for your /dev/vg00/lvol1 using "lvdisplay -v /dev/vg00/lvol(0)1"
See with ioscan -fnC disk if indeed one of these disks is reporting the errors. vgdisply -v vg00 should give you some more info as well.
Look up the /var/adm/syslog/syslog.log for detailed errors. You must be sure to replace the correct disk! Also always replace a disk with exactly the same type of disk. This prevents awfull things to happen.

An LVM manual is handy in these times. www.google.com is it as well. Good luck!

Regs David

Re: errors in syslog

V. Nyga — Wed, 14 Jan 2004 12:31:24 GMT

Hi Stuart Abramson

you should check this thread:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=351828 ;-)

Brian,
sorry for using your thread for chat.

Volkmar

Re: errors in syslog

Daniel Tan_3 — Fri, 11 Jun 2004 23:38:44 GMT

if it is for a tape device, what do you use to check the hardware address?

lssf /dev/???

Re: errors in syslog

Jakes Louw — Sat, 12 Jun 2004 00:41:50 GMT

/dev/dsk or /dev/rdsk