1830009 Members
2397 Online
109998 Solutions
New Discussion

errors in syslog

 
SOLVED
Go to solution
Brian Bientz
Advisor

errors in syslog

We are getting the following errors in syslog:

Jan 14 08:50:01 prodcl1b vmunix: SCSI: Request Timeout; Abort Tag -- lbolt: 829523321, dev: 1f036000, io_id: 389644d
Jan 14 08:50:02 prodcl1b vmunix: LVM: Recovered Path (device 0x1f036000) to PV 1 in VG 0.
Jan 14 08:50:02 prodcl1b vmunix: LVM: Restored PV 1 to VG 0.
Jan 14 08:50:04 prodcl1b vmunix: LVM: Failed to automatically resync PV 1f036000 error: 5


Can anyone provide some insight? At the time the messages were appearing in the syslog, performance on the node was pathetic. A simple ls command was taking many seconds to return. A number of processes were timing out.

Any insight would be appreciated. Thanks!
10 REPLIES 10
Sunil Sharma_1
Honored Contributor
Solution

Re: errors in syslog

Hi,

These are SCSI errors. it seems your one disk has problem.


0x1f036000 suppose to be minor number of disk. but it suppose to be only 8 digit hear it's 10 digit.

once again check the syslog and find out the disk whihc has problem.

Sunil
*** Dream as if you'll live forever. Live as if you'll die today ***
Jakes Louw
Trusted Contributor

Re: errors in syslog

Hi
You are experiencing hardware errors on a disk.
Do the following:

ll /dev/dsk | grep 1f036000

This will give the CTD file that is giving the errors.
Do a pvdisplay on this disk to see which VG, but judging from the output you sent, I reckon this is VG00.

I hope you have root mirrored!

The performance drop is caused by the system processing I/O resubmits due to excessive error recovery.
Trying is the first step to failure - Homer Simpson
Jakes Louw
Trusted Contributor

Re: errors in syslog

Just a correction to myself: you do have mirroring, as the message "Restored PV 1 to VG 0." indicates that a mirrored pair has had one of the disks come back from an error.
The failure to resync, however, indicates a terminal failure.

Report to HP immediately if you have a contract, or else tell your boss you need money for a Time and Material call!
Trying is the first step to failure - Homer Simpson
Dietmar Konermann
Honored Contributor

Re: errors in syslog

The dev_t 0x1f036000 is major# 0x1f (31), minor# 0x036000.

# lsdev -b 31
Character Block Driver Class
188 31 sdisk disk

So the disk at c3t6d0 looks suspicious.

Best regards...
Dietmar.
"Logic is the beginning of wisdom; not the end." -- Spock (Star Trek VI: The Undiscovered Country)
Cheryl Griffin
Honored Contributor

Re: errors in syslog

# ioscan -fnC disk
Make sure the disk shows "claimed" and has a valid description

Test the disk with dd.
# dd if=/dev/dsk/c3t6d0 of=/dev/null bs=64 count=1000
If successful, the results of this command will look similar to:
100+0 records in
100+0 records out

Look for any of the following:
*IO errors
*the number of blocks read in do not match was is read out
*the command hangs or immediately produces an error

# diskinfo /dev/rdsk/c3t6d0
Look for any of the following which indicate a HW problem:
*the command produces an error
*disk size reports zero or is incorrect
"Downtime is a Crime."
Stuart Abramson_2
Honored Contributor

Re: errors in syslog

1. Get the "dev:" entry from the lbolt:

# dmesg | grep lbolt | grep dev:

SCSI: Abort -- lbolt: 18346341, dev: e7015000, io_id: 122e9a3
SCSI: Request Timeout -- lbolt: 18351441, dev: e7015000
SCSI: Abort -- lbolt: 18351441, dev: e7015000, io_id: 122e9be
SCSI: Request Timeout -- lbolt: 18356641, dev: e7015000
SCSI: Abort -- lbolt: 18356641, dev: e7015000, io_id: 122e9cf
SCSI: Request Timeout -- lbolt: 18362141, dev: e7015000
SCSI: Abort -- lbolt: 18362141, dev: e7015000, io_id: 122e9e0
SCSI: Request Timeout -- lbolt: 74105435, dev: 1f000000
SCSI: Abort Tag -- lbolt: 74105435, dev: 1f000000, io_id: 4ead34

Here we have two:

1f
e7

2. This is the major number of the device in question. Convert the first
two digits of the device from hex to decimal:

# printf "%#d\n" 0x1f
31

3. find out what driver this major number is. It tells us the type of
device:

# lsdev 31

Character Block Driver Class
188 31 sdisk disk

So, this is probably a disk !


4. Find the device file entry from the remainder of the lbolt error:

SCSI: Abort Tag -- lbolt: 74105435, dev: 1f000000, io_id: 4ead34

This is the minor number for the device that is failing.

a. Block device:

# ll -R /dev/ | grep 31 | grep 0x000000

brw-r----- 1 bin sys 31 0x000000 Jul 15 16:25 c0t0d0

Or:

b. Character Device:

# ll -R /dev/ | grep 188 | grep 0x000000
crw-r----- 1 bin sys 188 0x000000 Oct 11 07:15 c0t0d0

5. Find the Hardware Address:

# lssf /dev/dsk/c0t0d0
sdisk card instance 0 SCSI target 0 SCSI LUN 0 section 0
at address 0/0/0.0.0 /dev/dsk/c0t0d0


6. Find the type of device:

# diskinfo /dev/rdsk/c0t0d0# diskinfo /dev/rdsk/c0t0d0
SCSI describe of /dev/rdsk/c0t0d0:
vendor: DGC
product id: C2300WDR1
type: direct access
size: 4102875 Kbytes
bytes per sector: 512


So, we have a Nike disk at hardware address 0/0/0.0.0, device file
/dev/dsk/c0t0d0
David_246
Trusted Contributor

Re: errors in syslog

Hi,

Looks like /stand is having problems, at least your underlying disk has.
As your root is mirrored there is no real problem, your system keeps running as it should. However, you lost your backup disk now. Ones your other disk will be failing too, you have a problem.

For now try to find out which disks are used for your /dev/vg00/lvol1 using "lvdisplay -v /dev/vg00/lvol(0)1"
See with ioscan -fnC disk if indeed one of these disks is reporting the errors. vgdisply -v vg00 should give you some more info as well.
Look up the /var/adm/syslog/syslog.log for detailed errors. You must be sure to replace the correct disk! Also always replace a disk with exactly the same type of disk. This prevents awfull things to happen.

An LVM manual is handy in these times. www.google.com is it as well. Good luck!

Regs David
@yourservice
V. Nyga
Honored Contributor

Re: errors in syslog

Hi Stuart Abramson

you should check this thread:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=351828 ;-)

Brian,
sorry for using your thread for chat.

Volkmar
*** Say 'Thanks' with Kudos ***
Daniel Tan_3
New Member

Re: errors in syslog

if it is for a tape device, what do you use to check the hardware address?

lssf /dev/???
Jakes Louw
Trusted Contributor

Re: errors in syslog

/dev/dsk or /dev/rdsk
Trying is the first step to failure - Homer Simpson