topic Re: SCSI problem help needed identifying disk in Operating System - HP-UX

SCSI problem help needed identifying disk

Tony Horton — Tue, 28 Oct 2003 20:09:59 GMT

Hi,

I have a scsi problem but I'm not sure which disk it is. The output from dmesg is as follows.

Oct 29 12:04
...
- lbolt: 1061326722, bus: 6
scb->cdb: 28 00 03 c5 f3 80 00 00 20 00
scb->cdb: 28 00 00 1a 3d a0 00 00 60 00
SCSI: Resetting SCSI -- lbolt: 1069236335, bus: 6
SCSI: Reset detected -- lbolt: 1069236335, bus: 6
scb->cdb: 2a 00 00 00 0c 40 00 00 02 00
scb->cdb: 28 00 00 5d f9 00 00 00 60 00
SCSI: Resetting SCSI -- lbolt: 1069793138, bus: 6
SCSI: Reset detected -- lbolt: 1069793138, bus: 6
scb->cdb: 28 00 00 13 76 b0 00 00 50 00
scb->cdb: 28 00 00 10 38 20 00 00 60 00
SCSI: Resetting SCSI -- lbolt: 1074152496, bus: 6
SCSI: Reset detected -- lbolt: 1074152496, bus: 6
scb->cdb: 28 00 00 3a c1 80 00 00 10 00
scb->cdb: 28 00 00 08 5d 00 00 00 80 00
SCSI: Resetting SCSI -- lbolt: 1093971586, bus: 6
SCSI: Reset detected -- lbolt: 1093971586, bus: 6
scb->cdb: 2a 00 00 bb 9b 3c 00 00 02 00
scb->cdb: 28 00 00 2d 25 80 00 00 80 00

I'm assuming that the bus 6 refers to the scsi devices with device files c6t2d0 (for example). There are only three disks on this channel. scsi id's 2 12 and 13.....

Any help in identifying the culprit greatly appreciated.

when these messages appear the system seems to freeze for 20-30 seconds, and then resumes normal operation.

Regards,

Tony.

Re: SCSI problem help needed identifying disk

KCS_1 — Tue, 28 Oct 2003 20:24:02 GMT

Hi,

Does it remain any messages in the syslog?

If installed the STM Diagnosic tool, have a look at the eventlog in /var/opt/resmon/log directory.

It will let you know and find more advanced symptoms.

Re: SCSI problem help needed identifying disk

Tony Horton — Tue, 28 Oct 2003 20:40:31 GMT

Hi Patrick,

The messages also appear in the syslog, but no more than the messages from dmesg (although different order).

Oct 29 11:12:06 ceeng vmunix: scb->cdb: 2a 00 00 00 cf f0 00 00 10 00
Oct 29 11:12:06 ceeng vmunix: scb->cdb: 28 00 00 5d f9 00 00 00 60 00
Oct 29 11:12:07 ceeng vmunix: SCSI: Resetting SCSI -- lbolt: 1171439936, bus: 6
Oct 29 11:12:07 ceeng vmunix: SCSI: Reset detected -- lbolt: 1171439936, bus: 6

I checked the event log as suggested, but the last entry was from August 3rd. It is however an entry for one of the disks I thought might be the problem, and was a recoverable read error.

I'll do some non-destructive STM tests on that disk and see if anything shows up. Thanks for the tip.

Regards,

Tony.

Re: SCSI problem help needed identifying disk

Rajeev Shukla — Tue, 28 Oct 2003 20:49:18 GMT

It appears to me that some disk is going offline and coming back online which is also freezing the system (because after coming online it is syncronizing which is freezing the system)

What you can parallely do is have a look at vgdisplay -v to see if you find any disk showing offline/ or stale LV

ioscan can also tell you when it happens.

Or have a look at STM logs if online diagnostics is installed.

Re: SCSI problem help needed identifying disk

KCS_1 — Tue, 28 Oct 2003 21:02:54 GMT

Hi,Tony

Then, what's the output of these commands which followed ?

# ioscan -funCdisk

Is there NO_HW or UNCLAIMED disk?

# vgdisplay -v | more

Any disk LV is Staled or not weird Symptoms of disk?

If there are some unnormal the output, take have this testing,

# dd if=/dev/rdsk/cXtYdZ of=/dev/null bs=1024

-Read/write looping test on specified disk

Good luck!

Re: SCSI problem help needed identifying disk

Tony Horton — Tue, 28 Oct 2003 23:26:34 GMT

Thanks Guys,

Tried all suggestions but turned up blank. Couldn't trigger the messages with the dd either. We seem to get the pauses multiple times a day, but the messages in the log seem to be less frequent, I suspect one of the disks is on the way out, but it is managing to read/write after a number of retries, and succeeding most times before the timeout period. I had a similar problem on a linux box before. Changing the offending disk fixed the problem.

There is actually a swap LV on this disk, could be the cause of the pauses. Maybe I should transfer the swap to another disk!

I couldn't see anywhere in the error messages where it specifically points to a particular disk, which made me wonder whether there was in fact a problem with the controller.

Regards,

Tony.

Re: SCSI problem help needed identifying disk

Rajeev Shukla — Tue, 28 Oct 2003 23:50:53 GMT

Hi Tony,
The best i suggest is since this is a production machine, and dont take a chance till the disk completely fails. Install "Event Monitoring Service" which is part of Online diagnostics, this will tell you about any powerfails or I/O error on disk with its hardware address and you can locate the disk easily and schedule some down time and replace the disk.
Dont let users get annoyed by slow preformance of the server.

Re: SCSI problem help needed identifying disk

Tony Horton — Wed, 29 Oct 2003 00:12:56 GMT

Hi Rajeev,

Unless it has died for some reason, EMS should be running, although it hasn't reported any problems since August. I agree with you about replacing before it dies, it's just a matter of knowing which one :-).... I ran STM and two of the 3 disks on that chanell showed read and write errors when I did an info, the ones playing up a quantum atlas 10K III drives, I have another four of them on the other channel, and all of them show zero errors. I checked the scsi cables, and one connector which went to the last disk on the chain didn't have it's thumb screws tightened up, have done that now just in case, but suspect it is something more than that.

Regards,

Tony.

Re: SCSI problem help needed identifying disk

Joe Short — Thu, 30 Oct 2003 09:16:33 GMT

It could also be the HBA. You said STM shows errors on 2 of the 3 disks. If the cables are all properly connected, so it may be somewhere else. Have you tried reseating the HBA? What model server is this? Is the HBA in the correct slot? Some servers have priorities for the expansion slots, higher performance cards should be installed in specific slots.

Re: SCSI problem help needed identifying disk

Tony Horton — Thu, 13 Nov 2003 18:51:32 GMT

Looks like it is one particular disk. I finally got some more meaningfull messages last night. A whole pile of write errors on one of the three disks.

I did pull the Controller out and reseat it, no change. It is in the correct slot (it's an L1000) so I don't have any choice but to put it in the "turbo" slots.

I suspect that the disk has been marginal for some time, and it's finally started to get to the point where the errors aren't recoverable. It's one of two disks in a mirror, so I should be able to reduce the mirror and do some more extensive tests on it now that I know which one :-)

Regards,

Tony.

Re: SCSI problem help needed identifying disk

Tony Horton — Thu, 13 Nov 2003 20:25:50 GMT

Now that I know which disk is suspect. Whats the best way of making sure its a media problem.

should I run mediainit, stm exerciser (this doesn't ever seem to do much). I'd like to get some hard evidence for warranty claim.

Regards,

Tony.

Re: SCSI problem help needed identifying disk

Tony Horton — Thu, 13 Nov 2003 22:42:26 GMT

Hmmm, Helps if you increase the timeout in the exerciser tool options, and set it to maximum coverage :-)

Re: SCSI problem help needed identifying disk

Michael Steele_2 — Thu, 13 Nov 2003 23:01:01 GMT

Attach LOGTOOL report and 'ioscan -fnkC disk':

STM > TOOLS > UTILITY > RUN > LOGTOOL > FILE > VIEW > RAW SUMMARY.

Re: SCSI problem help needed identifying disk

Tony Horton — Thu, 13 Nov 2003 23:21:47 GMT

Hi Michael,

I thought the problem was probably termination or cable or controller before, as all of the disks on the 0/4/0/1 path were showing similar numbers of errors.

I'm getting a new ultrium on Monday, and the IBM drive has already been replaced (it died completely). It's scsi ID 13 that seems to be the main culprit, ie the Atlas 10K3 73GB drive (based on this mornings syslog messages)

I have added the syslog messages from this morning on the end of the summary.

Regards,

Tony.

Re: SCSI problem help needed identifying disk

Tony Horton — Thu, 13 Nov 2003 23:23:42 GMT

Sorry, once again with a .txt extention

Re: SCSI problem help needed identifying disk

Steven E. Protter — Thu, 13 Nov 2003 23:26:42 GMT

If you have not blown away /etc/lvmtab, the script I'm attaching will help you identify the disk. It must have been configured into a volume group for this script to work.

SEP

Re: SCSI problem help needed identifying disk

Tony Horton — Thu, 13 Nov 2003 23:33:50 GMT

Sorry forgot the ioscan before.

Re: SCSI problem help needed identifying disk

Michael Steele_2 — Thu, 13 Nov 2003 23:34:20 GMT

LOGTOOL didn't have enough errors to indicate a failure but when c6t13d0 is Powerfailed "...LVM: vg[1]: pvnum=5 (dev_t=0x1f06d000) is POWERFAILED..." usually the firmware needs upgarding or the timeout needs to be increased.

Verify the disks with this command to find vg1 and pv 5.

# strings /etc/lvmtab

First vg is vg1.
5th disk is pv 5.

######################################

(239) 0/4/0/1.13.0 = QUANTUMATLAS10K3_73_WLS
(208) 0/4/0/1.2.0 = SEAGATEST318203LW
(216) 0/4/0/1.12.0 = QUANTUMATLAS10K3_73_WLS

Check the firmware on these. Increase the timeout:

# pvchange -t 160 /dev/rdsk/cXtYd0

Re: SCSI problem help needed identifying disk

Michael Steele_2 — Thu, 13 Nov 2003 23:40:42 GMT

Since all of your errors are isolated to the c6 controller I'd be interested to know whats on c6. Use 'pvdisplay -v' and 'bdf' to see whats on these disks.

Use 'diskinfo' to find the firmware version. See if there are upgrades.

# diskinfo -v /dev/rdsk/cXtYdZ

Please attach:

sar -d 5 5
sar -u 5 5

Re: SCSI problem help needed identifying disk

Tony Horton — Thu, 13 Nov 2003 23:58:35 GMT

Hi Michael,

There are only 3 devices on that controller they are an 18GB Quantum Atlas 10K3, a 73GB Quantum Atlas 10K3 (both in a kingston data silo case, they aren't "HP" disks). The third device is a segate 18GB disk which is a genuine HP external disk.

The 73GB drive has one 70GB lvol on it which is one half of a lotus notes mirror. When the errors started this morning database compacting was occuring.

The 18GB quantum drive has secondary swap, progress 4GL compiled programs, spool files from the progress system, and a D3 (pick) raw partition (note that all of these are mirror copies as well).

The 18GB seagate disk has a lvol for squid, an lvol for samba, and an lvol for the test progress database, none of these are particulary high volume, none are mirrored.

The sar output is probably a bit pointless at the moment as I'm running a non destructive read/write excerciser on the 73GB disk, but I've attached it anyway.

Unfortunately since the disks are non-hp, firmware upgrade is out of the question. Maxtor don't seem to release new versions.

Regards,

Tony.