Persistent "SCSI: Request Timeout" errors in syslog

[Stephan Grob]

Almost every morning around 4:10AM I see the following entry in syslog:

Aug 30 04:11:35 xxxxxxx vmunix: SCSI: Request Timeout; Abort Tag -- lbolt: 684229598, dev: cb044200, io_id: 46239f6
Aug 30 04:11:35 xxxxxxx syslog: FNSOD:SCSI interface returns SCTL_INCOMPLETE.

The device that's associated with this is a HP 2200MX optical jukebox. It is connected via a Y cable to another server which serves as the failover in this cluster. The failover server does no work against the jukebox unless the package is active there.

I have gone through the crontab many times over and cannot seem to find anything around 4:00AM that would cause this. The application running on this server is FileNet Image Services and the OS is HP-UX 11i (11.11).

Any ideas, thoughts, etc are welcome. This has been happening for about a year and a half and I just can't seem to put my finger on it.

Thanks,
Stephan Grob

[melvyn burnard]

Check ALL connections and terminations on ALL devices in the chain, looks lik eyou may have one of these issues.

[Simon Hargrave]

I have seen something similar before.

Running an ioscan when a tape backup is running can cause temporary SCSI errors.

Check your crontabs for any ioscan jobs, which may include SysInfo etc which will also call this.

[Pedro Cirne]

Hi Stephan,

This may be related with the jukebox software agging/archiving data...

Is one of the servers rebooting at 04:10AM?

Pedro

[Stephan Grob]

Melvyn - We actually had a problem with this that was just discovered a few months ago. We had an extra terminator in the chain. Unfortunately the problem persisted even after it was removed.

Simon - I did a grep on all cron jobs on both servers in the cluster and nothing is running ioscan or SysInfo.

Pedro - Nope. No reboots. The software itself is not doing any different work at 4AM than 4PM.

-Stephan

[Devender Khatana]

Hi,

It could be a result of ioscan run while collecting nickel outputs. Is nickel scheduled for approximately this time?

It should be some utility which internally runs ioscan.

HTH,
Devender

[Torsten.]

Hi Stephan,

I would suggest to NOT use a Y-cable, because you have an open SCSI port in the chain. Don't know how sensitive the lib is, but I guess this controller is very sensitive.

Use 2 cables instead.

[Stephan Grob]

Devender - nickel isn't on these servers.

Torsten - Now this might hold some promise. I cracked open the jukebox and saw that it has two interface cards in it. I always thought we only had one. I had already planned on performing a test which will have me disconnect the Y-cable from the failover box and see if the error persists. If that test is successful then I think the two cable option will prevent this error from happening.

If the above mentioned scenario does play out that way, it still perplexes me what is running around 4AM to cause this. Another admin here and an HP tech have looked and we all come up empty handed.

[Torsten.]

Hi Stephan,

if you are using the 2 SCSI controllers in the lib and a Y-cable, the result is a "star", not a BUS.

The only possible and working connection is this:

1 cable from server a to connector A,
1 cable from server b to connector B,
1 cable to bridge the 2 open ports.

What is your current layout?

[Stephan Grob]

Hey Torsten,

Both SCSI interfaces in the library aren't currently being used. Here's how it looks right now:

terminator on single-ended interface B
terminator on differential interface A
Y cable coming from server A & B to differential interface B

I've got our HP tech coming onsite this afternoon to hash this out with me. I read through the manual just a while ago and if I'm reading it correctly, we're missing two terminators on the controllers in the library. I could be misreading this so hopefully this guy will set me straight.

-Stephan

[Torsten.]

You are using only one scsi card in the lib because you have only 6 drives or less, I assume. It's alway refered to have one terminator on a unused bus. So you have to install 2 single ended terminators behind each other and 1 differential terminator on the unused differential pair of ports (back controller). Now you have one open port and the y-cable. But anyway, I don't like this cable being there. Try to use 2 cables.

A totally different idea is to check the firmware of the optical drives and update, if necessary.

[Steven E. Protter]

It is possible to increase the timeout parameters on the drives.

The HP response center once recommended I tried that and sent me the procedure. The procedure is lost.

You should be able to get it from your friendly, local Response Center.

I would say that unless the drives are very busy, you should not have to change the timeout figures.

Something else may be wrong, and a disk may be approaching failure. Perhaps mstm/cstm or xstm tests are in order.

SEP

[Matthew Ghofrani]

Here is the scoop;

1. You need to change the default time-out from 30 sec to something bigger then 90 sec.
2. If you have ISEE (HP's free hardware moniter) installed , the default ioscan time is at 4:00 AM.
3. There has been a recent patch recall specifically dealing with this issue of scsi cables & jBoxes.
4. There is also a known issue with dual hbas (if you use them) which will be fixed with higher driver firmware patch.

Matthew Ghofrani
From Boston

[Stephan Grob]

Well ... on the terminator issue I was reminded that we have self terminating cables. I can't use the other interface in the library since we're only using 6 drives; there's nothing connected to the second interface. So, I basically have no choice but to stick with the Y-cable unless I want to begin doing a manual failover for this cluster which I don't want to do.

I'll check on the other stuff some of you have mentioned.

-Stephan