"Tape Drive Status Change" alerts from 7.0 Insight Agents

[Steve Kadish]

Hi,

We have a HP MSL5026 tape library with two drives which we use with Arcserve. A while back, I was regularly getting the following errors during Arcserve backups:

"Tape Drive Status Change. The Storage Agent has detected a change in the status of a tape drive. The current physical drive status is indicated in the cpqTapePhyDrvStatus variable."

One of the drives would be reported as "offline," and then a few seconds later, as "ok."

When I went to the 6.40 agents, the errors disappeared. On Friday, I upgraded to the 7.0 support pack, and now they are back - I got four sets during the weekend backups.

Note that there are not any actual errors as reported by Arcserve.

Also, I am using the 6.1.540.100 version of the 64-Bit/66-Mhz Dual Channel Wide Ultra3 SCSI Controller Driver rather than the latest because an HP tech advised me that Arcserve users had had success with that driver.

Does anyone have any insight on what might be causing these messages?

Thanks,
- Steve

[James Kennedy_4]

If your MSL5026 is connected to a storage router, Fibre to SCSI (such as the NSR1200), make sure the firmware on the storage router is up to date.

Make sure the firmware on the MSL is up to date, and both tape drives themselves (if yours has 2).

Are you connected to a SAN or anything complex? or just directly to one server. Let me know more about your environment and I might be able to be more informative.

[Rob Buxton]

Steve,
I seem to recall seeing an HP Advisory regarding something like this.
Are you registered for any of the Active Answers stuff? You may want to look at it.

Alas, I'm a tad ruthless once I've read e-mail and I don't have a link for you.

[Steve Kadish]

Thanks, guys. The library is directly connected via SCSI to the server. No SAN switches or fiber involved. And it and the drives have, if not the lattest, extremely recent software.

Rob - I'm not familiar with "Active Answers," can you point me in that direction? I do recieve the weekly driver update / support bulletin email, but I may have missed this bulletin...

Thanks,
- Steve

[James Kennedy_4]

Steve,

It doesn't sound too complex. Here's what you may want to try.

Go to Control Panel -> HP Management Agents

In the services tab, remove SCSI information to the Inactive agents list.

Hopefully you are using ARCServe 9.0 or higher and have the latest patches for that. If you are not, you should really consider upgrading your version of ARCServe.

Check your event logs for any SCSI Timeout errors, or other strange events that occur during the backup process. They can sometimes be more informative than ARCServe's logs.

[Pat Wilson]

What will be missed if the 'SCSI' is moved to the inactive agents list?

[Rob Buxton]

Steve,
Good point, they've changed so much over the years it's hard to remember where I registered for what.

But the item I was thinking about came out as one of the Advisories and the e-mails are headed up:
Your Daily HP Driver and Support Alert/Notification
They contain details of new Drivers and Advisories of known issues.
I'll try and dig around and see if I can find out where and how I registered for it.

[Steve Kadish]

Hi all,

Thanks for the suggestions. Since I haven't been able to find any old bulletins on this issue, and I don't want to deactivate the SCSI agents, I have opened a ticket with HP and we're proceeding to troubleshoot starting with hardware.

If/when there's an eventual solution, I'll post it on this thread!

- Steve

[Pat Wilson]

As we were having the same issue (with the same model tape library), and also having some failed backups using BackupExec, I pushed the downgrade to the version 6.40 Management agent only. All the other versions from the Windows 2000 v7.00 management pack were retained.
I had also noticed some device timeouts in the event logs approximately every 5 minutes (coincidentally my Hardware Status Polling interval). So far the errors have disappeared, and the SCSI bus is still being monitored (but no more device timeouts).
We are doing some more testing/monitoring, and I'll keep you posted...

[Dan Raney]

I'm just wondering if this worked for folks. We're in a similar to situation to Pat Wilson - using BackupExec instead of ArcServe. Does downgrading the management agent work?

[Steve Kadish]

Sorry, I forgot to update the thread. I don't have a very satisfying resolution to relate. HP tech support told me that the SCSI agent constantly polls the SCSI bus for status and that if the bus was very busy - as it presumably is while we have two backups running simultaneously - that it was possible it would return false positives. They recommended that if there were no actual problems with the SCSI operation, which there weren't, that I simply deactivate the agent.

Thanks,
- Steve

[Dan Raney]

You're right. That's not very helpful response from HP, but it's more than we've been able to extract from them. It's a little troubling, because if you disable the agent and you actually DO have a problem, then you don't have a very proactive way to find out about it. Alternatively, I guess you could just ignore the warnings as well.

Anyway, I really appreciate you coming back and updating the thread. Thanks very much!

[T Bell]

Anyone have any news on when new management agents will be released to address this issue?

We're experiencing the same annoying "offline/online" storage status messages, snmp traps, etc. on two new DL380 servers connected to SSL 1016 autoloaders.

[Dan Raney]

Hey -

We followed up with the Storage Works folks, and got this reply:

"...[since] these are direct SCSI attached, you can disable the SCSI Info from the HP Management Agents. This can be accomplished by going to control panel, HP management, then select SCSI Info then Remove. This will put SCSI Info in the deactive window on the right side. We don't recommend stopping the storage agent because there are other agents that will restart it, like the foundation agent. This was a problem in the 6.40 agents as well.

Hp Storage Engineering is aware of this and I expect a more permanent solution will be forthcoming."

[J_139]

Hello,

I am having the same exact problem. If anyone hears about a fix for this please let me know and I'll do the same.

Thanks!
J.S.

[Alain Staelen]

I have reported the issue again to HP support, I opened a new call and after a week I basically got the same response back from HP :
- disable the storage agents if it bother you
- no we won't fix it for you
- we are planning a fix for mid 2005 when we change the agents architecture.

This is unacceptable for us and I would suggest that everybody with the same problem calls their HP contacts and asks for a more "responsable" answer.

This is not an answer for a company that wants to deliver enterprise class support.

[Michael Jay_2]

OK, am I missing something here? We are being flooded with the same annoying messages. So on the SIM server I went into Options-Events-SNMP Trap Settings and went to the cpqscsi.mib, located two event named: cpqTape4PhyDrvStatusChange and cpqTape5PhyDrvStatusChange (these are the only two I could find that had the exact text in the messages we're getting) and set value to "NO" on "Enable Trap Handling" but we're still gettings the messages.

In the SIM online help in this same area it says:

Enable Trap Handling - Most traps are enabled. Trap handling gives you control over the volume of messages. Turn off nuisance messages, such as unnecessary informational messages, or repeated trap messages for an event that has not been corrected.

Isn't this what we're trying to do?

Thanks.

[Steve Kadish]

Michael,

What version of the agents are you running? Since we upgraded to PSP 7.1, I get these errors occasionally, always during very heavy backup periods when our library is working overtime, but I'm no longer flooded with false alarms. Is it possible that there is an actual problem with your SCSI connections or subsystem?

- Steve

[Michael Maiolo]

We have the same problem with all our MSL libraries as well.

Most of ours are Fibre attached through and NSR. So it's not soley a SCSI based issued.

There is an HP article, customer advisory EX040107_CW01 which descibes the insight management agents generating event ID 1192 during a backup. I suspect it's also the same issue causing the flood of message in SIM.

To the agents in appears that the library is going offline, then online again.

The fix in the article is to upgrade to 7.1 version of the agents.

My temporary fix is to disable that portion of the SNMP agent. You can still have the SCSI agent enabled on the server to trap SCSI errors of another nature, but through SIM disable the status change portion of the MIB. I believe that the trap numbers 5018 & 5019. (May be a few others also)

[Michael Jay_2]

Steve,

We're running everything from 5.5 to 7.0. I did notice that none of the 5.5 backup servers generate this error but 1/2 of the 6.4 servers do and all of the 7.0 backup servers do.

No, I don't believe there are problems with the SCSI subsystems. I believe, like others have stated, that these annoying errors only occur during backups when the device is busy when it is queried and so SIM reports it as offline and then 10 minutes later it reports it OK.

BUT, my point of joining this thread is to find out why (either what I'm doing wrong, or to correct my understanding)configuring the SNMP Settings for the traps to NO - under the Enable Trap Handling option does not work as advertised in the online help.

I am glad, however, to hear you get these annoying messages less with 7.1.

Thanks.

MJ

[Michael Jay_2]

Hey, I did something different and was able to get rid of those annonying "Tape Drive Status Change" messages.

In addition to going in under Options-Events-SNMP Trap Settings and setting to "NO" the Enable Trap Handling for those two events I also changed both events to information and now we're NOT bothered!

BUT, I'm sure this is only because we've set out SIM server to not e-mail or page us for informational messages. I still think it should have worked regardless of the Severity setting, but I'll live with it.

MJ

[Steve Kadish]

Hi Michael,

The "approved" workaround for this problem involves a change on the server, not in SIM. Open the "HP Management Agents" control panel, and on the Services tab, move "SCSI information" from "Active Agents" to "Inactive Agents."

This will stop monitoring of the SCSI subsystem on the server. The best solution, though, is to update the agents to 7.10.

- Steve

[Michael Jay_2]

Seems kinda silly to me to
1) Have to reconfigure all the backup servers to deactivate the SCSI Information agent
and
2) Potentially miss out on other important scsi subsystem messages.

I like my way better but I wish I could get an answer as to why the snmp setting does not work the way I think it should.

MJ

[Michael Maiolo]

Disabling the SCSI agent means that you will not be alerted if a disk in your system fails.. I know which I'd prefer!

[Annette Jones_3]

Guys,

In the end, cause the traps are persistent and in the majority they are ignored, cause it's just busy tape libraries. I setup a managed task to clear the alerts.

If there is a problem with the library or drive it will manifest in other ways, hardware fault or software when loading/unloading tapes.