Syslog.log error question

[Marty Metras]

This morning I woke up to a whole mess of "Event Monitoring Service Event Notification" in my email. 7 of them claming Disk at hardware path.... : Software configuration error.
This was for 7 diferent hardware paths.
Also in syslog.log I had a mess of these messages:
"Apr 1 05:11:54 kncryvis vmunix: SCSI: Reset detected -- lbolt: 403891060, bus: 10"
and
"Apr 1 05:19:01 kncryvis EMS [2797]: ------ EMS Event Notification ------ Value: "MAJORWARNING (3)" for Resource: "/storage/events
/disks/default/0_12_0_0.10.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/
resdata -R 183304238 -r /storage/events/disks/default/0_12_0_0.10.0 -n 183304219 -a "

Other than this the system seams to be running fine.

Can you help me figture what is going on?
Marty

[John Poff]

Hi,

It looks like your SCSI bus at 0/12/0 freaked out for a second, and all the SCSI devices on that chain reset. Maybe a cable was jiggled, or maybe one of the SCSI devices is starting to act up. Are those SCSI devices disk drives? If so, are they all happy this morning?

JP

[john korterman]

Hi Marty,
try to execute the specified command:
# /opt/resmon/bin/
resdata -R 183304238 -r /storage/events/disks/default/0_12_0_0.10.0 -n 183304219 -a "

it gives you the details.

Check also for dead disks:
# vgdisplay -v | grep stale

If anything appears, check all your disks.

regards,
John K.

[Marty Metras]

John,
The other messages are on bus 1/4/0.
It shows 7 drives on 2 buses.
The system seems to be happy this morning.
What can I look for to see if there is a pending problem?
Marty

[Steven E. Protter]

Bad disk.

Do a make_tape_recovery tape if its a vg00 disk.

fbackup any databases with the databases down.

Call Support or make other arrangements to replace the disk.

Post back here if you need a procedure for making an Ignite tape or recoverying from it.

SEP

[Pete Randall]

Marty,

The fact that it happened to several disks on two different busses pretty much shoots down the bad disk theory - for me at least.

It seems more likely to be a random reset, maybe caused by power issues (spikes, drops) or something. I would monitor your syslog closely for the next couple of weeks. If there are no recurrences, wonderful. If there are, get HP involved to figure out exactly what's going on.


Pete

[Marty Metras]

Hi everybody,
Here is what I found.
We test the backup generator on Tuesday mornings. I got a message around 3:00am and 4:00am when the switch it on and off. This is an hour before the drive error.
Around 5:19am the messages about the drive/bus error happened.
Now that I cannot find any thing not working,
No errors than in syslog.log.
I traced these hardware addresses back to 7 Hard Drives on 2 busses that is one Volume Group.
This is one mount point where I store backups for just in case. I only use this.
I tried to get an error by writing a few gigs of data and reading it. No errors.
Here is what I decide for know. One of a few things happened. The system spike from the power switch that didn't show up for an hour.
7 old drives are about ready to die.
Or maybe it is an April Fool's joke the system is playing on me. Or tomorrow of the next day some thing will show up and I will be able to find what caused this.
In any case, I find no problems or cause at this point in time.

Thanks for your help people,
Marty

[john korterman · Accepted solution]

Hi again,
The detailed message indicates, as far as I can see, nothing serious. If there was only one message for a hot-swap disk, it could perhaps be explained by someone pulling the disk out and in. But of course not simultaneously for seven disks. Better check the "Execute the following command to.." for the other disks, although I guess they are similar (I assume that the vgdisplay -v | grep stale produced nothing). If the other messages are different then write back.

regards,
John K.

[Marty Metras]

vgdisplay -v | grep stale
Returned nothing.
Thanks,
Marty