I've got a real head-scratcher. The good news is that I have PLENTY of symptoms! Boy do I have symptoms.
It all started last week, I did a kernel parameter change and rebooted the box. Came up fine, but it never occurred to me to look and make sure the tape drives (2 DLTs in a TLX891 library) were still there, since why on earth would they disappear?
Fast forward to Sunday morning, our weekly backup (this is just a development server) fails when "robot move slot 0 drive 0" fails. I check it out, and run a hwmgr -show scsi and discover that the changer seems okay, but the drives have lost their device names (they were tape0 and tape1; now they're just nothing). I'd like to print a copy of the original hwmgr but it's long gone, sadly.
I'd run into a similar situation when we'd had to replace the entire library to fix a robot problem; I ended up deleting the drives, running hwmgr -scan component, and then using dsfmgr to move the device files to where they oughta be. So I said, no problem, I'll just delete 'em again, which I did.
Then I ran the hwmgr -scan scsi, and it just hung. Never came back. Eventually I left the data center; I assume it must have died, but I have no idea if it ever returned anything.
Anyway, today I finally get back to fiddling with it and trying to bring the drives back to life. A little research reveals I should have run "hwmgr -scan component," so I do that and monitor EVM for the success response, which never comes. I do a ps to see if the scan's still running, but it quit. I'm thinking maybe it wrote something to my mail, but unfortunately the mail file filled up, which is when I noticed the SECOND symptom: SCSI CAM errors, whatever those are, and tons of 'em, all looking like this:
Formatted Message:
SCSI event
Event Data Items:
Event Name : sys.unix.binlog.hw.scsi
Priority : 700
PID : 614
PPID : 1
Event Id : 110080
Timestamp : 25-Mar-2008 22:00:03
Host IP address : 170.212.26.37
Host Name : pdsdev
User Name : root
Format : SCSI event
Reference : cat:evmexp.cat:300
Variable Items:
subid_class (INT32) = 199
subid_num (INT32) = 1
subid_unit_num (INT32) = 96
subid_type (INT32) = 34
binlog_event (OPAQUE) = [OPAQUE VALUE: 1096 bytes]
============================ Translation =============================
Sequence number of error: 2593
Time of error entry: 25-Mar-2008 22:00:03
Host name: pdsdev
SCSI CAM ERROR PACKET
SCSI device class: DEC SIM
Bus Number: 1
Target number: 4
Lun Number: 0
Bus 1 is definitely where my drives and changer are, so this makes sense. No idea why they're throwing out errors, but they're coming fast and furious.
Then I go into scu, to see what I can see, and I run "scan edt," and it says:
Scanning all available buses, please be patient...
And then just sits there. I'm not a patient man by any means, but it's been almost 2 hours, I don't think it's coming back any time soon.
Any ideas? I'm wondering if I can just reboot the box to get it to come back to life, but that may not happen this week because the developers are working fast and furious. Even then, I'm not sure if a reboot will do a lick of good, since I'm guessing last week's reboot screwed it up. I find it hard to believe the kernel parameters (just a few things to line up with Oracle 10g, which we want to install in a few weeks) would cause a problem with the SCSI path.
I'm gonna send stuff over to the hardware vendor, but I'm hoping that this is just a funky OS thing I can fix myself.
THANKS!!!
