I have two drives on a 2nd ds2405 that aren't being seen by controller one. The drives and controller are listed in a "good" state through armdsp -a. I have cycled the DS2405 individually as well as cycled the VA7410 it is attached to.
Next steps were going to be a reseat controller 1 then replace one of the drives.
Attached the good armdsp -t output in the first message. When the array was cycled all drives were seen for a short time however after the rebuild was finished the drives in slot 7 and 15 disappeared from controller 1.
It could be the controller or the disks. Any warnings now? (armdiag -W ).
Go with the drives and monitor. You might want to perform a full diag capture and have it analyzed (armdiag -I(UC i) -if ). The file will contain a history of events that may indicate where the problem(s) are.
Check the disk error logs for signs of possible issues and compare against the other disks.
Disk drives have been replaced and the array is in the process of rebuilding. However all the drives associated with that controller are now missing.
I am going to swap controller boards C1<->C2 and see if the problems follow over to the 2nd controller. Hopefully they will. If that is the case I will swap the bad controller board.
Please see attached output for more information. I will run the diagnostics check once the rebuild is complete.
No disk Page Code logs were attached? You should see read/write/channel statistics when running the armlog -d 'fru' command...
FRU Location ---------------- = M/D1 Disk Address ---------------- = 111 Disk Capacity --------------- = 33.378 GB Disk Product Rev ------------ = HP07 Disk Product ID ------------- = ST336752FC Disk Vendor ----------------- = HP 36.4G Disk Volume Set Serial Number = Disk World Wide Name -------- =
Write Error Counter Page 0x02 Corrected write errors with delay = 0 Corrected write errors with retry = 0 Total Corrected write errors = 0 Total times write ECC used = 145 Total MegaBytes written = 20,668,273 Total Uncorrected write errors = 0
Read Error Counter Page 0x03 Corrected read errors without delay = 15,278,361 Corrected read errors with delay = 0 Corrected read errors with retry = 0 Total Corrected read errors = 15,278,361 Total times read ECC used = 15,278,361 Total MegaBytes read = 371,578,203 Total Uncorrected read errors = 0
Verify Error Counter Page 0x05 Corrected verify errors without delay = 2,189 Corrected verify errors with delay = 0 Corrected verify errors with retry = 0 Total Corrected verify errors = 2,189 Total times verify ECC used = 2,189 Total MegaBytes verified = 494,085 Total Uncorrected read errors = 0
Non-medium Error Counter Page 0x06 Total non-medium errors = 35,453
Temperature and FC link Error Counter Page 0x0D Temperature Data (Celsius) = 26 Temperature Reference Data (Celsius) = 65 Link Failure count, Port A. = 0 Loss of Synchronization count, Port A. = 14 Invalid Transmission word count, Port A. = 1,004 Invalid CRC count, Port A. = 0 Link Failure count, Port B. = 0 Loss of Synchronization count, Port B. = 12 Invalid Transmission word count, Port B. = 906 Invalid CRC count, Port B. = 0
Cache Statistics Page 0x37 Logical blocks sent to an initiator = 2,378,210,751 Logical blocks received from an initiator = 4,145,212,184 Logical blocks read from cache = 1,583,989,233 Read & writes; data lengths <= segment size = 1,643,133,198 Read & writes; data lengths > segment size = 124,079,008
Defect Data No entries in the Defect List.
FWIW:
Link cards and midplanes can also manifest themselves as disk channel errors, but further analysis would be required. I only see the 2nd enclosure reporting disks issues in the attachement and not the entire chain.
armlog -d seems to be having issues reporting everything. Looking into this.
#-> armlog -d JB2/D7
Disk Mechanism Log
FRU Location ---------------- = JB2/D7 Disk Address ---------------- = 38 Disk Capacity --------------- = 66.757 GB Disk Product Rev ------------ = HP0A Disk Product ID ------------- = ST373405FC Disk Vendor ----------------- = HP 73.4G Disk Volume Set Serial Number = Disk World Wide Name -------- =
Use the armdiag or armlog to review the events prior to the first rebuild failure and go forward, check for any messages pertaining to rebuild failures and reasons.
Think we have identified that particular array as causing us problems before. Somehow it was not labeled as being suspect bad and was since re-connected back into another va configuration.
I am going to pull that 2405 out of the current config to get things back to a "stable" state. We have another unit that is not being used that I will re-add.
I will post back on the progress in case anyone else runs into similar problems in the future.
This is the first time I have removed an entire enclosure. Any feedback about this process?
- Delete newly created LUNS - Down the drives in the enclosure - Modify the security table and remove the associated LUNS - Power off the 2405 and disconnected the fiber cables.
I did notice the following from a previously saved armdsp output. The loop id on the second controller is different from the first. Originally I ignored this but I noticed in the first enclosure the loop id's are the same. I am currently researching.
Controller At JA0/C1: Status:_______________________________Good Serial Number:________________________ Vendor ID:____________________________HP Product ID:___________________________A6255A Product Revision:_____________________HP05 Controller Type:______________________HP StorageWorks Disk System DS2405 Enclosure Switch Setting:_____________0 Drive Address Basis:__________________0 Enclosure ID:_________________________0 Loop Pair:____________________________0 Loop ID:______________________________15 Hard Address:_________________________15
Controller At JA0/C2: Status:_______________________________Good Serial Number:________________________ Vendor ID:____________________________HP Product ID:___________________________A6255A Product Revision:_____________________HP05 Controller Type:______________________HP StorageWorks Disk System DS2405 Enclosure Switch Setting:_____________0 Drive Address Basis:__________________0 Enclosure ID:_________________________0 Loop Pair:____________________________0 Loop ID:______________________________15 Hard Address:_________________________15
Controller At JB2/C1: Status:_______________________________Good Serial Number:________________________ Vendor ID:____________________________HP Product ID:___________________________A6255A Product Revision:_____________________HP05 Controller Type:______________________HP StorageWorks Disk System DS2405 Enclosure Switch Setting:_____________2 Drive Address Basis:__________________2 Enclosure ID:_________________________2 Loop Pair:____________________________1 Loop ID:______________________________47 Hard Address:_________________________47
Controller At JB2/C2: Status:_______________________________Good Serial Number:________________________ Vendor ID:____________________________HP Product ID:___________________________A6255A Product Revision:_____________________HP05 Controller Type:______________________HP StorageWorks Disk System DS2405 Enclosure Switch Setting:_____________2 Drive Address Basis:__________________2 Enclosure ID:_________________________2 Loop Pair:____________________________1 Loop ID:______________________________31 Hard Address:_________________________47
This was found to be a bad fiber cable. After replacing the cable ( next step before giving up and replacing unit ) the sporadic dis-appear and re-appear of drives on that controller has stopped. It has been 3 days with no issues so closing out this thread. Thanks for the suggestions!
