Disk Arrays
cancel
Showing results for 
Search instead for 
Did you mean: 

VA7410 few disks only seen by one controller

dave.s
Advisor

VA7410 few disks only seen by one controller

Anyone have any insight on the attached document?

I have two drives on a 2nd ds2405 that aren't being seen by controller one. The drives and controller are listed in a "good" state through armdsp -a. I have cycled the DS2405 individually as well as cycled the VA7410 it is attached to.

Next steps were going to be a reseat controller 1 then replace one of the drives.

15 REPLIES
dave.s
Advisor

Re: VA7410 few disks only seen by one controller

Attached the good armdsp -t output in the first message. When the array was cycled all drives were seen for a short time however after the rebuild was finished the drives in slot 7 and 15 disappeared from controller 1.

Please find the current state in this attachment.
cnb
Honored Contributor

Re: VA7410 few disks only seen by one controller

It could be the controller or the disks. Any warnings now? (armdiag -W ).

Go with the drives and monitor. You might want to perform a full diag capture and have it analyzed (armdiag -I(UC i) -if ). The file will contain a history of events that may indicate where the problem(s) are.

Check the disk error logs for signs of possible issues and compare against the other disks.

#armlog -d JB2/D7 or use the armperf command.


Rgds,

dave.s
Advisor

Re: VA7410 few disks only seen by one controller

Disk drives have been replaced and the array is in the process of rebuilding. However all the drives associated with that controller are now missing.

I am going to swap controller boards C1<->C2 and see if the problems follow over to the 2nd controller. Hopefully they will. If that is the case I will swap the bad controller board.

Please see attached output for more information. I will run the diagnostics check once the rebuild is complete.
cnb
Honored Contributor

Re: VA7410 few disks only seen by one controller

When did these problems first start?

To check the controller events since Jan 01 2010 at 00:01:

# armlog -c -s 010100012010 >

# more


Anything showing failed?

# armdsp -f
cnb
Honored Contributor

Re: VA7410 few disks only seen by one controller

No disk Page Code logs were attached? You should see read/write/channel statistics when running the armlog -d 'fru' command...

FRU Location ---------------- = M/D1
Disk Address ---------------- = 111
Disk Capacity --------------- = 33.378 GB
Disk Product Rev ------------ = HP07
Disk Product ID ------------- = ST336752FC
Disk Vendor ----------------- = HP 36.4G
Disk Volume Set Serial Number =
Disk World Wide Name -------- =

Supported Log Pages
Page Code 0x0
Page Code 0x2
Page Code 0x3
Page Code 0x5
Page Code 0x6
Page Code 0xd
Page Code 0x10
Page Code 0x2f
Page Code 0x32
Page Code 0x37
Page Code 0x3d
Page Code 0x3e

Write Error Counter Page 0x02
Corrected write errors with delay = 0
Corrected write errors with retry = 0
Total Corrected write errors = 0
Total times write ECC used = 145
Total MegaBytes written = 20,668,273
Total Uncorrected write errors = 0

Read Error Counter Page 0x03
Corrected read errors without delay = 15,278,361
Corrected read errors with delay = 0
Corrected read errors with retry = 0
Total Corrected read errors = 15,278,361
Total times read ECC used = 15,278,361
Total MegaBytes read = 371,578,203
Total Uncorrected read errors = 0

Verify Error Counter Page 0x05
Corrected verify errors without delay = 2,189
Corrected verify errors with delay = 0
Corrected verify errors with retry = 0
Total Corrected verify errors = 2,189
Total times verify ECC used = 2,189
Total MegaBytes verified = 494,085
Total Uncorrected read errors = 0

Non-medium Error Counter Page 0x06
Total non-medium errors = 35,453

Temperature and FC link Error Counter Page 0x0D
Temperature Data (Celsius) = 26
Temperature Reference Data (Celsius) = 65
Link Failure count, Port A. = 0
Loss of Synchronization count, Port A. = 14
Invalid Transmission word count, Port A. = 1,004
Invalid CRC count, Port A. = 0
Link Failure count, Port B. = 0
Loss of Synchronization count, Port B. = 12
Invalid Transmission word count, Port B. = 906
Invalid CRC count, Port B. = 0

Cache Statistics Page 0x37
Logical blocks sent to an initiator = 2,378,210,751
Logical blocks received from an initiator = 4,145,212,184
Logical blocks read from cache = 1,583,989,233
Read & writes; data lengths <= segment size = 1,643,133,198
Read & writes; data lengths > segment size = 124,079,008

Defect Data
No entries in the Defect List.

FWIW:

Link cards and midplanes can also manifest themselves as disk channel errors, but further analysis would be required. I only see the 2nd enclosure reporting disks issues in the attachement and not the entire chain.

Rgds,
dave.s
Advisor

Re: VA7410 few disks only seen by one controller

armlog shows a lot of repeated messages since the disk replacement. I've included the output in the attached document.

The events go back when the 2 ds2405 was added to the va7410 about a week ago. The controller reporting all the problems has never been stable.

The array is still in a rebuild state=0% which concerns me.

Thanks for your help thus far
dave.s
Advisor

Re: VA7410 few disks only seen by one controller

armlog -d seems to be having issues reporting everything. Looking into this.

#-> armlog -d JB2/D7

Disk Mechanism Log

FRU Location ---------------- = JB2/D7
Disk Address ---------------- = 38
Disk Capacity --------------- = 66.757 GB
Disk Product Rev ------------ = HP0A
Disk Product ID ------------- = ST373405FC
Disk Vendor ----------------- = HP 73.4G
Disk Volume Set Serial Number =
Disk World Wide Name -------- =

java.lang.ArrayIndexOutOfBoundsException: 3
cnb
Honored Contributor

Re: VA7410 few disks only seen by one controller

Backend Port issues reported on M/C1.J2, (disk channel events)which we already knew about.

What we need to find out is the reason the rebuild is stalled (maybe not enough available disks?)

# armrbld -p


Check that all disks are present and included with armdsp -a.

Check all Controller/Disk/Enclosure LEDs and logs.

http://bizsupport1.austin.hp.com/bizsupport/TechSupport/Document.jsp?lang=en&cc=us&taskId=110&prodSeriesId=3542195&prodTypeId=12169&objectID=lpg60204#A0


Use the armdiag or armlog to review the events prior to the first rebuild failure and go forward, check for any messages pertaining to rebuild failures and reasons.



cnb
Honored Contributor

Re: VA7410 few disks only seen by one controller

What version of Java are you using? Not sure if this is any help:

http://g4u0420c.houston.hp.com/en/JDKJRE50RN/jdk_rnotes_5.0.04.html

Rgds,
dave.s
Advisor

Re: VA7410 few disks only seen by one controller

Think we have identified that particular array as causing us problems before. Somehow it was not labeled as being suspect bad and was since re-connected back into another va configuration.

I am going to pull that 2405 out of the current config to get things back to a "stable" state. We have another unit that is not being used that I will re-add.

I will post back on the progress in case anyone else runs into similar problems in the future.

This is the first time I have removed an entire enclosure. Any feedback about this process?

- Delete newly created LUNS
- Down the drives in the enclosure
- Modify the security table and remove the associated LUNS
- Power off the 2405 and disconnected the fiber cables.
cnb
Honored Contributor

Re: VA7410 few disks only seen by one controller

Backups!

Might want to also check the array state (idle) to make sure no other major operations are in progress.

Verify the LCC Speed settings on the new DS2405 before connecting it up.

Some manuals that might help:
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/lpg60188/lpg60188.pdf

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c00084782/c00084782.pdf

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/lpg60195/lpg60195.pdf

Rgds,
dave.s
Advisor

Re: VA7410 few disks only seen by one controller

Very good information. Thanks for those links.

I did notice the following from a previously saved armdsp output. The loop id on the second controller is different from the first. Originally I ignored this but I noticed in the first enclosure the loop id's are the same. I am currently researching.

Controller At JA0/C1:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________0
Drive Address Basis:__________________0
Enclosure ID:_________________________0
Loop Pair:____________________________0
Loop ID:______________________________15
Hard Address:_________________________15

Controller At JA0/C2:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________0
Drive Address Basis:__________________0
Enclosure ID:_________________________0
Loop Pair:____________________________0
Loop ID:______________________________15
Hard Address:_________________________15

Controller At JB2/C1:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________2
Drive Address Basis:__________________2
Enclosure ID:_________________________2
Loop Pair:____________________________1
Loop ID:______________________________47
Hard Address:_________________________47

Controller At JB2/C2:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________2
Drive Address Basis:__________________2
Enclosure ID:_________________________2
Loop Pair:____________________________1
Loop ID:______________________________31
Hard Address:_________________________47
cnb
Honored Contributor

Re: VA7410 few disks only seen by one controller

Dave,

FWIW:

Just checked some of the ones here and they s/b configured like this:

Controller At JA0/C1:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________0
Drive Address Basis:__________________0
Enclosure ID:_________________________0
Loop Pair:____________________________0
Loop ID:______________________________15
Hard Address:_________________________15

Controller At JA0/C2:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________0
Drive Address Basis:__________________0
Enclosure ID:_________________________0
Loop Pair:____________________________0
Loop ID:______________________________15
Hard Address:_________________________15

Controller At JB1/C1:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________1
Drive Address Basis:__________________1
Enclosure ID:_________________________1
Loop Pair:____________________________1
Loop ID:______________________________31
Hard Address:_________________________31

Controller At JB1/C2:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________1
Drive Address Basis:__________________1
Enclosure ID:_________________________1
Loop Pair:____________________________1
Loop ID:______________________________31
Hard Address:_________________________31

Controller At JB2/C1:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________2
Drive Address Basis:__________________2
Enclosure ID:_________________________2
Loop Pair:____________________________1
Loop ID:______________________________47
Hard Address:_________________________47

Controller At JB2/C2:
Status:_______________________________Good
Serial Number:________________________
Vendor ID:____________________________HP
Product ID:___________________________A6255A
Product Revision:_____________________HP05
Controller Type:______________________HP StorageWorks Disk System DS2405
Enclosure Switch Setting:_____________2
Drive Address Basis:__________________2
Enclosure ID:_________________________2
Loop Pair:____________________________1
Loop ID:______________________________47
Hard Address:_________________________47
dave.s
Advisor

Re: VA7410 few disks only seen by one controller

This was found to be a bad fiber cable. After replacing the cable ( next step before giving up and replacing unit ) the sporadic dis-appear and re-appear of drives on that controller has stopped. It has been 3 days with no issues so closing out this thread. Thanks for the suggestions!
dave.s
Advisor

Re: VA7410 few disks only seen by one controller

bad fiber cable