topic Re: VA7410 logging occasonal errors in Disk Enclosures

VA7410 logging occasonal errors

Carl Houseman — Mon, 02 Jul 2007 13:35:13 GMT

This is a new installation of refurbed hardware, not yet under HP contract (or I'd ask HP...).

VA7410 w/2GB cache, 1.40 f/w, and a DS2405, both fully populated with ST373453FC (15K 73GB) at f/w HP05. Host (RP3440) has A6795A connected to each VA7410 controller. Software is CVSDM 1.09.02. Controller is configured for RAID 1+0.

Occasionally there is a series of errors in syslog, major highlights from a recent sequence as follows:

CVSDM; MAJOR WARNING Event Code=400; Description=265:Rebuild has been started either automatically or by command... Hardware Address=0/3/1/0.8.0.255.6.14.0; Unknown FRU Location;

CVSDM; MAJOR WARNING Event Code=400; Description=116: Target Sense Key = 0xff -- Target ASC = 0xff -- Target ASCQ = 0xff -- Target Physical LBA = 0x00 -- The host Pass through request was not successfull... Hardware Address=0/3/1/0.8.0.255.6.14.0; FRU Location=JA0/C1

CVSDM; MAJOR WARNING Event Code=400; Description=573:WARNING_ACTIVATED_EH One of the warning indicators has been actviated. Check the extended event information for more information.; State: bit on: 1: DEGRADED_WARNING; classid: 0030, inst: 0000; enclosureId/slot/component/subcomponent: 0x00/0xff/0xff/0xff; controller tick : 72755976623; serialNum/moduleId/processId : 00PR02C52500/0x5f/0xffffffff; ; Hardware Address=0/3/1/0.8.0.255.6.14.0; FRU Location=M

CVSDM; INFORMATION Event Code=400; Description=150: Target Sense Key = 0x0b -- Target ASC = 0x47 -- Target ASCQ = 0x00 -- Target Physical LBA = 0x164c730 -- DEV_ABORTED_COMMAND_EH The drive returned CHECK CONDITION status indicating an aborted I/O, which will be retried by the PDD's error recovery algorithms. This event can be seen during the normal operation of the Array, and only represents a problem when it occurs in rapid succession causing a slow down or DEV_IO_FAILED_EH.loop/enclosureId/slot : 0xff/0x00/0x1c; controller tick : 72756806219; serialNum/moduleId/processId : 00PR02C52500/0x3b/0x19e; ; Hardware Address=0/3/1/0.8.0.255.6.14.0; FRU Location=M/D13

CVSDM; INFORMATION Event Code=400; Description=150: Target Sense Key = 0x0b -- Target ASC = 0x47 -- Target ASCQ = 0x00 -- Target Physical LBA = 0x15bec00 -- DEV_ABORTED_COMMAND_EH The drive returned CHECK CONDITION status indicating an aborted I/O, which will be retried by the PDD's error recovery algorithms. This event can be seen during the normal operation of the Array, and only represents a problem when it occurs in rapid succession causing a slow down or DEV_IO_FAILED_EH.loop/enclosureId/slot : 0x00/0x00/0x00; controller tick : 72759196866; serialNum/moduleId/processId : 00PR02C52500/0x3b/0x1db; ; Hardware Address=0/3/1/0.8.0.255.6.14.0; FRU Location=JA0/D1

Followed by another UNKNOWN FRU LOCATION message followed by another DEGRADED WARNING.

When I look at the array status after this, all of the indicators are green - no failed drives. Nothing wrong.

The FRU locations being cited from the above and from prior errors logged include:

JA0/C1
M
JA0/D11
JA0/D5
M/D13
JA0/D1

So I'm having drive errors in both enclosures? A total of 4 flaky drives? Or something else?

thanks all...

Re: VA7410 logging occasonal errors

Andy_Stor — Tue, 03 Jul 2007 03:04:24 GMT

Hi carl,

The error may or may not be a subjected to a Hardware Issue,

Could you attach the armdsp -a output for the VA7410.

Andy

Re: VA7410 logging occasonal errors

Carl Houseman — Tue, 03 Jul 2007 10:47:49 GMT

I don't see anything terrible that jumps out at me, but armdsp.txt is attached.

The only point of possible conflict, is that under Controllers, we see that

Back Port At M/C1.J1:
...
Data Rate:__________________________2 GBit/sec

and later on under Ports, we see

Settings for port M/C1.J1:
Data Rate:____________________________1 GBit/sec

Seems one of these reports is not right, but if this was a serious problem, the thing wouldn't work at all.

Re: VA7410 logging occasonal errors

Torsten. — Tue, 03 Jul 2007 10:59:27 GMT

Hi Carl,

regarding the speed:
The Cx/Hx ports are at 2 GBit/sec (to the hosts), the Cx/Jx ports at 1 GBit/sec - this is pretty normal.

Maybe one of the disks is "going to be bad" and disturbs the loop. I would recommend to let hp support take a look into the logs.

Re: VA7410 logging occasonal errors

Carl Houseman — Tue, 03 Jul 2007 11:02:54 GMT

Torsten, I think need to re-read my posts...

1. No HP support yet.

2. Both reports are for the same port - M/C1.J1 - which I really want to be running at 2GBit/sec.

Re: VA7410 logging occasonal errors

Torsten. — Tue, 03 Jul 2007 11:42:50 GMT

OK, it is a while ago that I worked with this arrays.

The ports
M/C1.H2
are connected to host (H=Host).

Ports
M/C1.J1
are connected to the disks (J=JBOD).

The setting default is for J ports 1GB for VA7400 and 2 GB for VA7410 (yours).

To adjust this you need to shutdown the array, pull the JBOD controllers and switch the DIPs.

Regarding the disks - some special tools and experience may be needed to find the culprit.

Re: VA7410 logging occasonal errors

Carl Houseman — Tue, 03 Jul 2007 12:17:48 GMT

DIPs on the DS2405 already set to 2 GBit.

Re: VA7410 logging occasonal errors

Carl Houseman — Wed, 04 Jul 2007 12:38:25 GMT

Finally, an array drive is showing up as failed. It is M/D14 which wasn't on my list of problematic FRUs before. So this is getting stranger and stranger.

And now, I have a question for a new topic on redundancy and hot spares in a RAID 1+0 configured VA7410.

Re: VA7410 logging occasonal errors

Carl Houseman — Thu, 12 Jul 2007 09:59:33 GMT

The failed drive was in RG2 but all the errors I was seeing prior to that failure were from an assortment of drives in RG1. Consequently I suspected M/C1 as the culprit ahd had it replaced, and that seems to have resolved the problem.