HP VA 7410 Rebuild Failure / Missing Drive

[Chad Brindley]

Hi,

We have an old VA 7410 which although is in planning for replacement is still in Production, in the past few months we have had on average 2 disk failures per month, this month we have had thus far almost a catastrophic number of failures (9 over the course of 1 week). Our third party support have stated they only see disk errors and have no worries with the controller itself.

We have one VA 7410 with 3 Disk System 2405 attached. This contains a mix of 40 x 73GB disk and 20 x 146GB disks.

At this present time we have a VA where all disks are Green but the VA is showing the Warnings;

Redundancy Loss

Rebuild Failed

Missing Drive

Balancing appears to have been in Progress for a 3+ days.

We had two 73GB disks in enclosure JB1 fail and one in enclosure JA0, the order of failure below.

JB1/D15

JA0/D6

JB1/D1

During this period the Rebuild restarted on each failure. Our Hardware support vendor came out and advised we place the Array into Manual Rebuild, we replaced the drives in RG1 first - JB1/D15, waited for it to format and then JB1/D1 after.

Once complete we tried to run a rebuild which failed as it stated it needed the disk which originally lived in place of JB1/D1, physically I still have this disk.

As we had a failed disk already in JA0/D6, we decided to put the missing disk here just so the array could see it again, this meant we had a disk which wasnt optimally placed due to it belonging to RG1 but in a slot for RG2 but the Rebuild ran and the Array said we had 1 Drive Redundancy Level again, the disk however was showing as Failing and Eventually Failed, the Rebuild Failed and we are back to where we were. I replaced the disk in JA0/D6 with a new disk which appears to be fine. The array however is still reporting the missing drive that is physically not in the Array but wont rebuild without it.

I stopped all applications which are hosted from this VA as it appeared excessive IO and high usage during backups, application use then disks would fail, its become a vicious circle and although the issue may occur again I need to get the Array back to its redundant state before I allow use of the Applications again, this of course in the ideal situation. After days of head scratching I am unsure where to turn with this one. We are at the point of invoking disaster recovery although the Array maybe fine and the data on it may also be OK, it was working ok with the 3 failed disks before the application was stoppped.

Could I;

a) Pull out the new included disk in JB1/D1 and insert the missing drive. As the Array is showing no drive redundancy am I likely to lose data in this scenario. I did try to down the disk in JB1/D1 through the GUI but it came back with an illegal statement message possibly because I was trying to do with no data loss and had rebuild set to manual.

b) Could I bolt on another HP 2405 disk tray if I get hold of one to put the missing disk back in. 

c) Is there a way to forcibly ignore the missing drive.

I'm a little nervous to try anything, the two new replacement disks included in JB1 I dont believe are doing anything, there has been no activity on the disk since being placed.

I have attached armdsp and armlog output from 12th January. The logs for the recent failures are relevant really from 8th / 9th January.

Any help or advice appreciated.

Thanks,

Chad