Upgrading firmware beyond v1.66 on BL460G6 p410 controller causes never-ending cycle of rebuilding disk array.

[jdeavers919]

We are having a very interesting problem on all 16 of our new BL460 G6 blades. All of the new blades are configured with the HP p410 RAID controller with only 2 72gb SAS disks in a RAID 1 configuration and all have Windows Server 2008 x64 operating systems installed. It seems that if we upgrade the firmware on that p410 controller to any version higher than v1.66, the server begins a never-ending cycle of “rebuilding the disk array” (see attached messages for details). However, downgrading to v1.66 of the firmware solves the problem and the arrays are stable once again. We have repeatedly tried to install versions 2.00 and 2.50 of the “ROM Flash Component for Windows - Smart Array P212, P410, P410i, P411, and P712m” but with either version, the “rebuilding array” problem starts and continues indefinitely. The array rebuild seems to happen at least once or twice per day on the servers and we see messages that indicate “a drive was removed”. We have had a ticket open with HP for over a week and have spent many hours on the phone with the HP Blade systems support group running diagnostics, replacing hardware, uploading reports, etc. but HP has absolutely no idea what the problem is there is no indication of bad hardware whatsoever. The ticket has been escalated within HP and we’re waiting for more information from them at this time.