Bad sectors on MSA2312sa RAID-5

[alex-63]

We have a weird problem with the MSA2312sa disk array. 9 physical disks (ST3300656SS and ST3300657SS) are configured as RAID-5 vdisk with one volume (2.4GB). The connected server (Windows Server 2008) detects bad sectors  on this volume. Files don't read ("data error", "CRC error") and HDDScan shows this:

Open Disk
Test : RD-Read

HDDScan Drive Test Report 
Model: HP MSA2312sa 
Firmware: M112
Serial: C
LBA: 4682550272

Report By: HDDScan for Windows version 3.3
Report Date: 16.12.2012 22:12:10

(The image here contains nothing special except total count of bad blocks -- 5702)

Executing
Block start at 151203840 time 65ms
Block start at 188514304 time 162ms
Bad block found, start LBA : 915337216
Bad block found, start LBA : 915337472
Bad block found, start LBA : 915337728
...
Bad block found, start LBA : 915441664
Bad block found, start LBA : 1036977152
Bad block found, start LBA : 1036977408
Bad block found, start LBA : 1036977664
...
Bad block found, start LBA : 1252704256

There absolutely no problems are reported in the MSA2312sa itself. Vdisk is fault tolerant. Media scrub and verify have completed without errors. I only found  several messages in the informational log section like this:

An error was detected by a disk drive. (disk: channel: 0, ID: 10, SN: 3LM5RPM500009909EUUT, enclosure: 1, slot: 11)(Key,Code,Qual:0x1,0x17,0x3)(CDB:Rd 098c7280 0080)(Info:0x098C72CF)(CmdSpc:0x0, FRU:0x0, SnsKeySpc:0x80000E)(Recovered Error, recovered data with negative head offset)

 but this recovered error can't produce such disastrous result in RAID-5 (!) volume.

[Johan Guldmyr]

Hi,

is that error always showing for the same disk? Always the same SN/slot and error message? Looks that disk has some problems and you see many of those read errors it should be replaced.

[alex-63]

Johan Guldmyr, if you say about the message in the MSA2312sa log, yes it's for one physical disk. But it's informational, not even warning message. It definitely should not cause bad sectors on the higher abstraction level in the RAID-5 volume. What the words "fault tolerance" mean if a single disk error causes massive data loss?

[Johan Guldmyr]

In a RAID I'd presume that a read error should not cause problems - the RAID should just calculate/xor whatever it couldn't read from parity instead..

Perhaps these events are seen only when the LUN is tested by your software.

[Torsten.]

I doubt the results and assume the tool cannot handle the device. Remember, the server (and the tool) cannot see the disks, but logical drives only!

The array does such tests on its own (disk scrubbing) anyway. If the array reports disk problems, the disk is probably going bad.

[alex-63]

Please reread my first post! The main problem is unreadable files -- Windows and applications say "data error", "CRC error" because they don't know it's a Smart Array with RAID-5 volume :). And yes, HDDScan also sees logical drive only and detects errors on this level like the Windows does. But why I get errors on this level on healthy RAID-5?

Some additional information: chkdsk /r /f also finds bad sectors (clusters). Rewriting the damaged areas (up to 1024 sectors blocks) has no result -- writing error and subsequent  reading error.