CHKDSK errors on DL380 G7

[tricktracktruck]

Hello.

We have a problem with 2 out of 3 identical servers we bought together.

When running CHKDSK errors are shown which could not be repaired

in read only mode. After running CHKDSK on reboot errors are gone but

always reappear on the next day.

The servers are all DL380 G7 with P410i + 256 MB BBWC and two 72 GB

harddisks running in simple RAID 1 configuration. One server is also

using FC SAN storage on multiple P2000 G3 FC boxes.

They are all on current identical firmware (per DEC 2011) running Server 2003 R2 SP2 x64.

Patchlevel and Software of all servers are identical as they are provisioned completely

unattended by scripts.

Interrestingly the server which is also attached to the SAN only shows those

errors on the DAS drives. On the SAN devices CHKDSK runs flawlessly.

All Servers are covered by carepacks and we already opened an incident,

but I 'm quite confused about the answer we got from HP support after we

submitted the HPS report.

They told us that they could not find any hardware problems so this must

be a software problem and we should call Windows for support. They also

asked us to try running a newer version of CHKDSK (not sure what he was talking

about).

Besinde the fact that I don't know how to call Windows (I would try MS instead)

we are running 60 servers here where 45 are running Windows 2003 R2 SP2 x86

and 10 are running Windows 2003 R2 SP2 x64 an non of them are showing similar

problems.

Any suggestions what to do next?

Thanks for your suggestions.

Cheers.

[rguha]

Hi, 

Please provide the array diagnostics report for analysis.

You can download the utility from the link given below:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=428936&prodNameId=3288114&swEnvOID=4024&swLang=8&taskId=135&swItem=MTX-c3e7acb4f845425d9be5dea323&mode=3

[tricktracktruck]

Thank's for your response.

Here is the report.

[Johan Guldmyr]

Hey,

see the section "Monitor and Performance Statistics " for each disk.

There's no errors reported for either of the two disks.

So there doesn't appear to be a problem with the disks, physically.

Maybe the problem lies with what's writing to the disks (OS, drivers, bad applications).

Are you running chkdsk while the server is online - and then seeing errors?

And when you reboot they are not seen? Maybe you only see the errors because you are running them while the server is online?

Why are you running chkdsk? Are you having any actual issues?

[tricktracktruck]

Hi Johan.

Thanks for your response.

You are right there are no physical errors on the disks itself, so the question is why the filesystem on that disks get corrupted. So maybe a problem with conrtoller-cache, controller itself, cabeling, backplane?

We see that errors when running CHKDSK online.

Then we correct them by running CHKDSK on reboot.

After that running CHKDSK online does not show errors.

Next day running CHKDSK online show errors again.

Strange enough CHKDSK on attached FC LUNs works flawlessly.

Same problem on second machine (but there is no SAN attached)

Third machine has no problem.

All machines same OS same FW same support pack components.

We sometimes also get a Windows message that directory structure

has been corrupted and cannot be repaired.

We also sporadically see the following eventlog messages:

Source: bfad

ID: 11

Description: The driver has detected a controller failure on \Device\RaidPort1

and

Source: bfad

ID: 129

Description: Reset to device \Device\RaidPort1 was issued

I agree with you that the problem lies whith what is written to disk.

But from my understanding this is outside the scope of an application, so what's left is OS, driver, hardware.

Cheers.

Edit:

The bfad errors are only present in the event log two times and only on the machine with the FC controller so it does no seem to be related to the problem.

[Johan Guldmyr]

Hi,

usually when there's a backplane issue the 'bus faults' in the ADU reports are larger than zero.

One thing that's easy to check - as you are running RAID1 - is to run it without the BBWC.

I am not sure, but probably the disks from the FC HBA are using a different driver.

As you say that the error comes "the next day" - maybe you could try to find out after exactly how long before you see the errors? And then maybe it would be possible to figure out what the OS has been doing in between? Scheduled jobs, etc,

Is it possible to find out if it's always the same part of the disk / file that is having these errors via CHKDSK? Or is it always different?

[tricktracktruck]

>>One thing that's easy to check - as you are running RAID1 - is to run it without the BBWC.

That we checked meanwhile.

Problem persists.

Strange is that chkdsk states that errors were found and to run it in read write mode but I don't see any errors corrected when running in read write mode. After that chkdisk states that everything is ok.

Cheers.

[Johan Guldmyr]

Perhaps if you provided the logs from the scans that would help us, at least so we can see what these 'errors' are that you're referring to.

[tricktracktruck]

Stupid question but where do I find logs created by chkdsk?

As I said chkdsk moans to be run in write mode as it has detected errors but when run

in write mode on reboot I did not see any errors displayed on the screen?

Maybe this happens to fast so I did overlook it.

Thanks for your help.

Cheers.

[Johan Guldmyr]

Sorry for the delay but I don't know if you can find them, but google says that some logs are in the event viewer.

But anyway, could it be that at some point was something bad written on the filesystem and chkdsk cannot handle it?