Input/output error ProLiant DL380G3

[lastgreatone]

Here is the server specs:
HP ProLiant DL380G3 (3GB RAM)
OS: RHES 2.1_2.4.9-e.65smp
2 x 72.8GB 10K U320 Hot Swap, no RAID

Server was unexpectedly powered down along with UPS, no clue what the cause was. But since then the root disk's LED turns red after approximately 1 hr after reboot. And I get Input/output errors, and dmesg shows:
kernel: cciss: cmd f69a024c has CHECK CONDITION, sense key = 0x3
Sep 12 10:40:00 digir1 kernel: EXT3-fs error (device cciss0(104,5)) in ext3_new_inode: IO

Is this a RedHat kernel issue or a hard disk issue?

[HugoAlmeida]

That looks like a hard disk problem to me.
cciss is the RAID driver, and it is reporting a problem, so everything points to a failed disk.

[lastgreatone]

OK thanks.

I'm not using RAID, should I still reconfigure the new disk with cciss?

[HugoAlmeida]

Sorry, cciss is the hard disk controller driver :) .

[lastgreatone]

what diagnostic tool should I run to confirm it's a hd problem? On HP-UX I could run ioscan but I don't know what to run on RedHat.

[Torsten. · Accepted solution]

I guess there wasn't a proper shutdown, so the filesystem got a problem (EXT3-fs error).

I would check disks with linux exactly like with hp-ux:

dd if= of=/dev/null

if there are some I/O errors, you have a hardware problem.

[lastgreatone]

One more question:

The command lspci shows that there is a hardware raid configured on this server, done by the vendor. I did not request it but it's there.

Is it possible the raid controller is the problem, broke due to unclean shutdown? On the front panel of the server there is an image of a disk. Soon after rebooting this image lights up with a Red X. It's not flashing. And maybe the I/O errors are due to the Raid controller trying to rebuild, but what?

What diagnostic tool can I run to test for the Raid controller?

[Torsten.]

I don't know about the red hat onboard utilities, but try to download the smart start CD image, burn it and start from

http://h18023.www1.hp.com/support/files/server/us/download/23318.html

there is a tool named ACU (array config utility) to configure/view your array/RAID.
Check out!

[Prashant (I am Back)]

Hi,

Top to bottom led options :
1
Internal Health
Green = Normal Amber = System degraded. [ 1 ] Red = System critical. [ 1 ]

2
External health (Power Supply)
Red = Critical power supply failure Amber = Power redundancy failure Green = Normal

3
NIC 1 link/activity
Green = Network link Flashing = Network link/activity Off = No link/activity (if power is off, view rear panel RJ-45 led's)

4
NIC 2 link/activity

5
Front Unit Identification (UID) Switch
Blue = Activated Flashing = System remotely managed Off = Deactivated

6
Power On/Standby button & system power LED
Green = System on Amber = System shut down, but power still applied Off = Power cord not attached or power supply failure


------------------------------------------

To check the raid status and failure you can use.
ACU/ADU
avaible for linux also on site
and avaiable on SmartCd also

Regards,
Prashant s.

[lastgreatone]

Do I simply run cpqacuxe from the HP SmartCD? I don't see an acu-xe rpm to install from the cd. The README.TXT makes reference on how to configure with cpqacuxe but no reference on how to troubleshoot. I'm assuming it will be apparent from the gui?

[lastgreatone]

Standby. I booted off the cd and am running the diags and it appears there is a POST problem.

[lastgreatone]

I ran the server diagnostic utility, reports following problem:

POST Device: POST Test
1727:Drive Array: New Logical
Drive(s) Attachment Detected Ran on CPU 0

Anyone know what the problem is?

[lastgreatone]

Here the result of compaq survey utility:

CISS Physical Drive Information
----------------------------------------

Physical Disk (Controller 0 Bus 1 SCSI ID 0):
---------------------------------------------
Drive Model: COMPAQ BD07285A25
Size: 69460.84MB
Drive SN: 3HZ104ZA00007337QM8T
Drive FW Revision: HPB3
Compaq Part: Yes
Designated as Spare: No
Drive Failed: No
Drive Rebuilding: No
Last Failure Reason: Timeout
Physical Box On Bus: 0
Physical Bay in Box: 0

Physical Disk (Controller 0 Bus 1 SCSI ID 1):
---------------------------------------------
Drive Model: COMPAQ BD07285A25
Size: 69460.84MB
Drive SN: 3HZ0K0W3000073252X5Y
Drive FW Revision: HPB3
Compaq Part: Yes
Designated as Spare: No
Drive Failed: No
Drive Rebuilding: No
Last Failure Reason: Unknown
Physical Box On Bus: 0
Physical Bay in Box: 1

[Prashant (I am Back)]

HI,

Check cables for bent pins. If this does not resolve the issue, try "shocking" the drives.

If this does not resolve the issue, try removing ALL drives from the server, and run System Erase after verifying that all HDDs internal and external have been disconnected from this server. Then reinstall the drives, boot to a SmartStart CD, run SCU, set boot order, and reboot.

Regards,
Prashant S.

[lastgreatone]

What do you mean by "shocking the drives"?

[Prashant (I am Back)]

Hi,

This just checking the drive when it is offline.
Try to do the above and update. What is the outsome then we will what we can do.

Regards,
Prashant s.

[Prashant (I am Back)]

Hi,

If you see the last error reported :

-----------------------------------------
Physical Disk (Controller 0 Bus 1 SCSI ID 0):
---------------------------------------------
Drive Model: COMPAQ BD07285A25
Size: 69460.84MB
Drive SN: 3HZ104ZA00007337QM8T
Drive FW Revision: HPB3
Compaq Part: Yes
Designated as Spare: No
Drive Failed: No
Drive Rebuilding: No
Last Failure Reason: Timeout
-----------------------------------------
It is timeout so i told you to try same.

Regards,
Prashant S.

[lastgreatone]

I installed a new drive and I have a healthy system again. thanks.