topic Re: Input/output error ProLiant DL380G3 in ProLiant Servers (ML,DL,SL)

Input/output error ProLiant DL380G3

lastgreatone — Wed, 14 Sep 2005 07:36:28 GMT

Here is the server specs:
HP ProLiant DL380G3 (3GB RAM)
OS: RHES 2.1_2.4.9-e.65smp
2 x 72.8GB 10K U320 Hot Swap, no RAID

Server was unexpectedly powered down along with UPS, no clue what the cause was. But since then the root disk's LED turns red after approximately 1 hr after reboot. And I get Input/output errors, and dmesg shows:
kernel: cciss: cmd f69a024c has CHECK CONDITION, sense key = 0x3
Sep 12 10:40:00 digir1 kernel: EXT3-fs error (device cciss0(104,5)) in ext3_new_inode: IO

Is this a RedHat kernel issue or a hard disk issue?

Re: Input/output error ProLiant DL380G3

HugoAlmeida — Wed, 14 Sep 2005 10:17:43 GMT

That looks like a hard disk problem to me.
cciss is the RAID driver, and it is reporting a problem, so everything points to a failed disk.

Re: Input/output error ProLiant DL380G3

lastgreatone — Wed, 14 Sep 2005 10:38:55 GMT

OK thanks.

I'm not using RAID, should I still reconfigure the new disk with cciss?

Re: Input/output error ProLiant DL380G3

HugoAlmeida — Wed, 14 Sep 2005 10:44:46 GMT

Sorry, cciss is the hard disk controller driver :) .

Re: Input/output error ProLiant DL380G3

lastgreatone — Wed, 14 Sep 2005 13:21:33 GMT

what diagnostic tool should I run to confirm it's a hd problem? On HP-UX I could run ioscan but I don't know what to run on RedHat.

Re: Input/output error ProLiant DL380G3

Torsten. — Wed, 14 Sep 2005 13:57:19 GMT

I guess there wasn't a proper shutdown, so the filesystem got a problem (EXT3-fs error).

I would check disks with linux exactly like with hp-ux:

dd if= of=/dev/null

if there are some I/O errors, you have a hardware problem.

Re: Input/output error ProLiant DL380G3

lastgreatone — Wed, 14 Sep 2005 15:12:07 GMT

One more question:

The command lspci shows that there is a hardware raid configured on this server, done by the vendor. I did not request it but it's there.

Is it possible the raid controller is the problem, broke due to unclean shutdown? On the front panel of the server there is an image of a disk. Soon after rebooting this image lights up with a Red X. It's not flashing. And maybe the I/O errors are due to the Raid controller trying to rebuild, but what?

What diagnostic tool can I run to test for the Raid controller?

Re: Input/output error ProLiant DL380G3

Torsten. — Wed, 14 Sep 2005 15:56:56 GMT

I don't know about the red hat onboard utilities, but try to download the smart start CD image, burn it and start from

http://h18023.www1.hp.com/support/files/server/us/download/23318.html

there is a tool named ACU (array config utility) to configure/view your array/RAID.
Check out!

Re: Input/output error ProLiant DL380G3

Prashant (I am Back) — Wed, 14 Sep 2005 23:43:21 GMT

Hi,

Top to bottom led options :
1
Internal Health
Green = Normal Amber = System degraded. [ 1 ] Red = System critical. [ 1 ]

2
External health (Power Supply)
Red = Critical power supply failure Amber = Power redundancy failure Green = Normal

3
NIC 1 link/activity
Green = Network link Flashing = Network link/activity Off = No link/activity (if power is off, view rear panel RJ-45 led's)

4
NIC 2 link/activity

5
Front Unit Identification (UID) Switch
Blue = Activated Flashing = System remotely managed Off = Deactivated

6
Power On/Standby button & system power LED
Green = System on Amber = System shut down, but power still applied Off = Power cord not attached or power supply failure

------------------------------------------

To check the raid status and failure you can use.
ACU/ADU
avaible for linux also on site
and avaiable on SmartCd also

Regards,
Prashant s.

Re: Input/output error ProLiant DL380G3

lastgreatone — Thu, 15 Sep 2005 07:58:19 GMT

Do I simply run cpqacuxe from the HP SmartCD? I don't see an acu-xe rpm to install from the cd. The README.TXT makes reference on how to configure with cpqacuxe but no reference on how to troubleshoot. I'm assuming it will be apparent from the gui?

Re: Input/output error ProLiant DL380G3

lastgreatone — Thu, 15 Sep 2005 08:43:47 GMT

Standby. I booted off the cd and am running the diags and it appears there is a POST problem.

Re: Input/output error ProLiant DL380G3

lastgreatone — Thu, 15 Sep 2005 09:42:09 GMT

I ran the server diagnostic utility, reports following problem:

POST Device: POST Test
1727:Drive Array: New Logical
Drive(s) Attachment Detected Ran on CPU 0

Anyone know what the problem is?

Re: Input/output error ProLiant DL380G3

lastgreatone — Thu, 15 Sep 2005 11:00:52 GMT

Here the result of compaq survey utility:

CISS Physical Drive Information
----------------------------------------

Physical Disk (Controller 0 Bus 1 SCSI ID 0):
---------------------------------------------
Drive Model: COMPAQ BD07285A25
Size: 69460.84MB
Drive SN: 3HZ104ZA00007337QM8T
Drive FW Revision: HPB3
Compaq Part: Yes
Designated as Spare: No
Drive Failed: No
Drive Rebuilding: No
Last Failure Reason: Timeout
Physical Box On Bus: 0
Physical Bay in Box: 0

Physical Disk (Controller 0 Bus 1 SCSI ID 1):
---------------------------------------------
Drive Model: COMPAQ BD07285A25
Size: 69460.84MB
Drive SN: 3HZ0K0W3000073252X5Y
Drive FW Revision: HPB3
Compaq Part: Yes
Designated as Spare: No
Drive Failed: No
Drive Rebuilding: No
Last Failure Reason: Unknown
Physical Box On Bus: 0
Physical Bay in Box: 1

Re: Input/output error ProLiant DL380G3

Prashant (I am Back) — Fri, 16 Sep 2005 02:10:17 GMT

HI,

Check cables for bent pins. If this does not resolve the issue, try "shocking" the drives.

If this does not resolve the issue, try removing ALL drives from the server, and run System Erase after verifying that all HDDs internal and external have been disconnected from this server. Then reinstall the drives, boot to a SmartStart CD, run SCU, set boot order, and reboot.

Regards,
Prashant S.

Re: Input/output error ProLiant DL380G3

lastgreatone — Tue, 20 Sep 2005 13:17:28 GMT

What do you mean by "shocking the drives"?

Re: Input/output error ProLiant DL380G3

Prashant (I am Back) — Tue, 20 Sep 2005 23:12:26 GMT

Hi,

This just checking the drive when it is offline.
Try to do the above and update. What is the outsome then we will what we can do.

Regards,
Prashant s.

Re: Input/output error ProLiant DL380G3

Prashant (I am Back) — Tue, 20 Sep 2005 23:14:53 GMT

Hi,

If you see the last error reported :

-----------------------------------------
Physical Disk (Controller 0 Bus 1 SCSI ID 0):
---------------------------------------------
Drive Model: COMPAQ BD07285A25
Size: 69460.84MB
Drive SN: 3HZ104ZA00007337QM8T
Drive FW Revision: HPB3
Compaq Part: Yes
Designated as Spare: No
Drive Failed: No
Drive Rebuilding: No
Last Failure Reason: Timeout
-----------------------------------------
It is timeout so i told you to try same.

Regards,
Prashant S.

Re: Input/output error ProLiant DL380G3

lastgreatone — Thu, 22 Sep 2005 10:09:52 GMT

I installed a new drive and I have a healthy system again. thanks.