topic Re: Linux server cannot see volumes on the storage HP MSA1000 in Operating System - Linux

Linux server cannot see volumes on the storage HP MSA1000

kouyate Maury_1 — Mon, 19 Jan 2009 08:08:41 GMT

Hi,
After a big light issue in my datacenter which power off and on all the systems in my datacenter my Linux (RedHat)servers (DL380 G5) can no more connect to the storage MSA 1000 (Cluster).
This is the error message:
SCSI error : <2 0 0 4> return code = 0x8000002
Current sdd: sense key Hardware Error
Additional sense: Internal target failure
end_request: I/O error, dev sdd, sector 573496320
Buffer I/O error on device sdd, logical block 71687040
SCSI error : <2 0 0 4> return code = 0x8000002
Current sdd: sense key Hardware Error
Additional sense: Internal target failure
end_request: I/O error, dev sdd, sector 573496336
Buffer I/O error on device sdd, logical block 71687042
SCSI error : <2 0 0 4> return code = 0x8000002
Current sdd: sense key Hardware Error
Additional sense: Internal target failure
end_request: I/O error, dev sdd, sector 573496344
Buffer I/O error on device sdd, logical block 71687043
SCSI error : <2 0 0 4> return code = 0x8000002
Current sdd: sense key Hardware Error
Additional sense: Internal target failure
end_request: I/O error, dev sdd, sector 573496352
Buffer I/O error on device sdd, logical block 71687044
SCSI error : <2 0 0 4> return code = 0x8000002
-----------------------------------------------------------------------
the storage is not responding even if i trace the storage hard drives from the server. the coomand and output is given below.

[root@dbapp-node1 ~]# smartctl -a /dev/sda
smartctl version 5.33 [x86_64-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
Device: COMPAQ MSA1000 VOLUME Version: 4.48
Serial number: PB9840TX3V104A
Device type: disk
Local Time is: Fri Jan 16 21:05:23 2009 WAT
device Test Unit Ready [Input/output error]
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

NB: The storage is attached to the server via fiber cable and the fiber cables are working fine.
I have attached a file containing some output of the second cluster system.

Re: Linux server cannot see volumes on the storage HP MSA1000

Steven E. Protter — Mon, 19 Jan 2009 10:37:46 GMT

Shalom,

Suggest you use hp_rescan utility, which ships with PSP to further check storage.

Post results.

SEP

Re: Linux server cannot see volumes on the storage HP MSA1000

kouyate Maury_1 — Mon, 19 Jan 2009 11:16:40 GMT

Hi Steven
this is the output:

[root@webnode1 hp_fibreutils]# ./hp_rescan -a
Sending rescan signal to /proc/scsi/qla2xxx/1...
Sending rescan signal to /proc/scsi/qla2xxx/2...
Adding legacy tape devices to /proc/scsi/device_info
Scanning /proc/scsi/qla2xxx/1, target 0, LUN 2
Scanning /proc/scsi/qla2xxx/2, target 0, LUN 2

scsi2 00 00 00 COMPAQ MSA1000 4.48 RAID
scsi2 00 00 01 COMPAQ MSA1000 4.48 Direct-Access
scsi2 00 00 02 COMPAQ MSA1000 4.48 Direct-Access

Re: Linux server cannot see volumes on the storage HP MSA1000

Ivan Ferreira — Mon, 19 Jan 2009 12:42:16 GMT

Â¿If you start the array configuration utility, can you see that the logical drives in the msa1000 are in optimal state?

Re: Linux server cannot see volumes on the storage HP MSA1000

kouyate Maury_1 — Mon, 19 Jan 2009 16:18:19 GMT

Hi Ivan,
see bellow the output:
[root@webnode1 ~]# lssd -l
sda 2,0,0,1 COMPAQ MSA1000 VOLUME 4.48 600143800007f420-ab58d43d58bf0000
sdb 2,0,0,2 COMPAQ MSA1000 VOLUME 4.48 600143800007f420-a988c43d36950001

as u can the we have 2 partitions in the storage.

Re: Linux server cannot see volumes on the storage HP MSA1000

Matti_Kurkela — Tue, 20 Jan 2009 08:09:02 GMT

The SCSI errors say: "sense key Hardware Error ... Internal target failure".

To me, that looks like the connection to the MSA1000 is probably OK, but the _MSA1000 itself_ is screaming for "HELP!!!" in the only way it knows how.

You'll need to start the Array Configuration Utility or plug a computer/terminal into the serial port on the MSA1000 and take a look into the internal state of the MSA1000.

I think you may have more disk failures than the current configuration of the MSA can handle. I sure hope your backups are good...

MK

Re: Linux server cannot see volumes on the storage HP MSA1000

kouyate Maury_1 — Tue, 20 Jan 2009 10:16:24 GMT

Hi Matti,
Thanks for your reply, in fact I was able to connect trough ILO GUI (Find attached the print screen) I saw some faulty drive but I couldn't connect to the serial console of the storage to cross check, and I'm very new ACU command, the last administrator did not let any documention about the storage config, I'm trying to find my way. So if u can help with some command I can use to check the status of the disks.

Re: Linux server cannot see volumes on the storage HP MSA1000

Matti_Kurkela — Tue, 20 Jan 2009 12:08:10 GMT

Here's the manual for all the SmartArray configuration tools:
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c00729544/c00729544.pdf

(ORCA seems to be for the server's internal SmartArray only, and CPQONLIN is for Netware; ACU is what you'll want.)

Here's the webGUI-based ACU for Linux:

http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121516&swItem=MTX-5e60955d5bc5479a97bdca725e&prodNameId=3288134&swEnvOID=2025&swLang=8&taskId=135&mode=4&idx=0

Run "man cpqacuxe" for instructions on how to start it. When it's running, log in to the System Management Homepage ( https://yourserver:2381 ) to see the ACU link in the left side panel.

Here is the command-line version of ACU for Linux:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?lang=en&cc=us&prodTypeId=15351&prodSeriesId=1121516&prodNameId=3288134&swEnvOID=2025&swLang=8&mode=2&taskId=135&swItem=MTX-673eb0ec605b4a2bac35ebb18c

Here are the manuals for the MSA1000:
http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?contentType=SupportManual〈=en&cc=us&docIndexId=64179&taskId=101&prodTypeId=12169&prodSeriesId=377751

MK

Re: Linux server cannot see volumes on the storage HP MSA1000

kouyate Maury_1 — Tue, 20 Jan 2009 19:08:43 GMT

Thanks, My storages was haven more faulty disks than it could handle.