HELP ME PLEASE - MA8000 volume corrupt

[Nam, ill]

Hi all,

I have critical problem with MA8000 that connected with ML570 Server.

A few days ago, client failed to connect with the F: drive volume that on the MA8000 storage.

When I arrived server room, I found some problmes below.

1. Amber alarm led on 2 of SAN switch ports.
2. Warning alarm on the cache battery module.
3. Controller working stoped with the cache battery module fail.
4. Some of commands could not work on the MA8000 serial console.
5. Server could not access to the Disk2 - MA8000 array.
6. Found latest fail code 20090010 from the serial console.

So, I have performed below.

1. Shutdown controller, Server off, Storage off -> Storage on, Server on -> amber led on SAN switch are disappeared, but could not access to the storage volume.

2. shutdown controller, Storage off, Server off -> Replaced with new cache battery module -> Storage on, Server on -> controller time setup, "RUN FRUTIL" command at the console. -> after 20 min, warning message of the cache battery module is cleared. but could not access storage volume - Disk2 OFFLINE on the windows disk manager.

What should I do to recover data ???
Please let me know if you have any idea.

Thanks.

[pasquino bocci_1]

Hi

1) is this the only disk you can't see?
2) if the batteries are not charged, you won't be able to access the disk ( you must change your cache policy to datacenter)
3) what about disk presentation?

Pasquino

[Nam, ill]

This is current status.

HSG> show this full
Controller:
HSG80 ZG04105815 Software V85F-0, Hardware E12
NODE_ID = 5000-1FE1-000A-E080
ALLOCATION_CLASS = 0
SCSI_VERSION = SCSI-2
Not configured for dual-redundancy
Device Port SCSI address 7
Time: 02-APR-2007 15:16:09
Command Console LUN is disabled
Host PORT_1:
Reported PORT_ID = 5000-1FE1-000A-E081
PORT_1_TOPOLOGY = LOOP_SOFT (loop up)
PORT_1_AL_PA = 01 (negotiated)
Host PORT_2:
Reported PORT_ID = 5000-1FE1-000A-E082
PORT_2_TOPOLOGY = LOOP_SOFT (loop up)
PORT_2_AL_PA = 02 (negotiated)
NOREMOTE_COPY
Cache:
256 megabyte write cache, version 0022
Cache is GOOD
No unflushed data in cache
CACHE_FLUSH_TIMER = DEFAULT (10 seconds)
Mirrored Cache:
Not enabled
Battery:
NOUPS
FULLY CHARGED
Expires: 02-APR-2009
Extended information:
Terminal speed 9600 baud, eight bit, no parity, 1 stop bit
Operation control: 00000000 Security state code: 75200
Configuration backup disabled
HSG> show disks full
Name Type Port Targ Lun Used by
------------------------------------------------------------------------------

DISK10800 disk 1 8 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK20100 disk 2 1 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK20200 disk 2 2 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK20300 disk 2 3 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK20400 disk 2 4 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK40100 disk 4 1 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK40200 disk 4 2 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK40300 disk 4 3 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK40400 disk 4 4 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK60100 disk 6 1 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK60200 disk 6 2 0 SPARESET
COMPAQ BD03664553 3B08
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: 71114623 blocks
DISK60300 disk 6 3 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
DISK60400 disk 6 4 0 RAID1
COMPAQ BD036635C5 B017
Switches:
NOTRANSPORTABLE
TRANSFER_RATE_REQUESTED = 20MHZ (synchronous 20.00 MHZ negotiated)
Size: NOT YET KNOWN
HSG> show raidsets full
Name Storageset Uses Used by
------------------------------------------------------------------------------

RAID1 raidset DISK10800 D1
DISK20100
DISK20200
DISK20300
DISK20400
DISK40100
DISK40200
DISK40300
DISK40400
DISK60100
DISK60300
DISK60400
Switches:
POLICY (for replacement) = BEST_PERFORMANCE
RECONSTRUCT (priority) = FAST
CHUNKSIZE = 128 blocks
State:
RECONSTRUCT 0% complete
DISK20100 (member 0) is RECONSTRUCTING 0% complete
DISK20200 (member 1) is RECONSTRUCTING 0% complete
DISK20300 (member 2) is RECONSTRUCTING 0% complete
DISK20400 (member 3) is RECONSTRUCTING 0% complete
DISK40100 (member 4) is RECONSTRUCTING 0% complete
DISK40200 (member 5) is RECONSTRUCTING 0% complete
DISK40300 (member 6) is RECONSTRUCTING 0% complete
DISK40400 (member 7) is RECONSTRUCTING 0% complete
DISK60100 (member 8) is RECONSTRUCTING 0% complete
DISK10800 (member 9) is RECONSTRUCTING 0% complete
DISK60300 (member 10) is RECONSTRUCTING 0% complete
DISK60400 (member 11) is RECONSTRUCTING 0% complete
Size: NOT YET KNOWN
HSG> show d1 full
^
Command syntax error at or near here
HSG> show d1
LUN Uses Used by
------------------------------------------------------------------------------

D1 RAID1
LUN ID: 6000-1FE1-000A-E080-0009-0410-5815-001C
NOIDENTIFIER
Switches:
RUN NOWRITE_PROTECT READ_CACHE
READAHEAD_CACHE WRITEBACK_CACHE
MAXIMUM_CACHED_TRANSFER_SIZE = 32
Access:
ALL
State:
INOPERATIVE
Unit has lost data
WRITE_PROTECT - DATA SAFETY
Size: NOT YET KNOWN
Geometry (C/H/S): NOT YET KNOWN

[pasquino bocci_1]

hi,
1) The HSG80 ports are set to loop, should be fabric. What kind of switch do you have?
2) try clear_errors unit number lost_data.

Pasquino

[Rob Leadbeater]

Looking at the state of the unit:

State:
INOPERATIVE
Unit has lost data
WRITE_PROTECT - DATA SAFETY

I suspect that when the RAID array has finished reconstructing, that things *might* come back to life. It looks as though the controllers have disabled access to the disk to try and keep it in a consistent state whilst the rebuild takes place...

However if I were you, I'd be making sure that you can read the customer's back up tapes sooner rather than later... Given a 12 member RAID array it might be quicker to give up waiting on the rebuild and just get the data back off tape - assuming you can of course !

It's also worth noting that ACS 8.5-0 is rather old now. Latest release is 8.8-4.

Cheers,

Rob

[Nam, ill]

Thanks a lot Pasquino, Rob..

I just ran "clear_errors ... lost_data" command and after restart systems it came up good status. Now, I can access storage data from the server so it has running data backup.

Thank you so much!