rp3440 connected to msa30 cannot activate vg

[Gabriel Madoz]

Good morning: I have a customer who has problems on activiting a volume group with 2 146 GB mirrored disks just after we power off and bring up a rp3440 server connected to a msa30 where the disks are placed. I have found an article here that talks about missing disks on msa30 apparently as not attached to the controller after reboot.
http://h30499.www3.hp.com/t5/LVM-and-VxVM/missing-vg/m-p/5013237#M43458

I think this was a little different from the problem we had. In the problem presented
in the forum the SA was eventually able to activate his volume groups. Then he ran
fsck and was able to mount his filesystems. Our problem is different from this in that
we could not activate the volume group by any means. We could see the disks in the
array with sautil, but the volume group information seemed to be corrupted and could
not be restored/rebuilt.

I put info attached here, if anyone has an idea what could be the problem it would be helpful
Thanks and regards





I would want you to take a look at the following link http://h30499.www3.hp.com/t5/LVM-and-VxVM/missing-vg/m-p/5013237#M43458, in case when you troubleshooted you encountered similar behaviour

[Gabriel Madoz]

Has anyone an idea to help on this trouble ?
Thanks

[Torsten.]

This is the current status:

---------- LOGICAL DRIVE 1 ----------

Device File = c4t0d1
RAID Level = 1+0
Size = 140009 MB
Stripe Size = 128 KB
Status = FAILED

Participating Physical Drive(s):

Ch ID

2 0
2 1




Did you replace BOTH disks???

[Gabriel Madoz]

Yes, we replaced both and next to it everything ok, but this always happened as soon as we had to shut down and power on server and storage and we have to find why to prevent it

[Torsten.]

You activated the RAID again, re-created VG. LVOLs and filesystems?

And after a reboot the RAID is failed again?

[Gabriel Madoz]

Yes, but only in the case when we put new disks, if the system is rebooted and restarted, we could not activate the volume group by any means in spite of being able to see the disks in the array with sautil, but the volume group information seemed to be corrupted and could not be restored/rebuilt.

[Torsten.]

Once the RAID is in status FAILED and you replace BOTH disk, all the data is gone.

The RAID remains in FAILED status.

You need to run

# sautil /dev/ciss4 accept _media _xchg 1

to enable the RAID again, but the disks are empty now.

So you need to get rid of the old VG information stored in the system (e.g. vgexport); then pvcreate, vgcreate, lvcreate etc ...

[Gabriel Madoz]

Ok, but the question is : why this happen? why after that we reboot the info is corrupted and we see the disks but cannot activate de vg

[Torsten.]

If I understand you right, you replaced the 2 disk - there nothings remains in this RAID, so there is no data at all. For this reason the RAID remains in FAILED status, even with new, good disks.

[Torsten.]

Since this is a new "disk", you can also restore the LVM configuration on the LUN (vgcfgrestore); but in each case you need to use newfs to create a new file system on the LVOLs.

[Gabriel Madoz]

Ok, we know how to solve it but the question is WHY is it happening.... From the logs I attached, can you see any info to address it on the future? thanks

[Torsten.]

I don't know why you replaced both disks, but since you replaced both disks in this RAID all the data is gone as expected.

[Gabriel Madoz]

We changed both because it was impossible to activate the vg

[Torsten.]

Do you have any data collection from a point in time before replacing the 2 disks (sautil/saconfig)?

[Gabriel Madoz]

saconfig

******************** SmartArray RAID Controller /dev/ciss4 ********************

Auto-Fail Missing Disks at Boot = enabled
Cache Configuration Status = write cache temporarily disabled (code=1)
Cache Ratio = 50% Read / 50% Write

---------- PHYSICAL DRIVES ----------

Location Ch ID Size Status

Internal 1 0 146.8 GB OK
Internal 1 1 146.8 GB OK
External 2 0 146.8 GB OK
External 2 1 146.8 GB OK

---------- LOGICAL DRIVE 0 ----------

Device File = c4t0d0
RAID Level = 1+0
Size = 140009 MB
Stripe Size = 128 KB
Status = OK

Participating Physical Drive(s):

Ch ID

1 0
1 1

Participating Spare Drive(s):

None

---------- LOGICAL DRIVE 1 ----------

Device File = c4t0d1
RAID Level = 1+0
Size = 140009 MB
Stripe Size = 128 KB
Status = FAILED

Participating Physical Drive(s):

Ch ID

2 0
2 1

Participating Spare Drive(s):

None
-------------------------------------
sautil

******************************************************************************
**** ****
**** S A U T I L S u p p o r t U t i l i t y ****
**** ****
**** for the HP SmartArray RAID Controller Family ****
**** ****
**** version A.02.12 ****
**** ****
**** (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P. ****
******************************************************************************



---- DRIVER INFORMATION ------------------------------------------------------

Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

Controller Product Number........... A9891A
Controller Product Name............. HP PCI-X SmartArray 6404 (Ch A1, A2)
Hardware Path....................... 0/2/1/0/4/0
Device File......................... /dev/ciss4

---- DRIVER STATISTICS -------------------------------------------------------

Rover Statistics:
Rover busy.......................... 0
Rover cmd failed.................... 0
Rover invalid class................. 0
Rover invalid sub class............. 0
Rover invalid detail................ 0
IOCTL command failed................ 0
IOCTL call failed................... 0
Invalid IOCTL....................... 0
Internal driver cmd failed.......... 0
Resets.............................. 0
Probe failed........................ 0
Heartbeat lost...................... 0

Events Statistics:
Init................................ 1
FW ready............................ 1
Init ok............................. 1
Init failed......................... 0
Suspend............................. 0
Suspended........................... 0
Download............................ 0
IO quiesced......................... 0
HW quiet............................ 0
Reset ok............................ 0
Download ok......................... 0
IO quiesce timeout.................. 0
Download timeout.................... 0
HW quiet timeout.................... 0
Reset timeout....................... 0
FW lockup........................... 0
Invalid event....................... 0

OLAR Statistics:
Suspend requested................... 0
Suspend forced...................... 0
Resume request...................... 0
Resume failed....................... 0

Driver Command Statistics:
Completed after timeout............. 0
SM Command Failed................... 0
IOs in activeQ (suspd).............. 0
Driver not ready open failed........ 0
Current lun open count.............. 1
Request failed - drv state.......... 0
CMD allocation failure.............. 0
IOs queued on send Q................ 0
Zero sized IO....................... 33
DMA setup fail count................ 0
Out of sgle......................... 0
Inquiries to NE LUN 0............... 0
CAC with no sense data.............. 0
Non active powerfail IO............. 0
Underruns........................... 6
Overruns............................ 0
Command invalid..................... 0
Other IO errors..................... 0
Interrupt while state invalid....... 0
Spurious interrupts................. 5
Not SU failure...................... 0
External register read request...... 0
PCI config reads.................... 40
Invalid register type............... 0
Invalid register length............. 0
Invalid register offset............. 0
Invalid register alignment.......... 0
Delete lun while open............... 0
Data copy failed.................... 0
DMA map failed...................... 0
BAR map failed...................... 0

******************************************************************************
**** End of SAUTIL Output ****
******************************************************************************

Another log

CURRENT MONITOR DATA:

Event Time..........: Sun Nov 14 19:50:29 2010
Severity............: CRITICAL
Monitor.............: dm_raid_adapter
Event #.............: 2
System..............: mvdh05.lac.nsroot.net

Summary:
Adapter at hardware path 0/2/1/0/4/0 : CISS: RAID SA controller is now
online.



Description of Error:


lbolt value: 2900

CISS: RAID SA controller is now on-line.


Probable Cause / Recommended Action:

No Action required. Information message only.

Additional Event Data:
System IP Address...: 169.167.27.10
Event Id............: 0x4ce059a500000000
Monitor Version.....: B.01.00
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_dm_raid_adapter.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
0x4ce0580400000000
Additional System Data:
System Model Number.............: 9000/800/rp3440
OS Version......................: B.11.11
EMS Version.....................: A.04.20
STM Version.....................: A.57.00
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/dm_raid_adapter.htm#2

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v



I/O Log Event Data:

Driver Status Code..................: 0x00000002
Length of Logged Hardware Status....: 0 bytes.
Offset to Logged Manager Information: 0 bytes.
Length of Logged Manager Information: 12 bytes.

Manager-Specific Information:

Raw data from the SCSI RAID SA Controller CISS driver:
00000004 00000B54 00000000

[Torsten.]

This is the same as in your first post, but I don't know if this is made with the new or old disks...

[Gabriel Madoz]

With the old

[Torsten.]

Not enough information. You cannot say what the problem was.
Both disks are shown as good, but the RAID is failed.

[Gabriel Madoz]

Ok, this was always the situation and once we changed smart array without improval changes

[Torsten.]

But now the problem is solved?

I'm not sure what was in the past and what is the current status ...



Next time use "sainfo" to make a data collection in case of problems.

[Gabriel Madoz]

According to this question I have more info to give:

 Every time you do data
> center maintenance involving mvdh05, the array believes that one of the
> disks has been exchanged .. and I have to use the "accept_media_xchg"
> option
> of the sautil command ::
>
>      Sautil /dev/ciss4 accept_media_xchg 1
>
>   Once this is done, I am able to access the logical volume again .. and
> vary on the volume group.  This happens every time we have a complete
> shutdown of this sytem .. this cannot be normal, since we do not have the
> same problem with mvdh03. 

I attach sautil from server mvdh03 that has no problem when it boots and from mvdh05 that has this problem. I see that firmware on mvdh05 is older than mvdh03, could it be a problem?

Any help would be grateful