topic Re: rp3440 connected to msa30 cannot activate vg in Operating System - HP-UX

rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Thu, 25 Aug 2011 20:03:04 GMT

Good morning: I have a customer who has problems on activiting a volume group with 2 146 GB mirrored disks just after we power off and bring up a rp3440 server connected to a msa30 where the disks are placed. I have found an article here that talks about missing disks on msa30 apparently as not attached to the controller after reboot.
http://h30499.www3.hp.com/t5/LVM-and-VxVM/missing-vg/m-p/5013237#M43458

I think this was a little different from the problem we had. In the problem presented
in the forum the SA was eventually able to activate his volume groups. Then he ran
fsck and was able to mount his filesystems. Our problem is different from this in that
we could not activate the volume group by any means. We could see the disks in the
array with sautil, but the volume group information seemed to be corrupted and could
not be restored/rebuilt.

I put info attached here, if anyone has an idea what could be the problem it would be helpful
Thanks and regards

I would want you to take a look at the following link http://h30499.www3.hp.com/t5/LVM-and-VxVM/missing-vg/m-p/5013237#M43458 , in case when you troubleshooted you encountered similar behaviour

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Mon, 21 Mar 2011 11:42:21 GMT

Has anyone an idea to help on this trouble ?
Thanks

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Mon, 21 Mar 2011 12:08:27 GMT

This is the current status:

---------- LOGICAL DRIVE 1 ----------

Device File = c4t0d1
RAID Level = 1+0
Size = 140009 MB
Stripe Size = 128 KB
Status = FAILED

Participating Physical Drive(s):

Ch ID

2 0
2 1

Did you replace BOTH disks???

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Mon, 21 Mar 2011 12:16:58 GMT

Yes, we replaced both and next to it everything ok, but this always happened as soon as we had to shut down and power on server and storage and we have to find why to prevent it

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Mon, 21 Mar 2011 16:15:08 GMT

You activated the RAID again, re-created VG. LVOLs and filesystems?

And after a reboot the RAID is failed again?

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Mon, 21 Mar 2011 16:24:31 GMT

Yes, but only in the case when we put new disks, if the system is rebooted and restarted, we could not activate the volume group by any means in spite of being able to see the disks in the array with sautil, but the volume group information seemed to be corrupted and could not be restored/rebuilt.

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Mon, 21 Mar 2011 16:56:39 GMT

Once the RAID is in status FAILED and you replace BOTH disk, all the data is gone.

The RAID remains in FAILED status.

You need to run

# sautil /dev/ciss4 accept _media _xchg 1

to enable the RAID again, but the disks are empty now.

So you need to get rid of the old VG information stored in the system (e.g. vgexport); then pvcreate, vgcreate, lvcreate etc ...

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Mon, 21 Mar 2011 17:05:47 GMT

Ok, but the question is : why this happen? why after that we reboot the info is corrupted and we see the disks but cannot activate de vg

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Mon, 21 Mar 2011 17:15:13 GMT

If I understand you right, you replaced the 2 disk - there nothings remains in this RAID, so there is no data at all. For this reason the RAID remains in FAILED status, even with new, good disks.

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Mon, 21 Mar 2011 17:53:15 GMT

Since this is a new "disk", you can also restore the LVM configuration on the LUN (vgcfgrestore); but in each case you need to use newfs to create a new file system on the LVOLs.

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Mon, 21 Mar 2011 17:57:37 GMT

Ok, we know how to solve it but the question is WHY is it happening.... From the logs I attached, can you see any info to address it on the future? thanks

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Mon, 21 Mar 2011 18:04:46 GMT

I don't know why you replaced both disks, but since you replaced both disks in this RAID all the data is gone as expected.

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Mon, 21 Mar 2011 18:12:07 GMT

We changed both because it was impossible to activate the vg

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Mon, 21 Mar 2011 18:31:39 GMT

Do you have any data collection from a point in time before replacing the 2 disks (sautil/saconfig)?

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Mon, 21 Mar 2011 18:47:36 GMT

saconfig

******************** SmartArray RAID Controller /dev/ciss4 ********************

Auto-Fail Missing Disks at Boot = enabled
Cache Configuration Status = write cache temporarily disabled (code=1)
Cache Ratio = 50% Read / 50% Write

---------- PHYSICAL DRIVES ----------

Location Ch ID Size Status

Internal 1 0 146.8 GB OK
Internal 1 1 146.8 GB OK
External 2 0 146.8 GB OK
External 2 1 146.8 GB OK

---------- LOGICAL DRIVE 0 ----------

Device File = c4t0d0
RAID Level = 1+0
Size = 140009 MB
Stripe Size = 128 KB
Status = OK

Participating Physical Drive(s):

Ch ID

1 0
1 1

Participating Spare Drive(s):

None

---------- LOGICAL DRIVE 1 ----------

Device File = c4t0d1
RAID Level = 1+0
Size = 140009 MB
Stripe Size = 128 KB
Status = FAILED

Participating Physical Drive(s):

Ch ID

2 0
2 1

Participating Spare Drive(s):

None
-------------------------------------
sautil

******************************************************************************
**** ****
**** S A U T I L S u p p o r t U t i l i t y ****
**** ****
**** for the HP SmartArray RAID Controller Family ****
**** ****
**** version A.02.12 ****
**** ****
**** (C) Copyright 2003-2007 Hewlett-Packard Development Company, L.P. ****
******************************************************************************

---- DRIVER INFORMATION ------------------------------------------------------

Driver State........................ READY

---- CONTROLLER INFORMATION --------------------------------------------------

Controller Product Number........... A9891A
Controller Product Name............. HP PCI-X SmartArray 6404 (Ch A1, A2)
Hardware Path....................... 0/2/1/0/4/0
Device File......................... /dev/ciss4

---- DRIVER STATISTICS -------------------------------------------------------

Rover Statistics:
Rover busy.......................... 0
Rover cmd failed.................... 0
Rover invalid class................. 0
Rover invalid sub class............. 0
Rover invalid detail................ 0
IOCTL command failed................ 0
IOCTL call failed................... 0
Invalid IOCTL....................... 0
Internal driver cmd failed.......... 0
Resets.............................. 0
Probe failed........................ 0
Heartbeat lost...................... 0

Events Statistics:
Init................................ 1
FW ready............................ 1
Init ok............................. 1
Init failed......................... 0
Suspend............................. 0
Suspended........................... 0
Download............................ 0
IO quiesced......................... 0
HW quiet............................ 0
Reset ok............................ 0
Download ok......................... 0
IO quiesce timeout.................. 0
Download timeout.................... 0
HW quiet timeout.................... 0
Reset timeout....................... 0
FW lockup........................... 0
Invalid event....................... 0

OLAR Statistics:
Suspend requested................... 0
Suspend forced...................... 0
Resume request...................... 0
Resume failed....................... 0

Driver Command Statistics:
Completed after timeout............. 0
SM Command Failed................... 0
IOs in activeQ (suspd).............. 0
Driver not ready open failed........ 0
Current lun open count.............. 1
Request failed - drv state.......... 0
CMD allocation failure.............. 0
IOs queued on send Q................ 0
Zero sized IO....................... 33
DMA setup fail count................ 0
Out of sgle......................... 0
Inquiries to NE LUN 0............... 0
CAC with no sense data.............. 0
Non active powerfail IO............. 0
Underruns........................... 6
Overruns............................ 0
Command invalid..................... 0
Other IO errors..................... 0
Interrupt while state invalid....... 0
Spurious interrupts................. 5
Not SU failure...................... 0
External register read request...... 0
PCI config reads.................... 40
Invalid register type............... 0
Invalid register length............. 0
Invalid register offset............. 0
Invalid register alignment.......... 0
Delete lun while open............... 0
Data copy failed.................... 0
DMA map failed...................... 0
BAR map failed...................... 0

******************************************************************************
**** End of SAUTIL Output ****
******************************************************************************

Another log

CURRENT MONITOR DATA:

Event Time..........: Sun Nov 14 19:50:29 2010
Severity............: CRITICAL
Monitor.............: dm_raid_adapter
Event #.............: 2
System..............: mvdh05.lac.nsroot.net

Summary:
Adapter at hardware path 0/2/1/0/4/0 : CISS: RAID SA controller is now
online.

Description of Error:

lbolt value: 2900

CISS: RAID SA controller is now on-line.

Probable Cause / Recommended Action:

No Action required. Information message only.

Additional Event Data:
System IP Address...: 169.167.27.10
Event Id............: 0x4ce059a500000000
Monitor Version.....: B.01.00
Event Class.........: I/O
Client Configuration File...........:
/var/stm/config/tools/monitor/default_dm_raid_adapter.clcfg
Client Configuration File Version...: A.01.00
Qualification criteria met.
Number of events..: 1
Associated OS error log entry id(s):
0x4ce0580400000000
Additional System Data:
System Model Number.............: 9000/800/rp3440
OS Version......................: B.11.11
EMS Version.....................: A.04.20
STM Version.....................: A.57.00
Latest information on this event:
http://docs.hp.com/hpux/content/hardware/ems/dm_raid_adapter.htm#2

v-v-v-v-v-v-v-v-v-v-v-v-v D E T A I L S v-v-v-v-v-v-v-v-v-v-v-v-v

I/O Log Event Data:

Driver Status Code..................: 0x00000002
Length of Logged Hardware Status....: 0 bytes.
Offset to Logged Manager Information: 0 bytes.
Length of Logged Manager Information: 12 bytes.

Manager-Specific Information:

Raw data from the SCSI RAID SA Controller CISS driver:
00000004 00000B54 00000000

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Mon, 21 Mar 2011 18:58:51 GMT

This is the same as in your first post, but I don't know if this is made with the new or old disks...

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Mon, 21 Mar 2011 19:08:45 GMT

With the old

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Tue, 22 Mar 2011 04:38:55 GMT

Not enough information. You cannot say what the problem was.
Both disks are shown as good, but the RAID is failed.

Re: rp3440 connected to msa30 cannot activate vg

Gabriel Madoz — Tue, 22 Mar 2011 11:39:44 GMT

Ok, this was always the situation and once we changed smart array without improval changes

Re: rp3440 connected to msa30 cannot activate vg

Torsten. — Tue, 22 Mar 2011 11:51:57 GMT

But now the problem is solved?

I'm not sure what was in the past and what is the current status ...

Next time use "sainfo" to make a data collection in case of problems.