Issue with LVM Disk and Help in Creating Disk

[Jon Le]

Hello everyone. Here are some details beforehand.

ioscan:
I have a HP-UX 9000, class RP5430.

the following scan is for a DS2100 external disk enclosure that has 3 harddrive in place.

0/3/0/0.0.0
sdisk claimed device seagate ST318404LC
/dev/dsk/c7t0d0
/dev/rdsk/c7t0d0

0/3/0/0.2.0
sdisk claimed device HP 18.2GST318406LC
/dev/dsk/c7t2d0
/dev/rdsk/c7t2d0

0/3/0/0.4.0
sdisk claimed device HP 18.2GST318406LC
/dev/dsk/c7t4d0
/dev/rdsk/c7t4d0

I had a disk go bad or a configuration error but I was using c7t0d0. However that disk got replaced and I want to use c7t0d0.

When I go into SAM in Disk Devices, it shows all 3 "unused".

Here is what I am trying to do. Begin to use all 3 disk. Each of the disk is 18.2 gig so I would like to have a volume group of 54.6 gig and possibly add more at a later date.

Here is the problem that I am running into:

when I do the following, i get the this error:

# pvdisplay /dev/dsk/c7t0d0
pvdisplay: Warning: couldn't query physical volume "/dev/dsk/c7t0d0": This specified path does not correspond to physical volume attached to this volume group
pvdisplay: warning: couldn't query all of the physical volumes.
pvdisplay; couldn't retrieve the names of the physical volumes belonging to volume group "/dev/test"
pvdisplay: cannot display physical volume "/dev/dsk/c7t0d0"

#vgdisplay /dev/test
vgdisplay: volume group not activated
vgdisplay: cannot display volume group "/dev/test"

#lvdisplay /dev/test/lvol1
lvdisplay: couldn't query logical volume "/dev/test/lvol1"
volume group not activated.

How do I go about resolving this error??

Also if you can provide me with a step by step instruction on how to start using the 3 disk. I know the steps involved but would like to confirm it with you guys. Thank you for your help in this matter.

[James R. Ferguson]

Hi Jon:

> vgdisplay /dev/test
> vgdisplay: volume group not activated

This means that the volume group is not active, so do:

# vgchange -a y /dev/test

Regards!

...JRF...

[Jon Le]

tried to activate it however here is what i am getting:

# vgchange -a y /dev/test

vgchange: warning: couldn't attach to the volume group physical volume "/dev/dsk/c7t0d0":
Cross device link
vgchange: warning: couldn't query physical volume "/dev/dsk/c7t0d0":
the specified path does not correspond to physical volume attached to this volume group
vgchange: warning: couldn't query all of the physical volume.
vgchange: couldn't activate volume group "/dev/test"
Quorum not present, or some physical volume(s) are missing.

Thank you for the quick response.

[melvyn burnard · Accepted solution]

well if your disk has been replaced, you need to recreate the LVM headers on it.
Firstly though, do a strings /etc/lvmatb and verify what disks are in vg named test

If all three disks are listed, you can use vgcfgrestore to recreate the headers from the backupfile
man vgcfgrestore.

[Jon Le]

when I do a strings /etc/lvmtab

i get the following:

# strings /etc/lvmtab

/dev/vg01
/dev/dsk/c1t0d0
/dev/dsk/c2t0d0
/dev/dsk/c2t2d0
/dev/vg00
/dev/dsk/c1t2d0
/dev/test
/dev/dsk/c7t0d0

This is what is in my fstab file:

/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs delaylog 0 1
/dev/vg00/lvol4 /tmp vxfs delaylog 0 2
/dev/vg00/lvol5 /home vxfs delaylog 0 2
/dev/vg00/lvol6 /opt vxfs delaylog 0 2
/dev/vg00/lvol7 /usr vxfs delaylog 0 2
/dev/vg00/lvol8 /var vxfs delaylog 0 2

/dev/vg01/data /data vxfs rw,suid,largefiles,delaylog,datainlog 0 2

/dev/test/lvol1 /test vxfs rw,suid,largefiles,delaylog,datainlog 0 2

[Torsten.]

The disk is still empty, but you have a valid VG backup for this disk.

# vgcfgrestore -n vgtest /dev/rdsk/c7t0d0

will restore the config.

# vgchange -a y vgtest

will activate the VG.

# vgdisplay -v vgtest

Now you need to do a "newfs" to the LVOLs (if there are any) to make them useable.

[Jon Le]

how do I start from scratch. Delete everything and begin from nothing.

[Torsten.]

You need to vgexport the vgtest in order to remove it from /etc/lvmtab.

man vgexport

[Jon Le]

thank you, i would run the following:

# vgexport -v -f backup /dev/test


after I do that, I can start with

1. creating a PV (pvcreate)
2. vgcreate
3. lvcreate

[Tim Nelson]

Here is what I would do after a bad disk replacement ( you can vgexport as mentioned because the data is gone any way) but this is good practice.

Remove old drive, insert new one.
ioscan
ioscan -kfnC disk ( look for your c7t0d0 disk as claimed)

vgcfgrestore -l -n /dev/vgtest ( this lists what devices the vgtest thinks it has)
vgcfgrestore -n /dev/vgtest /dev/rdsk/c7t0d0
vgchange -a y /dev/vgtest
vgdisplay -v /dev/vgtest

( In this case because you only had one drive and it is replaced, data is gone so.. )

newfs /dev/vgtest/rlvol1
mount /test-filesystem

or just start over

vgexport /dev/vgtest
pvcreate /dev/rdsk/c7t0d0
mkdir /dev/vgtest
mknod /dev/vgtest/group c 64 0x0Y0000 (pick a unique minor number for Y )
vgcreate /dev/vgtest /dev/dsk/c7t0d0
lvcreate -L SIZE_MB -n NAME /dev/vgtest
newfs /dev/vgtest/rNAME
edit /etc/fstab
mount /new_test_filesstem

[Jon Le]

thank you very much for the quick response.

I did the vgexport and was successful.

If I wanted to use all 3 disk combined together. I would do the following: correct me if any.

# pvcreate /dev/rdsk/c7t0d0
# pvcreate /dev/rdsk/c7t2d0
# pvcreate /dev/rdsk/c7t4d0

# mkdir /dev/vgtest
# mknod /dev/vgtest/group c 64 0x020000

the vgcreate is the one that I am confused about. There are multiple PV.

Can you provide me with the rest of the command to utilize the full extent of the 3 disk. Thank you so much for everyone's help.

I do have the HP-UX CSA desk reference but would like to get your input.

[Torsten.]

Now it depends if you want to add a larger disk anytime in future and if you want to stripe the LVOLs across all disks or if you want to mirror 1 disk to another.

[Jon Le]

eventually i would like to mirror the disk. I have the mirror software but haven't had a chance to implement it yet.

[Torsten.]

If you mirror the question is: How about the third disk?

Depending on your hp-ux version you can either stripe **or** mirror the data.

Still the question is - would you like to add a larger disk in future?

BTW, are you running 11.11?

[James R. Ferguson]

Hi Jon:

# vgcreate /dev/vgtest /dev/dsk/c7t0d0 /dev/dsk/c7t2d0 /dev/dsk/c7t4d0

...will create a volume group (with defaults) containing the three physical disk provided as the arguments. The manpages for 'vgcreate' are your friend.

Regards!

...JRF...

[Torsten.]

I don't know how you want to use this VG in future ... but since a 73GB disk is the largest supported in this DS2100 I would extend the max_PE value to be able to include a 73GB disk later. Calculate the values depending on the PE_size you want to use.

[Jon Le]

If you mirror the question is: How about the third disk?

Depending on your hp-ux version you can either stripe **or** mirror the data.

Still the question is - would you like to add a larger disk in future?

BTW, are you running 11.11?

Well in the near future, I will be purchasing another DS that will have 4 drive in it so that I can mirror all the drives. Right now, there is not enough disk to mirror .

Currently there are 7 disk.

4 are internal disk having 18.2 gig each.
c1t0d0, c1t2d0, c2t0d0, c2t2d0.

/dev/vg01
/dev/dsk/c1t0d0
/dev/dsk/c2t0d0
/dev/dsk/c2t2d0
/dev/vg00
/dev/dsk/c1t2d0

those contain the boot and a majority of it is being used for storage of an application. I was going to use the DS2100 that I am configuring now to mirror those drive but I want to use it as an test environment.

On 2nd thought, I only really need to use one of the drive for this test environment. I don't need it to be mirrored as it is a test environment. Can I use the other 2 drive to mirror 2 of the internal drive? how do i go about doing that.

[Michael Steele_2]

Hi:

Regarding "...Can I use the other 2 drive to mirror 2 of the internal drive? how do i go about doing that. ..."

You can do either. But there are some rules to follow when you mirror. You should use different controllers.

c0t0d0
c1t0d0

Here c0 and c1 are controllers. If you use c0 and c0 then you have a single point of failure (* SPOF *).

The targets should be the same. Although this may no longer be necessary in disk arrays, for SCSI JBODS its a must due to write scheduling priorities.

c0t0d0 and c1t0d0
not
c0t1do and c1t3do

Here t0, t1 and t3 are targets. The first example (* c0t0d0 and c1t0d0 *) are fine, the second is not.

Always use strict allocation policy. This is the default. This keeps you from mirroring extents back onto the primary. Like in your example of three disks. Eventually you will not be able to mirror additional logical volumes because of the strict allocation policy. In short, you want alternates to be identical to primaries (* mirrors *) and a three disk vg breaks this rule. One disk will not be mirrored.

[Jon Le]

thank you. What if I get a 72 gb drive for the ds2100 to occupy the last opening.

Right now on the ds2100 there are 4 slots. 3 of which is being used by 18.2 gig drives. If I get a 72 gb drive for the 4th slot. Can I use that 72 gb to mirror the 4 internal drive?

Here is my configuration:

HP9000 (internal drives):
c1t0d0
c1t2d0
c2t0do
c2t2d0

DS2100:
c7t0d0
c7t2d0
c7t4d0

Here is the VG and LV for HP9000:

c1t0d0, c2t0d0, c2t2d0 = vg01
with only one LV name of "storage"
storage - /storage - 52080 MB

c1t2d0 = vg00
with 8 LV.
vol1 - /stand - 300 MB
vol2 - /swap - 3072 MB
vol3 - / - 200 MB
vol4 - /tmp - 200 MB
vol5 - /home - 20 MB
vol6 - /opt - 664 MB
vol7 - /usr - 1104 MB
vol8 - /var - 3572 MB

So total of 9132 MB of c1t2d0.

What is your recommendation?

[Michael Steele_2]

This may or may not work. You never get the full 72 GB. LVM will always subtract something for itself. For example, the VGRA header.

You also run into another rule. Any vg with a 73 GB disk will have to be the first disk added into the vg. If you add a 18 GB disk in first, and then vgextend in the 73 GB disk, then you'll only see the first 18 GB on this 73 GB disk.

Its worth a try but I think you'll end up with 2 complete mirrors and a 3rd mirror missing the last lvol.

[Michael Steele_2]

Wait a minute. 3 18 GB disks is only 54 GB. So you've got more than enough room. Just follow the 2nd comment and add the 72 GB disk into the vg first.

[Jon Le]

so I would order the 72 gb drive and place it into the 4th slot on the ds2100...which would give it a hardware path of

0/3/0/0.6.0
c7t6d0

When go to create the volume group, i would do the following:

#vgcreate /dev/vgtest /dev/dsk/c7t6d0

After doing that, I can use the 72 gig (c7t6d0) to mirror the 4 drive (c1t0d0, c1t2d0, c2t0d0, c2t2d0)

[Torsten.]

If you do something like

#vgcreate -s 4 -e 20000 /dev/vgtest ...

you can create the VG now and add the larger disk later.

[Jon Le]

trying to do a pvcreate on c7t2d0 and get the following

# pvcreate /dev/rdsk/c7t2d0
pvcreate: the physical volume already belongs to a volume group


I looked at the volume group and do not see anything using the PV c7t2d0

[Trevor Roddam_1]

Jon.

Noting that you may already have solved your issues.
Please read http://docs.hp.com/en/5991-1236/When_Good_Disks_Go_Bad.pdf

As this will explain what should be done when good disks go bad. It has a step by step instruction.

What looks like happened is that you didn't restore the LVM configuration to the replacement disk. This causes LVM to get out of sync with the /etc/lvmtab and reality.

In short, you can recreate the lvmtab by using vgscan, make sure you have a backup of /etc/lvmtab. (High risk but great rewards). This may be the only way out as you have done many things to fix the issue.

or replace the original dead HDD and remove it from the VG

or restore the LVM config to the new HDD.

Please read the docco even if you have resolved the issue as it will make it a lot easier if it happens again and outlines the best practice disk replacement stratagies.

Trevor.