Set up vg, lv - Getting started on RHEL5

[rsmith_1]

I am fairly new to Linux. Have very good experirnce in HP-UX lvm. I have this server where i have to set up VG, lv and start from scratch. It is a test box so we can erase and rebuild. We are using the Linux Native multipathing. I find the fdisk -l ouput confusing and not sure which device to start the pvcreate. I have posted lsscsi and fdisk -l output, assuming we have to create 1 VG on each individual path could someone guide me please?

#lsscsi
[0:2:0:0] disk DELL PERC 6/i 1.22 /dev/sda
[0:2:1:0] disk DELL PERC 6/i 1.22 /dev/sdb
[0:2:2:0] disk DELL PERC 6/i 1.22 /dev/sdc
[0:2:3:0] disk DELL PERC 6/i 1.22 /dev/sdd
[1:0:0:0] cd/dvd PLDS DVD+-RW DS-8A4S JD51 /dev/sr0
[3:0:0:0] storage HP HSV450 0953 -
[3:0:0:1] disk HP HSV450 0953 /dev/sde
[3:0:0:2] disk HP HSV450 0953 /dev/sdf
[3:0:1:0] disk EMC SYMMETRIX 5874 /dev/sdg
[3:0:1:1] disk EMC SYMMETRIX 5874 /dev/sdh
[3:0:1:2] disk EMC SYMMETRIX 5874 /dev/sdi
[5:0:0:0] storage HP HSV450 0953 -
[5:0:0:1] disk HP HSV450 0953 /dev/sdj
[5:0:0:2] disk HP HSV450 0953 /dev/sdk
[5:0:1:0] disk EMC SYMMETRIX 5874 /dev/sdl
[5:0:1:1] disk EMC SYMMETRIX 5874 /dev/sdm
[5:0:1:2] disk EMC SYMMETRIX 5874 /dev/sdn

# fdisk -l

Disk /dev/sda: 74.8 GB, 74859937792 bytes
255 heads, 63 sectors/track, 9101 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 33 265041 83 Linux
/dev/sda2 34 2122 16779892+ 83 Linux
/dev/sda3 2123 2775 5245222+ 83 Linux
/dev/sda4 2776 9101 50813595 5 Extended
/dev/sda5 2776 4733 15727603+ 83 Linux
/dev/sda6 4734 6691 15727603+ 83 Linux
/dev/sda7 6692 9101 19358293+ 83 Linux

Disk /dev/sdb: 74.8 GB, 74859937792 bytes
255 heads, 63 sectors/track, 9101 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 2089 16779861 82 Linux swap / Solaris

Disk /dev/sdc: 74.8 GB, 74859937792 bytes
255 heads, 63 sectors/track, 9101 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Disk /dev/sdd: 74.8 GB, 74859937792 bytes
255 heads, 63 sectors/track, 9101 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Disk /dev/sde: 3221 MB, 3221225472 bytes
100 heads, 62 sectors/track, 1014 cylinders
Units = cylinders of 6200 * 512 = 3174400 bytes

Disk /dev/sde doesn't contain a valid partition table

Disk /dev/sdf: 3221 MB, 3221225472 bytes
100 heads, 62 sectors/track, 1014 cylinders
Units = cylinders of 6200 * 512 = 3174400 bytes

Disk /dev/sdf doesn't contain a valid partition table

Disk /dev/sdg (Sun disk label): 15 heads, 128 sectors, 4 cylinders
Units = cylinders of 1920 * 512 bytes

Device Flag Start End Blocks Id System
/dev/sdg3 u 0 4 3840 5 Whole disk
/dev/sdg7 0 4 3840 4 SunOS usr

Disk /dev/sdg3 (Sun disk label): 15 heads, 128 sectors, 4 cylinders
Units = cylinders of 1920 * 512 bytes

Device Flag Start End Blocks Id System
/dev/sdg3p3 u 0 4 3840 5 Whole disk
/dev/sdg3p7 0 4 3840 4 SunOS usr

Disk /dev/sdg7 (Sun disk label): 15 heads, 128 sectors, 4 cylinders
Units = cylinders of 1920 * 512 bytes

Device Flag Start End Blocks Id System
/dev/sdg7p3 u 0 4 3840 5 Whole disk
/dev/sdg7p7 0 4 3840 4 SunOS usr

Disk /dev/sdh: 3221 MB, 3221422080 bytes
100 heads, 62 sectors/track, 1014 cylinders
Units = cylinders of 6200 * 512 = 3174400 bytes

Disk /dev/sdh doesn't contain a valid partition table

Disk /dev/sdi: 3221 MB, 3221422080 bytes
100 heads, 62 sectors/track, 1014 cylinders
Units = cylinders of 6200 * 512 = 3174400 bytes

Disk /dev/sdi doesn't contain a valid partition table

Disk /dev/sdj: 3221 MB, 3221225472 bytes
100 heads, 62 sectors/track, 1014 cylinders
Units = cylinders of 6200 * 512 = 3174400 bytes

Disk /dev/sdj doesn't contain a valid partition table

Disk /dev/sdk: 3221 MB, 3221225472 bytes
100 heads, 62 sectors/track, 1014 cylinders
Units = cylinders of 6200 * 512 = 3174400 bytes

Disk /dev/sdk doesn't contain a valid partition table

Disk /dev/sdl (Sun disk label): 15 heads, 128 sectors, 4 cylinders
Units = cylinders of 1920 * 512 bytes

Device Flag Start End Blocks Id System
/dev/sdl3 u 0 4 3840 5 Whole disk
/dev/sdl7 0 4 3840 4 SunOS usr

Disk /dev/sdl3 (Sun disk label): 15 heads, 128 sectors, 4 cylinders
Units = cylinders of 1920 * 512 bytes

Device Flag Start End Blocks Id System
/dev/sdl3p3 u 0 4 3840 5 Whole disk
/dev/sdl3p7 0 4 3840 4 SunOS usr

Disk /dev/sdl7 (Sun disk label): 15 heads, 128 sectors, 4 cylinders
Units = cylinders of 1920 * 512 bytes

Device Flag Start End Blocks Id System
/dev/sdl7p3 u 0 4 3840 5 Whole disk
/dev/sdl7p7 0 4 3840 4 SunOS usr

Disk /dev/sdm: 3221 MB, 3221422080 bytes
100 heads, 62 sectors/track, 1014 cylinders
Units = cylinders of 6200 * 512 = 3174400 bytes

Disk /dev/sdm doesn't contain a valid partition table

Disk /dev/sdn: 3221 MB, 3221422080 bytes
100 heads, 62 sectors/track, 1014 cylinders
Units = cylinders of 6200 * 512 = 3174400 bytes

Disk /dev/sdn doesn't contain a valid partition table

# pvs
#vgs
No volume groups found
#dmsetup ls
No devices found

Thanks

Rsmith.

[Matti_Kurkela]

The /dev/sd* devices are like HP-UX /dev/(r)dsk/c*t*d* devices: when you're using multipathing, multiple devices can refer to the same disk/LUN via different paths.

You'll want to use multipath devices instead. They will be named /dev/mapper/mpath* by default.

NOTE: as the fdisk output of /dev/sda indicates it's likely your /var is a separate filesystem, it's important that you implement the fix/workaround listed in this RedHat Knowledge Base article before going any further with multipath disk setup:

https://access.redhat.com/kb/docs/DOC-17650

Basically:
- mkdir /etc/multipath
- mv /var/lib/multipath/bindings /etc/multipath/bindings
- ln -s /etc/multipath/bindings /var/lib/multipath/bindings
- add the line "bindings_file /etc/multipath/bindings" to the defaults section of /etc/multipath.conf.

This will avoid trouble later if/when you change your SAN disk configuration.


If your multipathing has not been initialized yet, first comment out the "denied everything" entry from the beginning of /etc/multipath.conf. On RHEL 5.x and older, you might want to add denied entries for your known non-multipathed disks, to minimize cluttering up your configuration.

Then run "multipath -v2" to detect the multipathed devices (-v2 just increases verbosity to tell you what is going on). Then make sure the multipathd daemon is running and will be started automatically at every boot:

service multipathd start
chkconfig multipathd on

Then, run "multipath -l" to see how many paths each actual disk/LUN has and which device maps to which. The "dmsetup ls --tree" command can do pretty much the same thing.

You'll also want to edit /etc/lvm/lvm.conf a little.
By default, it includes "preferred_names = [ ]" setting. There is also another commented-out preferred_names line, which is suitable for multipathed use:
-----
preferred_names = [ ]

# Try to avoid using undescriptive /dev/dm-N names, if present.
# preferred_names = [ "^/dev/mpath/", "^/dev/mapper/mpath", "^/dev/[hs]d" ]
-----

Comment out the default preferred_names setting and uncomment the alternative line below it. Then run "vgscan" to refresh LVM device cache.

Now you're ready to start setting up multipathed disks for LVM.

You can use the disks HP-UX style, and pvcreate the "whole disk" device. Or you can use fdisk to create a partition that encompasses the entire disk, and use pvcreate on that. Both ways will work. Creating a PC-style partition allows Windows sysadmins to notice that the LUN is already in use, if it's ever accidentally presented to a Windows system. But on the other hand, it wastes one cylinder's worth of disk capacity and makes the setup slightly more complex. The choice is yours.

You can use either the /dev/mapper/mpath* device or one of the single-path /dev/sd* devices in the pvcreate step: because pvcreate just writes a PV identifier to the actual disk/LUN, you can use any path you like at this step.

After this, you can proceed about the same as with HP-UX. One note though: although Linux LVM looks similar to HP-UX LVM, the internal workings are very different at some points. In particular, exporting/importing a VG does not mean at all the same thing as on HP-UX, and is used much less. On the other hand, Linux vgscan is essentially safe at any situation. (In fact, I think the udev subsystem runs "multipath" and "vgscan" automatically each time new disk devices are detected.)

MK

[rsmith_1]

MK

Thanks much. I denied the devices and ran multipath -v2. The /var/lib/multipath/bindings file was never present so mv did not work!

Now my question is

should i pvcreate on /dev/mapper/mpath devices? Here is the ouput after running multipath -v2

# ll /dev/mapper
total 0
crw------- 1 root root 10, 63 May 19 19:35 control
brw-rw---- 1 root disk 253, 0 Jun 16 08:53 mpath2
brw-rw---- 1 root disk 253, 1 Jun 16 08:53 mpath3
brw-rw---- 1 root disk 253, 2 Jun 16 08:53 mpath4
brw-rw---- 1 root disk 253, 3 Jun 16 08:53 mpath5
brw-rw---- 1 root disk 253, 4 Jun 16 08:53 mpath6
brw-rw---- 1 root disk 253, 5 Jun 16 08:53 mpath7
brw-rw---- 1 root disk 253, 6 Jun 16 08:53 mpath8
# ll /dev/mpath
total 0
lrwxrwxrwx 1 root root 7 Jun 16 08:53 mpath2 -> ../dm-0
lrwxrwxrwx 1 root root 7 Jun 16 08:53 mpath3 -> ../dm-1
lrwxrwxrwx 1 root root 7 Jun 16 08:53 mpath4 -> ../dm-2
lrwxrwxrwx 1 root root 7 Jun 16 08:53 mpath5 -> ../dm-3
lrwxrwxrwx 1 root root 7 Jun 16 08:53 mpath6 -> ../dm-4
lrwxrwxrwx 1 root root 7 Jun 16 08:53 mpath7 -> ../dm-5
lrwxrwxrwx 1 root root 7 Jun 16 08:53 mpath8 -> ../dm-6
# multipath -l
mpath2 (36842b2b00d276d001426440d09714243) dm-0 DELL,PERC 6/i
[size=70G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 0:2:2:0 sdc 8:32 [active][undef]
mpath8 (360000970000192601379533030363639) dm-6 EMC,SYMMETRIX
[size=3.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 3:0:1:2 sdi 8:128 [active][undef]
\_ 5:0:1:2 sdn 8:208 [active][undef]
mpath7 (360000970000192601379533030363638) dm-5 EMC,SYMMETRIX
[size=3.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 3:0:1:1 sdh 8:112 [active][undef]
\_ 5:0:1:1 sdm 8:192 [active][undef]
mpath6 (360000970000192601379533030303534) dm-4 EMC,SYMMETRIX
[size=5.6M][features=1 queue_if_no_path][hwhandler=0][ro]
\_ round-robin 0 [prio=0][active]
\_ 3:0:1:0 sdg 8:96 [active][undef]
\_ 5:0:1:0 sdl 8:176 [active][undef]
mpath5 (36001438005ded0a80000800008a10000) dm-3 HP,HSV450
[size=3.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 3:0:0:2 sdf 8:80 [active][undef]
\_ round-robin 0 [prio=0][enabled]
\_ 5:0:0:2 sdk 8:160 [active][undef]
mpath4 (36001438005ded0a800008000089d0000) dm-2 HP,HSV450
[size=3.0G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 5:0:0:1 sdj 8:144 [active][undef]
\_ round-robin 0 [prio=0][enabled]
\_ 3:0:0:1 sde 8:64 [active][undef]
mpath3 (36842b2b00d276d001426441d0a6ba0ce) dm-1 DELL,PERC 6/i
[size=70G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=0][active]
\_ 0:2:3:0 sdd 8:48 [active][undef]
#

Thanks

Rsmith.

[Matti_Kurkela · Accepted solution]

> The /var/lib/multipath/bindings file was never present so mv did not work!

Yeah, I tried write my instructions to cover any possible situation, since I didn't know if you already had removed the denied or not. As you hadn't, the multipath subsystem should now generate the bindings file directly to /etc/multipath/bindings, which was the desired result of that exercise.

> should i pvcreate on /dev/mapper/mpath devices?

In principle, yes... but since pvcreate will be writing to the *disk* at the end of the paths, any way to refer to a particular disk works.

For example, if you want to pvcreate the EMC LUN with the WWID ...363639, you can run either "pvcreate /dev/mapper/mpath8", "pvcreate /dev/sdi" or "pvcreate /dev/sdn" - all these commands will have exactly the same result: the disk will be assigned a PV UUID and the PV header (including the UUID) will be written to the disk.

If you made the /etc/lvm/lvm.conf change I suggested, you'll find LVM will now prefer the "mpath*" devices if the same PV UUID can be detected on multiple disk devices. For example, if you run "pvcreate /dev/sdn" and then run "pvs", it will tell you that "mpath8" is now a LVM PV that is not yet used in any VG.


If you want, you could also set up a PC-style partition table on each disk. For example:
- run "fdisk /dev/mapper/mpath8" (or "fdisk /dev/sdn" if you want), and create a single partition that encompasses the whole disk
- run "kpartx" to create the multipathed partition device /dev/mapper/mpath8p1: normally it would be detected automatically along with the disk itself, but not now since you created the partition after the disk was already detected
- run "pvcreate /dev/mapper/mpath8p1"

MK

[rsmith_1]

MK

Thanks a lot!

Best Regards

Roger.