Linux UDEV for ASM oracle database

[gjajuga]

 I would like to know exactly how to configure udev in Linux Redhat for multipath and connect to oracle ASM for linux 5.9 or  Linux 6.4

I saw that some linux administrator use enviroment key such as

 ENV {DM_NAME}
 env{DM_TYPE}
 ENV{DM_MPATH}
 ENV{DM_UUID}

Where can I find them a description and how to use them in scripts?

It is only example, but I want to know what exactly mean   ENV {DM_NAME} etc..

[Matti_Kurkela]

What is "the script" you're talking about?

Without knowing that, it will be very difficult to answer your question in any meaningful way.

[gjajuga]

For example in udev I have rule

file name /etc/udev/rules.d/64-device-mapper.rules

---------------------------------------------------------------------------------------

SUSE 10.

----------------------------------------------------------------------------------------

# device mapper links hook into "online" event, when the dm table
# is available, while some table types must be ignored

KERNEL=="dm-*", ACTION=="add|online", GOTO="device_mapper_do"
GOTO="device_mapper_end"

LABEL="device_mapper_do"
IMPORT{program}="/sbin/dmsetup info -j %M -m %m --export"
ENV{DM_STATE}!="ACTIVE", GOTO="device_mapper_end"
PROGRAM!="/sbin/dmsetup status -j %M -m %m", GOTO="device_mapper_end"
RESULT=="|*snapshot*|*error*", GOTO="device_mapper_end"

IMPORT{program}="/sbin/vol_id --export $tempnode"
ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_UUID}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID}"
ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_SAFE}=="?*", SYMLINK+="disk/by-label/$env{ID_FS_LABEL_SAFE}"

ENV{DM_TABLE}=="*SMOK*", PROGRAM="/bin/chown oracle:dba /dev/mapper/$env{DM_TABLE}"
ENV{DM_TABLE}=="*vv_morwahp1*", PROGRAM="/bin/chown oracle:dba /dev/mapper/$env{DM_TABLE}"

LABEL="device_mapper_end"
----------------------------------------------------------------------------------------------------------------------------------

example linux redhat  6

-----------------------------------------------------------------------------------------------------------------------------------

if [ -z "$1" ]
then
        echo ""
        echo "Usage:        $0 username"
        echo ""
        echo "Description:  Generate udev rules from multipath.conf Oracle RAC LUNs"
        echo "              Please name LUNs on multipath.conf begining with asm_* "
        echo ""
        echo "username:    The user name should be the oracle username or equivalent."
        echo ""
                echo ""
        exit 1
fi
ORA_USER="$1"
ORA_UID=`/usr/bin/id --user "${ORA_USER}"`
ORA_GID=`/usr/bin/id --group "${ORA_USER}"`

# Refresh partitions
#echo "Running partprobe"
#/sbin/partprobe

# Generating udev rules
echo "Generating udev rules on /etc/udev/rules.d/99-asm.rules"
for x in `/sbin/multipath -l | grep asm | awk '{print $1}'`
do for z in `/sbin/dmsetup ls | grep ${x} | grep p | awk '{ print $1}'`
do echo 'ENV{DM_NAME}=="'${z}'", NAME+="oracle/'${z}\"', OWNER="'${ORA_UID}'", GROUP="'${ORA_GID}'", MODE="0660", ACTION=="add|change"'
done
done | sort -r > /etc/udev/rules.d/99-asm.rules

[Matti_Kurkela]

Note that earlier udev versions (like in SuSE 10) held all the udev rules at /etc/udev/rules.d.

Newer versions (like the version in RHEL 6) separates the "standard" rules to /lib/udev/rules.d, and reserves /etc/udev/rules.d for auto-generated and/or custom rules only.

In SuSE 10, the ENV{DM_*} variables are generated by this line in the 64-device-mapper.rules file:

IMPORT{program}="/sbin/dmsetup info -j %M -m %m --export"

 When a new device-mapper device (/dev/dm-*) appears in the system, this rules file runs the "dmsetup info" command. The %M is replaced by the major device number, and %m with the minor device number of the /dev/dm-* device, respectively. The --export option makes "dmsetup info" output its results in format that is importable as environment variables - and that is exactly what the IMPORT{program} rule does.

Because "dmsetup info" only displays information and does nothing else, it is safe to run it any time.

In my system, there are three device-mapper devices in total:

linux:~ # ls -l /dev/dm-*
brw-r----- 1 root disk 253, 0 Jun 13 11:47 /dev/dm-0
brw-r----- 1 root disk 253, 1 Jun 13 11:47 /dev/dm-1
brw-r----- 1 root disk 253, 2 Jun 13 11:47 /dev/dm-2

linux:~ # ls -l /dev/mapper/
total 0
lrwxrwxrwx 1 root root     16 Jun 13 11:47 control -> ../device-mapper
brw------- 1 root root 253, 0 Jun 13 11:47 vg00-lv0
brw------- 1 root root 253, 1 Jun 13 11:47 vg00-lv1
brw------- 1 root root 253, 2 Jun 13 11:47 vg00-lv2

You can see that the minor/major numbers of the /dev/mapper/vg00-* devices are the same as the /dev/dm-* devices, so both sets of device nodes will actually map to the same kernel devices. So you can identify the devices this way now.

But let's see how the udev rule set sees the first one (/dev/dm-0 or /dev/mapper/vg00-lv0):

linux:~ # dmsetup info -j 253 -m 0 --export
DM_TABLE=vg00-lv0
DM_STATE=ACTIVE
DM_TBLSTATE=LIVE
DM_OPENCOUNT=2
DM_EVENT=0
DM_MAJOR=253
DM_MINOR=0
DM_TARGETS=1
DM_UUID=LVM-jKp3sT0RKXfsLkb1GZH19Bli0NUvkKz2qKuNTTJyVGHX1tcpFORRQoRX41t7CHYr

These are exactly the ENV{DM_*} variables you were looking for.

If you want a little more human-readable display, just omit the --export option:

linux:~ # dmsetup info -j 253 -m 0
Name:              vg00-lv0
State:             ACTIVE
Tables present:    LIVE
Open count:        2
Event number:      0
Major, minor:      253, 0
Number of targets: 1
UUID: LVM-jKp3sT0RKXfsLkb1GZH19Bli0NUvkKz2qKuNTTJyVGHX1tcpFORRQoRX41t7CHYr

If you want to know more about these, see the dmsetup man page and the device-mapper documentation.

Likewise, the second IMPORT{program} rule adds the output of "/sbin/vol_id --export" to the list of variables available to the udev rules. I'm not sure if vol_id can read a meaningful identifier from a ASM disk. The easiest way to find out is to try it: run "/sbin/vol_id --export <some ASM device>" and see what it outputs.

RHEL 5.x is tricky.

On RHEL 5.x, the standard multipath device rules in /etc/udev/rules.d/40-multipath.rules include a "OPTIONS=last_rule" setting, so if your custom rule file name has a number higher than 40, any rules for multipath devices in it will be ignored. You may want to copy the standard multipath rules file to e.g. 39-multipath.rules and modify it to include your customizations. That way, your customized rules file will be processed first, and if it includes "OPTIONS=last_rule" for multipath devices, it will prevent the standard multipath rules from taking effect. But the standard rules file will still exist unmodified, so you won't have any problems upgrading the system with yum or "rpm -Uvh".

If your ASM multipath device names are all like /dev/mapper/asm_*, then this might be the 39-multipath.rules file you'll need for RHEL 5.x:

# multipath wants the devmaps presented as meaninglful device names
# so name them after their devmap name
SUBSYSTEM!="block", GOTO="end_mpath"
KERNEL!="dm-[0-9]*", GOTO="end_mpath"
PROGRAM!="/sbin/mpath_wait %M %m", GOTO="end_mpath"
PROGRAM!="/sbin/dmsetup info -c --noheadings -j %M -m %m", GOTO="end_mpath"
RESULT!="*:*:*:*:*:*:*:mpath-*", GOTO="kpartx_check"
PROGRAM!="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", GOTO="end_mpath"
RESULT!="?*", GOTO="end_mpath"
NAME="%k", SYMLINK="mpath/%c"
# the next line added for ASM
RESULT=="asm_?*", SYMLINK+="asmdisk/%c", OWNER="oracle", GROUP="dba", MODE="660"
# end of ASM addition #1
ACTION!="change", GOTO="skip_kpartx"
RUN+="/bin/bash -c '/sbin/mpath_wait /dev/mapper/%c; /sbin/kpartx -a -p p /dev/mapper/%c'"
LABEL="skip_kpartx"
OPTIONS="last_rule"
LABEL="kpartx_check"
RESULT!="*:*:*:*:*:*:*:part*-mpath-*", GOTO="end_mpath"
PROGRAM=="/sbin/dmsetup info -c --noheadings -o name -j %M -m %m", RESULT=="?*", NAME="%k", SYMLINK="mpath/%c"
# the next line added for ASM
RESULT=="asm_?*", SYMLINK+="asmdisk/%c", OWNER="oracle", GROUP="dba", MODE="660"
# end of ASM addition #2
OPTIONS="last_rule"
LABEL="end_mpath"

 (Disclaimer: This is not tested, but I adapted it from an older setup that has been used in production.)

This should create all the multipath devices as usual, but if the multipath name begins with "asm_", then an additional symlink should be created as /dev/asmdisk/asm_*, and the owner, group and mode of the actual device file should be set to appropriate values. (The permissions of the symlinks don't matter: only the permissions of the actual device file are meaningful.)

In RHEL 6, the "OPTIONS=last_rule" is not used in the standard udev rules files, so you can use a script to generate a set of rules like you seem to be planning to do.

You might want to search for string "IMPORT{program}" in all the standard rules files (located in /lib/udev/rules.d in RHEL 6) and read the man pages of each command used, to find information on the various ENV variables available.