Shared RAW devices on Linux

[TJ_16]

We are trying to install Oracle RAC 10G on a couple RedHat Linux 3.0 servers. We are using HP XP128 SAN and Oracle requires 2 raw devices be set up for the RAC install.

I am having an issue with the raw devices. I can fdisk, make file systems, write to the shared storage from one host, but I can't from the other host. Also when I set up my raw devices on both host, the Oracle install bombs.

Any ideas or suggestions on where the problem may lie?

Thanks,
TJ

[Ivan Ferreira]

Wait, you said:

"I am having an issue with the raw devices. I can fdisk, make file systems"

Raw devices does not have filesystems. To use a shared device with a filesystem, you need to install de OCFS kernel module or buy Red Hat Global Filesystem. Without them, you cannot use a filesystem simultaneously on both nodes.

[TJ_16]

OK, let me rephrase this:

For Oracle RAC installation it requires 2 raw partitions to be set up on each host that is shared between each host. I fdisk the LUN, create two partitions, set up those partitions in /etc/sysconfig/rawdevices, restart rawdevices. I am able to dd to one of the raw devices on one host, but I can't on the other.

I am thinking it is a reservation conflict or something...

I am not trying to create any filesystem on the raw devices... just creating two partitions, and making those two partitions raw devices.

[Ivan Ferreira]

I think that you need to reboot the other system so the kernel can use the new partition table. Also, ensure that the rawdevices file is configured in this node also.

[TJ_16]

Done that already.

My /etc/sysconfig/rawdevices file look the same on both hosts:

# raw device bindings
# format:
#
# example: /dev/raw/raw1 /dev/sda1
# /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/sda1
/dev/raw/raw2 /dev/sda2

[Ivan Ferreira]

Can you at least see the device from the other node? Is the disk presented to both nodes? What is the error that you have. If you run fdisk -l, you should get the same information on both nodes.

[TJ_16]

Good host:
fdisk -l /dev/sda

Disk /dev/sda: 14.5 GB, 14580449280 bytes
255 heads, 63 sectors/track, 1772 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 125 1004031 83 Linux
/dev/sda2 126 250 1004062+ 83 Linux
[]# dd if=/dev/zero of=/dev/sda1 bs=512 count=1
1+0 records in
1+0 records out

#############################################
Bad Host:
fdisk -l /dev/sda

Disk /dev/sda: 14.5 GB, 14580449280 bytes
255 heads, 63 sectors/track, 1772 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 125 1004031 83 Linux
/dev/sda2 126 250 1004062+ 83 Linux
[]# dd if=/dev/zero of=/dev/sda1 bs=512 count=1
1+0 records in
1+0 records out
[]# tail /var/log/messages

Oct 26 09:27:02 xxxx kernel: scsi1 (0,0,0) : RESERVATION CONFLICT
Oct 26 09:27:02 xxxx kernel: SCSI disk error : host 1 channel 0 id 0 lun 0 return code = 18
Oct 26 09:27:02 xxxx kernel: I/O error: dev 08:01, sector 0

So when I try to dd to it on the bad host I get the RESERVATION CONFLICT, where I don't get it on the good host.

Could this be a SAN issue?

[Ivan Ferreira]

I think it could be. Do you have the same error when the good host is powered down?

Las time we had problems with FC reservations we had to power down the storage and switches to clear the reservation. You should try that also.

[Jeff McKenzie]

The attached .bmp shows some settings you may want to try changing on the XP. You might also get some ideas from the OS Config Guide at:

http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?contentType=SupportManual&locale=en_US&docIndexId=179911&taskId=101&prodTypeId=12169&prodSeriesId=82540

or directly:

http://h200001.www2.hp.com/bc/docs/support/SupportManual/c00472212/c00472212.pdf

Jeff McKenzie