topic Re: Shared RAW devices on Linux in Operating System - Linux

Shared RAW devices on Linux

TJ_16 — Tue, 25 Oct 2005 09:43:10 GMT

We are trying to install Oracle RAC 10G on a couple RedHat Linux 3.0 servers. We are using HP XP128 SAN and Oracle requires 2 raw devices be set up for the RAC install.

I am having an issue with the raw devices. I can fdisk, make file systems, write to the shared storage from one host, but I can't from the other host. Also when I set up my raw devices on both host, the Oracle install bombs.

Any ideas or suggestions on where the problem may lie?

Thanks,
TJ

Re: Shared RAW devices on Linux

Ivan Ferreira — Tue, 25 Oct 2005 10:38:26 GMT

Wait, you said:

"I am having an issue with the raw devices. I can fdisk, make file systems"

Raw devices does not have filesystems. To use a shared device with a filesystem, you need to install de OCFS kernel module or buy Red Hat Global Filesystem. Without them, you cannot use a filesystem simultaneously on both nodes.

Re: Shared RAW devices on Linux

TJ_16 — Tue, 25 Oct 2005 10:58:06 GMT

OK, let me rephrase this:

For Oracle RAC installation it requires 2 raw partitions to be set up on each host that is shared between each host. I fdisk the LUN, create two partitions, set up those partitions in /etc/sysconfig/rawdevices, restart rawdevices. I am able to dd to one of the raw devices on one host, but I can't on the other.

I am thinking it is a reservation conflict or something...

I am not trying to create any filesystem on the raw devices... just creating two partitions, and making those two partitions raw devices.

Re: Shared RAW devices on Linux

Ivan Ferreira — Tue, 25 Oct 2005 13:08:44 GMT

I think that you need to reboot the other system so the kernel can use the new partition table. Also, ensure that the rawdevices file is configured in this node also.

Re: Shared RAW devices on Linux

TJ_16 — Tue, 25 Oct 2005 15:40:27 GMT

Done that already.

My /etc/sysconfig/rawdevices file look the same on both hosts:

# raw device bindings
# format:
#
# example: /dev/raw/raw1 /dev/sda1
# /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/sda1
/dev/raw/raw2 /dev/sda2

Re: Shared RAW devices on Linux

Ivan Ferreira — Tue, 25 Oct 2005 15:55:51 GMT

Can you at least see the device from the other node? Is the disk presented to both nodes? What is the error that you have. If you run fdisk -l, you should get the same information on both nodes.

Re: Shared RAW devices on Linux

TJ_16 — Wed, 26 Oct 2005 09:31:04 GMT

Good host:
fdisk -l /dev/sda

Disk /dev/sda: 14.5 GB, 14580449280 bytes
255 heads, 63 sectors/track, 1772 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 125 1004031 83 Linux
/dev/sda2 126 250 1004062+ 83 Linux
[]# dd if=/dev/zero of=/dev/sda1 bs=512 count=1
1+0 records in
1+0 records out

#############################################
Bad Host:
fdisk -l /dev/sda

Disk /dev/sda: 14.5 GB, 14580449280 bytes
255 heads, 63 sectors/track, 1772 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 125 1004031 83 Linux
/dev/sda2 126 250 1004062+ 83 Linux
[]# dd if=/dev/zero of=/dev/sda1 bs=512 count=1
1+0 records in
1+0 records out
[]# tail /var/log/messages

Oct 26 09:27:02 xxxx kernel: scsi1 (0,0,0) : RESERVATION CONFLICT
Oct 26 09:27:02 xxxx kernel: SCSI disk error : host 1 channel 0 id 0 lun 0 return code = 18
Oct 26 09:27:02 xxxx kernel: I/O error: dev 08:01, sector 0

So when I try to dd to it on the bad host I get the RESERVATION CONFLICT, where I don't get it on the good host.

Could this be a SAN issue?

Re: Shared RAW devices on Linux

Ivan Ferreira — Wed, 26 Oct 2005 10:05:39 GMT

I think it could be. Do you have the same error when the good host is powered down?

Las time we had problems with FC reservations we had to power down the storage and switches to clear the reservation. You should try that also.

Re: Shared RAW devices on Linux

Jeff McKenzie — Wed, 26 Oct 2005 10:28:00 GMT

The attached .bmp shows some settings you may want to try changing on the XP. You might also get some ideas from the OS Config Guide at:

http://h20000.www2.hp.com/bizsupport/TechSupport/DocumentIndex.jsp?contentType=SupportManual&locale=en_US&docIndexId=179911&taskId=101&prodTypeId=12169&prodSeriesId=82540

or directly:

http://h200001.www2.hp.com/bc/docs/support/SupportManual/c00472212/c00472212.pdf

Jeff McKenzie