topic Re: Oracle RAC w/ raw devices in Operating System - HP-UX

Oracle RAC w/ raw devices

MRahman — Wed, 20 Feb 2008 21:41:18 GMT

Hello All,

We recently had power work in our server room and had to shutdown all of our equipment. This included SAN and servers.

After we brought up the machines, our HRAC nodes do not see the raw SAN disks presented to it in the same order before the shutdown.

We believe the SAN environment had a change introduced that is causing the disks to not be presented correctly to the HRAC. SAN guys say otherwise. They say no change was intriduced.

The SAN is a Clariion CX500 with a Brocade 3200. Before the shutdown, the raw disks were presented to one node as such:

LUN 0 = /dev/rdsk/c13t0d0 {ocr disk}
LUN 1 = /dev/rdsk/c12t0d1 {vote disk}
LUN 2 = /dev/rdsk/c12t0d2 {data}
LUN 3 = /dev/rdsk/c13t0d0 {data}

They are now showing up as:

LUN 0 = /dev/rdsk/c12t0d0
LUN 1 = /dev/rdsk/c9t0d1
LUN 2 = /dev/rdsk/c9t0d2 {data}
LUN 3 = /dev/rdsk/c12t0d3 {data}

Cluster Ready Services (crs) wont start because the disks are not where it wants them. Question is,

1) How can I determine which one is the vote disk since both LUN 0 and LUN 1 are 1GB?

2) OS: Do I have some way of presenting the raw devices in the lcoations before they changed? Running HP-UX 11.11

3) If I take a shot and select one of the 1GB disk as vote disk and it happens to be the ocr disk, will I loose all my ocr registry?

4) What is the best way to proceed?

Thanks.

Re: Oracle RAC w/ raw devices

skt_skt — Thu, 21 Feb 2008 02:30:48 GMT

the controller number is defined by the combination of the FCID+VBUS and they are purly at the SAN side and how/where the LUNs are presented.

But make sure there is no change in the h/w for the FC cards.

Do a compare with earlier "ioscan -funC fc" o/p.

Re: Oracle RAC w/ raw devices

Duncan Edmonstone — Thu, 21 Feb 2008 08:25:38 GMT

Whilst SAN changes can explain away changes in the cX part of a device special file, any changes in the tYdZ part are going to come from the disk array. Here's you're before and after:

>LUN 0 = /dev/rdsk/c13t0d0 {ocr disk}
>LUN 1 = /dev/rdsk/c12t0d1 {vote disk}
>LUN 2 = /dev/rdsk/c12t0d2 {data}
>LUN 3 = /dev/rdsk/c13t0d0 {data}
>
>They are now showing up as:
>
>LUN 0 = /dev/rdsk/c12t0d0
>LUN 1 = /dev/rdsk/c9t0d1
>LUN 2 = /dev/rdsk/c9t0d2 {data}
>LUN 3 = /dev/rdsk/c12t0d3 {data}

So normally I'd look at this and say that c9t0d1 is now your voting disk, but in this case I'm concerned, as the the tXdY component of the device special file has changed for one of the data disks. This points to changes on the Clariion rather than (or as well as) changes in the SAN.

I guess another possibility is that something here is still operating in FCAL mode... but haven't seen any of that in a while.

Do you by any chance have an ioscan output from before the change? Post that with the new ioscan output and we might be able to say more... At the very least post your new 'ioscan -f' output.

HTH

Duncan

Re: Oracle RAC w/ raw devices

MRahman — Thu, 21 Feb 2008 15:05:19 GMT

> Do you by any chance have an ioscan output
> from before the change? Post that with the
> new ioscan output and we might be able to say
> more... At the very least post your new
> 'ioscan -f' output.

Unfortunately there is no previous output of ioscan.

Below is the current output of ioscan -f:

bash-2.03# ioscan -f
Class I H/W Path Driver S/W State H/W Type Description
==============================================================================
root 0 root CLAIMED BUS_NEXUS
cell 0 0 cell CLAIMED BUS_NEXUS
ioa 0 0/0 sba CLAIMED BUS_NEXUS System Bus Adapter (804)
ba 0 0/0/0 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
tty 2 0/0/0/0/0 func0 CLAIMED INTERFACE PCI BaseSystem (103c128d)
tty 0 0/0/0/0/1 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
ext_bus 0 0/0/0/3/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD A6793-60001
target 0 0/0/0/3/0.7 tgt CLAIMED DEVICE
ctl 0 0/0/0/3/0.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 1 0/0/0/3/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra Wide A6793-60001
target 1 0/0/0/3/1.7 tgt CLAIMED DEVICE
ctl 1 0/0/0/3/1.7.0 sctl CLAIMED DEVICE Initiator
ba 1 0/0/1 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 2 0/0/2 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 3 0/0/4 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 4 0/0/6 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 5 0/0/8 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 6 0/0/8/0/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge
ext_bus 2 0/0/8/0/0/1/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 2 0/0/8/0/0/1/0.7 tgt CLAIMED DEVICE
ctl 2 0/0/8/0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 3 0/0/8/0/0/1/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 3 0/0/8/0/0/1/1.7 tgt CLAIMED DEVICE
ctl 3 0/0/8/0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
lan 0 0/0/8/0/0/4/0 igelan CLAIMED INTERFACE HP A6794-60001 PCI 1000Base-T
ba 7 0/0/10 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
fc 0 0/0/10/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
fcp 0 0/0/10/0/0.1 fcp CLAIMED INTERFACE FCP Domain
ext_bus 12 0/0/10/0/0.1.20.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 4 0/0/10/0/0.1.20.0.0.0 tgt CLAIMED DEVICE
disk 4 0/0/10/0/0.1.20.0.0.0.0 sdisk CLAIMED DEVICE DGC CX500WDR5
disk 5 0/0/10/0/0.1.20.0.0.0.1 sdisk CLAIMED DEVICE DGC CX500WDR5
disk 7 0/0/10/0/0.1.20.0.0.0.2 sdisk CLAIMED DEVICE DGC CX500WDR5
disk 8 0/0/10/0/0.1.20.0.0.0.3 sdisk CLAIMED DEVICE DGC CX500WDR5
ext_bus 10 0/0/10/0/0.1.20.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 5 0/0/10/0/0.1.20.255.0.0 tgt CLAIMED DEVICE
ctl 16 0/0/10/0/0.1.20.255.0.0.0 sctl CLAIMED DEVICE DGC CX500
ctl 17 0/0/10/0/0.1.20.255.0.0.1 sctl CLAIMED DEVICE DGC CX500
ctl 18 0/0/10/0/0.1.20.255.0.0.2 sctl CLAIMED DEVICE DGC CX500
ctl 19 0/0/10/0/0.1.20.255.0.0.3 sctl CLAIMED DEVICE DGC CX500
ctl 20 0/0/10/0/0.1.20.255.0.0.4 sctl CLAIMED DEVICE DGC CX500
ctl 21 0/0/10/0/0.1.20.255.0.0.5 sctl CLAIMED DEVICE DGC CX500
ctl 22 0/0/10/0/0.1.20.255.0.0.6 sctl CLAIMED DEVICE DGC CX500
ctl 23 0/0/10/0/0.1.20.255.0.0.7 sctl CLAIMED DEVICE DGC CX500
ba 8 0/0/12 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 1 0/0/12/0/0 igelan CLAIMED INTERFACE HP A6847-60101 PCI 1000Base-SX Adapter
ba 9 0/0/14 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
memory 0 0/5 memory CLAIMED MEMORY Memory
processor 0 0/10 processor CLAIMED PROCESSOR Processor
processor 1 0/11 processor CLAIMED PROCESSOR Processor
processor 2 0/12 processor CLAIMED PROCESSOR Processor
processor 3 0/13 processor CLAIMED PROCESSOR Processor
cell 1 1 cell CLAIMED BUS_NEXUS
ioa 1 1/0 sba CLAIMED BUS_NEXUS System Bus Adapter (804)
ba 10 1/0/0 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
tty 3 1/0/0/0/0 func0 CLAIMED INTERFACE PCI BaseSystem (103c128d)
tty 1 1/0/0/0/1 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
ext_bus 4 1/0/0/3/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD A6793-60001
target 6 1/0/0/3/0.6 tgt CLAIMED DEVICE
disk 0 1/0/0/3/0.6.0 sdisk CLAIMED DEVICE HP 73.4GST373405LC
target 7 1/0/0/3/0.7 tgt CLAIMED DEVICE
ctl 4 1/0/0/3/0.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 5 1/0/0/3/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra Wide Single-Ended A6793-60001
target 8 1/0/0/3/1.2 tgt CLAIMED DEVICE
disk 1 1/0/0/3/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 305
target 9 1/0/0/3/1.7 tgt CLAIMED DEVICE
ctl 5 1/0/0/3/1.7.0 sctl CLAIMED DEVICE Initiator
ba 11 1/0/1 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 12 1/0/1/0/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge
ext_bus 6 1/0/1/0/0/1/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 10 1/0/1/0/0/1/0.7 tgt CLAIMED DEVICE
ctl 6 1/0/1/0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 7 1/0/1/0/0/1/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 11 1/0/1/0/0/1/1.7 tgt CLAIMED DEVICE
ctl 7 1/0/1/0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
lan 2 1/0/1/0/0/4/0 igelan CLAIMED INTERFACE HP A6794-60001 PCI 1000Base-T
ba 13 1/0/2 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
fc 1 1/0/2/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
fcp 1 1/0/2/0/0.1 fcp CLAIMED INTERFACE FCP Domain
ext_bus 13 1/0/2/0/0.1.16.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 12 1/0/2/0/0.1.16.0.0.0 tgt CLAIMED DEVICE
disk 3 1/0/2/0/0.1.16.0.0.0.0 sdisk CLAIMED DEVICE DGC CX500WDR5
ext_bus 11 1/0/2/0/0.1.16.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 13 1/0/2/0/0.1.16.255.0.0 tgt CLAIMED DEVICE
ctl 24 1/0/2/0/0.1.16.255.0.0.0 sctl CLAIMED DEVICE DGC CX500
ctl 25 1/0/2/0/0.1.16.255.0.0.1 sctl CLAIMED DEVICE DGC CX500
ctl 26 1/0/2/0/0.1.16.255.0.0.2 sctl CLAIMED DEVICE DGC CX500
ctl 27 1/0/2/0/0.1.16.255.0.0.3 sctl CLAIMED DEVICE DGC CX500
ctl 28 1/0/2/0/0.1.16.255.0.0.4 sctl CLAIMED DEVICE DGC CX500
ctl 29 1/0/2/0/0.1.16.255.0.0.5 sctl CLAIMED DEVICE DGC CX500
ctl 30 1/0/2/0/0.1.16.255.0.0.6 sctl CLAIMED DEVICE DGC CX500
ctl 31 1/0/2/0/0.1.16.255.0.0.7 sctl CLAIMED DEVICE DGC CX500
ba 14 1/0/4 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 3 1/0/4/0/0 igelan CLAIMED INTERFACE HP A6847-60101 PCI 1000Base-SX Adapter
ba 15 1/0/6 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 16 1/0/8 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 17 1/0/10 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 18 1/0/12 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 19 1/0/14 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
memory 1 1/5 memory CLAIMED MEMORY Memory
processor 4 1/10 processor CLAIMED PROCESSOR Processor
processor 5 1/11 processor CLAIMED PROCESSOR Processor
processor 6 1/12 processor CLAIMED PROCESSOR Processor
processor 7 1/13 processor CLAIMED PROCESSOR Processor

Re: Oracle RAC w/ raw devices

Duncan Edmonstone — Thu, 21 Feb 2008 15:21:32 GMT

Can you check this data your originally provided:

LUN 0 = /dev/rdsk/c13t0d0 {ocr disk}
LUN 1 = /dev/rdsk/c12t0d1 {vote disk}
LUN 2 = /dev/rdsk/c12t0d2 {data}
LUN 3 = /dev/rdsk/c13t0d0 {data}

They are now showing up as:

LUN 0 = /dev/rdsk/c12t0d0
LUN 1 = /dev/rdsk/c9t0d1
LUN 2 = /dev/rdsk/c9t0d2 {data}
LUN 3 = /dev/rdsk/c12t0d3 {data}

I note that in the old data the ocr disk and 2nd data disk are the same (c13t0d0). It's hard to give an accurate response if you don't provide accurate data. Please check again - what were the devices before, what are they now?

Duncan

Re: Oracle RAC w/ raw devices

Duncan Edmonstone — Thu, 21 Feb 2008 15:51:00 GMT

Also

Are you using PowerPath?

Whay do I only see one disk out of the 2nd HBA (1/0/2/0) whilst I see 4 out of the other (0/0/10/0)

This ioscan doesn't add up with the LUN IDs you have posted - can you post an "ioscan -fn" as well?

Thanks

Duncan

Re: Oracle RAC w/ raw devices

MRahman — Thu, 21 Feb 2008 17:26:27 GMT

Thanks to ALL that replied. And I appologize for fat fingering the LUN assignments. Here is what it was:

LUN 0 = /dev/rdsk/c13t0d0 {ocr disk}
LUN 1 = /dev/rdsk/c12t0d1 {vote disk}
LUN 2 = /dev/rdsk/c12t0d2 {data}
LUN 3 = /dev/rdsk/c13t0d3 {data}

After going back to our SAN guy and asking what might have been experimented with we found that ALL LUNs were moved to one controller (SPA) thus causing the host to see them as:

LUN 0 = /dev/rdsk/c12t0d0
LUN 1 = /dev/rdsk/c12t0d1
LUN 2 = /dev/rdsk/c12t0d2
LUN 3 = /dev/rdsk/c12t0d3

When we moved the LUNs to SPB, the host saw them as:

LUN 0 = /dev/rdsk/c13t0d0
LUN 1 = /dev/rdsk/c13t0d1
LUN 2 = /dev/rdsk/c13t0d2
LUN 3 = /dev/rdsk/c13t0d3

So now we had proof that it was a change introduced from the SAN. We balanced the LUNs across SPA & SPB on the CX500 based on our original output. Once we did that and rebooted the hosts, everything came up just fine.

I want to Thanks everyone here for helping.

Out of curiosity, is there a way to look at the contents of a RAW device?

Re: Oracle RAC w/ raw devices

Duncan Edmonstone — Thu, 21 Feb 2008 18:54:59 GMT

Sure you can - just use dd:

To look at the first two 1K blocks

dd if=/dev/rdsk/c12t0d0 of=/tmp/mydisk bs=1k count=2

You can then use soemthing like xd to look at the contents of those blocks

Whether you can out anything usefull in there depends on what Oracle write into the starting blcoks of their disks.

HTH

Duncan

Re: Oracle RAC w/ raw devices

Bryan Rooney — Thu, 24 Apr 2008 14:16:05 GMT

It is possible on HP-UX Itanium 11.23 (Oracle 10.2.0.2) to look at raw disks, not certain for other versioning. You can look at raw ASM disks (and maybe ocr and voting, but not sure) with Oracle tools that can be compiled called kfed and kfod.

$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk ikfed

$ kfed -h
as/mlib ASM Library [asmlib='lib']aun/um AU number to examine or update [AUNUM=number]
aus/z Allocation Unit size in bytes [AUSZ=number]
blkn/um Block number to examine or update [BLKNUM=number]
blks/z Metadata block size in bytes [BLKSZ=number]
ch/ksum Update checksum before each write [CHKSUM=YES/NO]
cn/t Count of AUs to process [CNT=number]
d/ev ASM device to examine or update [DEV=string]
o/p KFED operation type [OP=READ/WRITE/MERGE/NEW/FORM/FIND/STRUCT]
p/rovnm Name for provisioning purposes [PROVNM=string]
te/xt File name for translated block text [TEXT=string]
ty/pe ASM metadata block type number [TYPE=number]

$ kfod -h
_asm_a/llow_only_raw_disks KFOD allow only raw devices [_asm_allow_only_raw_disks=TRUE/(FALSE)]
_asm_l/ibraries ASM Libraries[_asm_libraries='lib1','lib2',...]
_asms/id ASM Instance[_asmsid=sid]
a/sm_diskstring ASM Diskstring [asm_diskstring='discoverystring', 'discoverystring' ...]
d/isks Disks to discover [disks=raw,asm,all]
g/roup Group discover [group=controlfile]
n/ohdr KFOD header suppression [nohdr=TRUE/(FALSE)]
o/p KFOD options type [OP=DISKS/GROUPS/ALL]
p/file ASM parameter file [pfile='parameterfile']
s/tatus Include disk header status [status=TRUE/(FALSE)]
v/erbose KFOD verbose errors [verbose=TRUE/(FALSE)]

Re: Oracle RAC w/ raw devices

Rasheed Tamton — Sat, 03 May 2008 10:55:42 GMT

Hi,

If you have EMC, is it not possible to use inq or syminq to check the disk ids/serial nos and compare it with the old ones or something like that.

I think your SAN guys can help on this in this kind of situation.

Regards.