- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: Oracle RAC w/ raw devices
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-20-2008 01:41 PM
тАО02-20-2008 01:41 PM
Oracle RAC w/ raw devices
We recently had power work in our server room and had to shutdown all of our equipment. This included SAN and servers.
After we brought up the machines, our HRAC nodes do not see the raw SAN disks presented to it in the same order before the shutdown.
We believe the SAN environment had a change introduced that is causing the disks to not be presented correctly to the HRAC. SAN guys say otherwise. They say no change was intriduced.
The SAN is a Clariion CX500 with a Brocade 3200. Before the shutdown, the raw disks were presented to one node as such:
LUN 0 = /dev/rdsk/c13t0d0 {ocr disk}
LUN 1 = /dev/rdsk/c12t0d1 {vote disk}
LUN 2 = /dev/rdsk/c12t0d2 {data}
LUN 3 = /dev/rdsk/c13t0d0 {data}
They are now showing up as:
LUN 0 = /dev/rdsk/c12t0d0
LUN 1 = /dev/rdsk/c9t0d1
LUN 2 = /dev/rdsk/c9t0d2 {data}
LUN 3 = /dev/rdsk/c12t0d3 {data}
Cluster Ready Services (crs) wont start because the disks are not where it wants them. Question is,
1) How can I determine which one is the vote disk since both LUN 0 and LUN 1 are 1GB?
2) OS: Do I have some way of presenting the raw devices in the lcoations before they changed? Running HP-UX 11.11
3) If I take a shot and select one of the 1GB disk as vote disk and it happens to be the ocr disk, will I loose all my ocr registry?
4) What is the best way to proceed?
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-20-2008 06:30 PM
тАО02-20-2008 06:30 PM
Re: Oracle RAC w/ raw devices
But make sure there is no change in the h/w for the FC cards.
Do a compare with earlier "ioscan -funC fc" o/p.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-21-2008 12:25 AM
тАО02-21-2008 12:25 AM
Re: Oracle RAC w/ raw devices
>LUN 0 = /dev/rdsk/c13t0d0 {ocr disk}
>LUN 1 = /dev/rdsk/c12t0d1 {vote disk}
>LUN 2 = /dev/rdsk/c12t0d2 {data}
>LUN 3 = /dev/rdsk/c13t0d0 {data}
>
>They are now showing up as:
>
>LUN 0 = /dev/rdsk/c12t0d0
>LUN 1 = /dev/rdsk/c9t0d1
>LUN 2 = /dev/rdsk/c9t0d2 {data}
>LUN 3 = /dev/rdsk/c12t0d3 {data}
So normally I'd look at this and say that c9t0d1 is now your voting disk, but in this case I'm concerned, as the the tXdY component of the device special file has changed for one of the data disks. This points to changes on the Clariion rather than (or as well as) changes in the SAN.
I guess another possibility is that something here is still operating in FCAL mode... but haven't seen any of that in a while.
Do you by any chance have an ioscan output from before the change? Post that with the new ioscan output and we might be able to say more... At the very least post your new 'ioscan -f' output.
HTH
Duncan
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-21-2008 07:05 AM
тАО02-21-2008 07:05 AM
Re: Oracle RAC w/ raw devices
> from before the change? Post that with the
> new ioscan output and we might be able to say
> more... At the very least post your new
> 'ioscan -f' output.
Unfortunately there is no previous output of ioscan.
Below is the current output of ioscan -f:
bash-2.03# ioscan -f
Class I H/W Path Driver S/W State H/W Type Description
==============================================================================
root 0 root CLAIMED BUS_NEXUS
cell 0 0 cell CLAIMED BUS_NEXUS
ioa 0 0/0 sba CLAIMED BUS_NEXUS System Bus Adapter (804)
ba 0 0/0/0 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
tty 2 0/0/0/0/0 func0 CLAIMED INTERFACE PCI BaseSystem (103c128d)
tty 0 0/0/0/0/1 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
ext_bus 0 0/0/0/3/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD A6793-60001
target 0 0/0/0/3/0.7 tgt CLAIMED DEVICE
ctl 0 0/0/0/3/0.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 1 0/0/0/3/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra Wide A6793-60001
target 1 0/0/0/3/1.7 tgt CLAIMED DEVICE
ctl 1 0/0/0/3/1.7.0 sctl CLAIMED DEVICE Initiator
ba 1 0/0/1 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 2 0/0/2 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 3 0/0/4 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 4 0/0/6 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 5 0/0/8 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 6 0/0/8/0/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge
ext_bus 2 0/0/8/0/0/1/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 2 0/0/8/0/0/1/0.7 tgt CLAIMED DEVICE
ctl 2 0/0/8/0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 3 0/0/8/0/0/1/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 3 0/0/8/0/0/1/1.7 tgt CLAIMED DEVICE
ctl 3 0/0/8/0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
lan 0 0/0/8/0/0/4/0 igelan CLAIMED INTERFACE HP A6794-60001 PCI 1000Base-T
ba 7 0/0/10 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
fc 0 0/0/10/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
fcp 0 0/0/10/0/0.1 fcp CLAIMED INTERFACE FCP Domain
ext_bus 12 0/0/10/0/0.1.20.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 4 0/0/10/0/0.1.20.0.0.0 tgt CLAIMED DEVICE
disk 4 0/0/10/0/0.1.20.0.0.0.0 sdisk CLAIMED DEVICE DGC CX500WDR5
disk 5 0/0/10/0/0.1.20.0.0.0.1 sdisk CLAIMED DEVICE DGC CX500WDR5
disk 7 0/0/10/0/0.1.20.0.0.0.2 sdisk CLAIMED DEVICE DGC CX500WDR5
disk 8 0/0/10/0/0.1.20.0.0.0.3 sdisk CLAIMED DEVICE DGC CX500WDR5
ext_bus 10 0/0/10/0/0.1.20.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 5 0/0/10/0/0.1.20.255.0.0 tgt CLAIMED DEVICE
ctl 16 0/0/10/0/0.1.20.255.0.0.0 sctl CLAIMED DEVICE DGC CX500
ctl 17 0/0/10/0/0.1.20.255.0.0.1 sctl CLAIMED DEVICE DGC CX500
ctl 18 0/0/10/0/0.1.20.255.0.0.2 sctl CLAIMED DEVICE DGC CX500
ctl 19 0/0/10/0/0.1.20.255.0.0.3 sctl CLAIMED DEVICE DGC CX500
ctl 20 0/0/10/0/0.1.20.255.0.0.4 sctl CLAIMED DEVICE DGC CX500
ctl 21 0/0/10/0/0.1.20.255.0.0.5 sctl CLAIMED DEVICE DGC CX500
ctl 22 0/0/10/0/0.1.20.255.0.0.6 sctl CLAIMED DEVICE DGC CX500
ctl 23 0/0/10/0/0.1.20.255.0.0.7 sctl CLAIMED DEVICE DGC CX500
ba 8 0/0/12 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 1 0/0/12/0/0 igelan CLAIMED INTERFACE HP A6847-60101 PCI 1000Base-SX Adapter
ba 9 0/0/14 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
memory 0 0/5 memory CLAIMED MEMORY Memory
processor 0 0/10 processor CLAIMED PROCESSOR Processor
processor 1 0/11 processor CLAIMED PROCESSOR Processor
processor 2 0/12 processor CLAIMED PROCESSOR Processor
processor 3 0/13 processor CLAIMED PROCESSOR Processor
cell 1 1 cell CLAIMED BUS_NEXUS
ioa 1 1/0 sba CLAIMED BUS_NEXUS System Bus Adapter (804)
ba 10 1/0/0 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
tty 3 1/0/0/0/0 func0 CLAIMED INTERFACE PCI BaseSystem (103c128d)
tty 1 1/0/0/0/1 asio0 CLAIMED INTERFACE PCI Serial (103c1048)
ext_bus 4 1/0/0/3/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD A6793-60001
target 6 1/0/0/3/0.6 tgt CLAIMED DEVICE
disk 0 1/0/0/3/0.6.0 sdisk CLAIMED DEVICE HP 73.4GST373405LC
target 7 1/0/0/3/0.7 tgt CLAIMED DEVICE
ctl 4 1/0/0/3/0.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 5 1/0/0/3/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra Wide Single-Ended A6793-60001
target 8 1/0/0/3/1.2 tgt CLAIMED DEVICE
disk 1 1/0/0/3/1.2.0 sdisk CLAIMED DEVICE HP DVD-ROM 305
target 9 1/0/0/3/1.7 tgt CLAIMED DEVICE
ctl 5 1/0/0/3/1.7.0 sctl CLAIMED DEVICE Initiator
ba 11 1/0/1 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 12 1/0/1/0/0 PCItoPCI CLAIMED BUS_NEXUS PCItoPCI Bridge
ext_bus 6 1/0/1/0/0/1/0 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 10 1/0/1/0/0/1/0.7 tgt CLAIMED DEVICE
ctl 6 1/0/1/0/0/1/0.7.0 sctl CLAIMED DEVICE Initiator
ext_bus 7 1/0/1/0/0/1/1 c8xx CLAIMED INTERFACE SCSI C1010 Ultra160 Wide LVD
target 11 1/0/1/0/0/1/1.7 tgt CLAIMED DEVICE
ctl 7 1/0/1/0/0/1/1.7.0 sctl CLAIMED DEVICE Initiator
lan 2 1/0/1/0/0/4/0 igelan CLAIMED INTERFACE HP A6794-60001 PCI 1000Base-T
ba 13 1/0/2 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
fc 1 1/0/2/0/0 td CLAIMED INTERFACE HP Tachyon XL2 Fibre Channel Mass Storage Adapter
fcp 1 1/0/2/0/0.1 fcp CLAIMED INTERFACE FCP Domain
ext_bus 13 1/0/2/0/0.1.16.0.0 fcparray CLAIMED INTERFACE FCP Array Interface
target 12 1/0/2/0/0.1.16.0.0.0 tgt CLAIMED DEVICE
disk 3 1/0/2/0/0.1.16.0.0.0.0 sdisk CLAIMED DEVICE DGC CX500WDR5
ext_bus 11 1/0/2/0/0.1.16.255.0 fcpdev CLAIMED INTERFACE FCP Device Interface
target 13 1/0/2/0/0.1.16.255.0.0 tgt CLAIMED DEVICE
ctl 24 1/0/2/0/0.1.16.255.0.0.0 sctl CLAIMED DEVICE DGC CX500
ctl 25 1/0/2/0/0.1.16.255.0.0.1 sctl CLAIMED DEVICE DGC CX500
ctl 26 1/0/2/0/0.1.16.255.0.0.2 sctl CLAIMED DEVICE DGC CX500
ctl 27 1/0/2/0/0.1.16.255.0.0.3 sctl CLAIMED DEVICE DGC CX500
ctl 28 1/0/2/0/0.1.16.255.0.0.4 sctl CLAIMED DEVICE DGC CX500
ctl 29 1/0/2/0/0.1.16.255.0.0.5 sctl CLAIMED DEVICE DGC CX500
ctl 30 1/0/2/0/0.1.16.255.0.0.6 sctl CLAIMED DEVICE DGC CX500
ctl 31 1/0/2/0/0.1.16.255.0.0.7 sctl CLAIMED DEVICE DGC CX500
ba 14 1/0/4 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
lan 3 1/0/4/0/0 igelan CLAIMED INTERFACE HP A6847-60101 PCI 1000Base-SX Adapter
ba 15 1/0/6 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 16 1/0/8 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 17 1/0/10 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 18 1/0/12 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
ba 19 1/0/14 lba CLAIMED BUS_NEXUS Local PCI Bus Adapter (782)
memory 1 1/5 memory CLAIMED MEMORY Memory
processor 4 1/10 processor CLAIMED PROCESSOR Processor
processor 5 1/11 processor CLAIMED PROCESSOR Processor
processor 6 1/12 processor CLAIMED PROCESSOR Processor
processor 7 1/13 processor CLAIMED PROCESSOR Processor
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-21-2008 07:21 AM
тАО02-21-2008 07:21 AM
Re: Oracle RAC w/ raw devices
LUN 0 = /dev/rdsk/c13t0d0 {ocr disk}
LUN 1 = /dev/rdsk/c12t0d1 {vote disk}
LUN 2 = /dev/rdsk/c12t0d2 {data}
LUN 3 = /dev/rdsk/c13t0d0 {data}
They are now showing up as:
LUN 0 = /dev/rdsk/c12t0d0
LUN 1 = /dev/rdsk/c9t0d1
LUN 2 = /dev/rdsk/c9t0d2 {data}
LUN 3 = /dev/rdsk/c12t0d3 {data}
I note that in the old data the ocr disk and 2nd data disk are the same (c13t0d0). It's hard to give an accurate response if you don't provide accurate data. Please check again - what were the devices before, what are they now?
Duncan
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-21-2008 07:51 AM
тАО02-21-2008 07:51 AM
Re: Oracle RAC w/ raw devices
Are you using PowerPath?
Whay do I only see one disk out of the 2nd HBA (1/0/2/0) whilst I see 4 out of the other (0/0/10/0)
This ioscan doesn't add up with the LUN IDs you have posted - can you post an "ioscan -fn" as well?
Thanks
Duncan
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-21-2008 09:26 AM
тАО02-21-2008 09:26 AM
Re: Oracle RAC w/ raw devices
LUN 0 = /dev/rdsk/c13t0d0 {ocr disk}
LUN 1 = /dev/rdsk/c12t0d1 {vote disk}
LUN 2 = /dev/rdsk/c12t0d2 {data}
LUN 3 = /dev/rdsk/c13t0d3 {data}
After going back to our SAN guy and asking what might have been experimented with we found that ALL LUNs were moved to one controller (SPA) thus causing the host to see them as:
LUN 0 = /dev/rdsk/c12t0d0
LUN 1 = /dev/rdsk/c12t0d1
LUN 2 = /dev/rdsk/c12t0d2
LUN 3 = /dev/rdsk/c12t0d3
When we moved the LUNs to SPB, the host saw them as:
LUN 0 = /dev/rdsk/c13t0d0
LUN 1 = /dev/rdsk/c13t0d1
LUN 2 = /dev/rdsk/c13t0d2
LUN 3 = /dev/rdsk/c13t0d3
So now we had proof that it was a change introduced from the SAN. We balanced the LUNs across SPA & SPB on the CX500 based on our original output. Once we did that and rebooted the hosts, everything came up just fine.
I want to Thanks everyone here for helping.
Out of curiosity, is there a way to look at the contents of a RAW device?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО02-21-2008 10:54 AM
тАО02-21-2008 10:54 AM
Re: Oracle RAC w/ raw devices
To look at the first two 1K blocks
dd if=/dev/rdsk/c12t0d0 of=/tmp/mydisk bs=1k count=2
You can then use soemthing like xd to look at the contents of those blocks
Whether you can out anything usefull in there depends on what Oracle write into the starting blcoks of their disks.
HTH
Duncan
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО04-24-2008 07:16 AM
тАО04-24-2008 07:16 AM
Re: Oracle RAC w/ raw devices
$ cd $ORACLE_HOME/rdbms/lib
$ make -f ins_rdbms.mk ikfed
$ kfed -h
as/mlib ASM Library [asmlib='lib']aun/um AU number to examine or update [AUNUM=number]
aus/z Allocation Unit size in bytes [AUSZ=number]
blkn/um Block number to examine or update [BLKNUM=number]
blks/z Metadata block size in bytes [BLKSZ=number]
ch/ksum Update checksum before each write [CHKSUM=YES/NO]
cn/t Count of AUs to process [CNT=number]
d/ev ASM device to examine or update [DEV=string]
o/p KFED operation type [OP=READ/WRITE/MERGE/NEW/FORM/FIND/STRUCT]
p/rovnm Name for provisioning purposes [PROVNM=string]
te/xt File name for translated block text [TEXT=string]
ty/pe ASM metadata block type number [TYPE=number]
$ kfod -h
_asm_a/llow_only_raw_disks KFOD allow only raw devices [_asm_allow_only_raw_disks=TRUE/(FALSE)]
_asm_l/ibraries ASM Libraries[_asm_libraries='lib1','lib2',...]
_asms/id ASM Instance[_asmsid=sid]
a/sm_diskstring ASM Diskstring [asm_diskstring='discoverystring', 'discoverystring' ...]
d/isks Disks to discover [disks=raw,asm,all]
g/roup Group discover [group=controlfile]
n/ohdr KFOD header suppression [nohdr=TRUE/(FALSE)]
o/p KFOD options type [OP=DISKS/GROUPS/ALL]
p/file ASM parameter file [pfile='parameterfile']
s/tatus Include disk header status [status=TRUE/(FALSE)]
v/erbose KFOD verbose errors [verbose=TRUE/(FALSE)]
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО05-03-2008 03:55 AM
тАО05-03-2008 03:55 AM
Re: Oracle RAC w/ raw devices
If you have EMC, is it not possible to use inq or syminq to check the disk ids/serial nos and compare it with the old ones or something like that.
I think your SAN guys can help on this in this kind of situation.
Regards.