Oracle 10g and SAN volumes

[Darrell G]

Here at my current job we are starting a project to move JDE to a new HP-UX server, and in the process upgrading to Oracle 10g.

I've been out of the HPUX/Oracle world for a while and back in those days we used RAW volumes and direct attached storage. Fast forward a few years - now everyone uses SAN and seems to love filesystems to hold Oracle databases.

So this leads me to question a couple things (I don't like to "follow the crowd". I tend to ask "why" and try to do things the right way for the given situation).

1) All of our database volumes are mangaged by the SAN. All the mirroring, striping, etc, is done there. So - why do I need to use LVM of VxVM to manage these volumes? It seems like its just an extra layer of overhead that I simply don't need. So does anyone see any downside to skipping LVM or VxVM and allowing the SAN to manage the volume?

2) I hate to bring up RAW - I know how contentious it is. But the argument against RAW because it is "hard to manage" just doesn't work for me. If its better, then its your job to figure it out. I know a lot of people use RAW for performance reasons, but to me RAW seems better because Oracle now has direct control of what is written to the volume, and when it is written. Its been a while, but back in the day I was pretty sure Oracle recommended raw volumes over filesystems because data integrity could be guanteed better in a RAW volume - because Oracle does ALL the IO. Data integrity is really important to me. I can always make things run faster, but outages are just not acceptable. So are RAW volumes better from an integrity standpoint?

[Jash_1]

Darrell,

Glad to have you back to the hpux/oracle scene!

We use both raw and cooked filesystems for our oracle databases here. It is a lot easier to use raw files if you have 11i v3, as any multi-pathing to the luns is automatically taken car of by the operating system. If you are using non HP san disks then multi-pathing can be a real issue with raw partitions, hence cooked is easier.

Just my humble opinion, hope all goes well.

Jash

[Steven E. Protter]

Shalom,

1) You are better off managing mirroring on the SAN. LVM is used to create raw disk areas unless you use Oracle ASM services which puts disk management in Oracles hands. Bad idea ASM.

2)We create soft links to make raw disk areas visible to the root and oracle user. Managing raw disk areas is not a big deal, it merely in this case requires the use of oracle utilities.

Raw disk volumes offer slightly (less than 5%) improvement in performance. Some product like Oracle 10g RAC require raw disk areas.

There is no difference in data integrity between raw disk and cooked file systems when using oracle. Reliability is exactly the same.

SEP

[Yogeeraj_1]

hi Darrell,

i suggest that you also have a look at Oracle ASM.

Worth it!

kind regards
yogeeraj

[Oviwan]

hey

1) with lvm you are more flexible to add more pv etc...

2) check this metalink note: 578455.1 in oracle 12g raw devices are de-supported...

regards

[Darrell G]

I agree that using the SAN to manage volumes is better. Adding a volume or any of the LVM related things are handled there, invisible to the OS.

This is the first I have heard of RAW volumes not being supported in the future. That would seem to be a very big change -I'm sure a lot of people are hating that.

I have seen conflicting opinions on ASM, some say it great, others hate it. What are the issues with ASM? I can't see that ASM really gives me anything. I don't have a cluster, and the SAN takes care of balancing any IO.

Multipathing to the SAN could be an issue, but we plan to use port agregation to overcome the multipathing issues in 11.23. It is a method supported by our SAN, so should work.

[Darrell G]

We have decided to go with ASM, but this brings a whole new question.

ASM provides a lot of fucntionality such as striping, mirroring, load balancing, etc. BUT, these are the exact same things already done by the SAN. It makes little sense to me to let ASM stripe and load balance, then send all this data off to the SAN have it do the same. And certainly the SAN will do this process much more efficiently that ASM could and not use my CPU cycles for disk management.

So can these things be disabled in ASM?

I also note that Oracle recommends multiple drives for performance. But in our case, we have two aggregated lan adapters going to one SAN - so one path to one device. Do we really gain anything by having mulitple LUNS when they all go to the same SAN via the same path? We are only talking about 1TB of data, so is there a real difference between one 1TB LUN or four 250 GB LUNS?

One Admin here has concern about the I/O queueing because he feels that a single large LUN will have queue contention. But we have a SAN with a huge number of spindles behind it, so how much is I/O queueing going to play into this?

[Deepak Kr]

>>there a real difference between one 1TB LUN or four 250 GB LUNS?

Yes, we had same concern in the past while implementing ORacle 10g RAC cluster.

Finally we took muliple luns instead of a large big lun from SAN Box...

For ASM multiple luns will do better.

[Oviwan]

Hey

when you create a diskgroup in asm you can tell asm what kind of redundancy you want. In your case you should use "external". check this:
http://download.oracle.com/docs/cd/B28359_01/server.111/b28286/statements_5008.htm#SQLRF01114

Maybe you can displace some bottlenecks if you use four 250GB Lun instead of one 1TB lun.

Regards

[Duncan Edmonstone · Accepted solution]

Darrel,

This has been a very "vanilla" conversation so far, but if you want to get into the whys and wherefores of ASM, to give some real advice we're going to need to know a bit more about your SAN... You seem to indicate you have 2 HBAs attached to your SAN, but what sort of storage is it? HP EVA? HP XP? EMC Clariion? EMC DMX? These specifics start to make a lot of difference when deciding how to configure your LUNs on the host.

As to concerns about queueing - yes there can be issues - each LUN has a SCSI queue associated with it which contains all the outstanding IOs to that LUN. The default queue depth on HP-UX is 8, so if you only use 1 LUN by default you will only ever have a maximum of 8 concurrent IOs outstanding at any point in time (most disk arrays can manage a lot more than that!) - if you have 8 LUNs, you can have 64 concurrent IOs outstanding. Of course this queue depth can be tuned if you need to.

Tell us a little more about your environment and we should be able to make some better recommendations for you.

HTH

Duncan

[Darrell G]

We have an Equallogic SAN with 3 LAN adapters, and an HP 4440 with 2 LAN adapters. The SAN is connected via a seperate LAN, but of course there are multiple servers outside the HP-UX enviroment using the SAN. All lan adapters are Gigabit. We have the iSCSI initiator from HP installed to communincate with the Equallogic. HPUX 11.23 and Oracle 10g will be running on these servers.

As iSCSI initiator does not suppport mulipath, will probably be using HP's Auto Port Aggregator to utilize the bandwidth of both HP lan adapters and to gain the ability to survive the loss of a single land adapter.

One item I recently discovered is that an iSCSI will be bound to a single lan adapter on the SAN. So this means that in order to get the maximum throuput to the SAN we will need multple LUNS. The obvious answer is a minimum 3 LUNS (one per adapter), although with only two HP lan adapters I suppose someone could argue for two. And since iSCSI does not load balance, then using ASM to spread the load would seem the right answer.

[Duncan Edmonstone]

Darrel,

Ahhh iSCSI... Equallogic... and HP-UX iSCSI I think you may find yourself ploughing a fairly lonely furrow on this one. I've never come across these devices outside of a Windows Departmental environment, and I don't see the HP-UX iSCSI initiator used much either. So I'm afraid I have no specific reccomendations for you at this point. What you outlined seems fairly sensible though.

I'm not totally sure where you'll stand from a support perspective with this either - have Equallogic(Dell) commited to support this configuration? I don't see it on the HP support matrix for iSCSI:

http://www.docs.hp.com/en/iSCSI_SM/iSCSI_SM.htm

Have you done any sort of performance testing on this config yet?

HTH

Duncan

[Darrell G]

Eqaullogic recommends the use of the HP iSCSI initiator, but then again, there are not a lot of choices out there. HP also supports it -apparently - and were helpful in overcoming some iSCSI/Equallogic obstacles. You are right that it does not show up in the matrix which concerned me when I was given this project. But HP claims to support Equalogic in a lot of literature. My concern has been noted by those that make these decisions and so we move on.

As for ploughing a lonely furrough. Its a lot like plowing in a desert. There is occasional help, and the plowing seems really easy, but you have this nagging feeling that the whole thing just maybe isn't supposed to be done this way...

I have done some performance testing. The results are much faster than the older system that is running. I am doing a bit more extensive testing right now to how different layouts perform.

[Yogeeraj_1]

Hi again Darrell,

I would also spend some time testing the ASM+Oracle database combination. The later itself should give you a boost in performance if properly configured.

Dont forget to size the area carefully. Important issues will also include the RMAN disk backup area/the Flash Recovery Area, etc

You will also need benchmarks from the old environment so that you can compare with the new one. I would collect statspack reports of the previous database...

Perform some backup and recovery tests as well.

good luck!
kind regards
yogeeraj

[Gokul Chandola]

Hi,
Number of LAN cards are not main concern if these are more than one, thing is that network designing shoud be perfect regading collession etc.
Other thing are in main streem.

Regards,
Gokul Chandola