Operating System - HP-UX
1748113 Members
3404 Online
108758 Solutions
New Discussion юеВ

Oracle ASM - more exacting, load on IO?

 
SOLVED
Go to solution
Alzhy
Honored Contributor

Oracle ASM - more exacting, load on IO?

We recently moved to ASM. And we're seeing more load on our SAN Storage Array's ports than when we were on Cooked VxFS DierctIO'd filesystems.

Before, Our Filesystems (24 total) used to be on 8-way stripes (8 physical/SAN disks per).

On our new ASM Layout, we provide our DBAs with 48 disks. ASM suppsoedly stripes accross all these 48 disks.

Sar and Disk stats indeed point a uniform load accross these 48 disks but my service times jumped from sub-20ms to ~130ms. And my Array front end now show its port processors are doubly busy than before.

So is ASM more heavy, exacting on Storage Arrays? That it merits a relayout frontend. channel and even array group wise on the SAN Array end?

Hakuna Matata.
19 REPLIES 19
Steven E. Protter
Exalted Contributor

Re: Oracle ASM - more exacting, load on IO?

Shalom,

I've seen these kind of results from ASM in the lab as well. ASM acts like it owns the disks, knows better, and takes over certain functions normally done by the OS.

ASM exacts a higher price an any kind of disk you give it. I believe the SAN/Disk array configuration is the better place to control disk, and am not a believer in ASM as a solution.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Hein van den Heuvel
Honored Contributor

Re: Oracle ASM - more exacting, load on IO?

As ASM wasintroduced on the system, it possibly switched from a cached file system to raw. (was it using direct-io/).
The file system may have been helping more than realized.. by using memory as cache.
In other words.. do you see less memory used in glance/vmstat since the switch?
Consider giving that now unused memory directly to oracle (SGA)?

20ms is not great.
130 ms is pretty darn slow.
Do you get user complaint now?
Is that response time measured with iostat (averaged over all disks)? Maybe this is mostly cause for example by spikes writing to archive logs which the end users do not feel? Time for redo and arch log (writer/buffer) tuning?

Can you check current and historical performance reports (statspack, awr) to figure out how Oracle perceives the IO response times? Does it 'add up'? Check read vs write, check whether particular object have vastly different Io response times.

Good luck,
Hein van den Heuvel
HvdH Performance Consulting.
Yogeeraj_1
Honored Contributor

Re: Oracle ASM - more exacting, load on IO?

hi,

Just think of ASM as a filesystem for database stuff. That is, at the core, what it is.

A special purpose filesystem.

ASM was designed to take all of the devices and stripe over all of them.

here you have the CHOICE of redundancy: either you use ASM's mirror and stripe everything approach or external hardware redundancy.

So it you are providing your DBA with 48 disks (which is already in a RAID), you should also inform the DBA NOT to define any further level of redundancy.

can you confirm your configuration?

(RDBMS Instance: SELECT * FROM V$ASM_DISKGROUP;)

kind regards
yogeeraj

Ps. also read: http://download-west.oracle.com/docs/cd/B19306_01/server.102/b14220/mgmt_db.htm#DCGGGJC

No person was ever honoured for what he received. Honour has been the reward for what he gave (clavin coolidge)

Re: Oracle ASM - more exacting, load on IO?

Nelson,

I have to say my first thought was also that the DBAs still had redundancy enabled in ASM (effectively doubling the IO rate).

So your service times have increased by a factor of 6, what about your throughput (MB/s) and IOs per second? Do you have any stats for changes there? (e.g. from Measureware looking at GBL_DISK_PHYS_BYTE_RATE and GBL_DISK_PHYS_RATE)

Also what about MPIO - how were you doing MPIO before and how are you doing it now? Any changes (cos ASM of course does no MPIO itself).

HTH

Duncan

I am an HPE Employee
Accept or Kudo
Alzhy
Honored Contributor

Re: Oracle ASM - more exacting, load on IO?

Thanks for all your inputs.

I am still in the process of gathering stats from my XP12000, System Ststs (SAR and GlancePlus/OVPA) -- but it really appears we're over the top and those service times are worrisome.

And yes my DBA says he uses external redudancy in ASM:

SQL> @$ORACLE_BASE/local/asm/asm_diskgroups

Disk Group Sector Block Allocation
Name Size Size Unit Size State Type Total Size (MB) Used Size (MB) Pct. Used
-------------------- ------- ------- ------------ ----------- ------ --------------- -------------- ---------
PRDB001_DATA 1,024 4,096 1,048,576 MOUNTED EXTERN 2,555,376 2,531,335 99.06
--------------- --------------
Grand Total: 2,555,376 2,531,335


We are using VxVM underneath each component ASM "volume" -- which is a VxVM Volume that is wholly contained on one XP12K Disk. I can monitor the entire member disks using vxtools and I see ALL 48 member volumes active ALL the time as if data is striped 48-way (which is suppsoed to be how ASM works). I can validate the cXtYdZ dveices of these volumes via SAR and the service times for reads and writes are indeed breaching 100+ ms service times.


Now compare that to our OLD cooked Filesystems which were 8-way stripes per Filesystem/Volume. The database hardly stress HALF of however many disks we had as I.O jumps (interleaves) from FS to FS. But with ASM -- it seems ALL 48 disks are always busy (striped) that I am noticing my XP12000 front end processors to be far more utilized/sterssed than we were on Cooked or even simple RAW (with 8-way stripes)..

Hakuna Matata.
Alzhy
Honored Contributor

Re: Oracle ASM - more exacting, load on IO?

I guess what I am saying (claiming) is with ASM -- all component dsisks are ALWAYS active -- hence more I/O load on a Storage Array comparatively with a COoked Filesystem or Simple/Admin cotnrolled RAW Database Layout where striping and placemnt of data is totally controlled.

Do you agree?

I am still poring through stats though.

Hakuna Matata.
Solution

Re: Oracle ASM - more exacting, load on IO?

This isn't a config I see a lot of - mostly if people are using ASM I see them using a simple MPIO tool (I'd assume HDLM in your case - you have HDS disk arrays yes?) I can't think I've come across anyone doing this with VxVM used just to get the DMP functions before - I wouldn't expect it to cause any issues, but its an extra variable I don't normally see... I know Symantec have qualified it and support it but I just don't see much of it. Sorry I'm not sure that helps any!

What sort of IO is the DB doing? Are there a lot of full table scans generating sequential IO? Doing that over 48 disks on multiple ports might make that sequential IO harder for the XP to spot (confusing the cache algorithms) I'd be interested in trying a test with your DMP turned off as well (i.e. all IO to just a primary path)

HTH

Duncan

I am an HPE Employee
Accept or Kudo
Alzhy
Honored Contributor

Re: Oracle ASM - more exacting, load on IO?

Duncan, so you're aware of the Oracle validated "recipe" eh? Yeap -- we own VxVM already and view using it as a plus as it makes tracking and managing things things easier... going to HDLM was cost prohibitive and we can't go to HP-UX 11.31 (yet) on these PARISC 11.11 ecossytems we have.

But do you agree with my assumptions? I will hopefully soon have stats to support this claim.

Hakuna Matata.

Re: Oracle ASM - more exacting, load on IO?

Nelson,

I'm not sure I *do* agree... is the database doing anything different? Is the DBA seeing much improved performance/throughput? The total amount of IO generated regardless of filesystems, raw or ASM shouldn't be significantly different (I guess the number of IOPS could change if ASM is reading/writing with different block sizes than VxFS was? In fact that gets me thinking... maybe with VxVM in the equation the ASM IO block sizes for reads/writes aren't matching the VxVM IO raw disk block sizes for reads writes? You'd know this if your total IOPS have gone up, but your throughput in MB/s has stayed largely the same. IIRC ASM will simply read in the sizes determined by the database parameters (i.e. database block size * multi-block read count) - how does that comnpare to the sizes that VxVM is reading?)

Also are you sure that ASM isn't doing any rebalancing? It will do thgis automatically if disks are added to an ASM disk group. Does the table V$ASM_OPERATION show anr jobs running? That would generate significant additional IO.

Finally waht about the internals of the XP? When you were on VxVM with 8 way stripes, you probvably had it configured so that each LUN in a strip came from a different array group and maybe even a different DKA/ACP... now you may have LUNs from the same array group in the same disk groupo - again if you have a lot of sequential IO this could cause some challenges as the disk heads thrash back and forth between the different ASM stripes on the same physical disks (if you follow me)

Anyway some things to think about. It'll be interesting to see those IO numbers.

HTH

Duncan

I am an HPE Employee
Accept or Kudo