topic Re: IO bottleneck on hp-ux system in Operating System - HP-UX

IO bottleneck on hp-ux system

SUDHAKAR_18 — Sun, 17 Jul 2011 06:56:00 GMT

Hi , I am having a hp-ux 11.31 system, with 16 CPU, 36Gb memory & hyperthreading enabled.

This system is used to run batches on specific time & we are looking into improve the performance of the batches.

Recently we have added flash drives to our Entrprise storage (EMC DMX4) & configured mountpoints for this system. data files moved accordingly. but we have not got any performance benefits.

Also look into below o/p

---------------------

==> from glance

                                                                   PROCESS LIST                                                         Users=    8
                         User      CPU %    Thrd Disk        Memory       Block
Process Name     PID     Name   (1600% max) Cnt IOrate      RSS/VSS      On
--------------------------------------------------------------------------------
ora_p040_MIS        3393 oradb        56.0     1 17.8   229.8mb 238.3mb    IO
ora_p043_MIS        3399 oradb        55.6     1 17.5   230.6mb 238.3mb    IO
ora_p044_MIS        3401 oradb        54.3     1 17.6   229.3mb 238.3mb   PRI
ora_p041_MIS        3395 oradb        54.3     1 17.8   230.3mb 238.3mb   PRI
ora_p047_MIS        3407 oradb        54.1     1 17.6   230.1mb 238.3mb    IO
ora_p042_MIS        3397 oradb        53.3     1 17.1   230.6mb 238.3mb    IO
ora_p046_MIS        3405 oradb        53.3     1 17.5   229.8mb 238.3mb    IO

-------------------------

mysystem:root-/>sar -d 1 1

HP-UX mysystem B.11.31 U ia64 07/17/11

12:18:52   device   %busy   avque   r+w/s blks/s avwait avserv
12:18:53   disk61 100.00    0.50      93   59612    0.00   34.74
           disk62   35.64    0.50       6    6020    0.00   60.69
           disk63 100.00    0.50      30   19485    0.00   34.29
           disk64   58.42    0.50      14   13972    0.00   42.08
           disk65 100.00    0.50      50   34503    0.00   36.83
           disk67 100.00    0.50      54   37259    0.00   36.47
           disk68   11.88    0.50       5    2044    0.00   24.70
           disk69 100.00    0.50      79   58170    0.00   44.69
           disk72 100.00    0.50      35   29149    0.00   38.26
           disk73   76.24    0.50      14   12071    0.00   54.82
           disk75 100.00    0.50      50   41156    0.00   69.98
           disk76    1.98    0.50       2      48    0.00    9.66
           disk77    0.99    0.50       1       2    0.00    5.50
           disk78   61.39    0.50      19   12895    0.00   32.95
           disk79   98.02    0.50      31   20309    0.00   32.04
           disk80   52.48    0.50      16   10075    0.00   33.32
           disk81 100.00    0.50     216    3438    0.00   75.14
           disk82 100.00    0.50     216    3453    0.00   76.36
          disk150   25.74    0.50       7    4150    0.00   36.20

mysystem:root-/>sar -H 1 1

HP-UX mysystem B.11.31 U ia64 07/17/11

12:22:02     ctlr   util t-put IO/s   r/s   w/s   read write avque avwait avserv
                   %age   MB/s   num   num   num   MB/s   MB/s   num   msec   msec
12:22:03    fclp0   100 36.38   270   269     1 36.36   0.02     1      0     36
            fclp1   100 32.09   262   260     2 32.07   0.02     1      0     36
            fclp2    93 29.94   266   262     4 29.87   0.06     1      0     31
            fclp3    97 32.89   269   266     3 32.85   0.04     1      0     30

--------------

is my average service time is too high ? or it can be normal ?

Appreciate your valuable suggestion to tune the system & get the max performance.

:smileyhappy:

Regards,

Sudhakar

Re: IO bottleneck on hp-ux system

SUDHAKAR_18 — Sun, 17 Jul 2011 07:00:08 GMT

also hav a look @ kernel parameters attached

>> Oracle 11g used

>> system running on SD2

Regards,

Sudhakar

Re: IO bottleneck on hp-ux system

Arunabha Banerjee — Sun, 17 Jul 2011 16:16:30 GMT

Hi Sudhakar,

Please review cpu utilization (sar -d 5 5) and check the %wio status report. If it is very high then your performnace will be very slow. Also from Glance check the IO By Disk & IO By Logical Volume (u & v). If you are suspecting the kernel parameter issue then check the kcusage (query the usage of kernel resources). like kcusage -h, kcusage -d, kcusage -m or kcusage -y (man kcusage for details). You can get this details from Glance (t) also.

If possible please run the following comands in business hours and share the output

# sar -d 5 5

# sar -v 5 5

# kcusage -h

# kcusage -d

# kcusage -m

Re: IO bottleneck on hp-ux system

Turgay Cavdar — Mon, 18 Jul 2011 05:58:25 GMT

>>Recently we have added flash drives to our Entrprise storage (EMC DMX4) & configured mountpoints for this system. data files moved accordingly. but we have not got any performance benefits.

So are you sure you had a performance bottlenecks on disks on the past? May be the bottleneck was somewhere else?

Which disks are flash drives?

For me your service times are high, check your storage config and performance...

Re: IO bottleneck on hp-ux system

Duncan Edmonstone — Mon, 18 Jul 2011 06:41:43 GMT

Well those disk service times are pretty shocking - on a "well behaved" system you should be looking at times ideally below 10ms, and definitely below 20ms... however this was one sample (using sar -d 1 1), so I have no idea if that was in any way representative.

So have EMC provided any advice on how many front-end ports you should be using? It is of course possible to upgrade the IO capability of your back-end disks, but if the front-end ports on the DMX can't handle the load, you might not see any advantage.

Re: IO bottleneck on hp-ux system

SUDHAKAR_18 — Mon, 18 Jul 2011 07:12:17 GMT

Hi,

Thanks for replying to my query.

As per my analysis, there is a mountpoint (/oracle11) which is used as oracle temp. which is heavily utilized for transactions. and avg service time also reaching 40ms sometimes.

We have configured direct IO, it has also given some amount of performance boost.

Now i am planing to move the oracle temp (/oracle11) to flash drives.

Re: IO bottleneck on hp-ux system

SUDHAKAR_18 — Mon, 01 Aug 2011 12:46:44 GMT

Adding flash drives did not helped me. Number of IO of mountpoints got reduced but no improvement from the process end.

I came to know that , we can tune the kernel parameter base_pagesize to reduce the IO & improve the memory management.

Can some one help me for base_pagesize & its benefits.

Regards,

Sudhakar

Re: IO bottleneck on hp-ux system

SUDHAKAR_18 — Mon, 08 Aug 2011 12:13:09 GMT

My confusions got cleared after reading below documents.

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01916157/c01916157.pdf

Re: IO bottleneck on hp-ux system

Hein van den Heuvel — Mon, 08 Aug 2011 14:41:41 GMT

Yeah, that's a good whitepaper.

Good to see that it helped, but it did not 'solve' the base problem did it?

You really really really want to focus on Oracle performance stats first, and seek confirmation in the OS stats next.

Work with your DBA (is that you? :-) to get to AWR information going (if licensed, good old statspack if not).

Or drill down with oracle Enterprise Manager views to see whether IO is an issue.

Oracle can identify the IO objects (which table!) and often more importantly the SQL (ordered by Phy Reads) and the HASH value for those top queries allowing you to get to the execution plans.

The OS stats to me suggest a lot of table (or index) scanning.

Not too many IO/sec but lots of blocks/se with an average of 700 blocks/IO.

Given that, I suppose the avserv time is not too bad.

If my thinking is correct, then there was NO reason to add flash drives and I would actually raise hell to have someone explain why the investment in time an money to make that change was warrented.

Flash drives can improve SEEK and ROTATIONAL delays dramaticallto to low and zero,

but from the OS data available here, this application seems to the have a throughput bottleneck, not a response time bottleneck. ( What were they thinking?! Where they thinking other than $$$ ? )

Regards,

Hein

HvdH Performance Consulting.

Re: IO bottleneck on hp-ux system

chindi — Mon, 14 Sep 2015 16:31:59 GMT

Hi Sudhakar,
Can you share what did you do to resolve your issue ??