Online Expert Day - HPE Data Storage - Live Now
April 24/25 - Online Expert Day - HPE Data Storage - Live Now
Read more
System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

IO bottleneck on hp-ux system

SOLVED
Go to solution
SUDHAKAR_18
Trusted Contributor

IO bottleneck on hp-ux system

Hi , I am having a hp-ux 11.31 system, with 16 CPU, 36Gb memory & hyperthreading enabled.

 

This system is used to run batches on specific time & we are looking into improve the performance of the batches.

Recently we have added flash drives to our Entrprise storage (EMC DMX4) & configured mountpoints for this system. data files moved accordingly. but we have not got any performance benefits.

 

Also look into below o/p

---------------------

 ==> from glance

                                                                   PROCESS LIST                                                         Users=    8
                         User      CPU %    Thrd Disk        Memory       Block
Process Name     PID     Name   (1600% max) Cnt  IOrate      RSS/VSS      On
--------------------------------------------------------------------------------
ora_p040_MIS        3393 oradb        56.0     1  17.8   229.8mb  238.3mb    IO
ora_p043_MIS        3399 oradb        55.6     1  17.5   230.6mb  238.3mb    IO
ora_p044_MIS        3401 oradb        54.3     1  17.6   229.3mb  238.3mb   PRI
ora_p041_MIS        3395 oradb        54.3     1  17.8   230.3mb  238.3mb   PRI
ora_p047_MIS        3407 oradb        54.1     1  17.6   230.1mb  238.3mb    IO
ora_p042_MIS        3397 oradb        53.3     1  17.1   230.6mb  238.3mb    IO
ora_p046_MIS        3405 oradb        53.3     1  17.5   229.8mb  238.3mb    IO

-------------------------

 

mysystem:root-/>sar -d 1 1

HP-UX mysystem B.11.31 U ia64    07/17/11

12:18:52   device   %busy   avque   r+w/s  blks/s  avwait  avserv
12:18:53   disk61  100.00    0.50      93   59612    0.00   34.74
           disk62   35.64    0.50       6    6020    0.00   60.69
           disk63  100.00    0.50      30   19485    0.00   34.29
           disk64   58.42    0.50      14   13972    0.00   42.08
           disk65  100.00    0.50      50   34503    0.00   36.83
           disk67  100.00    0.50      54   37259    0.00   36.47
           disk68   11.88    0.50       5    2044    0.00   24.70
           disk69  100.00    0.50      79   58170    0.00   44.69
           disk72  100.00    0.50      35   29149    0.00   38.26
           disk73   76.24    0.50      14   12071    0.00   54.82
           disk75  100.00    0.50      50   41156    0.00   69.98
           disk76    1.98    0.50       2      48    0.00    9.66
           disk77    0.99    0.50       1       2    0.00    5.50
           disk78   61.39    0.50      19   12895    0.00   32.95
           disk79   98.02    0.50      31   20309    0.00   32.04
           disk80   52.48    0.50      16   10075    0.00   33.32
           disk81  100.00    0.50     216    3438    0.00   75.14
           disk82  100.00    0.50     216    3453    0.00   76.36
          disk150   25.74    0.50       7    4150    0.00   36.20

 

mysystem:root-/>sar -H 1 1

HP-UX mysystem B.11.31 U ia64    07/17/11

12:22:02     ctlr   util t-put  IO/s   r/s   w/s   read  write avque avwait avserv
                   %age   MB/s   num   num   num   MB/s   MB/s   num   msec   msec
12:22:03    fclp0   100  36.38   270   269     1  36.36   0.02     1      0     36
            fclp1   100  32.09   262   260     2  32.07   0.02     1      0     36
            fclp2    93  29.94   266   262     4  29.87   0.06     1      0     31
            fclp3    97  32.89   269   266     3  32.85   0.04     1      0     30

 

--------------

 

is my average service time is too high ? or it can be normal ?

Appreciate your valuable suggestion to tune the system & get the max performance.

 

:smileyhappy:

Regards,

Sudhakar

 

9 REPLIES
SUDHAKAR_18
Trusted Contributor

Re: IO bottleneck on hp-ux system

also hav a look @ kernel parameters attached

 

>> Oracle 11g used

>> system running on SD2

 

Regards,

Sudhakar

Arunabha Banerjee
Valued Contributor

Re: IO bottleneck on hp-ux system

Hi Sudhakar,

 

Please review cpu utilization (sar -d 5 5) and check the %wio status report. If it is very high then your performnace will be very slow. Also from Glance check the IO By Disk & IO By Logical Volume (u & v). If you are suspecting the kernel parameter issue then check the kcusage (query the usage of kernel resources). like kcusage -h, kcusage -d, kcusage -m or kcusage -y (man kcusage for details). You can get this details from Glance (t) also.

 

If possible please run the following comands in business hours and share the output

 

# sar -d 5 5

# sar -v 5 5

# kcusage -h

# kcusage -d

# kcusage -m

 

 

AB
Turgay Cavdar
Honored Contributor

Re: IO bottleneck on hp-ux system

>>Recently we have added flash drives to our Entrprise storage (EMC DMX4) & configured mountpoints for this system. data files moved accordingly. but we have not got any performance benefits.

So are you sure you had a performance bottlenecks on disks on the past? May be the bottleneck was somewhere else?

Which disks are flash drives?

For me your service times are high, check your storage config and performance...

Duncan Edmonstone
Honored Contributor

Re: IO bottleneck on hp-ux system

Well those disk service times are pretty shocking - on a "well behaved" system you should be looking at times ideally below 10ms, and definitely below 20ms... however this was one sample (using sar -d 1 1), so I have no idea if that was in any way representative.

 

So have EMC provided any advice on how many front-end ports you should be using? It is of course possible to upgrade the IO capability of your back-end disks, but if the front-end ports on the DMX can't handle the load, you might not see any advantage.

 

Other questions:

 

What sort of IO are the batch jobs generating? Presumably large sequential reads, and possibly large sequential writes?

What IO load-balancing algorithms are you using? Are you using PowerPath or native HP-UX load balancing?

You mention mount points, so these systems are obviously on filesystems - what mount options are you using?


HTH

Duncan
SUDHAKAR_18
Trusted Contributor

Re: IO bottleneck on hp-ux system

Hi,

 

Thanks for replying to my query.

 

As per my analysis, there is a mountpoint (/oracle11) which is used as oracle temp. which is heavily utilized for transactions. and avg service time also reaching 40ms sometimes.

 

We have configured direct IO, it has also given some amount of performance boost.

 

Now i am planing to move the oracle temp (/oracle11) to flash drives. 

SUDHAKAR_18
Trusted Contributor

Re: IO bottleneck on hp-ux system

Adding flash drives did not helped me. Number of IO of mountpoints got reduced but no improvement from the process end.

 

I came to know that , we can tune the kernel parameter base_pagesize to reduce the IO & improve the memory management.

 

Can some one help me for base_pagesize & its benefits.

 

Regards,

Sudhakar

SUDHAKAR_18
Trusted Contributor
Solution

Re: IO bottleneck on hp-ux system

My confusions got cleared after reading below documents.

 

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01916157/c01916157.pdf

 

 

Hein van den Heuvel
Honored Contributor

Re: IO bottleneck on hp-ux system

Yeah, that's a good whitepaper.

Good to see that it helped, but it did not 'solve' the base problem did it?

 

You really really really want to focus on Oracle performance stats first, and seek confirmation in the OS stats next.

 

Work with your DBA (is that you? :-) to get to AWR  information going (if licensed, good old statspack if not).

Or drill down with oracle Enterprise Manager views to see whether IO is an issue.

Oracle can identify the IO objects (which table!) and often more importantly the SQL (ordered by Phy Reads) and the HASH value for those top queries allowing you to get to the execution plans.

 

The OS stats to me suggest a lot of table (or index) scanning.

Not too many IO/sec but lots of blocks/se with an average of 700 blocks/IO.

Given that, I suppose the avserv time is not too bad.

 

If my thinking is correct, then there was NO reason to add flash drives and I would actually raise hell to have someone explain why the investment in time an money  to make that  change was warrented.

Flash drives can improve SEEK and ROTATIONAL delays dramaticallto to low and zero,

but from the OS data available here, this application seems to the have a throughput bottleneck, not a response time bottleneck. ( What were they thinking?! Where they thinking other than $$$ ? )

 

Regards,

Hein 

HvdH Performance Consulting.

 

 

chindi
Respected Contributor

Re: IO bottleneck on hp-ux system

Hi Sudhakar,
Can you share what did you do to resolve your issue ??