- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: IO bottleneck on hp-ux system
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-16-2011 11:56 PM
07-16-2011 11:56 PM
Hi , I am having a hp-ux 11.31 system, with 16 CPU, 36Gb memory & hyperthreading enabled.
This system is used to run batches on specific time & we are looking into improve the performance of the batches.
Recently we have added flash drives to our Entrprise storage (EMC DMX4) & configured mountpoints for this system. data files moved accordingly. but we have not got any performance benefits.
Also look into below o/p
---------------------
==> from glance
PROCESS LIST Users= 8
User CPU % Thrd Disk Memory Block
Process Name PID Name (1600% max) Cnt IOrate RSS/VSS On
--------------------------------------------------------------------------------
ora_p040_MIS 3393 oradb 56.0 1 17.8 229.8mb 238.3mb IO
ora_p043_MIS 3399 oradb 55.6 1 17.5 230.6mb 238.3mb IO
ora_p044_MIS 3401 oradb 54.3 1 17.6 229.3mb 238.3mb PRI
ora_p041_MIS 3395 oradb 54.3 1 17.8 230.3mb 238.3mb PRI
ora_p047_MIS 3407 oradb 54.1 1 17.6 230.1mb 238.3mb IO
ora_p042_MIS 3397 oradb 53.3 1 17.1 230.6mb 238.3mb IO
ora_p046_MIS 3405 oradb 53.3 1 17.5 229.8mb 238.3mb IO
-------------------------
mysystem:root-/>sar -d 1 1
HP-UX mysystem B.11.31 U ia64 07/17/11
12:18:52 device %busy avque r+w/s blks/s avwait avserv
12:18:53 disk61 100.00 0.50 93 59612 0.00 34.74
disk62 35.64 0.50 6 6020 0.00 60.69
disk63 100.00 0.50 30 19485 0.00 34.29
disk64 58.42 0.50 14 13972 0.00 42.08
disk65 100.00 0.50 50 34503 0.00 36.83
disk67 100.00 0.50 54 37259 0.00 36.47
disk68 11.88 0.50 5 2044 0.00 24.70
disk69 100.00 0.50 79 58170 0.00 44.69
disk72 100.00 0.50 35 29149 0.00 38.26
disk73 76.24 0.50 14 12071 0.00 54.82
disk75 100.00 0.50 50 41156 0.00 69.98
disk76 1.98 0.50 2 48 0.00 9.66
disk77 0.99 0.50 1 2 0.00 5.50
disk78 61.39 0.50 19 12895 0.00 32.95
disk79 98.02 0.50 31 20309 0.00 32.04
disk80 52.48 0.50 16 10075 0.00 33.32
disk81 100.00 0.50 216 3438 0.00 75.14
disk82 100.00 0.50 216 3453 0.00 76.36
disk150 25.74 0.50 7 4150 0.00 36.20
mysystem:root-/>sar -H 1 1
HP-UX mysystem B.11.31 U ia64 07/17/11
12:22:02 ctlr util t-put IO/s r/s w/s read write avque avwait avserv
%age MB/s num num num MB/s MB/s num msec msec
12:22:03 fclp0 100 36.38 270 269 1 36.36 0.02 1 0 36
fclp1 100 32.09 262 260 2 32.07 0.02 1 0 36
fclp2 93 29.94 266 262 4 29.87 0.06 1 0 31
fclp3 97 32.89 269 266 3 32.85 0.04 1 0 30
--------------
is my average service time is too high ? or it can be normal ?
Appreciate your valuable suggestion to tune the system & get the max performance.
:smileyhappy:
Regards,
Sudhakar
Solved! Go to Solution.
- Tags:
- I/O bottleneck
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2011 12:00 AM
07-17-2011 12:00 AM
Re: IO bottleneck on hp-ux system
also hav a look @ kernel parameters attached
>> Oracle 11g used
>> system running on SD2
Regards,
Sudhakar
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2011 09:16 AM
07-17-2011 09:16 AM
Re: IO bottleneck on hp-ux system
Hi Sudhakar,
Please review cpu utilization (sar -d 5 5) and check the %wio status report. If it is very high then your performnace will be very slow. Also from Glance check the IO By Disk & IO By Logical Volume (u & v). If you are suspecting the kernel parameter issue then check the kcusage (query the usage of kernel resources). like kcusage -h, kcusage -d, kcusage -m or kcusage -y (man kcusage for details). You can get this details from Glance (t) also.
If possible please run the following comands in business hours and share the output
# sar -d 5 5
# sar -v 5 5
# kcusage -h
# kcusage -d
# kcusage -m
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2011 10:58 PM
07-17-2011 10:58 PM
Re: IO bottleneck on hp-ux system
>>Recently we have added flash drives to our Entrprise storage (EMC DMX4) & configured mountpoints for this system. data files moved accordingly. but we have not got any performance benefits.
So are you sure you had a performance bottlenecks on disks on the past? May be the bottleneck was somewhere else?
Which disks are flash drives?
For me your service times are high, check your storage config and performance...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-17-2011 11:41 PM
07-17-2011 11:41 PM
Re: IO bottleneck on hp-ux system
Well those disk service times are pretty shocking - on a "well behaved" system you should be looking at times ideally below 10ms, and definitely below 20ms... however this was one sample (using sar -d 1 1), so I have no idea if that was in any way representative.
So have EMC provided any advice on how many front-end ports you should be using? It is of course possible to upgrade the IO capability of your back-end disks, but if the front-end ports on the DMX can't handle the load, you might not see any advantage.
Other questions:
What sort of IO are the batch jobs generating? Presumably large sequential reads, and possibly large sequential writes?
What IO load-balancing algorithms are you using? Are you using PowerPath or native HP-UX load balancing?
You mention mount points, so these systems are obviously on filesystems - what mount options are you using?
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2011 12:12 AM
07-18-2011 12:12 AM
Re: IO bottleneck on hp-ux system
Hi,
Thanks for replying to my query.
As per my analysis, there is a mountpoint (/oracle11) which is used as oracle temp. which is heavily utilized for transactions. and avg service time also reaching 40ms sometimes.
We have configured direct IO, it has also given some amount of performance boost.
Now i am planing to move the oracle temp (/oracle11) to flash drives.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-01-2011 05:46 AM
08-01-2011 05:46 AM
Re: IO bottleneck on hp-ux system
Adding flash drives did not helped me. Number of IO of mountpoints got reduced but no improvement from the process end.
I came to know that , we can tune the kernel parameter base_pagesize to reduce the IO & improve the memory management.
Can some one help me for base_pagesize & its benefits.
Regards,
Sudhakar
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2011 05:13 AM
08-08-2011 05:13 AM
SolutionMy confusions got cleared after reading below documents.
http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01916157/c01916157.pdf
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2011 07:41 AM
08-08-2011 07:41 AM
Re: IO bottleneck on hp-ux system
Yeah, that's a good whitepaper.
Good to see that it helped, but it did not 'solve' the base problem did it?
You really really really want to focus on Oracle performance stats first, and seek confirmation in the OS stats next.
Work with your DBA (is that you? :-) to get to AWR information going (if licensed, good old statspack if not).
Or drill down with oracle Enterprise Manager views to see whether IO is an issue.
Oracle can identify the IO objects (which table!) and often more importantly the SQL (ordered by Phy Reads) and the HASH value for those top queries allowing you to get to the execution plans.
The OS stats to me suggest a lot of table (or index) scanning.
Not too many IO/sec but lots of blocks/se with an average of 700 blocks/IO.
Given that, I suppose the avserv time is not too bad.
If my thinking is correct, then there was NO reason to add flash drives and I would actually raise hell to have someone explain why the investment in time an money to make that change was warrented.
Flash drives can improve SEEK and ROTATIONAL delays dramaticallto to low and zero,
but from the OS data available here, this application seems to the have a throughput bottleneck, not a response time bottleneck. ( What were they thinking?! Where they thinking other than $$$ ? )
Regards,
Hein
HvdH Performance Consulting.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-14-2015 09:31 AM
09-14-2015 09:31 AM
Re: IO bottleneck on hp-ux system
Can you share what did you do to resolve your issue ??