topic Re: Performance Problems in Operating System - HP-UX

Performance Problems

David Bell_1 — Wed, 12 Feb 2003 19:31:44 GMT

All,

I'm having some performance issues with a particular HP9000/L2000. I've looked at a lot of things but haven't found anything in particular, perhaps I'm missing something! I know that running 32 bit Oracle on a 64 bit OS is part of the problem along with the memory implications, however, I believe the performance should be better. Could you have a look at the parameters in the attached file and see if you see anything glaring? By the way, each instance is running in it's own memory window. The problem si users reporting sluggish response even when the SPU shows a high percentage idle. In addition, they occasionally will get booted off the host (seession closed). Thanks for your time,

Dave

Re: Performance Problems

Ken Hubnik_2 — Wed, 12 Feb 2003 19:39:42 GMT

What speed are your network interface cards set to 10 or 100 ??? Half or Full duples?? It could be a network issue??

Re: Performance Problems

Jeff Schussele — Wed, 12 Feb 2003 19:41:26 GMT

Hi Dave,

One thing I'd do is change swap priority.

Make the vg00 swap device priority = 1 & the external devices priority = 0 & let them round-robin if they need to.
You may be slowing normal OS activity by swapping on the same device the OS resides.

Rgds,
Jeff

Re: Performance Problems

David Bell_1 — Wed, 12 Feb 2003 19:44:17 GMT

Ken,

The network interface is set to 100FD manually on the Host as well as the switch. It is running at 100FD. I'm not ruling out network problems at this time, howver, I'm looking elsewhere first. This could potentially be a layer2/layer3 problem, however, there are several other hosts of the same type on this switch. Further, I've changed from LAN 0 to LAN 1 with the same problem. The LAN patches are all up to date per HP.

Thanks,

Dave

Re: Performance Problems

David Bell_1 — Wed, 12 Feb 2003 19:48:36 GMT

Jeff,

Thanks for the tip. I will make that change as soon as I can schedule a reboot. I see your point, however, the primary swap (lvol2) is typically configured as part of the root volume group during installation. Are you saying that the swap should be installed on a different disk (away from the OS) when possible?

Thanks,

Dave

Re: Performance Problems

Vincent Fleming — Wed, 12 Feb 2003 19:57:35 GMT

Dave,

You look a bit I/O bound, too - your wio% is too high. Best is < 10.

You could probably help this by striping the drives; it looks like you're pounding 2 drives only. If you can spread this load out, you'll see lower wio times and better performance overall.

Can you tell us about how you have your LVM configured and what you're using for disks?

-Vince

Re: Performance Problems

Jeff Schussele — Wed, 12 Feb 2003 19:58:53 GMT

Hi Dave,

No, you're correct, you must have a swap partition in vg00, but there's nothing that says you can't adjust priorities such that it's the last used.

The fact that users occasionally get dropped though, does tend to point to the network being involved in some fashion however. I just wanted to point out to you that you are swapping to vg00 & that alone can have an impact.

Cheers,
Jeff

Re: Performance Problems

RAC_1 — Wed, 12 Feb 2003 20:01:07 GMT

I would do pri=1 for secondary swaps. (Not all but at least two)

What I achive eith this?
Interleaving should start with this. Swapping load gets divided between primary swap under vg00 and secondary swap.

I would also increase dbc_max_pct to 10 and watch.

As mentioned not to forget about NIC and switch speed settings.

Re: Performance Problems

Vincent Fleming — Wed, 12 Feb 2003 20:01:28 GMT

Dave,

I just noticed that you have 2 VA7100's... Would they be c9t0d2 and c15t0d2?

Are you sharing these arrays with other hosts?

Is it in AutoRAID mode?

-Vince

Re: Performance Problems

David Bell_1 — Wed, 12 Feb 2003 21:04:44 GMT

Vince,

I believe I may have some I/O bottle necking going on. I agree with your assumptions of %wio, hence the start of this thread. The configuration is as follows:

Array 1:

(15) 73GB mechanisms
1024 cache
(6)LUNS - (4)140GB + (2)75GB

Array 2:

(12) 73GB mechanisms
1024 cache
(4)LUNS - (4)140GB

The Volume groups are configured to utilize one LUN from each array using a 64K stripe for all LVs. There is the primary link and three alternate links. The exception is that the two 75GB LUNS are configured in the same VG as they are not often utilized. The reason you see higher I/O on that particular LUN(s) is due to the fact that that VG represents a particular instance of Oracle and others are either not utilized or nearly idle at this time. I'll admit that the free space (not assigned to LUNs)utilization is minimal and could stand some improvement, however, it does have the Hot Spare for overhead of 66GB.

Jeff,

Thanks for the clarification on the "swap".

Anil,

I have thought about the increase of dbc_max_pct, however, after reading many posts on this subject, I tend to lean closer to the 5 and below.

Thanks for all the replies so far.

Dave

Re: Performance Problems

Vincent Fleming — Wed, 12 Feb 2003 21:23:13 GMT

Dave,

Have you made sure that the primary PV-link is controller MC/1 for all LUNs on both arrays?

You can also increase your queue depth and see what happens - the VA can handle a higher queue depth than other arrays. ('scsictl -m queue_depth=16 /dev/rdsk/c5t0d0') I would suggest between 9 and 12, but I think you could go as high as 30, if your access patterns are bursty.

Could you post the output from armdsp -e and armdsp -a for the arrays?

Also... if I get what you're saying, the load changes as the DBs become active - so the two I pointed out was just coincidence... others become active when/if these subside?

Thanks,

Vince

Re: Performance Problems

David Bell_1 — Wed, 12 Feb 2003 21:57:30 GMT

Vince,

First, since I missed it on my last post, NO I'm not sharing these arrays with any other hosts.

The Primary PV-Link for each LUN is NOT MC/1. This is because I spoke with several folks about this including some HP folks. I know that all writes occur on MC/1, however, it was felt that the I/O going through MC/2 across the 800MB internal backplane would be faster than having ALL data (including reads) going through MC/1 only. If this is incorrect, I have some changes to make. As to the Q-Depth, I've heard it tossed around but I've never actually made any adjustments to it.

Our access is based upon 3 development instances of Oracle. Some days are relatively light and other days are excessively heavy. This makes it difficult to gauge where the difficulties lie. Basically, the instances are active but may not be being accessed all the time. So if DEV is being accessed, the devices you saw (LUN2/Array1 + LUN2/Array2)may be relatively high while LUNS 3,4,5, and 6 are relatively low. The next time it could be CRM fairly high so LUN 4 from each array may be high. Sometimes, they're all being used so it's all quite high. When that happens, typically the CPUs will max out. No one expects perfect performance out of these instances as they are for "development". However, they need them to be relatively proficient which apparently they have not been!

Can I adjust the q-depth while the array is accessed or should it be quiesced?

I appreciate your insight Vince. I've attached the requested information. Let me know if there's problems with it.

Thanks,

Dave

Re: Performance Problems

David Bell_1 — Thu, 13 Feb 2003 15:09:14 GMT

Dave

Re: Performance Problems

Dave Chamberlin — Thu, 13 Feb 2003 15:51:01 GMT

Hi Dave,

A agree that you do have some IO issues. I am not sure what you are running though - oracle 11.5.8 is an oracle applications level - not a database instance. If you are running database instances - what do the initORA parameters look like? The 32 bit Oracle may not be an issue. I run 32 bit Oracle on my machine with 3.5G RAM,only using about 900M for my SGA. I would recommend adding more disk cache buffering - bringing it closer to 300M (but not higher) by changing dbc_max_pct to 5.

Re: Performance Problems

James Odak — Thu, 13 Feb 2003 16:13:21 GMT

One thing that may or may not help is your max inodes ..its huge 18k and you are using only 5k .. from what i have gathered ..inodes are only needed for NFS file systems and since you are using VxFS(except for stand of course) you do not need this so high ..8k would be better ... as explained to me having such a high max may effect performace as it reserves some resources in the event that max is achived

Good luck

Jim

FYI i believe that is controlled by the ninode kernel parameter

Re: Performance Problems

David Bell_1 — Thu, 13 Feb 2003 16:55:45 GMT

Dave,

With regards to the Oracle, excuse my ignorance with the numbers. While I have some exposure, the DBAs do the primary work here. The Oracle Database being utilized is 8.1.7.4. That being said, I'm not sure what the initORA paramaters look like as they were set up prior to my arrival by the DBAs and previous admin. I will look into this as well. Thanks for the insight.

As to your point about dbc_max_pct, I will give the 5 % marker a try. As I explained to Anil in a previous response to this thread, I have read a lot on this and the answer seems to vary a bit. However, overwhelming it appears that 5 and below is best. I will bring it up to 5 and see what benefit it has. Sorry about the "4" points, it should have been "5" I didn't notice it until it was already in.

James,

Thanks for the tip on max inodes. I know it's pretty big as well. It does run quite a bit higher when all the databases are active. I don't know at this point whether it bumps the 18K or not, I'm guessing it's a bit high. I'll check that as I'm going and once I have opportunity to make some changes and reboot.

Dave

Re: Performance Problems

Vincent Fleming — Mon, 17 Feb 2003 22:10:21 GMT

Dave;

Sorry it's taken a while to get back to this;

OK... start out by getting the VAs to the current FW revision; it has performance enhancements in it. (I think it's HP18 this week).

Any LUNs that you access via MC/2 will be slower than the ones accessed via MC/1. I would try all the LUNs through MC/1 and see how it goes. If, as you say, the databases are bursty (usually not becoming active at the same time), then your performance will probably improve.

Load balancing VA controllers should only be used when throughput is more important than latency (not usually the case with databases). (ie: use both controllers for sequential bandwidth, but not for databases)

Let us know how you make out.

Vince

Re: Performance Problems

Steven E. Protter — Tue, 18 Feb 2003 03:15:28 GMT

maybe a little bigger
msgseg 2048

maybe a lot bigger
nflocks 3000

maybe a lot more shared memory segments for oracle
shmseg 300

Oracle is a MASSIVE user of shared memory resources.

Long term, think about a 64 bit version of Oracle. I know I have done it it hurts but ours is purring quite nicely during load testing.

SEP

Re: Performance Problems

Giri Sekar. — Tue, 18 Feb 2003 03:22:45 GMT

Hi:

OK. Here is one thought. Sometimes the switch OR SSR may be set to ip-level instead of data level. Make sure that is not the case.

Thanks

Giri Sekar.

Re: Performance Problems

David Bell_1 — Wed, 19 Feb 2003 00:21:42 GMT

Vince,

Sorry about the points. That scrolling mouse seems to get me in trouble in these windows. I will try switching them over after I have performed the upgrades to firmware. Thanks very much for the insight.

Steven,

Thank you for the kernel suggestions, I'll be making some tuning changes hopefully this weekend.

Giri,

I'm not too sure which switch you were referencing. If you could be a bit more detailed I would appreciate it.

Thanks all,

Dave