ias 10G application server, how do you do it?

[Steven E. Protter]

Environment.
rp5450 server 8 GB RAM, tuned by me.

Databse 9.2.0.4 9i Rel 2 database server, patched current and often.

ias 10g as (application server) No patches from oracle yet. All required OS patches and kernel parameter changes implemented.

Problem:

Poor performance.

as 10g has two major components. A database for the as server and the application server.

Our dba set up the database to be owned and started by user oracle, the owner of the 9.2.0.4 database binaries. We change the ORACLE_SID and the ORACLE_HOME but we start with the oracle user.

Peformance is miserable for the AS server.

I was asked to write a startup script for the product suite. I made a mistake, possibly a fortitous one. I accidently used the ias10g user, the owner of the application server binaries, and the psuedo as database server binaries to start the database.

So, we do this:
oracle user starts the as database
ias10g user starts the application user.

Bad peformance.

ias10g user starts both, good performance, but performance degrades over time.

How do you do it? Anyone had a similar issue? If so, what did you do to solve it.

Note, this case is specific to the product sets listed above. OS is 11i v1, also patched to near perfection (so I hope).

Peformance monitoring shows an ocean of shared memory and no resource issues. The box has swap reserved but is not paging at all.

SEP

[Steven E. Protter]

Preliminary results say (duh), gotta start the applicaton servers "psuedo" database with the owner of the binaries.

Current theory has the perfomance problems arising from confusion at just what user owns the shared memory.

Anyone have experience or insight to add?

Have a nice weekend.

SEP

[Victor BERRIDGE]

Hi Steven,
I havent installed IAS10g yes so I dont know what to say with 9 you will have to wait till I return to work and look in my notes...
But I can say this:
We never use oracle to launch anything it is the keeper UID of the software...
we create individual dba who is owner of instance and apps... since we had issues on servers running many instances I struggled to find in shared memory semaphores etc who was who...
this helps alot when it comes to ipcrm...
I remember seiing a lot of open sockets not beeing cleaned properly...

Good luck
Have a nice week-end
All the best
Victor

[Gary Cantwell]

updated subject line as requestewd,

Cheers,

Gary

[Steven E. Protter]

Bumping with prejudice.

Here is additional information. The problem seems to get worse over time. Just letting the box sit there seems to cause issues.

10G seems to have a web server based on Apache 2.0.x

HP-UX 11i has a built in web server based on Apache 1.3.x

Can these two web servers be corrupting each other shared memory? Should i upgrade and replace the Apache 1.3. server. I do actually have some internal docs sitting on the Apache server.

Also, anyone have trouble with shared memory/semaphores slowing down or getting corrupted and have a patch they can recommend.

This is getting annoying.

We seem to be able to duplicate the problem like this:

reports server starts right after the rest of the 10G architecture.

We stop the reports server according to Oracle guidelines. We start it again, via the web interface or command line.

Then we have the problem.

Someone must have experienced this before. I've never thought myself bleeding edge.

SEP

[Alzhy]

SEP,

If you've Measureware.. call it to action. You should be able to track resource use (memory, cpu, IPC, etc..) over time and see which application group (yes you need to group them) is exhibiting the obvious.

With Oracle9i Rel. 2, we've uncovered a number of slow leakers that have been remedied by patches from Oracle.

Also, can't you avoid have both Application Server and Database server on the same boxen so you can tune separately?

I've always opposed Application and DB serving to be handled by a single OS instance (ie. single server). Can't you vPar your box so one instance is DB server and the other the app server?

[Steven E. Protter]

Note: App server and Database server ARE on the same box.

I've been running glance/gpm for days. When the application server slows to a crawl the system blissfully reports no performance problems whatsover.

Quite vexing. Current theory is Apache 1.3 which shipped with the OS is bothering the Apache 2.0 based httpd server that shipped with ias 10G.

Getting ready to open a software call. Lets see if ITRC can beat the response center.

Using a modified version of HP's performance data collection scripts (the one i post all over itrc forums). Not too much I know how to do to collect data on shared memory performance and corruption.

SEP

[Steven E. Protter]

Nelson note: rp5450 servers can't vpar.

SEP

[Alzhy]

Oh.. I thought it was an 5470.. ;-(


Same other suggestions though.


You can try using WLM/PSETs if you've the license..

[Victor BERRIDGE]

Hi Steven,
I unfortunatly cleaned up my mailbox so I have no info left and have to wait till a DBA will be kind enough to spare some time with me...
but here are some perf I collected with your famous script...
You will not see anything of the swinstall because using my own account (that still works... yes hard times lately...)

[Victor BERRIDGE]

Hi Steven,
I unfortunatly cleaned up my mailbox so I have no info left and have to wait till a DBA will be kind enough to spare some time with me...
but here are some perf I collected with your famous script...
You will not see anything of the swinstall because using my own account (that still works... yes hard times lately...)

Sorry - previous post missed the click attachment..

All the best
Victor

[Alzhy]

Stephen,

I've polled all my Compadres in the industry and all those that have Oracle 10 Infrastructure have their Application server running on a separate boxen - another UNIX, Linux or Windows.

Although not impossible, it will be tricky to tune your environment so it is friendly to the Database as well as the Application server (IAS).

With 10G.. I think all tuning rules that applied to 9i still is valid and the deep division between cooked and raw still exists.

[Steven E. Protter]

Victor,

Well, maybe it shows you are getting decent peformance. Thats true.

When I ran these reports myself there was nothing on them. The performance issue seems somewhat related to shared memory performance.

I have yet to be able to measure that. I need an addition to my famous script set.

I am confused and frustrated.

We seem to be able to develop the performance problem, but we have not identified an exact trigger. Maybe its apache interference, but at least once we had the problem with apache shut. It appears 10G is based on apaceh 1.3, which directly contradicts what they told us in teleconference. Surprise.

SEP

[wayne_104]

Hi Steven

How did you eventually sort out your problem i have the same problem.

[Steven E. Protter]

It was pretty much oracle Q&A problems all the way. Installed fixes from Oracle, tried to keep our support calls out of certain inefficient call centers. Vanilla all the way.

SEP

[Eric Antunes]

Hi Steven,

Did you saw Metalink Note 260449.1?

Best Regards,

Eric Antunes

[wayne_104]

Thanks You are a star

[TwoProc]

Steven, I apologize for amount of different directions that this response it taking - but please allow me just to come up with possibles that have occured to me...

Just wondering, database 9.2.0.4 had a lot of problems with latch contention on HPUX (maybe others too) - which we got out of by patching up to 9.2.0.5. I'm guessing this could be it - although you said that this seems to slow down over time with nothing running...

As a point of reference - on a Linux box we've setup a test server to get a feel for Oracle's Portal product - it's got the same setup that you're describing - but I'm not seeing this performance problem.

Have you run a statspack set to see what the waits are for?

Also, in an effort to isolate whether or not it is the Apache Server or the database - why not just create a few tests that are going to hit the Apache Server only, then starting hitting individual modules - plsql module, portal module, etc. May with a bit of repetition you could identify it is Apache.

Another thought the portal runs in Java, and I'm counting four different ones running on our server right now. Isee that most start with a -Xmx256m ,and one starts with -Xmx512m. In an effort to rule out garbage collection in the java server (I've seen this tear up java servers and their associated servlets) can you increase all of the startup parameters for each one to -Xmx1024m ?

From what I've seen with out java servlets running on servers started with too little memory - you could be invoking garbage collection for memory issues at a heavy rate, and this could be causing your problem. I've noticed this type of behavior in Oracle Product Configurator when memory sizes were set too low. If this turns out to be part of your problem, then I would think that some process is repeatedly running in the java servlets and the servlet is slowly leaking/fragmenting memory.

[Steven E. Protter]

Good post.

I'll look into it.

SEP