Need help troubleshooting performance issue

[Tony Williams]

Over most of the day and during this time the number of processes are steady from 3700 to 3900. I'm attaching a Excel 2007 spreadsheet with a lot of the Global metrics I'm looking at. B4 the CPU and Global priority queue go up, The root disk become very busy (400% utilization) page requests and free memory start to go down,

[Michael Steele_2]

Hi

Need to see a breakdown by process - Please include the reports requested above

[Tony Williams]

I don't seem to be able toopen the file so I, going to try a excell 2003 format.

[Tony Williams]

Thanks Michael,

I have included all but the sar -d there are several thousand disks so the output is very long.

[Michael Steele_2]

1049892 R 18018 1 java
966304 S 6595 10378 dw.sapTPQ_DVEBMGS82
716076 S 28075 8116 dw.sapEWD_DVEBMGS32
609112 S 24831 24816 dw.sapERQ_DVEBMGS29
565144 S 10060 24816 dw.sapERQ_DVEBMGS29

So keep an eye on these processes.

Put the UNIX95 commands in a 15 cron and save the data for at least two days unless you see vsz process growth greater than the above.

Include sar -c -u and -v.

Attach the sar -d Totals: Note any avwait > aserv disks and pvdisplay -v those disks and note the file systems.

[Tony Williams]

Thanks Michael,

This problem happened this past Friday and the 1st order of business is always to free up resources, its only after everything is back to normal that we start looking to see what the problem was. Your suggestions will help if we have the problem again.

[Raj D.]

Tony,

>>The problem was 100% CPU utilization with a global priority queue of 120, a memory queue of 60, and a disk queue of 20.

The 100% cpu utilization and pri.queue 120 says it all. And login takes 10 minute as priority queue was high with cpu bottleneck.


Here the question would be:
- Did you see any increased load at that time. i.e may be more oracle process or more java process or more application than usual scenario, or more batch was executed.
- How many cpu do you have . What is the model of the server.
- How many process wa runningduring that time, and how many process runs at usual load.
- what was the load factor at that time. Obviously it would be more than 1, 2 ..
- What measureware 'extract' report shows the historical data of cpu/mem/io/swap/network in/out etc.
From above we can narrow down the cause,

Hth,
Raj.

[Michael Steele_2]

HI

What is this process?

1049892 R 18018 1 java : First in virtual memory and gone to init. Is that normal for it to go to init or should it have a parent pid?

What is this process?

90.82 R 18669 18375 jlaunch : 2nd in cpu activity only behind the kernel.

Java login?

Question to Others:

Is it normal for 'kernel' to be consumming the most CPU time?

[Michael Steele_2]

HI

What HP-UX version?

Is this a virtual server or what?

[Raj D.]

Tony,


>>I have included all but the sar -d there are several thousand disks so the output is very long.

Well, to get a clear idea quickly , if the disks are hitting heavily ,you can check with a small scrpt (sar -d) to find out disk and their correspoding vgs . (check the attached one : find_high_io_wait_11iv2.sc ) . Then if you see the avwait is more you can try to locate the cause of the problem.

Hope this helps..,
Raj.