system performance suddenly slow

[Timothy P. Jackson]

We are running a process that is run early every morning. For some reason the process it taking at least twice as long as it should be and not done yet. The first thing that gets blamed is the OS and hardware. I have run the performance monitoring script that has been posted in the forums by Clay but I cannot seem to find any real problems. Maybe I am missing something. I have attached the reports from the performance script.

I did modify the script a little bit so that I would only get the reports that I wanted.

Any help would be greatly appreciated!

Tim

[Yogeeraj_1]

hi Tim,

based on the top output:
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 pts/tj 8988 root 239 20 75812K 7240K run 334:18 100.01 99.84 udt
3 pts/to 12109 ma100273 154 20 71660K 7784K sleep 14:42 44.94 44.86 udt



Seems like the udt process is consuming much of the server resources. It could be waiting on a remote resource as well. Is there an Oracle database at the backend?


what is the nature of the process in question here?

revert

kind regards,
yogeeraj

[Timothy P. Jackson]

Thanks for the response Yogeeraj!

The nature of the process is MRP. It is running native Unidata 6.0 and the programming is in Unibasic. The ERP package that this is part of is called Manage-2000.

Yesterday this process ran in just over 3 hours. Today it is looking like it will finish in about about 9.5 hours. So I started to look at the OS to see if I was hitting any limits or thresholds.

I have just got authorization from the director to reboot the system and do some file maintenance, maybe that will help.

If you have any other ideas I would sure appreciate hearing them.

Thanks,
Tim

[Yogeeraj_1]

hi,

Am not knowledgeable to such a system. Did you have a look at the database? log files? error files? Any changes made recently? new software installed?

Sorry cannot help much.

I assume that there is no errors at the OS level.

good luck!
kind regards
yogeeraj

[Timothy P. Jackson]

Thanks again yogeeraj!

I have no OS errors or Database errors that I can see at moment. The first thing that gets blamed is the HP system and the OS. I believe that it is a Database file problem, but I had to do my due diligence and check out the OS before I can get the DBA to start looking at database files.

Thanks for taking a second look at my OS stats!

Tim

[Hein van den Heuvel]

The attached stats show at first glance nothing special other than the 1 'udt' process possibly being in an infinite loop.
Plenty of memory and cpu resources.
The disks barely loaded... The only thing to check there is c6t0d3. How is it used? While not extremely loaded, and showing good response times it also shows deep queues at time. As if IO's come in bunches.

Is the system dependent on information residing on other systems? Some database connection or RPC/TPC tasks? How is the network behaving?

>> I have no OS errors or Database errors that I can see at moment.

Good.

>> The first thing that gets blamed is the HP system and the OS.

Why? It did not get changed did it?

As yogeeraj asked... what changed?
More likely you'll find it was a business taks, or some business volume.

>> I believe that it is a Database file problem, but I had to do my due diligence and check out the OS before I can get the DBA to start looking at database files.

If it 'used to work reasonable' then you probably should start looking higher still: The application. True, the DB/DBA can possibly give a hint why more or different resources are used. Did anything 'happen' in DB space? recent new data loaded? Index rebuilds?...

IF a reboot help, it is more likely due to the implied application re-starts (java) then for an OS reason.

Good luck!

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Timothy P. Jackson]

Thanks for the reply,

Nothing from the OS side should have changed. I am suppose to be the only one making OS changes and updates and I didn't do anything. Unfortunately this process can only be run at a certian time. So I will have to wait until morning to see if the reboot helped.

Thanks again!

Tim

[Dennis Handly]

If you think these udt processes are runaway, one thing you can do is to renice them so they are at the bottom:
renice -n 20 8988 12109 27068 12448 12040 12045

If you know they are doing useful work, then you don't want to do that.

>> The first thing that gets blamed is the HP system and the OS.

>Hein: Why? It did not get changed did it?

Because the OS is suppose to adapt to changing requirements and do the right thing. Which is defined by the customer, of course. :-)

[Jeeshan]

Hi

Did you check the buffer cache, disk or number of open file usage?

Most of these are related to kernel parameters.

Did you change anything in your script?
Or did you increase your load?

[Ilkka Seittenranta]

Hi Tim,

You system uses about 60% of memory to swapping (see swapinfo).

You must disable pseudoswap if you like find what kind performance problems you have. After pseudoswap uses all free memory you can't get any error messages in the syslog-file, there is not enough resources to open file handles!

After pseudoswap uses all free memory the system change to "idle-process"-state, the state is something "waiting until there is more recourses free". The nonsystem-process are that time to "halt"-state and that time load is something like 0.02

Best Regard
Ilkka