Operating System - HP-UX
1748102 Members
5009 Online
108758 Solutions
New Discussion юеВ

Re: slow performance, run a process takes up 25 hours

 
Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

result of "top" the first two processes are sleeping.

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 1939 root 148 24 3129M 3588K sleep 1:15 2.48 2.48 data_export1
1 ? 17045 root 148 24 3129M 3588K sleep 12:13 1.31 1.31 data_export2
3 ? 51 root 152 20 1568K 1568K run 203:41 1.22 1.22 vxfsd
1 ? 2831 root 152 20 267M 60480K run 131:30 0.91 0.91 java
2 ? 1643 root 152 20 703M 70252K run 31:57 0.23 0.23 java
2 ? 1485 root 152 20 16936K 2524K run 0:17 0.18 0.18 rpcd
3 ? 2616 root 152 20 98092K 10448K run 2:33 0.16 0.16 vxsvc
3 ? 1850 root 152 20 34840K 5060K run 6:11 0.14 0.14 ovcd
3 ? 1886 root 152 20 69108K 9980K run 1:16 0.14 0.14 coda
1 ? 1820 root -16 20 39748K 16828K run 59:08 0.14 0.14 midaemon
TTr
Honored Contributor

Re: slow performance, run a process takes up 25 hours

You have two export jobs, data_export1 and data_export2. Is this normal or they are conficting jobs?
Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

normal. each dump data from different table, they can be run in parallel. there is no impact to performance.
TTr
Honored Contributor

Re: slow performance, run a process takes up 25 hours

What about my other questions above? database locks, other database jobs and syslog disk errors?
Rita C Workman
Honored Contributor

Re: slow performance, run a process takes up 25 hours

ninode and vx_ninode create roll-over tables in memory. Letting the parm algorhythm do it creates tooooo big a table and wastes memory. Ditto for vx_ninode.

Drop your ninode value to 2048. Requires a reboot.
If your vx_ninode is set to 0 then change it to maybe anywhere from 20,000 to 100,000.

I don't see how much memory you have on this box, just swap on lvol2. So I would at least add 8Gb more of swapspace and use SAM to set it up immediately. You can still do that since your only at 11.23.

It may be an old process didn't clean out of memory correctly, so from top you may not see it. You can do a reboot and see if it cleans up anything left over your not able to identify.
You can use that reboot as an opportunity to make these couple of simple parm changes (if you like).

Just a couple real quick thoughts.
Rgrds,
Rita

Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

Have you added any data recently to this database environment? --> no.
Are there other processes running that conflict in i/o with the export job? --> no.
Are there other processes running that cause locking in the data that would suspend the export job completely until the locking is cleared? --> no locking. export data is not like modify, no exclusive lock is required.
Check the syslog if there are any issues with the c4t0d1 disk. Maybe read or write errors and retries. --> nothing about "c4t0d1" nor sys error.
Check around in the database for anything else that niormally does not run during the export process. --> checked database level, nothing changed recently and it's working normal. export process normally take 2 hours, but it is too long this time (and still hanging).
TTr
Honored Contributor

Re: slow performance, run a process takes up 25 hours

Is there an output file that these two jobs write to? If so does it grow in size? If it does not change in size, it would mean that the two jobs are hung somehow. You might consider killing them and restarting them.
Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

yes, i am monitoring the output data. the file size keep growing. That meaning the process is runing but it is too slow. anything elso to check at the OS level?
Bill Hassell
Honored Contributor

Re: slow performance, run a process takes up 25 hours

Unfortunately, there isn't much you can do with the OS because you did not de-tune the OS. In other words, it used to run in a couple of hours and without (I assume) changes to the OS, the process is running at a much slower rate.

When I see a massive slowdown like this, I immediately check for NFS. If you have any NFS activities, shut them down completely (no automounter, nothing related to network filesystems. I would expect bdf and logins are running slowly too.


Bill Hassell, sysadmin
Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

actually login and bdf is normal speed.