slow performance, run a process takes up 25 hours

[Grace Li]

I am running a data export process. Normally it takes about 2 hours but now more 25 hours past, it is still running. The result of "top" shows 154 processes of 179 are sleeping. How to tune hp-ux?

#top
Load averages: 0.01, 0.02, 0.02
179 processes: 154 sleeping, 25 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.01 0.0% 3.0% 0.0% 97.0% 0.0% 0.0% 0.0% 0.0%
1 0.02 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
2 0.01 0.0% 1.0% 0.0% 99.0% 0.0% 0.0% 0.0% 0.0%
3 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.01 0.0% 1.0% 0.0% 99.0% 0.0% 0.0% 0.0% 0.0%


#swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol2
reserve - 3606 -3606
total 8192 3606 4586 44% - 0 -
#sar -v 2 4

HP-UX phobos B.11.23 U 9000/800 08/25/09

09:36:23 text-sz ov proc-sz ov inod-sz ov file-sz ov
09:36:25 N/A N/A 178/6420 0 1799/264576 0 2163/300000 0
09:36:27 N/A N/A 178/6420 0 1799/264576 0 2163/300000 0
09:36:29 N/A N/A 178/6420 0 1799/264576 0 2163/300000 0
09:36:31 N/A N/A 178/6420 0 1799/264576 0 2163/300000 0


# vmstat 4 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 2 0 917931 558681 82 14 0 0 0 0 0 1127 3804 522 2 2 96
1 2 0 918404 558620 0 0 0 0 0 0 0 1581 612 426 1 0 99
1 2 0 918404 558473 346 61 0 0 0 0 0 1620 8907 592 4 4 92
1 2 0 1123017 558477 142 24 0 0 0 0 0 1592 3986 494 0 0 99
1 2 0 1123017 558470 62 9 0 0 0 0 0 1601 2075 462 0 0 100

Tunable Value Expression Changes
dbc_max_pct 8 8 Immed
Tunable Value Expression Changes
dbc_min_pct 5 Default Immed

[Tingli]

How about io stat such as:
sar -d 5 10
or
iostat 5 5?

[Grace Li]

#sar -d 5 10

HP-UX phobos B.11.23 U 9000/800 08/25/09

10:29:57 device %busy avque r+w/s blks/s avwait avserv
10:30:02 c2t0d0 18.40 0.64 42 570 0.25 6.92
c2t1d0 14.20 0.66 36 483 0.39 6.30
c4t0d1 99.40 0.69 362 8689 0.20 6.37
10:30:07 c2t0d0 4.19 0.50 8 107 0.00 7.90
c2t1d0 2.99 0.50 6 96 0.00 7.67
c4t0d1 100.00 0.61 362 8305 0.13 6.24
10:30:12 c2t0d0 3.39 0.50 7 97 0.00 10.76
c2t1d0 2.40 0.50 5 83 0.00 7.94
c4t0d1 99.80 0.56 320 7082 0.09 6.69
10:30:17 c2t0d0 7.00 15.42 24 430 21.52 11.49
c2t1d0 4.40 17.68 20 359 29.64 10.49
c4t0d1 100.00 0.64 332 7440 0.17 6.80
10:30:22 c2t0d0 1.60 0.50 3 41 0.00 7.56
c2t1d0 1.40 0.50 2 38 0.00 6.74
c4t0d1 100.00 0.58 337 7618 0.13 6.66
10:30:27 c2t0d0 2.40 0.50 5 81 0.00 9.90
c2t1d0 1.60 0.50 5 84 0.00 7.02
c4t0d1 100.00 0.75 371 9364 0.31 6.62
10:30:32 c2t0d0 1.20 0.50 2 26 0.00 7.35
c2t1d0 0.80 0.50 1 22 0.00 7.37
c4t0d1 99.80 0.68 387 10448 0.20 6.31
10:30:37 c2t0d0 3.80 0.50 7 84 0.00 9.09
c2t1d0 2.20 0.50 4 73 0.00 7.10
c4t0d1 100.00 0.65 373 9176 0.21 6.41
10:30:42 c2t0d0 1.80 0.50 3 42 0.00 10.01
c2t1d0 1.20 0.50 2 38 0.00 6.63
c4t0d1 99.80 0.63 374 9063 0.18 6.34
10:30:47 c2t0d0 2.00 0.50 4 46 0.00 7.38
c2t1d0 1.00 0.50 2 41 0.00 5.49
c4t0d1 99.60 0.56 352 8610 0.12 6.47

Average c2t0d0 4.58 3.89 11 152 4.90 8.71
Average c2t1d0 3.22 4.68 8 132 7.26 7.59
Average c4t0d1 99.86 0.64 357 8580 0.18 6.48

[Grace Li]

# iostat 5 5

device bps sps msps

c2t0d0 0 0.0 1.0
c2t1d0 0 0.0 1.0
c4t0d1 0 0.0 1.0
c5t1d1 0 0.0 1.0

c2t0d0 183 21.0 1.0
c2t1d0 204 20.4 1.0
c4t0d1 3447 323.8 1.0
c5t1d1 0 0.0 1.0

c2t0d0 50 6.4 1.0
c2t1d0 62 4.8 1.0
c4t0d1 3350 336.3 1.0
c5t1d1 0 0.0 1.0

c2t0d0 42 6.2 1.0
c2t1d0 40 5.0 1.0
c4t0d1 4907 362.5 1.0
c5t1d1 0 0.0 1.0

c2t0d0 55 7.2 1.0
c2t1d0 50 4.6 1.0
c4t0d1 4451 350.5 1.0
c5t1d1 0 0.0 1.0

[Yogesh M Puranik]

Hi,

Ususally it happens some times,

Your checks should be,

1]CPU :

Check out the top processes which are using most of the CPU.

#UNIX95= ps -ef -o pid,ruser,pcpu,args|tr -d "%" |awk '{if($4>=90) print $0}'

Also check


#ps -ef |grep -i NO (For oracle process)

Well if you find almost all processes (oracle)are in sleep state since long time,then u can take the decisions to kill the same with proper approvals,


But after looking at vmstat output,cpu is idle and there is hardly utilisation.


2]Memory :

It is most important parameter to check if you are getting very slow response.

in vmstat output, i am looking lots of page out, it means u are going short of memory,and swap utilisation is more.check,
#swapinfo -tam,

Action :

If you see more swap utilisation,then,

run below command,

#ipcs -mob
ex.

#ipcs -mob
IPC status from /dev/kmem as of Tue Aug 25 13:34:26 2009
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411c0189 --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 2 61760
m 2 0x41201f8a --rw-rw-rw- root root 1 8192
m 3 0x00a5c581 --rw------- sfmdb users 4 10469376
m 32772 0x06347849 --rw-rw-rw- root root 2 65544
m 65541 0x0c6629c9 --rw-r----- root root 2 17868736
m 163846 0x491823a2 --rw-r--r-- root root 0 22908

If you see 7th Column (NATTCH )any process is having 0 value, that means thease processes are completed and still they didn't release the memory segment

You can release thorough

#ipcrm -m (Be careful while running this command)


3]Disk utlisation :

#Using iostat command,check any perticular disk is showing any abnormal behavior,


4]Network

On critical server,usually export,back thease activities should do when there will be less load on the server.these can be depend on LAN speed.

Check thease statictics as well,

Hope above points may help u in some manner.

Rgds
Yogesh

[Tingli]

Only thing I can say so far is that c4t0d1 is fairly busy, although its avque is not that high. Maybe there is one job with a lot of io.

[Dennis Handly]

>but now more 25 hours past, it is still running.

The system seems idle. What's still running?

[Grace Li]

i am exporting data to disk, thus I/O is expected to "c4t0d1".

I can eliminate issues at the network area because the process does not related to network.

Also the "swap" space and memory, it's peak usage is 44%, not reaching the alert level.

Regarding cpu, I can see no backgroup process is high up the usage, but my export process sleep very frequently.

[TTr]

The c4t0d1 disk is rather busy. That alone however does not justify the 25 hours. Maybe this disk was just as busy when the export job took only 2 hours. You have to check a few more things.

Have you added any data recently to this database environment?
Are there other processes running that conflict in i/o with the export job?
Are there other processes running that cause locking in the data that would suspend the export job completely until the locking is cleared?
Check the syslog if there are any issues with the c4t0d1 disk. Maybe read or write errors and retries.
Check around in the database for anything else that niormally does not run during the export process.

[Grace Li]

"the system is idle, what is still running" that is the key issue here.

My processes (export data) are suposed to run but they sleep (thus taking long time). they are still hanging there. how to expedite?

[Grace Li]

result of "top" the first two processes are sleeping.

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 ? 1939 root 148 24 3129M 3588K sleep 1:15 2.48 2.48 data_export1
1 ? 17045 root 148 24 3129M 3588K sleep 12:13 1.31 1.31 data_export2
3 ? 51 root 152 20 1568K 1568K run 203:41 1.22 1.22 vxfsd
1 ? 2831 root 152 20 267M 60480K run 131:30 0.91 0.91 java
2 ? 1643 root 152 20 703M 70252K run 31:57 0.23 0.23 java
2 ? 1485 root 152 20 16936K 2524K run 0:17 0.18 0.18 rpcd
3 ? 2616 root 152 20 98092K 10448K run 2:33 0.16 0.16 vxsvc
3 ? 1850 root 152 20 34840K 5060K run 6:11 0.14 0.14 ovcd
3 ? 1886 root 152 20 69108K 9980K run 1:16 0.14 0.14 coda
1 ? 1820 root -16 20 39748K 16828K run 59:08 0.14 0.14 midaemon

[TTr]

You have two export jobs, data_export1 and data_export2. Is this normal or they are conficting jobs?

[Grace Li]

normal. each dump data from different table, they can be run in parallel. there is no impact to performance.

[TTr]

What about my other questions above? database locks, other database jobs and syslog disk errors?

[Rita C Workman]

ninode and vx_ninode create roll-over tables in memory. Letting the parm algorhythm do it creates tooooo big a table and wastes memory. Ditto for vx_ninode.

Drop your ninode value to 2048. Requires a reboot.
If your vx_ninode is set to 0 then change it to maybe anywhere from 20,000 to 100,000.

I don't see how much memory you have on this box, just swap on lvol2. So I would at least add 8Gb more of swapspace and use SAM to set it up immediately. You can still do that since your only at 11.23.

It may be an old process didn't clean out of memory correctly, so from top you may not see it. You can do a reboot and see if it cleans up anything left over your not able to identify.
You can use that reboot as an opportunity to make these couple of simple parm changes (if you like).

Just a couple real quick thoughts.
Rgrds,
Rita

[Grace Li]

Have you added any data recently to this database environment? --> no.
Are there other processes running that conflict in i/o with the export job? --> no.
Are there other processes running that cause locking in the data that would suspend the export job completely until the locking is cleared? --> no locking. export data is not like modify, no exclusive lock is required.
Check the syslog if there are any issues with the c4t0d1 disk. Maybe read or write errors and retries. --> nothing about "c4t0d1" nor sys error.
Check around in the database for anything else that niormally does not run during the export process. --> checked database level, nothing changed recently and it's working normal. export process normally take 2 hours, but it is too long this time (and still hanging).

[TTr]

Is there an output file that these two jobs write to? If so does it grow in size? If it does not change in size, it would mean that the two jobs are hung somehow. You might consider killing them and restarting them.

[Grace Li]

yes, i am monitoring the output data. the file size keep growing. That meaning the process is runing but it is too slow. anything elso to check at the OS level?

[Bill Hassell]

Unfortunately, there isn't much you can do with the OS because you did not de-tune the OS. In other words, it used to run in a couple of hours and without (I assume) changes to the OS, the process is running at a much slower rate.

When I see a massive slowdown like this, I immediately check for NFS. If you have any NFS activities, shut them down completely (no automounter, nothing related to network filesystems. I would expect bdf and logins are running slowly too.

[Grace Li]

actually login and bdf is normal speed.