Operating System - HP-UX
1752587 Members
4002 Online
108788 Solutions
New Discussion юеВ

Re: slow performance, run a process takes up 25 hours

 
Grace Li
Frequent Advisor

slow performance, run a process takes up 25 hours

I am running a data export process. Normally it takes about 2 hours but now more 25 hours past, it is still running. The result of "top" shows 154 processes of 179 are sleeping. How to tune hp-ux?

#top
Load averages: 0.01, 0.02, 0.02
179 processes: 154 sleeping, 25 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.01 0.0% 3.0% 0.0% 97.0% 0.0% 0.0% 0.0% 0.0%
1 0.02 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
2 0.01 0.0% 1.0% 0.0% 99.0% 0.0% 0.0% 0.0% 0.0%
3 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.01 0.0% 1.0% 0.0% 99.0% 0.0% 0.0% 0.0% 0.0%


#swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol2
reserve - 3606 -3606
total 8192 3606 4586 44% - 0 -
#sar -v 2 4

HP-UX phobos B.11.23 U 9000/800 08/25/09

09:36:23 text-sz ov proc-sz ov inod-sz ov file-sz ov
09:36:25 N/A N/A 178/6420 0 1799/264576 0 2163/300000 0
09:36:27 N/A N/A 178/6420 0 1799/264576 0 2163/300000 0
09:36:29 N/A N/A 178/6420 0 1799/264576 0 2163/300000 0
09:36:31 N/A N/A 178/6420 0 1799/264576 0 2163/300000 0


# vmstat 4 5
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 2 0 917931 558681 82 14 0 0 0 0 0 1127 3804 522 2 2 96
1 2 0 918404 558620 0 0 0 0 0 0 0 1581 612 426 1 0 99
1 2 0 918404 558473 346 61 0 0 0 0 0 1620 8907 592 4 4 92
1 2 0 1123017 558477 142 24 0 0 0 0 0 1592 3986 494 0 0 99
1 2 0 1123017 558470 62 9 0 0 0 0 0 1601 2075 462 0 0 100

Tunable Value Expression Changes
dbc_max_pct 8 8 Immed
Tunable Value Expression Changes
dbc_min_pct 5 Default Immed
19 REPLIES 19
Tingli
Esteemed Contributor

Re: slow performance, run a process takes up 25 hours

How about io stat such as:
sar -d 5 10
or
iostat 5 5?
Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

#sar -d 5 10

HP-UX phobos B.11.23 U 9000/800 08/25/09

10:29:57 device %busy avque r+w/s blks/s avwait avserv
10:30:02 c2t0d0 18.40 0.64 42 570 0.25 6.92
c2t1d0 14.20 0.66 36 483 0.39 6.30
c4t0d1 99.40 0.69 362 8689 0.20 6.37
10:30:07 c2t0d0 4.19 0.50 8 107 0.00 7.90
c2t1d0 2.99 0.50 6 96 0.00 7.67
c4t0d1 100.00 0.61 362 8305 0.13 6.24
10:30:12 c2t0d0 3.39 0.50 7 97 0.00 10.76
c2t1d0 2.40 0.50 5 83 0.00 7.94
c4t0d1 99.80 0.56 320 7082 0.09 6.69
10:30:17 c2t0d0 7.00 15.42 24 430 21.52 11.49
c2t1d0 4.40 17.68 20 359 29.64 10.49
c4t0d1 100.00 0.64 332 7440 0.17 6.80
10:30:22 c2t0d0 1.60 0.50 3 41 0.00 7.56
c2t1d0 1.40 0.50 2 38 0.00 6.74
c4t0d1 100.00 0.58 337 7618 0.13 6.66
10:30:27 c2t0d0 2.40 0.50 5 81 0.00 9.90
c2t1d0 1.60 0.50 5 84 0.00 7.02
c4t0d1 100.00 0.75 371 9364 0.31 6.62
10:30:32 c2t0d0 1.20 0.50 2 26 0.00 7.35
c2t1d0 0.80 0.50 1 22 0.00 7.37
c4t0d1 99.80 0.68 387 10448 0.20 6.31
10:30:37 c2t0d0 3.80 0.50 7 84 0.00 9.09
c2t1d0 2.20 0.50 4 73 0.00 7.10
c4t0d1 100.00 0.65 373 9176 0.21 6.41
10:30:42 c2t0d0 1.80 0.50 3 42 0.00 10.01
c2t1d0 1.20 0.50 2 38 0.00 6.63
c4t0d1 99.80 0.63 374 9063 0.18 6.34
10:30:47 c2t0d0 2.00 0.50 4 46 0.00 7.38
c2t1d0 1.00 0.50 2 41 0.00 5.49
c4t0d1 99.60 0.56 352 8610 0.12 6.47

Average c2t0d0 4.58 3.89 11 152 4.90 8.71
Average c2t1d0 3.22 4.68 8 132 7.26 7.59
Average c4t0d1 99.86 0.64 357 8580 0.18 6.48
Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

# iostat 5 5

device bps sps msps

c2t0d0 0 0.0 1.0
c2t1d0 0 0.0 1.0
c4t0d1 0 0.0 1.0
c5t1d1 0 0.0 1.0

c2t0d0 183 21.0 1.0
c2t1d0 204 20.4 1.0
c4t0d1 3447 323.8 1.0
c5t1d1 0 0.0 1.0

c2t0d0 50 6.4 1.0
c2t1d0 62 4.8 1.0
c4t0d1 3350 336.3 1.0
c5t1d1 0 0.0 1.0

c2t0d0 42 6.2 1.0
c2t1d0 40 5.0 1.0
c4t0d1 4907 362.5 1.0
c5t1d1 0 0.0 1.0

c2t0d0 55 7.2 1.0
c2t1d0 50 4.6 1.0
c4t0d1 4451 350.5 1.0
c5t1d1 0 0.0 1.0
Yogesh M Puranik
Valued Contributor

Re: slow performance, run a process takes up 25 hours

Hi,

Ususally it happens some times,

Your checks should be,

1]CPU :

Check out the top processes which are using most of the CPU.

#UNIX95= ps -ef -o pid,ruser,pcpu,args|tr -d "%" |awk '{if($4>=90) print $0}'

Also check


#ps -ef |grep -i NO (For oracle process)

Well if you find almost all processes (oracle)are in sleep state since long time,then u can take the decisions to kill the same with proper approvals,


But after looking at vmstat output,cpu is idle and there is hardly utilisation.


2]Memory :

It is most important parameter to check if you are getting very slow response.

in vmstat output, i am looking lots of page out, it means u are going short of memory,and swap utilisation is more.check,
#swapinfo -tam,

Action :

If you see more swap utilisation,then,

run below command,

#ipcs -mob
ex.

#ipcs -mob
IPC status from /dev/kmem as of Tue Aug 25 13:34:26 2009
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411c0189 --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 2 61760
m 2 0x41201f8a --rw-rw-rw- root root 1 8192
m 3 0x00a5c581 --rw------- sfmdb users 4 10469376
m 32772 0x06347849 --rw-rw-rw- root root 2 65544
m 65541 0x0c6629c9 --rw-r----- root root 2 17868736
m 163846 0x491823a2 --rw-r--r-- root root 0 22908

If you see 7th Column (NATTCH )any process is having 0 value, that means thease processes are completed and still they didn't release the memory segment

You can release thorough

#ipcrm -m (Be careful while running this command)


3]Disk utlisation :

#Using iostat command,check any perticular disk is showing any abnormal behavior,


4]Network

On critical server,usually export,back thease activities should do when there will be less load on the server.these can be depend on LAN speed.

Check thease statictics as well,

Hope above points may help u in some manner.

Rgds
Yogesh







Tingli
Esteemed Contributor

Re: slow performance, run a process takes up 25 hours

Only thing I can say so far is that c4t0d1 is fairly busy, although its avque is not that high. Maybe there is one job with a lot of io.
Dennis Handly
Acclaimed Contributor

Re: slow performance, run a process takes up 25 hours

>but now more 25 hours past, it is still running.

The system seems idle. What's still running?
Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

i am exporting data to disk, thus I/O is expected to "c4t0d1".

I can eliminate issues at the network area because the process does not related to network.

Also the "swap" space and memory, it's peak usage is 44%, not reaching the alert level.

Regarding cpu, I can see no backgroup process is high up the usage, but my export process sleep very frequently.
TTr
Honored Contributor

Re: slow performance, run a process takes up 25 hours

The c4t0d1 disk is rather busy. That alone however does not justify the 25 hours. Maybe this disk was just as busy when the export job took only 2 hours. You have to check a few more things.

Have you added any data recently to this database environment?
Are there other processes running that conflict in i/o with the export job?
Are there other processes running that cause locking in the data that would suspend the export job completely until the locking is cleared?
Check the syslog if there are any issues with the c4t0d1 disk. Maybe read or write errors and retries.
Check around in the database for anything else that niormally does not run during the export process.
Grace Li
Frequent Advisor

Re: slow performance, run a process takes up 25 hours

"the system is idle, what is still running" that is the key issue here.

My processes (export data) are suposed to run but they sleep (thus taking long time). they are still hanging there. how to expedite?