Unable to allocate priority for the running processes

[GSK_2]

Hi guys,

I am facing a problem with changing or allocating different priorities. Can someone help me. Just to relate, here is how our setup is:
we have a DB server which is rx7620 model with 6 CPU's and 16GB of Ram and 20Gb swap space and is housing 7 databases of different sizes. We have some batch processes running for all the databases during night time and at the same time we have backup going on on backup another server to which DB server Filesystems are nfs shared. We use backup express as the backup solution. During night the backup client processes are consuming more CPU causing delay for the processing of other DB batch jobs. These backup client process is running as root owner. We tried to decrease its priority by increasing its nice value, it did not helped. We increased the priority of the batch processes by decreasing its nice value, still two backup client processes are consuming an individual CPU % of 45 each. Can we have any solution for this. We have shifted these batch jobs and as well as backup job to the most ideal time w.r.t to server usage. Is it that changing the priority of backup process only getting the top priority always is because it was running as root user? We have some bottle neck with disk and memory at the moment. Please find some output details which might help you in guiding me.

System: ORACLE Thu Jul 19 04:23:47 2007
Load averages: 0.73, 0.80, 0.94
678 processes: 631 sleeping, 46 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.79 36.2% 3.5% 24.8% 35.4% 0.0% 0.0% 0.0% 0.0%
1 0.63 16.5% 1.4% 31.1% 51.0% 0.0% 0.0% 0.0% 0.0%
2 0.69 35.2% 1.6% 16.1% 47.0% 0.0% 0.0% 0.0% 0.0%
3 0.80 25.2% 2.0% 25.2% 47.5% 0.0% 0.0% 0.0% 0.0%
4 0.72 23.9% 2.6% 18.1% 55.4% 0.0% 0.0% 0.0% 0.0%
5 0.73 21.3% 3.7% 22.3% 52.7% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.73 26.3% 2.6% 23.0% 48.1% 0.0% 0.0% 0.0% 0.0%

Memory: 10761728K (4760380K) real, 14642180K (6858956K) virtual, 1031424K free Page# 1/16

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
4 pts/2 717 root 255 35 17088K 5224K run 230:18 51.91 51.82 dmbs
2 pts/2 680 root 255 35 13264K 1464K run 154:00 35.68 35.62 dmbs
1 pts/2 162 root 154 20 12816K 1784K sleep 72:14 19.26 19.22 evthnd
5 ? 9993 oracle 216 20 2393M 25608K run 13:14 17.88 17.85 oracleamerp
4 ? 10044 oracle 149 20 2393M 24200K sleep 12:48 17.31 17.28 oracleamerp
5 ? 20320 oracle 148 5 2379M 9784K sleep 86:04 8.28 8.26 oracletrdamr
5 ? 26791 oracle 154 20 1774M 6548K sleep 0:35 7.57 7.55 oraclehp101b
4 ? 20318 oracle 149 5 2379M 10476K sleep 73:49 7.46 7.44 oracletrdamr
4 ? 20334 oracle 149 5 2371M 6000K sleep 73:09 7.05 7.04 oracletrdamr
2 ? 10080 oracle 148 20 2380M 15032K sleep 6:15 6.91 6.89 oracleamerp
4 ? 24321 oracle 154 20 1774M 7440K sleep 0:55 6.27 6.26 oraclehp101b
0 ? 20337 oracle 149 5 2371M 6256K run 85:32 5.63 5.62 oracletrdamr
3 ? 20291 oracle 148 5 2371M 5984K sleep 73:08 5.35 5.34 oracletrdamr
3 ? 20309 oracle 149 5 2371M 5928K sleep 73:37 5.29 5.28 oracletrdamr
4 ? 20324 oracle 148 5 2371M 5952K sleep 74:01 5.21 5.20 oracletrdamr
5 ? 20272 oracle 154 20 816M 5712K sleep 31:19 5.01 5.00 oracleSTN920
4 ? 9435 oracle 148 5 2371M 6008K sleep 3:19 4.45 4.44 oracletrdamr
5 ? 20331 oracle 148 5 2379M 10728K sleep 85:27 4.19 4.18 oracletrdamr
2 ? 24326 oracle 204 20 1773M 6420K run 0:19 4.06 4.05 oraclehp101b
0 ? 3059 oracle 154 20 815M 5452K sleep 12:46 3.96 3.95 oracleSTN920
4 ? 20314 oracle 148 5 2379M 11060K sleep 73:55 3.95 3.94 oracletrdamr
4 ? 20301 oracle 148 5 2379M 10488K sleep 73:58 3.63 3.62 oracletrdamr
3 ? 1882 root 152 20 239M 108M run 1321:23 3.41 3.40 opcle
2 ? 1528 root -16 20 40820K 16860K run 1697:56 3.30 3.30 midaemon

GlancePlus C.03.86.00 03:32:28 oradb ia64 Current Avg High
------------------------------------------------------------------------------------------------------------------------------------------------------
Cpu Util S SRU U | 97% 94% 97%
Disk Util F F |100% 100% 100%
Mem Util S SU UB B | 94% 94% 95%
Networkil U UR R | 73% 73% 73%
------------------------------------------------------------------------------------------------------------------------------------------------------
MEMORY REPORT Users= 5
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 14164 18117 1748.6 933.8 1748.6
Page In 7226 8351 892.0 430.4 892.0
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 4kb 0.0 0.2 0.8
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 1 0.0 0.0 0.1
VM Writes 8 14 0.9 0.7 0.9

Total VM : 14.1gb Sys Mem : 2.8gb User Mem: 11.4gb Phys Mem: 15.9gb
Active VM: 7.9gb Buf Cache: 812mb Free Mem: 904mb

Thanks,
Satish

[Ralf Seefeldt · Accepted solution]

Hello Satish,

The Problem is, that the priority increaces over the time. by default, it will be increaced any 40 mseconds until it gets to high. Then the process will start sleeping and another process starts running. During sleeping the priority will be decreaced constantly.

I can see some ways of optimisation (assuming, you have an CPU bottleneck and not one with RAM, disk, LAN or sth else):
- spread the nicevalues even faster. The nice-value should have some impact on the speed, the priority is in- and decreaced (I think)

- When a process goes to sleep, that consumes time. This is controled by the kernelparameter timeslice. It has an optimal value of 10 in nearly any situation. SAM's OLTP/Database server reduces this value to 1.

ncsice is often set to 33000 by ORACLE. a value between 2048 and 8192 should be sufficient, even for oracle. (may be, this is only true for 1 database instance)

- use the Process-Ressource-Manager (I think, it is not free)

- by default, any process will be governd by the SCHED_HPUX scheduling system. There are others available: HP-UX realtime (SCHED_RTPRIO) and POSIX REALTIME . The latter offers the fastest priorities. You will find details at: http://techsolutions.hp.com/en/5964-5283/ch05s22.html

I hope, this is at least some starting point for you.

Bye
Ralf

[Steven Schweda]

I know nothing, but ...

> We have some bottle neck with disk and
> memory at the moment.

If the data base stuff is I/O-limited and not
CPU-limited, then reducing the CPU priority
of the backup jobs may not matter much. If
there is always free CPU time, even a
low-priority job can get a lot of the CPU.

Backup to what? Tape? Another disk?

How are the disks (and tapes?) connected to
the system? One big SCSI bus?

[GSK_2]

Thanks for the inputs Ralf.

Steven, How can we find that the DB stuff is I/O limited or CPU limited? Is there any KTP or any configuraion file?
We were taking it into tape. Disks are comming from HP EVA storage and tape is over the network. Would running the backup with another user other than root prevent from taking high priority? Any suggestions are welcome. Thanks in advance.

Regards,
Satish