Some recommendation to improve the perform?

[Miguel Hernández]

Hello to all:

I have a rx2600 running two oracle instances and a lot of Java services. The machine is very satured (CPU, memory, disk).

ds01:#model
ia64 hp server rx2600
ds01:#uname -a
HP-UX ds01 B.11.23 U ia64 2940943301 unlimited-user license
ds01:#kctune -q dbc_max_pct
Tunable Value Expression Changes
dbc_max_pct 10 10 Immed
ds01:#kctune -q dbc_min_pct
Tunable Value Expression Changes
dbc_min_pct 5 Default Immed
ds01:#swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 62 4034 2% 0 - 1 /dev/vg00/lvol2
dev 4096 61 4035 1% 0 - 1 /dev/vg00/swap
reserve - 5870 -5870
memory 8181 2432 5749 30%
total 16373 8425 7948 51% - 0 -


Can somebody give me any idea to improve the performance?

Thanks

[Ludovic Derlyn]

hi,

there are few informations
what is the result of sar -d for example
and what are values for others parameters in kernel

what are the options for mountpoint for oracle ?
for database, you can specify mincache=direct for example...

Regards
L-DERLYN

[RAC_1]

First try to understand what kind of bottlenek do you see on your system? CPU, DISK, MEMEORY, SWAP, NETWORK or combination of these? run glance and check cpu,mem and didk utils. Check how efficiently you are using your buffer cache. glance -m will help and also sar -b 1 5

Find disk botlnecks - glance -i and sar -d, check disk wait i/os.

[Steven E. Protter]

Shalom,

I suggest identifying the problem prior to trying to solve it.

http://www.hpux.ws/system.perf.sh

SEP

[Miguel Hernández]

Alarms on Glance:

09:14:27 RED CPU Bottleneck probability= 100.00%
09:14:27 YELLOW Disk Bottleneck probability= 60.79%
09:15:03 END End of Disk Bottleneck Alert
09:15:28 END End of CPU Bottleneck Alert
09:16:26 RED Disk Bottleneck probability= 100.00%
09:22:01 YELLOW Disk Bottleneck probability= 78.21%
09:28:27 END End of Disk Bottleneck Alert
09:30:28 RED Disk Bottleneck probability= 100.00%
09:31:59 END End of Disk Bottleneck Alert
09:38:27 YELLOW Disk Bottleneck probability= 83.89%
09:39:38 YELLOW Disk Bottleneck probability= 70.91%
09:40:33 END End of Disk Bottleneck Alert
09:41:39 RED CPU Bottleneck probability= 100.00%
09:44:00 END End of CPU Bottleneck Alert
09:48:22 YELLOW Disk Bottleneck probability= 72.41%
09:52:30 END End of Disk Bottleneck Alert
09:54:41 YELLOW Disk Bottleneck probability= 72.74%
09:59:28 END End of Disk Bottleneck Alert
10:01:29 RED Disk Bottleneck probability= 100.00%
10:01:54 YELLOW Disk Bottleneck probability= 62.42%
10:03:40 END End of Disk Bottleneck Alert
10:06:37 RED CPU Bottleneck probability= 100.00%
10:07:22 END End of CPU Bottleneck Alert
10:08:53 YELLOW Disk Bottleneck probability= 77.50%
10:12:04 END End of Disk Bottleneck Alert
10:12:15 RED CPU Bottleneck probability= 100.00%
10:14:21 END End of CPU Bottleneck Alert
10:18:53 RED CPU Bottleneck probability= 100.00%
10:20:39 END End of CPU Bottleneck Alert


The average use of memory is 96%

sar -d

HP-UX ds01 B.11.23 U ia64 06/29/06

00:00:05 device %busy avque r+w/s blks/s avwait avserv
01:00:00 c2t1d0 2.12 0.56 4 60 0.25 7.68
c2t0d0 1.49 0.59 3 38 0.35 8.55
c6t5d1 46.20 0.94 88 15813 1.85 22.90
c6t5d2 49.40 1.19 102 21592 2.83 23.53
c6t5d3 56.42 4.65 150 26033 11.61 20.66
c6t5d4 59.91 1.53 96 4345 1.33 14.24
02:00:00 c2t1d0 1.61 0.53 3 34 0.14 9.06
c2t0d0 1.16 0.53 2 29 0.17 9.19
c6t5d1 12.41 0.50 27 641 0.03 5.80
c6t5d2 10.93 0.50 24 556 0.00 4.79
c6t5d3 27.50 0.50 48 789 0.00 5.96
c6t5d4 81.72 2.08 93 1533 2.02 18.08
03:00:00 c2t1d0 1.61 0.56 3 35 0.29 9.23
c2t0d0 1.15 0.59 2 30 0.36 9.36
c6t5d1 3.44 0.50 32 809 0.00 1.19
c6t5d2 2.71 0.50 27 733 0.00 1.10
c6t5d3 12.69 0.50 62 1005 0.01 2.16
c6t5d4 65.78 1.45 83 1368 0.46 12.81
04:00:00 c2t1d0 1.63 0.61 3 47 0.43 8.55
c2t0d0 1.14 0.66 2 31 0.73 9.42
c6t5d1 1.78 0.50 29 746 0.00 0.69
c6t5d2 1.34 0.50 24 674 0.00 0.62
c6t5d3 7.25 0.51 53 858 0.02 1.48
c6t5d4 60.34 1.21 86 1414 0.26 10.41
05:00:00 c2t1d0 1.86 0.70 3 51 0.56 9.97
c2t0d0 1.33 0.71 2 35 0.74 11.01
c6t5d1 0.50 0.50 7 183 0.00 0.86
c6t5d2 0.41 0.50 7 177 0.01 0.85
c6t5d3 2.24 0.50 14 236 0.01 1.60
c6t5d4 45.81 1.13 63 1051 0.22 9.83
06:00:00 c2t1d0 1.76 0.60 3 51 0.47 8.79
c2t0d0 1.23 0.63 2 35 0.66 9.66
c6t5d1 0.19 0.50 3 61 0.01 0.78
c6t5d2 0.20 0.50 3 68 0.00 0.66
c6t5d3 1.51 0.50 8 119 0.00 2.07
c6t5d4 43.82 0.76 61 1008 0.11 9.41
07:00:01 c2t1d0 1.73 0.59 3 41 0.45 9.33
c2t0d0 1.24 0.62 2 33 0.54 9.70
c6t5d1 15.04 0.57 324 7186 0.06 0.84
c6t5d2 12.59 0.50 313 5227 0.00 0.43
c6t5d3 7.92 0.51 28 461 0.02 3.03
c6t5d4 53.96 0.96 79 1344 0.20 10.17
08:00:00 c2t1d0 1.82 0.67 3 55 0.72 8.60
c2t0d0 1.26 0.75 2 35 1.00 9.59
c6t5d1 1.27 0.50 19 428 0.00 0.72
c6t5d2 1.15 0.50 17 392 0.00 0.74
c6t5d3 5.04 0.50 38 610 0.01 1.39
c6t5d4 66.02 1.09 126 2151 0.19 8.71
08:20:00 c2t1d0 2.15 0.77 4 68 0.98 9.25
c2t0d0 1.50 0.93 3 51 1.57 9.84
c6t5d1 3.52 0.90 28 1422 0.41 2.99
c6t5d2 2.56 0.50 21 758 0.01 2.04
c6t5d3 7.34 0.50 58 932 0.01 1.38
c6t5d4 62.43 1.40 122 2109 0.33 8.35
08:40:00 c2t1d0 2.50 0.61 5 58 0.43 9.54
c2t0d0 1.71 0.65 3 47 0.65 9.86
c6t5d1 11.98 1.17 106 6524 0.77 3.97
c6t5d2 8.41 0.52 75 2761 0.05 2.41
c6t5d3 14.11 0.50 103 2238 0.00 1.54
c6t5d4 80.96 5.35 266 4999 3.43 8.21
09:00:01 c2t1d0 2.84 0.78 7 175 1.01 8.37
c2t0d0 1.89 0.91 4 144 1.28 9.11
c6t5d1 2.73 0.50 37 817 0.00 0.80
c6t5d2 2.35 0.50 33 760 0.00 0.76
c6t5d3 12.02 0.51 52 843 0.02 2.51
c6t5d4 72.70 2.66 160 2737 0.98 8.83
09:20:00 c2t1d0 2.71 0.75 5 92 0.88 9.12
c2t0d0 1.89 0.86 3 64 1.24 9.79
c6t5d1 2.50 0.50 34 746 0.00 0.80
c6t5d2 2.11 0.50 30 691 0.00 0.77
c6t5d3 10.65 0.50 71 1141 0.01 1.61
c6t5d4 73.48 2.12 153 2642 0.72 9.46
09:40:02 c2t1d0 3.61 1.85 7 94 4.94 12.56
c2t0d0 2.61 2.22 5 82 6.06 13.29
c6t5d1 23.37 0.52 62 9318 0.03 11.25
c6t5d2 4.85 0.50 29 647 0.02 1.87
c6t5d3 12.50 0.50 62 995 0.01 2.15
c6t5d4 68.40 3.23 143 2512 1.18 8.71
10:00:01 c2t1d0 4.24 2.47 9 124 6.92 13.42
c2t0d0 3.13 2.81 7 109 8.29 14.07
c6t5d1 15.07 0.50 44 6856 0.01 9.52
c6t5d2 1.78 0.50 20 484 0.00 0.98
c6t5d3 6.45 0.50 47 757 0.01 1.42
c6t5d4 63.75 3.36 130 2210 1.12 8.58
10:20:00 c2t1d0 5.44 2.19 14 348 5.79 10.63
c2t0d0 4.05 2.70 10 258 7.52 11.87
c6t5d1 1.82 0.50 21 494 0.00 0.93
c6t5d2 1.75 0.50 19 468 0.00 1.00
c6t5d3 5.61 0.50 46 741 0.01 1.25
c6t5d4 63.66 3.73 122 2078 1.17 9.09
10:40:01 c2t1d0 3.81 2.01 8 110 5.45 13.57
c2t0d0 2.87 2.40 6 93 7.40 14.75
c6t5d1 2.42 0.61 24 834 0.12 1.77
c6t5d2 1.89 0.50 21 545 0.00 1.03
c6t5d3 5.68 0.50 47 757 0.00 1.24
c6t5d4 63.47 3.58 128 2244 1.17 8.62
11:00:00 c2t1d0 3.58 1.88 7 97 4.86 13.11
c2t0d0 2.67 2.30 5 83 6.62 14.25
c6t5d1 1.81 0.50 20 471 0.00 0.94
c6t5d2 1.64 0.50 18 439 0.00 0.94
c6t5d3 5.58 0.50 48 765 0.00 1.22
c6t5d4 65.92 3.09 126 2152 0.95 8.86
11:20:00 c2t1d0 3.48 1.98 7 89 5.53 13.57
c2t0d0 2.65 2.39 5 80 7.44 15.21
c6t5d1 1.77 0.50 19 447 0.00 0.99
c6t5d2 1.66 0.50 17 418 0.00 1.02
c6t5d3 5.57 0.50 50 795 0.00 1.16
c6t5d4 62.86 3.44 122 2091 1.04 8.52
11:40:00 c2t1d0 3.38 1.87 7 120 5.02 11.69
c2t0d0 2.46 2.47 5 80 7.69 14.53
c6t5d1 1.81 0.50 18 434 0.01 1.09
c6t5d2 1.64 0.50 16 403 0.01 1.15
c6t5d3 5.86 0.51 48 779 0.02 1.31
c6t5d4 67.48 2.85 133 2294 0.96 9.15
12:00:03 c2t1d0 5.43 1.90 14 373 4.71 9.37
c2t0d0 3.96 2.65 9 249 7.48 11.70
c6t5d1 1.51 0.50 17 400 0.00 0.93
c6t5d2 1.33 0.50 16 374 0.00 0.93
c6t5d3 4.83 0.50 43 684 0.00 1.16
c6t5d4 57.53 3.08 135 2377 0.79 7.26
12:20:01 c2t1d0 4.46 2.18 10 172 5.69 11.38
c2t0d0 3.47 2.76 7 136 7.94 13.12
c6t5d1 1.63 0.50 21 476 0.00 0.81
c6t5d2 1.49 0.50 20 452 0.00 0.85
c6t5d3 4.94 0.50 45 725 0.00 1.13
c6t5d4 82.35 1.76 275 4519 0.41 5.84
12:40:01 c2t1d0 5.12 1.64 11 166 4.03 10.53
c2t0d0 4.17 2.16 8 137 6.21 11.94
c6t5d1 1.57 0.50 24 525 0.00 0.69
c6t5d2 1.48 0.50 23 500 0.00 0.73
c6t5d3 6.04 0.50 40 652 0.01 1.58
c6t5d4 76.46 3.21 228 3594 1.00 6.99
13:00:01 c2t1d0 4.26 1.13 11 219 2.34 9.93
c2t0d0 3.35 1.41 8 173 3.73 11.87
c6t5d1 2.04 0.50 31 684 0.00 0.74
c6t5d2 1.83 0.50 29 646 0.00 0.73
c6t5d3 8.18 2.00 51 830 0.48 1.78
c6t5d4 71.24 7.51 202 3344 2.50 7.36
13:20:00 c2t1d0 3.30 1.67 7 90 4.36 13.19
c2t0d0 2.69 2.40 5 79 7.94 16.05
c6t5d1 3.37 0.68 37 1422 0.18 2.23
c6t5d2 2.39 0.51 31 853 0.02 1.33
c6t5d3 10.84 0.50 60 969 0.01 1.96
c6t5d4 69.04 5.78 182 3058 1.76 6.95
13:40:00 c2t1d0 3.31 1.12 6 85 2.63 12.87
c2t0d0 2.51 1.47 5 72 4.40 15.10
c6t5d1 7.53 1.19 74 5737 0.75 3.67
c6t5d2 4.78 0.50 48 2096 0.02 1.85
c6t5d3 13.75 0.50 144 2303 0.00 1.18
c6t5d4 66.92 4.90 168 2833 1.56 7.30

Average c2t1d0 2.59 1.31 5 88 2.88 10.40
Average c2t0d0 1.89 1.65 4 67 4.27 11.75
Average c6t5d1 8.02 0.65 54 2809 0.33 4.45
Average c6t5d2 6.83 0.61 49 2475 0.43 4.36
Average c6t5d3 12.23 1.38 54 2613 2.37 5.79
Average c6t5d4 63.43 2.69 118 2205 1.04 9.57


Kernel parameters:
ds01:#kctune -q maxtsiz
Tunable Value Expression Changes
maxtsiz 134217728 134217728 Immed
ds01:#kctune -q maxssiz
Tunable Value Expression Changes
maxssiz 134217728 134217728 Immed
ds01:#kctune -q maxdsiz
Tunable Value Expression Changes
maxdsiz 1073741824 Default Immed
ds01:#kctune -q nproc
Tunable Value Expression Changes
nproc 4200 Default Immed
ds01:#kctune -q create_fastlinks
Tunable Value Expression
create_fastlinks 0 Default
ds01:#kctune -q fs_async
Tunable Value Expression
fs_async 0 Default
ds01:#kctune -q maxfiles
Tunable Value Expression
maxfiles 2048 2048
ds01:#kctune -q maxfiles_lim
Tunable Value Expression Changes
maxfiles_lim 4096 Default Immed


I believe that those are the most important.

Thanks

[Ludovic Derlyn]

hi,

download and try SEP script, it's very complete for debugging

regards

L-DERLYN

[Miguel Hernández]

See attached file with the result of Steve's script.

I see this mensaje on dmesg:

msgcnt 16609 vxfs: mesg 001: vx_nospace - /dev/ceessdg7/vol29 file system full (1 block extent)

What's the meaning?? I made bdf and vol29 is not full!

Thank you very much

[Miguel Hernández]

I have watched the error of the last message and it was solving by a companion.

Today the machine goes at 100% of CPU:

ds01:#sar -u 1 10

HP-UX ds01 B.11.23 U ia64 06/30/06

12:30:49 %usr %sys %wio %idle
12:30:50 82 18 0 0
12:30:51 81 19 0 0
12:30:52 81 19 0 0
12:30:53 84 16 0 0
12:30:54 85 15 0 0
12:30:55 84 16 0 0
12:30:56 81 19 0 0
12:30:57 85 15 0 0
12:30:58 87 13 0 0
12:30:59 90 10 0 0

Average 84 16 0 0

would improve something change fs_async to 1?

any idea?

[Chan 007]

Hi,

I guess you need this patch PHKL_33694.

Check this one and install it. Also keep your system up to date with QPK & Patch levels..!!!

Regards
Chan

[Ludovic Derlyn]

Hi Miquel

in your case , it's difficult to etablish a diag

it's necessary to identifies bottlenecks

for cpu bootlenecks, check if wait queue is important
for memory , see high cpu utilisation, high disk utilisation or high memory swapping with swapping (ypur case)
for disk, see high cpu utilisation, high disk utilisation

For disk , avwait and averv must be equal to 20 ms , in your case what is the value ?
if you have database , you can specify in mount option mincache=direct (delaylog in your case ?)

if you set fs_async=1 , speed must be faster, but you increase risk of file system corruption in cases of crash

In your case, have you trying to increase swap if there are not bootlenecks on disks ?
I/O are distributed or not ? (see with glance if some filesystem or disks have many I/O or load)

Else, see if PRM is applicable for increase:decrease priority of some applications

regards
L-DERLYN

[Ludovic Derlyn]

hi again,

An sugestion again, which not corrected directly your problem,

define a block size ,

8K recommended for large file database

large block size for sequential I/O application

small block size for random I/O application

regards
L-DERLYN

[Miguel Hernández]

Chan:
I will install the patch next week. Thank you very much.

Ludovic, thank you very much to answer:
Look this:

ds01:#sar -q 1 10

HP-UX ds01 B.11.23 U ia64 06/30/06

14:56:55 runq-sz %runocc swpq-sz %swpocc
14:56:56 8.5 99 0.0 0
14:56:57 3.0 101 0.0 0
14:56:58 3.0 99 0.0 0
14:56:59 5.0 99 0.0 0
14:57:00 3.0 100 0.0 0
14:57:01 4.0 100 0.0 0
14:57:02 8.5 101 0.0 0
14:57:03 10.5 100 0.0 0
14:57:04 4.0 99 0.0 0
14:57:05 4.0 101 0.0 0

Average 5.3 100 0.0 0


Of sar -d 1 10 I only write the averages:

ds01:#sar -d 1 10

HP-UX ds01 B.11.23 U ia64 06/30/06

15:01:54 device %busy avque r+w/s blks/s avwait avserv
Average c2t1d0 2.29 0.50 4 41 0.00 5.76
Average c2t0d0 2.09 0.50 4 38 0.00 6.17
Average c6t5d1 1.79 0.50 25 577 0.00 0.74
Average c6t5d2 1.40 0.50 22 531 0.00 0.65
Average c6t5d3 8.97 0.50 71 1176 0.00 1.60
Average c6t5d4 59.42 0.52 110 1910 0.01 8.87


Also we are going to prove to mount option mincache=direct

This machine only have one fc card and we can't put another (the SAN switches are full).

And about fs_async... is a machine test. I would do it, but... if my boss allows it!!!


Thank you very very much

[Ludovic Derlyn]

Hi

Number of jobs in queue for cpu for best performance are 3 maximal... for you 5.3

ideal is to have a second Fc for load balancing IO by controller ...

You can try to add a swap more important

regards

L-DERLYN