weblogic 8.1.4 performance issue on HP_UX 11.11i

[renzhuoren]

We deployed weblogic 8.1.4 on HP_UX 11.11i.
There is very bizarrerie performance issue.
Please see the pic from httpwatch on cliend IE side.
It seems it takes a very long time to wait a 304 response from Weblogic server. But according to the monitor result, Webllogic and HP_UX are very idle, no high CPU used ,no high RAM used,no high queue, and the network is Okay, no packets lost.
Anybody know the reason for this performance issue, please share your experience with us.
Is there some thing wrong with some paremters with HP_UX?

[TTr]

Check if there are name resolution issues on the server. Try resolving some of the names in the attachment within the server.

[renzhuoren]

thanks a lot!
I have checked , no problem with file name.
and This issue happens sometimes, for everytime not on the same JSP entry.
Please also give me some advice on how to monitor the latency time? for example , monitor how hp_ux hanlder with tcp packet. I guess HP_UX get the http request tcp packet from client on time , but put it in one qeune for a very long time , and then pass it to weblogic and weblogic gives the response.

[TTr]

> no problem with the filename?

What exactly did you check? I was refering to host/domain name resolution.

> But according to the monitor result, Webllogic and HP_UX are very idle, no high CPU used ,no high RAM used,no high queue, and the network is Okay, no packets lost.

What did you use to check this?

Is the https connection that you show on the attachment only between the browser and the server? No other servers are involved? Is there an application layer?

Check the weblogic logs on the server for any errors.

Search in this forum for "weblogic", there are many threads that may help you.

[renzhuoren]

Thanks a lot to TTr's support.
I think I misunderstand your suggestion.
Here I will list more information.
in HP_UX server ,we deployed a web application based on Struts framework on weblogic 8.1.4, and we use HTTPS mode.
Now the problem is that End user feel the performance is very poor, but We find the Server Java application runs very fast by java log.
CPU monitor:
10:46:53 %usr %sys %wio %idle
10:46:58 8 3 0 89
10:47:03 9 2 0 89
10:47:08 0 0 0 99
10:47:13 0 0 0 99
10:47:18 7 1 0 91
10:47:23 3 0 0 97
10:47:28 10 1 2 87
10:47:33 13 2 0 85
10:47:38 11 3 1 85
10:47:43 22 5 0 73
10:47:48 23 5 0 72
10:47:53 11 3 0 87
10:47:58 6 1 0 93
10:48:03 3 2 0 95
10:48:08 6 1 1 92
10:48:13 3 2 0 95
10:48:18 2 0 0 97
10:48:23 2 2 0 96
10:48:28 4 1 0 95
10:48:33 10 1 0 89
10:48:38 9 2 0 89
10:48:43 5 1 0 94
10:48:48 3 1 0 96
10:48:53 14 5 0 81
10:48:58 22 6 0 72
10:49:03 20 5 0 75
10:49:08 7 3 0 90
10:49:13 3 1 0 96
10:49:18 3 1 0 96
10:49:23 4 1 0 94
10:49:28 6 2 0 92
10:49:33 8 1 0 91
10:49:38 5 1 0 94
10:49:43 3 1 0 95
10:49:48 5 3 0 92
10:49:53 10 3 0 87
10:49:58 7 2 2 90
10:50:03 8 3 2 87
10:50:08 14 5 0 81
10:50:13 20 6 0 74
10:50:18 19 5 0 76
RAM monitor:
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
15 0 0 1659771 766783 0 0 0 0 0 0 0 2417 8355 5038 80 2 19
25 0 0 1642540 766783 0 0 0 0 0 0 0 2384 8376 4748 78 3 19
25 0 0 1642540 766783 0 0 0 0 0 0 0 2221 6780 4045 80 1 19
15 0 0 1642229 766783 0 0 0 0 0 0 0 2163 6981 4166 79 2 18
15 0 0 1642229 766783 0 0 0 0 0 0 0 1955 6441 3514 80 1 19
27 1 0 1629329 766783 0 0 0 0 0 0 0 1957 4990 3248 85 1 14
27 1 0 1629329 766783 0 0 0 0 0 0 0 1321 2433 1224 25 0 75
2 0 0 1641161 766783 0 0 0 0 0 0 0 1115 1060 503 25 0 75
2 0 0 1641161 766783 0 0 0 0 0 0 0 2032 8602 3537 82 4 14
16 0 0 1671407 766783 0 0 0 0 0 0 0 2014 7632 3695 92 2 6
16 0 0 1671407 766783 2 0 0 0 0 0 0 1857 6520 3218 86 2 12
14 0 0 1677691 766783 0 0 0 0 0 0 0 1794 5367 2680 81 1 18
14 0 0 1677691 766783 0 0 0 0 0 0 0 1627 5855 2749 92 1 7
13 1 0 1649110 766783 0 0 0 0 0 0 0 1653 5414 3036 89 2 8
13 1 0 1649110 766783 0 0 0 0 0 0 0 1432 2992 1879 51 0 49
2 0 0 1643487 766783 0 0 0 0 0 0 0 1142 1264 729 25 0 75
2 0 0 1643487 766791 8 0 0 0 0 0 0 1482 3218 1164 43 1 56
21 0 0 1626786 766791 1 0 0 0 0 0 0 2093 6935 3451 90 3 7
21 0 0 1626786 766791 0 0 0 0 0 0 0 1887 9094 4874 97 3 0
10 1 0 1641415 766791 0 0 0 0 0 0 0 2052 8065 4979 97 3 0
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
10 1 0 1641415 766791 0 0 0 0 0 0 0 2041 7812 4708 98 2 0
8 0 0 1671498 766791 0 0 0 0 0 0 0 1425 4506 2511 91 1 8
8 0 0 1671498 766791 0 0 0 0 0 0 0 1400 4473 2283 99 1 0
2 0 0 1659771 766791 0 0 0 0 0 0 0 1368 3686 1884 75 1 25
2 0 0 1659771 766791 0 0 0 0 0 0 0 1745 5608 2171 12 1 87
1 0 0 1643715 766791 0 0 0 0 0 0 0 1626 3746 1445 3 1 96
1 0 0 1643715 766791 0 0 0 0 0 0 0 1430 2857 1051 4 0 96
1 0 0 1640753 766791 0 0 0 0 0 0 0 1524 2645 916 6 1 93
1 0 0 1640753 766783 0 0 0 0 0 0 0 1606 2558 768 1 0 98
1 0 0 1626678 766783 0 0 0 0 0 0 0 1540 2288 714 2 0 97
1 0 0 1626678 766783 0 0 0 0 0 0 0 1565 3300 1148 7 1 92
1 0 0 1672033 766783 0 0 0 0 0 0 0 1506 3015 1097 4 0 95
1 0 0 1672033 766783 0 0 0 0 0 0 0 1640 2609 827 3 1 96
1 0 0 1672938 766783 0 0 0 0 0 0 0 2078 8421 4005 19 5 76
1 0 0 1672938 766783 1 0 0 0 0 0 0 2084 8796 4359 45 4 50
2 0 0 1680411 766783 0 0 0 0 0 0 0 2037 9571 4805 20 4 76
2 0 0 1680411 766783 0 0 0 0 0 0 0 1692 4815 2159 26 1 73
1 0 0 1649027 766783 0 0 0 0 0 0 0 2434 5099 1390 10 0 90
1 0 0 1649027 766783 0 0 0 0 0 0 0 1833 4184 1485 18 2 80
1 0 0 1640698 766783 0 0 0 0 0 0 0 1577 2594 784 18 0 82
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 0 0 1640698 766783 0 0 0 0 0 0 0 1740 4370 1779 10 2 88
2 0 0 1626623 766783 0 0 0 0 0 0 0 1684 4089 1552 25 1 75
2 0 0 1626623 766754 0 0 0 0 0 0 0 1626 3213 1243 2 1 96
2 0 0 1642946 766783 0 0 0 0 0 0 0 1785 4165 1289 29 1 70
2 0 0 1642946 766783 0 0 0 0 0 0 0 1884 5453 2243 23 2 75
2 0 0 1675719 766783 0 0 0 0 0 0 0 1610 4444 1738 8 1 91
2 0 0 1675719 766783 0 0 0 0 0 0 0 1582 4059 1644 28 2 70
1 0 0 1661286 766783 0 0 0 0 0 0 0 1594 3319 1310 2 1 98
1 0 0 1661286 766783 0 0 0 0 0 0 0 1895 7288 3752 37 4 58
2 0 0 1642901 766783 0 0 0 0 0 0 0 2303 10922 5765 28 5 67
2 0 0 1642901 766783 0 0 0 0 0 0 0 2056 10308 5480 42 4 53
1 0 0 1641059 766783 0 0 0 0 0 0 0 1680 5053 2253 5 1 94
1 0 0 1641059 766783 0 0 0 0 0 0 0 1546 3101 1246 11 0 89
2 0 0 1629699 766783 0 0 0 0 0 0 0 1398 3982 1881 19 1 81
2 0 0 1629699 766754 0 0 0 0 0 0 0 1864 4037 1465 4 0 95
1 0 0 1644030 766783 0 0 0 0 0 0 0 1667 4931 1688 7 2 91
1 0 0 1644030 766783 0 0 0 0 0 0 0 1418 3610 1317 4 1 95
1 0 0 1671407 766783 0 0 0 0 0 0 0 1483 3082 1086 3 1 96
1 0 0 1671407 766783 1 0 0 0 0 0 0 1559 2830 969 10 1 89
1 0 0 1677691 766783 0 0 0 0 0 0 0 1461 3769 1539 2 1 97
Disk monitor
09:06:40 device %busy avque r+w/s blks/s avwait avserv
10:23:50 c0t6d0 4.01 0.50 7 62 0.00 8.81
c3t6d0 2.40 0.50 4 48 0.00 7.98
10:23:55 c0t6d0 1.40 0.50 3 17 0.00 8.45
c3t6d0 1.20 0.50 2 13 0.00 7.88
10:24:00 c0t6d0 52.69 2.87 168 1831 4.42 5.36
c3t6d0 27.94 4.48 93 1271 6.04 5.63
10:24:05 c0t6d0 36.60 0.92 114 1312 1.08 5.61
c3t6d0 21.00 1.20 68 979 1.54 5.31
10:24:10 c0t6d0 3.40 0.50 7 61 0.27 14.10
c3t6d0 2.20 0.50 5 54 0.02 9.71
10:24:15 c0t6d0 0.60 0.50 1 5 0.00 5.50
c3t6d0 0.20 0.50 1 4 0.00 4.86
10:24:20 c0t6d0 4.78 0.50 9 102 0.00 8.58
c3t6d0 2.79 0.50 6 87 0.00 5.96
10:24:25 c0t6d0 3.01 0.50 5 51 0.00 7.42
c3t6d0 1.41 0.50 3 46 0.00 5.27
10:24:30 c0t6d0 12.00 18.82 49 479 33.43 13.61
c3t6d0 9.60 19.17 45 466 28.45 12.18
10:24:35 c0t6d0 17.17 15.25 61 566 28.08 10.46
c3t6d0 11.38 17.62 50 502 29.39 10.45
10:24:40 c0t6d0 2.20 0.50 5 61 0.00 7.84
c3t6d0 1.40 0.50 3 63 0.00 6.26
10:24:45 c0t6d0 78.80 0.96 290 3348 1.01 3.40
c3t6d0 43.40 1.40 141 2113 1.49 3.82
10:24:50 c0t6d0 6.60 0.50 19 252 0.00 4.50

We use httpwatch to monitor performance of Client side.

We use Weblogic performance monitor console to monitor weblogic.

We find when the user feel the performance is poor, just because sometimes his IE want to load some JS or PICTURE object from server(not really load object, just want check the version to get 304 from server), the process is very long.

We also monitor the network by using ehtereal to pick up packets on HP_UX server side.
We find when client IE send Request Packets to Server, and Servers give ACK packet very fast, but after 3-15 seconds,server will send 304 reponse packets to client.

[renzhuoren]

Mostly, Server should reponse very fast, We do not know why it is so long time between ACK and Application Data packet from server to Client.Please see the attachment pic for detail information. This issue happens by accident, sometimes on this web objects sometimes on the other web objects.

[renzhuoren]

some result about network:

1 B_AP_A1#[/]netstat -an | grep tcp | awk '//{print $6}' | sort | uniq -c
17 CLOSE_WAIT
11 CLOSING
194 ESTABLISHED
12 FIN_WAIT_1
37 FIN_WAIT_2
47 LISTEN
150 SYN_RCVD
13 TIME_WAIT

2 B_AP_A1#[/]netstat -p tcp
tcp:
2778345692 packets sent
2609479420 data packets (4014062408 bytes)
13869569 data packets (953309084 bytes) retransmitted
168860266 ack-only packets (36735303 delayed)
0 URG only packets
129724 window probe packets
54155 window update packets
66445124 control packets
2683634063 packets received
2186358972 acks (for 1424237058 bytes)
8826260 duplicate acks ##############
0 acks for unsent data
2098398150 packets (1141444970 bytes) received in-sequence
362 completely duplicate packets (235330 bytes)
19619 packets with some dup, data (22186776 bytes duped)
531224 out of order packets (673032026 bytes)
11498 packets (816344714 bytes) of data after window
456 window probes
24374649 window update packets
59481 packets received after close
24 segments discarded for bad checksum
0 bad TCP segments dropped due to state change
1970031 connection requests
21285384 connection accepts #################
23255415 connections established (including accepts)
27881571 connections closed (including 4626880 drops)
1462436 embryonic connections dropped
2166480205 segments updated rtt (of 2166480205 attempts)
23766534 retransmit timeouts
1796724 connections dropped by rexmit timeout
129724 persist timeouts
2779255 keepalive timeouts
2756757 keepalive probes sent
7185 connections dropped by keepalive
6375 connect requests dropped due to full queue ##############
1116834 connect requests dropped due to no listener

[renzhuoren]

Parameter Value
max_thread_proc 3600
maxfiles 2048
maxusers 2048
nkthread 6000
nproc 2068
ncallout 6000
tcp_conn_request_max 4096