server is very slow few days aftetr reboot

[chandana_1]

User complains the server getting slower and slower within few days after a reboot. Normal proviusioning takes less than 5 mins. but when the problem occurs it will take around 20 mins.

sar output showed both mirrored disks are busy and CPUs also busy

11:44:19 device %busy avque r+w/s blks/s avwait avserv
Average c2t0d0 90.53 1.78 413 4805 0.45 4.82
Average c2t1d0 77.32 3.12 256 3764 8.03 10.80

11:32:25 %usr %sys %wio %idle

Average 17 17 59 6

The server is Rp34440 with 2 CPUs and 4G memory.

Can any expert advice me on how to check for any hardware failure or how to solve this slow issue

Thanks in advance
chandana

[SoorajCleris]

Hi Chandana,

The disk c2t1d0 got avwait.

Run the same for little more time and past the output.

Could you please paste the output of swapinfo -tam


Regards,
Sooraj

[Matti_Kurkela]

Try restarting the application without rebooting the server. If that restores the response time to < 5 mins, the problem is in the application.

Such a behaviour is a common symptom of a memory or other resource leak, i.e. the application allocates some resource, then somehow "forgets about it" and allocates a new resource instead of reusing the old one.

In this situation, the old resource is still allocated to the application, but the application does not really use it any more.

When the application is stopped, the OS clears away all the "forgotten" allocations made by the application, and the resources are again available for use. When the application is restarted, it will again work fast, until the amount of "forgotten" allocations again becomes too big.

Such resource leaks are *always* a software bug: they should never be accepted in production software.

If the application restart does not help, but rebooting the system does, it might be because the application generates huge numbers of temporary files in /tmp or /var/tmp, and your server is configured to clean those directories at every reboot.

MK

[chandana_1]

Thanks a lot for the few sugesstions. I'll try and reply with the results.
Chandana

[Emil Velez]

check if you are low on memory after a few days.

Your buffer cache could be set too big

DBC_MAX make it slower

if it is 11.31 consider the filecache_max

..

[Kapil Jha]

Just restart the application or if u pretty low on memory part reboot the server again.

Make sure if some setting was changed recently.

BR,
Kapil+

[chandana_1]

Hi Sooraj,

Here the sar and vmstat outputs;

11:28:35 device %busy avque r+w/s blks/s avwait avserv
11:28:37 c2t0d0 98.00 0.52 346 9084 0.02 6.62
c2t1d0 97.00 0.61 240 7344 0.66 15.55
11:28:39 c2t0d0 91.04 0.52 327 4494 0.01 4.42
c2t1d0 81.09 0.53 232 3823 0.14 8.51
11:28:41 c2t0d0 99.50 0.98 341 5758 0.48 7.43
c2t1d0 93.97 1.25 234 4598 3.14 12.18
11:28:43 c2t0d0 98.01 0.65 324 4917 0.13 7.65
c2t1d0 96.02 0.82 244 4519 1.31 13.62
11:28:45 c2t0d0 100.00 0.59 343 8567 0.06 9.84
c2t1d0 100.00 0.71 232 7539 0.97 17.06
11:28:47 c2t0d0 99.00 0.63 368 10686 0.07 8.03
c2t1d0 99.50 0.79 217 9228 1.24 22.04
11:28:49 c2t0d0 98.49 0.87 499 9279 0.41 6.94
c2t1d0 100.00 2.03 243 7085 5.73 18.67
11:28:51 c2t0d0 95.52 0.51 557 11106 0.03 4.36
c2t1d0 98.51 0.82 217 8354 2.13 27.01
11:28:53 c2t0d0 99.50 0.58 682 10087 0.05 4.84
c2t1d0 100.00 1.12 234 5848 2.75 20.94
11:28:55 c2t0d0 96.00 1.04 342 6267 0.32 5.42
c2t1d0 88.50 1.34 236 5599 2.92 9.19
11:28:57 c2t0d0 96.50 0.70 288 3452 0.13 7.12
c2t1d0 88.50 0.72 238 3194 1.01 9.15
11:28:59 c2t0d0 97.01 0.81 279 4094 0.29 7.42
c2t1d0 85.07 0.77 255 4067 1.09 9.79
11:29:01 c2t0d0 100.00 0.68 345 8374 0.19 7.83
c2t1d0 96.98 0.85 285 7715 1.73 16.26
11:29:03 c2t0d0 100.00 0.64 346 7198 0.14 9.13
c2t1d0 100.00 1.37 229 6116 3.79 23.48
11:29:05 c2t0d0 99.50 0.52 435 8942 0.01 5.32
c2t1d0 98.50 0.53 258 7400 0.23 15.95
11:29:07 c2t0d0 96.50 0.82 792 12741 0.45 2.79
c2t1d0 97.50 1.41 336 8903 3.75 11.06
11:29:09 c2t0d0 100.00 0.50 505 9656 0.01 5.11
c2t1d0 96.02 0.66 387 8511 1.01 10.69
11:29:11 c2t0d0 100.00 0.52 809 15757 0.04 3.82
c2t1d0 100.00 1.38 228 10519 5.48 31.10
11:29:13 c2t0d0 99.00 0.82 425 10430 0.85 8.14
c2t1d0 99.50 1.70 225 8139 6.18 27.56
11:29:15 c2t0d0 100.00 1.74 343 8981 2.25 11.67
c2t1d0 100.00 2.31 239 7835 7.50 22.33
11:29:17 c2t0d0 96.50 0.66 292 4451 0.07 7.05
c2t1d0 92.50 0.83 265 4331 1.44 9.43
11:29:19 c2t0d0 97.50 0.56 358 4222 0.04 5.60
c2t1d0 90.50 0.56 360 4422 0.31 5.91
11:29:21 c2t0d0 98.00 0.55 379 3924 0.05 6.37
c2t1d0 95.00 0.57 322 3664 0.34 8.42
11:29:23 c2t0d0 100.00 0.57 524 8159 0.06 6.77
c2t1d0 98.00 0.79 243 5618 1.54 17.80
11:29:25 c2t0d0 98.01 0.52 454 7120 0.02 6.39
c2t1d0 93.53 0.57 270 5449 0.40 12.49
11:29:27 c2t0d0 100.00 0.60 506 9785 0.03 7.84
c2t1d0 100.00 1.29 219 7291 2.89 24.63
11:29:29 c2t0d0 99.50 0.58 570 10833 0.11 7.08
c2t1d0 99.50 2.96 227 8208 14.05 34.42
11:29:31 c2t0d0 100.00 0.69 470 10672 0.17 8.46
c2t1d0 99.50 2.43 224 8823 9.53 29.85
11:29:33 c2t0d0 99.00 1.64 355 7482 2.15 9.24
c2t1d0 99.00 2.14 265 6558 6.99 16.16
11:29:35 c2t0d0 100.00 0.51 313 4473 0.03 8.15
c2t1d0 97.98 0.51 246 3934 0.07 10.47

Average c2t0d0 98.45 0.70 430 8032 0.26 6.54
Average c2t1d0 96.15 1.11 255 6487 2.85 16.35


11:31:10 runq-sz %runocc swpq-sz %swpocc
11:31:12 4.0 74 0.0 0
11:31:14 9.7 75 0.0 0
11:31:16 3.2 100 0.0 0
11:31:18 3.5 101 0.0 0
11:31:20 4.2 100 0.0 0
11:31:22 10.2 100 0.0 0
11:31:24 6.2 101 0.0 0
11:31:26 7.5 100 0.0 0
11:31:28 4.5 100 0.0 0
11:31:30 4.5 100 0.0 0
11:31:32 5.5 100 0.0 0
11:31:34 10.0 100 0.0 0
11:31:36 4.0 75 0.0 0
11:31:38 1.7 75 0.0 0
11:31:40 3.8 101 0.0 0
11:31:42 3.0 100 0.0 0
11:31:44 1.5 50 0.0 0
11:31:46 4.2 100 0.0 0
11:31:48 2.5 50 0.0 0
11:31:50 1.3 75 0.0 0
11:31:52 6.3 75 0.0 0
11:31:54 3.0 75 0.0 0
11:31:56 2.7 75 0.0 0
11:31:58 3.3 75 0.0 0
11:32:00 2.3 75 0.0 0
11:32:02 4.0 100 0.0 0
11:32:04 2.2 100 0.0 0
11:32:06 1.5 100 0.0 0
11:32:08 7.8 100 0.0 0
11:32:10 11.0 50 0.0 0

Average 4.7 87 0.0 0


11:37:09 swpin/s bswin/s swpot/s bswot/s pswch/s
11:37:11 2.48 0.0 0.00 0.0 2552
11:37:13 0.00 0.0 0.00 0.0 2762
11:37:15 2.00 0.0 0.00 0.0 2459
11:37:17 0.00 0.0 0.00 0.0 2368
11:37:19 0.00 0.0 0.00 0.0 2430
11:37:21 2.50 0.0 0.00 0.0 2553
11:37:23 0.00 0.0 0.00 0.0 2387
11:37:25 2.50 0.0 0.00 0.0 2067
11:37:27 0.00 0.0 0.00 0.0 2075
11:37:29 0.00 0.0 0.00 0.0 2353
11:37:31 2.50 0.0 0.00 0.0 2247
11:37:33 0.00 0.0 0.00 0.0 2480
11:37:35 2.49 0.0 0.00 0.0 2452
11:37:37 0.00 0.0 0.00 0.0 2212
11:37:39 0.00 0.0 0.00 0.0 2462
11:37:41 2.51 0.0 0.00 0.0 2350
11:37:43 0.00 0.0 0.00 0.0 2004
11:37:45 2.50 0.0 0.00 0.0 2788
11:37:47 0.00 0.0 0.00 0.0 2387
11:37:49 0.00 0.0 0.00 0.0 2221
11:37:51 2.49 0.0 0.00 0.0 2698
11:37:53 0.00 1.0 0.00 0.0 2062
11:37:55 2.50 2.0 0.00 0.0 2632
11:37:57 0.00 2.0 0.00 0.0 2753
11:37:59 0.00 2.0 0.00 0.0 2429
11:38:01 2.49 3.5 0.00 0.0 2335
11:38:03 0.00 2.5 0.00 0.0 2467
11:38:05 2.50 3.5 0.00 0.0 2501
11:38:07 0.00 2.0 0.00 0.0 2858
11:38:09 0.00 2.5 0.00 0.0 2441

Average 0.98 0.7 0.00 0.0 2426

procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
4 3 0 1919642 23641 1202 200 117 4 3 0 91 1007 59870 1850 24 15 61
4 3 0 1919642 23229 945 161 69 0 0 0 0 1501 62337 1615 15 11 74
4 9 0 1958611 22737 674 112 88 0 0 0 4 1499 55262 1519 22 10 68
4 9 0 1958611 22346 1084 178 214 0 0 0 0 1612 63299 1764 19 19 62
1 21 0 1964665 21754 701 115 123 0 0 0 4 1006 31268 1135 4 6 90
1 21 0 1964665 21472 597 102 82 0 0 0 0 764 26417 1016 6 9 85
2 8 0 1889675 21915 1596 240 199 0 0 0 3 1105 57772 1740 20 19 61
2 8 0 1889675 21686 891 136 109 0 0 0 0 1162 56420 1635 30 10 60
5 6 0 1930695 23600 1079 184 112 0 0 0 0 1412 70240 1812 39 16 45
5 6 0 1930695 23831 512 88 54 0 0 0 0 1479 53350 1501 42 10 48
7 1 0 1938595 23080 546 91 122 0 0 0 0 1580 55882 1566 47 15 38
7 1 0 1938595 21508 481 57 64 0 0 0 0 1706 50651 1478 37 10 53
11 6 0 1881567 24848 930 132 135 0 0 0 0 1648 78207 1941 52 21 27
11 6 0 1881567 17314 1845 304 260 0 0 0 0 1658 91499 2312 48 29 23
2 2 0 1922533 18835 1214 184 162 0 0 0 0 1720 53942 1840 25 18 57
2 2 0 1922533 23153 1234 199 127 0 0 0 0 1499 62253 1771 21 19 60
1 8 0 2007286 23576 1254 214 120 0 0 0 0 1725 58860 1799 14 14 73
1 8 0 2007286 23242 708 121 64 0 0 0 0 1506 37431 1502 9 8 83
1 3 0 1934417 23233 552 91 38 0 0 0 0 1512 31636 1420 8 9 83
1 3 0 1934417 23211 555 92 32 0 0 0 0 1242 33023 1418 8 9 83


And I noticed some errors in event log and syslog. What can we deduce from this? Thanks for the advice.

>------------ Event Monitoring Service Event Notification ------------<

Notification Time: Mon Jun 7 23:04:14 2010

nocrm001 sent Event Monitor notification information:

/storage/events/disks/default/0_1_1_0.1.0 is >= 1.
Its current value is CRITICAL(5).



Event data from monitor:

Event Time..........: Mon Jun 7 23:04:13 2010
Severity............: CRITICAL
Monitor.............: disk_em
Event #.............: 100337
System..............: nocrm001

Summary:
Disk at hardware path 0/1/1/0.1.0 : Media failure


SYSLOG
======
Jun 7 23:04:14 nocrm001 EMS [7957]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/enclosures/gazemon/0_1_1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 521469960 -r /storage/events/enclosures/gazemon/0_1_1_0.1.0 -n 521469956 -a
Jun 7 23:03:05 nocrm001 su: + tty?? root-bmml
Jun 7 23:04:14 nocrm001 above message repeats 2 times
Jun 7 23:04:14 nocrm001 EMS [5445]: ------ EMS Event Notification ------ Value: "CRITICAL (5)" for Resource: "/storage/events/disks/default/0_1_1_0.1.0" (Threshold: >= " 3") Execute the following command to obtain event details: /opt/resmon/bin/resdata -R 356843525 -r /storage/events/disks/default/0_1_1_0.1.0 -n 356843524 -a

Thanks,
Chandana

[Torsten.]

Receive the details:

/opt/resmon/bin/resdata -R 521469960 -r /storage/events/enclosures/gazemon/0_1_1_0.1.0 -n 521469956 -a

/opt/resmon/bin/resdata -R 356843525 -r /storage/events/disks/default/0_1_1_0.1.0 -n 356843524 -a