topic Re: High Internal Disk Load in Operating System - HP-UX

High Internal Disk Load

Pedro Cirne — Mon, 19 Mar 2007 11:07:09 GMT

Hi,

I'm experiencing very high disk load on internal disks which is really a bottleneck.

I've checked with Glance and vxfsd is with a very high I/O rate, allways greater than 700, and the filesystems greater I/O rate are allways /opt/oracle and /usr. I don't think this is a normal situation.

The box is:
root@porsx011:/# model
9000/800/rp7420

There is no swap activity:

root@porsx011:/# sar -w

HP-UX porsx011 B.11.11 U 9000/800 03/19/07

00:30:14 swpin/s bswin/s swpot/s bswot/s pswch/s
00:35:14 0.00 0.0 0.00 0.0 3555
00:40:14 0.00 0.0 0.00 0.0 4951
00:45:14 0.00 0.0 0.00 0.0 6289
00:50:14 0.00 0.0 0.00 0.0 3653
00:55:14 0.00 0.0 0.00 0.0 3330
01:00:14 0.00 0.0 0.00 0.0 3577
01:05:14 0.00 0.0 0.00 0.0 5100
01:10:14 0.00 0.0 0.00 0.0 2830
01:15:14 0.00 0.0 0.00 0.0 3269
01:20:14 0.00 0.0 0.00 0.0 4383
01:25:14 0.00 0.0 0.00 0.0 5102
01:30:00 0.00 0.0 0.00 0.0 5050
01:35:00 0.00 0.0 0.00 0.0 5000
01:40:00 0.00 0.0 0.00 0.0 4335
01:45:00 0.00 0.0 0.00 0.0 3816
01:50:00 0.00 0.0 0.00 0.0 3696
01:55:00 0.00 0.0 0.00 0.0 4376
02:00:00 0.00 0.0 0.00 0.0 4252
02:05:00 0.00 0.0 0.00 0.0 6336
02:10:00 0.00 0.0 0.00 0.0 3116
02:15:00 0.00 0.0 0.00 0.0 2854
02:20:00 0.00 0.0 0.00 0.0 5369
02:25:00 0.00 0.0 0.00 0.0 7090
02:30:01 0.00 0.0 0.00 0.0 6046
02:35:01 0.00 0.0 0.00 0.0 2892
02:40:01 0.00 0.0 0.00 0.0 3476
02:45:01 0.00 0.0 0.00 0.0 3803
02:50:01 0.00 0.0 0.00 0.0 2982
02:55:01 0.00 0.0 0.00 0.0 3509
03:00:01 0.00 0.0 0.00 0.0 3406
03:05:01 0.00 0.0 0.00 0.0 4311
03:10:01 0.00 0.0 0.00 0.0 3140
03:15:01 0.00 0.0 0.00 0.0 2759
03:20:01 0.00 0.0 0.00 0.0 3320
03:25:01 0.00 0.0 0.00 0.0 2420
03:30:02 0.00 0.0 0.00 0.0 3988
03:35:02 0.00 0.0 0.00 0.0 2486
03:40:02 0.00 0.0 0.00 0.0 3400
03:45:02 0.00 0.0 0.00 0.0 3640
03:50:02 0.00 0.0 0.00 0.0 4166
03:55:02 0.00 0.0 0.00 0.0 3922
04:00:02 0.00 0.0 0.00 0.0 3215
04:05:02 0.00 0.0 0.00 0.0 5202
04:10:02 0.00 0.0 0.00 0.0 3681
04:15:02 0.00 0.0 0.00 0.0 2749
04:20:02 0.00 0.0 0.00 0.0 4738
04:25:02 0.00 0.0 0.00 0.0 3275
04:30:00 0.00 0.0 0.00 0.0 5210
04:35:00 0.00 0.0 0.00 0.0 3499
04:40:00 0.00 0.0 0.00 0.0 3742
04:45:00 0.00 0.0 0.00 0.0 5020
04:50:00 0.00 0.0 0.00 0.0 4167
04:55:00 0.00 0.0 0.00 0.0 4586
05:00:00 0.00 0.0 0.00 0.0 4574
05:05:00 0.00 0.0 0.00 0.0 5295
05:10:00 0.00 0.0 0.00 0.0 4306
05:15:00 0.00 0.0 0.00 0.0 3739
05:20:00 0.00 0.0 0.00 0.0 4297
05:25:00 0.00 0.0 0.00 0.0 6403
05:30:02 0.00 0.0 0.00 0.0 7139
05:35:02 0.00 0.0 0.00 0.0 3802
05:40:02 0.00 0.0 0.00 0.0 3189
05:45:02 0.00 0.0 0.00 0.0 4641
05:50:02 0.00 0.0 0.00 0.0 4168
05:55:02 0.00 0.0 0.00 0.0 5237
06:00:02 0.00 0.0 0.00 0.0 3702
06:05:02 0.00 0.0 0.00 0.0 8603
06:10:02 0.00 0.0 0.00 0.0 1544
06:15:02 0.00 0.0 0.00 0.0 2968
06:20:02 0.00 0.0 0.00 0.0 4512
06:25:02 0.00 0.0 0.00 0.0 4123
06:30:02 0.00 0.0 0.00 0.0 4118
06:35:02 0.00 0.0 0.00 0.0 4125
06:40:02 0.00 0.0 0.00 0.0 5171
06:45:02 0.00 0.0 0.00 0.0 3178
06:50:02 0.00 0.0 0.00 0.0 3673
06:55:02 0.00 0.0 0.00 0.0 3510
07:00:02 0.00 0.0 0.00 0.0 3285
07:05:02 0.00 0.0 0.00 0.0 6063
07:10:02 0.00 0.0 0.00 0.0 4205
07:15:02 0.00 0.0 0.00 0.0 4240
07:20:02 0.00 0.0 0.00 0.0 3425
07:25:02 0.00 0.0 0.00 0.0 3655
07:30:02 0.00 0.0 0.00 0.0 4462
07:35:02 0.00 0.0 0.00 0.0 3868
07:40:02 0.00 0.0 0.00 0.0 3593
07:45:02 0.00 0.0 0.00 0.0 2851
07:50:02 0.00 0.0 0.00 0.0 4786
07:55:02 0.00 0.0 0.00 0.0 3853
08:00:02 0.00 0.0 0.00 0.0 3875
08:05:02 0.00 0.0 0.00 0.0 5611
08:10:02 0.00 0.0 0.00 0.0 4668
08:15:02 0.00 0.0 0.00 0.0 4168
08:20:02 0.00 0.0 0.00 0.0 3603
08:25:02 0.00 0.0 0.00 0.0 6795
08:30:01 0.00 0.0 0.00 0.0 6503
08:35:01 0.00 0.0 0.00 0.0 5298
08:40:01 0.00 0.0 0.00 0.0 3604
08:45:01 0.00 0.0 0.00 0.0 2715
08:50:01 0.00 0.0 0.00 0.0 3114
08:55:01 0.00 0.0 0.00 0.0 3940
09:00:01 0.00 0.0 0.00 0.0 2548
09:05:01 0.00 0.0 0.00 0.0 3958
09:10:01 0.00 0.0 0.00 0.0 2202
09:15:01 0.00 0.0 0.00 0.0 12581
09:20:01 0.00 0.0 0.00 0.0 15996
09:25:01 0.00 0.0 0.00 0.0 12339
09:30:01 0.00 0.0 0.00 0.0 13052
09:35:01 0.00 0.0 0.00 0.0 13828
09:40:01 0.00 0.0 0.00 0.0 10749
09:45:01 0.00 0.0 0.00 0.0 3539
09:50:01 0.00 0.0 0.00 0.0 2928
09:55:01 0.00 0.0 0.00 0.0 3905
10:00:01 0.00 0.0 0.00 0.0 3712
10:05:01 0.00 0.0 0.00 0.0 4900
10:10:01 0.00 0.0 0.00 0.0 3197
10:15:01 0.00 0.0 0.00 0.0 4126
10:20:01 0.00 0.0 0.00 0.0 4683
10:25:01 0.00 0.0 0.00 0.0 7458
10:30:00 0.00 0.0 0.00 0.0 6020
10:35:00 0.00 0.0 0.00 0.0 4428
10:40:00 0.00 0.0 0.00 0.0 4153
10:45:00 0.00 0.0 0.00 0.0 4015
10:50:00 0.00 0.0 0.00 0.0 4660
10:55:00 0.00 0.0 0.00 0.0 3662
11:00:00 0.00 0.0 0.00 0.0 3309
11:05:00 0.00 0.0 0.00 0.0 4871
11:10:00 0.00 0.0 0.00 0.0 3369
11:15:00 0.00 0.0 0.00 0.0 2781
11:20:00 0.00 0.0 0.00 0.0 3163
11:25:00 0.00 0.0 0.00 0.0 6749
11:30:01 0.00 0.0 0.00 0.0 5195
11:35:01 0.00 0.0 0.00 0.0 3847
11:40:01 0.00 0.0 0.00 0.0 4024
11:45:01 0.00 0.0 0.00 0.0 5324
11:50:01 0.00 0.0 0.00 0.0 4250
11:55:01 0.00 0.0 0.00 0.0 4753
12:00:01 0.00 0.0 0.00 0.0 8819
12:05:01 0.00 0.0 0.00 0.0 8336
12:10:01 0.00 0.0 0.00 0.0 6366
12:15:01 0.00 0.0 0.00 0.0 5221
12:20:01 0.00 0.0 0.00 0.0 7279
12:25:01 0.00 0.0 0.00 0.0 6830
12:30:10 0.00 0.0 0.00 0.0 9037
12:35:10 0.00 0.0 0.00 0.0 6665
12:40:10 0.00 0.0 0.00 0.0 6601
12:45:10 0.00 0.0 0.00 0.0 6098
12:50:10 0.00 0.0 0.00 0.0 7615
12:55:10 0.00 0.0 0.00 0.0 4919
13:00:10 0.00 0.0 0.00 0.0 7956
13:05:10 0.00 0.0 0.00 0.0 10068
13:10:10 0.00 0.0 0.00 0.0 6545
13:15:10 0.00 0.0 0.00 0.0 6516
13:20:10 0.00 0.0 0.00 0.0 7212
13:25:10 0.00 0.0 0.00 0.0 7704
13:30:00 0.00 0.0 0.00 0.0 9109
13:35:00 0.00 0.0 0.00 0.0 5922
13:40:00 0.00 0.0 0.00 0.0 6189
13:45:00 0.00 0.0 0.00 0.0 6690
13:50:00 0.00 0.0 0.00 0.0 6007
13:55:00 0.00 0.0 0.00 0.0 6363
14:00:00 0.00 0.0 0.00 0.0 7877
14:05:00 0.00 0.0 0.00 0.0 9023
14:10:00 0.00 0.0 0.00 0.0 7370
14:15:00 0.00 0.0 0.00 0.0 5807
14:20:00 0.00 0.0 0.00 0.0 7414
14:25:00 0.00 0.0 0.00 0.0 7235
14:30:04 0.00 0.0 0.00 0.0 9940
14:35:04 0.00 0.0 0.00 0.0 8504
14:40:04 0.00 0.0 0.00 0.0 5565
14:45:04 0.00 0.0 0.00 0.0 9626
14:50:04 0.00 0.0 0.00 0.0 7650
14:55:04 0.00 0.0 0.00 0.0 8985
15:00:04 0.00 0.0 0.00 0.0 11504
15:05:04 0.00 0.0 0.00 0.0 6875
15:10:04 0.00 0.0 0.00 0.0 3239
15:15:04 0.00 0.0 0.00 0.0 7598
15:20:04 0.00 0.0 0.00 0.0 9199
15:25:04 0.00 0.0 0.00 0.0 9362
15:30:04 0.00 0.0 0.00 0.0 14211
15:35:04 0.00 0.0 0.00 0.0 11053
15:40:04 0.00 0.0 0.00 0.0 11229
15:45:04 0.00 0.0 0.00 0.0 6730
15:50:04 0.00 0.0 0.00 0.0 11657
15:55:04 0.00 0.0 0.00 0.0 10074
16:00:04 0.00 0.0 0.00 0.0 10692

Average 0.00 0.0 0.00 0.0 5458

From sar (the same in Glance):

Average c0t6d0 80.25 56.48 301 2474 117.34 19.45
Average c3t6d0 74.07 35.02 286 2373 74.69 18.62
Average c13t0d1 11.24 0.97 88 5265 0.34 3.25
Average c12t0d2 8.60 1.11 67 4297 0.37 3.30
Average c13t0d4 11.42 1.03 89 5215 0.30 3.05
Average c13t2d4 14.21 0.51 75 3426 0.02 2.80
Average c12t2d5 16.06 0.50 75 3479 0.02 3.12
Average c13t0d3 13.74 0.50 45 1775 0.01 3.93
Average c12t0d5 12.78 0.50 44 1851 0.01 3.73
Average c13t0d6 17.63 0.50 95 3856 0.01 2.66
Average c12t0d7 11.75 0.50 40 1389 0.01 3.61
Average c13t1d6 16.29 0.50 94 3866 0.01 2.51
Average c12t1d7 12.52 0.50 40 1357 0.01 3.83
Average c13t2d0 12.42 0.50 42 1497 0.01 3.70
Average c12t2d1 11.96 0.50 43 1551 0.01 3.44
Average c13t2d2 11.81 0.50 41 1434 0.01 3.54

Any ideia of what might be causing this very high I/O rate on internal disks?

Thank you.

Re: High Internal Disk Load

Steven E. Protter — Mon, 19 Mar 2007 11:12:50 GMT

Shalom,

The most common cause of this is the applications being run on the server. Database's and such are known to generate a lot of i/o.

Another common cause is users, running tests, creating and destroying large files. Compiles and such.

Take a look at what the system is running and who is doing what how.

SEP

Re: High Internal Disk Load

Jaime Bolanos Rojas. — Mon, 19 Mar 2007 11:40:30 GMT

Hi Pedro,

If you already know that you have a disk bottleneck, the next logical step would be to know which is the application causing it, and perhaps see why is the application producing so much I/O, if it's normal or maybe upgrade your hardware.

You can check this out by using Glance ( if you do not have it you could always get a trial version) then from the main screen just type "i" command to go to the I/O by filesystem screen.

Regards,

Jaime.

Re: High Internal Disk Load

Pedro Cirne — Mon, 19 Mar 2007 12:40:20 GMT

Hi,

The filesystem is /home, I've several symbolic links that point from /home on internal disks to other filesystems on symmetrix.

Are filesystems generating I/O when using the symbolic links on internal disks to write on symmetrix volumes?

Re: High Internal Disk Load

A. Clay Stephenson — Mon, 19 Mar 2007 12:55:29 GMT

No, softlinks should cause almost 0 i/o on the filesystem housing the link. A directory look up will be done that but that's it and it would generally be cached anyway. However, a few softlinks that were "backwards" (i.e pointing to your home directory would cause your kinds of problems). If /home is extremely busy, it usually means that files are being created, updated, or read repeatedly. Use Glance to drill down on processes which have high i/o rates.

Re: High Internal Disk Load

DavidJ — Tue, 20 Mar 2007 01:51:52 GMT

Hi Pedro,

Check the %used of your fie systems and try and get them to an "acceptable" level. Check /home, what free space do you have in this file system. Below is a little script which I find quite handy to get the 10 highest users of /home, and then ask them to clean up.

Hope it helps.

homefolders=/home
users=`ls -1 $homefolders`
limit=100

printf " %-10.10s\t%s\n" " User " " MB "
printf " %-10.10s\t%s\n" "==========" "===="

for user in $users
do
diskuse=`du -sk $homefolders/$user | awk '{ print $1 }' -`
megs=`expr $diskuse / 1024`
if [ $megs -gt $limit ]
then
printf " %-10.10s\t %s\n" $user $megs
fi
done

Re: High Internal Disk Load

Dennis Handly — Tue, 20 Mar 2007 02:38:37 GMT

>David: Check the %used of your file systems

Why would the fullness of a FS cause a high I/O rate? Perhaps for HFS with fragmentation it might.

Re: High Internal Disk Load

KRI — Tue, 20 Mar 2007 02:59:31 GMT

Hi !

Look in /var/adm/syslog/syslog.log for possible errors.

Regards!
KRI

Re: High Internal Disk Load

Pedro Cirne — Tue, 20 Mar 2007 04:15:47 GMT

Hi All,

The filesystem ocupation is not too much:

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol5 10240000 5031176 5168480 49% /home

How can I check what directories/files are being changed/acessed on /home?

I hava Glance installed on this server..

Re: High Internal Disk Load

Dennis Handly — Tue, 20 Mar 2007 04:24:27 GMT

>How can I check what directories/files are being changed/acessed on /home?

$ find /home $ -atime 0 -o -mtime 0 $

But this will give only a resolution in terms of a day. For finer granularity, you can touch a file (ref-file) and then use:
$ find /home $ -newer ref-file -o -newera ref-file $

Re: High Internal Disk Load

KRI — Tue, 20 Mar 2007 04:27:16 GMT

Tray:
find /home -mtime -1
find /home -atime -1
find /home -ctime -1

-atime = access time
-mtime = modification time
-ctime = last change

Tray also:

fuser /home

Regards
KRI

Re: High Internal Disk Load

Luis Ernesto Angulo — Wed, 30 Jul 2008 23:07:18 GMT

I have a similar situation. Did you find a way to fix this issue?

In my case seems that the vxfsd process is doing a lot of raw writes to the /usr filesystem. I haven't been able to identify which process is triggering this behavior yet.

Any idea will be appreciated...

Regards,

Re: High Internal Disk Load

Dennis Handly — Thu, 31 Jul 2008 00:06:05 GMT

>Luis: I have a similar situation.

You should create your own thread with your new problem, so you can assign points.
You can include a link to this thread if there is useful info here.

Re: High Internal Disk Load

Luis Ernesto Angulo — Thu, 31 Jul 2008 02:28:15 GMT

Dennis, I created a thread yesterday, it would be great if you were willing to participate there:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1254214

However, as long as I can see, this thread hasn't been closed, and I think that if this issue was solved it would be helpful if we could get a feedback. After all, the purpose of this forum is to keep the knowledge base growing based on shared experiences...

Regards,

Re: High Internal Disk Load

Pedro Cirne — Thu, 31 Jul 2008 06:48:17 GMT

Closing this case.