1835062 Members
1939 Online
110073 Solutions
New Discussion

High Internal Disk Load

 
Pedro Cirne
Esteemed Contributor

High Internal Disk Load

Hi,

I'm experiencing very high disk load on internal disks which is really a bottleneck.

I've checked with Glance and vxfsd is with a very high I/O rate, allways greater than 700, and the filesystems greater I/O rate are allways /opt/oracle and /usr. I don't think this is a normal situation.

The box is:
root@porsx011:/# model
9000/800/rp7420

There is no swap activity:

root@porsx011:/# sar -w

HP-UX porsx011 B.11.11 U 9000/800 03/19/07

00:30:14 swpin/s bswin/s swpot/s bswot/s pswch/s
00:35:14 0.00 0.0 0.00 0.0 3555
00:40:14 0.00 0.0 0.00 0.0 4951
00:45:14 0.00 0.0 0.00 0.0 6289
00:50:14 0.00 0.0 0.00 0.0 3653
00:55:14 0.00 0.0 0.00 0.0 3330
01:00:14 0.00 0.0 0.00 0.0 3577
01:05:14 0.00 0.0 0.00 0.0 5100
01:10:14 0.00 0.0 0.00 0.0 2830
01:15:14 0.00 0.0 0.00 0.0 3269
01:20:14 0.00 0.0 0.00 0.0 4383
01:25:14 0.00 0.0 0.00 0.0 5102
01:30:00 0.00 0.0 0.00 0.0 5050
01:35:00 0.00 0.0 0.00 0.0 5000
01:40:00 0.00 0.0 0.00 0.0 4335
01:45:00 0.00 0.0 0.00 0.0 3816
01:50:00 0.00 0.0 0.00 0.0 3696
01:55:00 0.00 0.0 0.00 0.0 4376
02:00:00 0.00 0.0 0.00 0.0 4252
02:05:00 0.00 0.0 0.00 0.0 6336
02:10:00 0.00 0.0 0.00 0.0 3116
02:15:00 0.00 0.0 0.00 0.0 2854
02:20:00 0.00 0.0 0.00 0.0 5369
02:25:00 0.00 0.0 0.00 0.0 7090
02:30:01 0.00 0.0 0.00 0.0 6046
02:35:01 0.00 0.0 0.00 0.0 2892
02:40:01 0.00 0.0 0.00 0.0 3476
02:45:01 0.00 0.0 0.00 0.0 3803
02:50:01 0.00 0.0 0.00 0.0 2982
02:55:01 0.00 0.0 0.00 0.0 3509
03:00:01 0.00 0.0 0.00 0.0 3406
03:05:01 0.00 0.0 0.00 0.0 4311
03:10:01 0.00 0.0 0.00 0.0 3140
03:15:01 0.00 0.0 0.00 0.0 2759
03:20:01 0.00 0.0 0.00 0.0 3320
03:25:01 0.00 0.0 0.00 0.0 2420
03:30:02 0.00 0.0 0.00 0.0 3988
03:35:02 0.00 0.0 0.00 0.0 2486
03:40:02 0.00 0.0 0.00 0.0 3400
03:45:02 0.00 0.0 0.00 0.0 3640
03:50:02 0.00 0.0 0.00 0.0 4166
03:55:02 0.00 0.0 0.00 0.0 3922
04:00:02 0.00 0.0 0.00 0.0 3215
04:05:02 0.00 0.0 0.00 0.0 5202
04:10:02 0.00 0.0 0.00 0.0 3681
04:15:02 0.00 0.0 0.00 0.0 2749
04:20:02 0.00 0.0 0.00 0.0 4738
04:25:02 0.00 0.0 0.00 0.0 3275
04:30:00 0.00 0.0 0.00 0.0 5210
04:35:00 0.00 0.0 0.00 0.0 3499
04:40:00 0.00 0.0 0.00 0.0 3742
04:45:00 0.00 0.0 0.00 0.0 5020
04:50:00 0.00 0.0 0.00 0.0 4167
04:55:00 0.00 0.0 0.00 0.0 4586
05:00:00 0.00 0.0 0.00 0.0 4574
05:05:00 0.00 0.0 0.00 0.0 5295
05:10:00 0.00 0.0 0.00 0.0 4306
05:15:00 0.00 0.0 0.00 0.0 3739
05:20:00 0.00 0.0 0.00 0.0 4297
05:25:00 0.00 0.0 0.00 0.0 6403
05:30:02 0.00 0.0 0.00 0.0 7139
05:35:02 0.00 0.0 0.00 0.0 3802
05:40:02 0.00 0.0 0.00 0.0 3189
05:45:02 0.00 0.0 0.00 0.0 4641
05:50:02 0.00 0.0 0.00 0.0 4168
05:55:02 0.00 0.0 0.00 0.0 5237
06:00:02 0.00 0.0 0.00 0.0 3702
06:05:02 0.00 0.0 0.00 0.0 8603
06:10:02 0.00 0.0 0.00 0.0 1544
06:15:02 0.00 0.0 0.00 0.0 2968
06:20:02 0.00 0.0 0.00 0.0 4512
06:25:02 0.00 0.0 0.00 0.0 4123
06:30:02 0.00 0.0 0.00 0.0 4118
06:35:02 0.00 0.0 0.00 0.0 4125
06:40:02 0.00 0.0 0.00 0.0 5171
06:45:02 0.00 0.0 0.00 0.0 3178
06:50:02 0.00 0.0 0.00 0.0 3673
06:55:02 0.00 0.0 0.00 0.0 3510
07:00:02 0.00 0.0 0.00 0.0 3285
07:05:02 0.00 0.0 0.00 0.0 6063
07:10:02 0.00 0.0 0.00 0.0 4205
07:15:02 0.00 0.0 0.00 0.0 4240
07:20:02 0.00 0.0 0.00 0.0 3425
07:25:02 0.00 0.0 0.00 0.0 3655
07:30:02 0.00 0.0 0.00 0.0 4462
07:35:02 0.00 0.0 0.00 0.0 3868
07:40:02 0.00 0.0 0.00 0.0 3593
07:45:02 0.00 0.0 0.00 0.0 2851
07:50:02 0.00 0.0 0.00 0.0 4786
07:55:02 0.00 0.0 0.00 0.0 3853
08:00:02 0.00 0.0 0.00 0.0 3875
08:05:02 0.00 0.0 0.00 0.0 5611
08:10:02 0.00 0.0 0.00 0.0 4668
08:15:02 0.00 0.0 0.00 0.0 4168
08:20:02 0.00 0.0 0.00 0.0 3603
08:25:02 0.00 0.0 0.00 0.0 6795
08:30:01 0.00 0.0 0.00 0.0 6503
08:35:01 0.00 0.0 0.00 0.0 5298
08:40:01 0.00 0.0 0.00 0.0 3604
08:45:01 0.00 0.0 0.00 0.0 2715
08:50:01 0.00 0.0 0.00 0.0 3114
08:55:01 0.00 0.0 0.00 0.0 3940
09:00:01 0.00 0.0 0.00 0.0 2548
09:05:01 0.00 0.0 0.00 0.0 3958
09:10:01 0.00 0.0 0.00 0.0 2202
09:15:01 0.00 0.0 0.00 0.0 12581
09:20:01 0.00 0.0 0.00 0.0 15996
09:25:01 0.00 0.0 0.00 0.0 12339
09:30:01 0.00 0.0 0.00 0.0 13052
09:35:01 0.00 0.0 0.00 0.0 13828
09:40:01 0.00 0.0 0.00 0.0 10749
09:45:01 0.00 0.0 0.00 0.0 3539
09:50:01 0.00 0.0 0.00 0.0 2928
09:55:01 0.00 0.0 0.00 0.0 3905
10:00:01 0.00 0.0 0.00 0.0 3712
10:05:01 0.00 0.0 0.00 0.0 4900
10:10:01 0.00 0.0 0.00 0.0 3197
10:15:01 0.00 0.0 0.00 0.0 4126
10:20:01 0.00 0.0 0.00 0.0 4683
10:25:01 0.00 0.0 0.00 0.0 7458
10:30:00 0.00 0.0 0.00 0.0 6020
10:35:00 0.00 0.0 0.00 0.0 4428
10:40:00 0.00 0.0 0.00 0.0 4153
10:45:00 0.00 0.0 0.00 0.0 4015
10:50:00 0.00 0.0 0.00 0.0 4660
10:55:00 0.00 0.0 0.00 0.0 3662
11:00:00 0.00 0.0 0.00 0.0 3309
11:05:00 0.00 0.0 0.00 0.0 4871
11:10:00 0.00 0.0 0.00 0.0 3369
11:15:00 0.00 0.0 0.00 0.0 2781
11:20:00 0.00 0.0 0.00 0.0 3163
11:25:00 0.00 0.0 0.00 0.0 6749
11:30:01 0.00 0.0 0.00 0.0 5195
11:35:01 0.00 0.0 0.00 0.0 3847
11:40:01 0.00 0.0 0.00 0.0 4024
11:45:01 0.00 0.0 0.00 0.0 5324
11:50:01 0.00 0.0 0.00 0.0 4250
11:55:01 0.00 0.0 0.00 0.0 4753
12:00:01 0.00 0.0 0.00 0.0 8819
12:05:01 0.00 0.0 0.00 0.0 8336
12:10:01 0.00 0.0 0.00 0.0 6366
12:15:01 0.00 0.0 0.00 0.0 5221
12:20:01 0.00 0.0 0.00 0.0 7279
12:25:01 0.00 0.0 0.00 0.0 6830
12:30:10 0.00 0.0 0.00 0.0 9037
12:35:10 0.00 0.0 0.00 0.0 6665
12:40:10 0.00 0.0 0.00 0.0 6601
12:45:10 0.00 0.0 0.00 0.0 6098
12:50:10 0.00 0.0 0.00 0.0 7615
12:55:10 0.00 0.0 0.00 0.0 4919
13:00:10 0.00 0.0 0.00 0.0 7956
13:05:10 0.00 0.0 0.00 0.0 10068
13:10:10 0.00 0.0 0.00 0.0 6545
13:15:10 0.00 0.0 0.00 0.0 6516
13:20:10 0.00 0.0 0.00 0.0 7212
13:25:10 0.00 0.0 0.00 0.0 7704
13:30:00 0.00 0.0 0.00 0.0 9109
13:35:00 0.00 0.0 0.00 0.0 5922
13:40:00 0.00 0.0 0.00 0.0 6189
13:45:00 0.00 0.0 0.00 0.0 6690
13:50:00 0.00 0.0 0.00 0.0 6007
13:55:00 0.00 0.0 0.00 0.0 6363
14:00:00 0.00 0.0 0.00 0.0 7877
14:05:00 0.00 0.0 0.00 0.0 9023
14:10:00 0.00 0.0 0.00 0.0 7370
14:15:00 0.00 0.0 0.00 0.0 5807
14:20:00 0.00 0.0 0.00 0.0 7414
14:25:00 0.00 0.0 0.00 0.0 7235
14:30:04 0.00 0.0 0.00 0.0 9940
14:35:04 0.00 0.0 0.00 0.0 8504
14:40:04 0.00 0.0 0.00 0.0 5565
14:45:04 0.00 0.0 0.00 0.0 9626
14:50:04 0.00 0.0 0.00 0.0 7650
14:55:04 0.00 0.0 0.00 0.0 8985
15:00:04 0.00 0.0 0.00 0.0 11504
15:05:04 0.00 0.0 0.00 0.0 6875
15:10:04 0.00 0.0 0.00 0.0 3239
15:15:04 0.00 0.0 0.00 0.0 7598
15:20:04 0.00 0.0 0.00 0.0 9199
15:25:04 0.00 0.0 0.00 0.0 9362
15:30:04 0.00 0.0 0.00 0.0 14211
15:35:04 0.00 0.0 0.00 0.0 11053
15:40:04 0.00 0.0 0.00 0.0 11229
15:45:04 0.00 0.0 0.00 0.0 6730
15:50:04 0.00 0.0 0.00 0.0 11657
15:55:04 0.00 0.0 0.00 0.0 10074
16:00:04 0.00 0.0 0.00 0.0 10692

Average 0.00 0.0 0.00 0.0 5458

From sar (the same in Glance):

Average c0t6d0 80.25 56.48 301 2474 117.34 19.45
Average c3t6d0 74.07 35.02 286 2373 74.69 18.62
Average c13t0d1 11.24 0.97 88 5265 0.34 3.25
Average c12t0d2 8.60 1.11 67 4297 0.37 3.30
Average c13t0d4 11.42 1.03 89 5215 0.30 3.05
Average c13t2d4 14.21 0.51 75 3426 0.02 2.80
Average c12t2d5 16.06 0.50 75 3479 0.02 3.12
Average c13t0d3 13.74 0.50 45 1775 0.01 3.93
Average c12t0d5 12.78 0.50 44 1851 0.01 3.73
Average c13t0d6 17.63 0.50 95 3856 0.01 2.66
Average c12t0d7 11.75 0.50 40 1389 0.01 3.61
Average c13t1d6 16.29 0.50 94 3866 0.01 2.51
Average c12t1d7 12.52 0.50 40 1357 0.01 3.83
Average c13t2d0 12.42 0.50 42 1497 0.01 3.70
Average c12t2d1 11.96 0.50 43 1551 0.01 3.44
Average c13t2d2 11.81 0.50 41 1434 0.01 3.54


Any ideia of what might be causing this very high I/O rate on internal disks?

Thank you.
14 REPLIES 14
Steven E. Protter
Exalted Contributor

Re: High Internal Disk Load

Shalom,

The most common cause of this is the applications being run on the server. Database's and such are known to generate a lot of i/o.

Another common cause is users, running tests, creating and destroying large files. Compiles and such.

Take a look at what the system is running and who is doing what how.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Jaime Bolanos Rojas.
Honored Contributor

Re: High Internal Disk Load

Hi Pedro,

If you already know that you have a disk bottleneck, the next logical step would be to know which is the application causing it, and perhaps see why is the application producing so much I/O, if it's normal or maybe upgrade your hardware.

You can check this out by using Glance ( if you do not have it you could always get a trial version) then from the main screen just type "i" command to go to the I/O by filesystem screen.

Regards,

Jaime.
Work hard when the need comes out.
Pedro Cirne
Esteemed Contributor

Re: High Internal Disk Load

Hi,

The filesystem is /home, I've several symbolic links that point from /home on internal disks to other filesystems on symmetrix.

Are filesystems generating I/O when using the symbolic links on internal disks to write on symmetrix volumes?
A. Clay Stephenson
Acclaimed Contributor

Re: High Internal Disk Load

No, softlinks should cause almost 0 i/o on the filesystem housing the link. A directory look up will be done that but that's it and it would generally be cached anyway. However, a few softlinks that were "backwards" (i.e pointing to your home directory would cause your kinds of problems). If /home is extremely busy, it usually means that files are being created, updated, or read repeatedly. Use Glance to drill down on processes which have high i/o rates.
If it ain't broke, I can fix that.
DavidJ
Regular Advisor

Re: High Internal Disk Load

Hi Pedro,

Check the %used of your fie systems and try and get them to an "acceptable" level. Check /home, what free space do you have in this file system. Below is a little script which I find quite handy to get the 10 highest users of /home, and then ask them to clean up.

Hope it helps.

homefolders=/home
users=`ls -1 $homefolders`
limit=100

printf " %-10.10s\t%s\n" " User " " MB "
printf " %-10.10s\t%s\n" "==========" "===="

for user in $users
do
diskuse=`du -sk $homefolders/$user | awk '{ print $1 }' -`
megs=`expr $diskuse / 1024`
if [ $megs -gt $limit ]
then
printf " %-10.10s\t %s\n" $user $megs
fi
done
Everyday I beat my own previous record for number of consecutive days I've stayed alive.
Dennis Handly
Acclaimed Contributor

Re: High Internal Disk Load

>David: Check the %used of your file systems

Why would the fullness of a FS cause a high I/O rate? Perhaps for HFS with fragmentation it might.
KRI
Advisor

Re: High Internal Disk Load

Hi !


Look in /var/adm/syslog/syslog.log for possible errors.

Regards!
KRI
Pedro Cirne
Esteemed Contributor

Re: High Internal Disk Load

Hi All,

The filesystem ocupation is not too much:

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol5 10240000 5031176 5168480 49% /home

How can I check what directories/files are being changed/acessed on /home?

I hava Glance installed on this server..
Dennis Handly
Acclaimed Contributor

Re: High Internal Disk Load

>How can I check what directories/files are being changed/acessed on /home?

$ find /home \( -atime 0 -o -mtime 0 \)

But this will give only a resolution in terms of a day. For finer granularity, you can touch a file (ref-file) and then use:
$ find /home \( -newer ref-file -o -newera ref-file \)
KRI
Advisor

Re: High Internal Disk Load

Tray:
find /home -mtime -1
find /home -atime -1
find /home -ctime -1

-atime = access time
-mtime = modification time
-ctime = last change

Tray also:

fuser /home

Regards
KRI
Luis Ernesto Angulo
Occasional Advisor

Re: High Internal Disk Load

I have a similar situation. Did you find a way to fix this issue?

In my case seems that the vxfsd process is doing a lot of raw writes to the /usr filesystem. I haven't been able to identify which process is triggering this behavior yet.

Any idea will be appreciated...

Regards,
Dennis Handly
Acclaimed Contributor

Re: High Internal Disk Load

>Luis: I have a similar situation.

You should create your own thread with your new problem, so you can assign points.
You can include a link to this thread if there is useful info here.
Luis Ernesto Angulo
Occasional Advisor

Re: High Internal Disk Load

Dennis, I created a thread yesterday, it would be great if you were willing to participate there:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1254214

However, as long as I can see, this thread hasn't been closed, and I think that if this issue was solved it would be helpful if we could get a feedback. After all, the purpose of this forum is to keep the knowledge base growing based on shared experiences...

Regards,
Pedro Cirne
Esteemed Contributor

Re: High Internal Disk Load

Closing this case.