File system performance issues

[Chris Naudé]

I have been trying to track down a problem which has been causing backups to run extremely slow. I believe I have found the bottleneck to be file system related. Here is a little background on the issue.

I have an ESL712 tape library with ten LTO3 drives. I have this connected to two brocade fibre switches which are then connected and zoned to several machines. My backups are currently running at around 5~10MB/sec.

I have discovered in my testing that I can't read my vxfs file systems any faster than 5~10MB/sec. I validated this using tar, dd and /dev/null.

Client:
bash-3.2# du -sk /opt
2727496 /opt
bash-3.2# time tar cf /dev/null /opt
real 12m39.753s
user 0m25.480s
sys 0m42.110s
3.5MB/sec

Backup Server:
bash-3.2# du -sk /home
807938 /home
bash-3.2# time tar cf /dev/null /home
real 1m13.610s
user 0m9.320s
sys 0m13.820s
10MB/sec

In addition to my file system backups running slow all of my Oracle RMAN backups run slow as well.

The disks we are reading from are on an XP512. I am aware that there are probably quite a few things that could be causing these issues. I have a case open with HP support as well. Hopefully I can get to the bottom of this start making good use of these LTO3 drives.

[A. Clay Stephenson]

You are starting your tests at much too high a level of abstraction.

I would start by umounting the filesystem and doing a test of the underlying disks or LUN's.

e.g.
First test the raw (character) device:

timex dd if=/dev/rdsk/c5t1d2 bs=1024k count=1000 of=/dev/null

Next test the buffer cache:
timex dd if=/dev/dsk/c5t1d2 bs=1024k count=1000 of=/dev/null

Now repeat the tests at the LVM or VxVM level:
timex dd if=/dev/vg05/rlvol1 bs=1024k count=1000 of=/dev/null


timex dd if=/dev/vg05/lvol1 bs=1024k count=1000 of=/dev/null

Finally remount the file and repeat for a cooked 1GiB file:

timex dd if=/aaa/bbb/My1GiBFile bs=1024k count=1000 of=/dev/null

I would repeat the tests a number of times and repeated tests of the block devices should show buffer cache improvements.

This will help pinpoint where in the chain the problem actually lies.

[whiteknight]

Chris,

Are you running on 11.23 on online JFS ?
i suspected you need to tune your JFS using vxtunesfs parameter, as there are few known issue reported


WK