Increasing LVM performance.

 
SOLVED
Go to solution
Tom Bies
Occasional Advisor

Increasing LVM performance.

Hello,

I have a HP N4000-55, attached to a EMC Clarrion FC4700. I am experiencing progressively slow I/O performance on a filesystem with 130,000+ files inside of it. We have tried defragmenting the filesystem with little success or improvement on performance. We have also tried removing the filesystem, re-creating it and restoring the data. This did help I/O performance, but only for a few weeks before we were crawling again. I ran the following defragment commands several times to see if they made a difference:

fsadm -F vxfs -d -D -e -E /var/opt/oneworld/PrintQueue

fsadm -F vxfs -a 1 -d /var/opt/oneworld/PrintQueue

fsadm -F vxfs -l 2048 -e /var/opt/oneworld/PrintQueue

I did these at lease 10-20 times over a 2 hour period, but this didn't seem to do much. The filesystem in question is configured as PVG-strict/distributed.

Here's some more details.

HP N4000-55
Processors: 8
Clock Frequency: 550 MHz
Kernel Width Support: 64
Physical Memory: 16398.7 MB
OS Identification: B.11.11 U

uteudc11[/root]# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol3
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol4
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol5
dev 944 0 944 0% 0 - 1 /dev/vg00/lvol20
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol21
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol22
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol23
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol24
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol25
reserve - 8606 -8606
memory 12600 3218 9382 26%
total 28904 11824 17080 41% - 0 -

Filesystem in question:

uteudc11[/opt/jde/oneworld/app/PrintQueue]# bdf .
Filesystem kbytes used avail %used Mounted on
/dev/vgmcjde/lvol4 32768000 23586956 8607247 73% /opt/jde/oneworld/app/PrintQueue

Lvdisplay -v of /dev/vgmcjde/lvol4:

--- Logical volumes ---
LV Name /dev/vgmcjde/lvol4
VG Name /dev/vgmcjde
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 32000
Current LE 4000
Allocated PE 4000
Stripes 0
Stripe Size (Kbytes) 0
Bad block NONE
Allocation PVG-strict/distributed
IO Timeout (Seconds) 180

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c12t0d6 1863 1863
/dev/dsk/c12t0d7 1867 1867
/dev/dsk/c7t1d4 68 68
/dev/dsk/c12t1d5 68 68
/dev/dsk/c7t1d6 67 67
/dev/dsk/c12t1d7 67 67

--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c12t0d6 02201 current
00001 /dev/dsk/c12t0d7 02199 current
00002 /dev/dsk/c12t0d6 02202 current
00003 /dev/dsk/c12t0d7 02200 current
00004 /dev/dsk/c12t0d6 02203 current
00005 /dev/dsk/c12t0d7 02201 current
00006 /dev/dsk/c12t0d6 02204 current
00007 /dev/dsk/c12t0d7 02202 current
00008 /dev/dsk/c12t0d6 02205 current
00009 /dev/dsk/c12t0d7 02203 current
00010 /dev/dsk/c12t0d6 02206 current
etc, etc...

Volume group info:

--- Volume groups ---
VG Name /dev/vgmcjde
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 5
Open LV 5
Max PV 96
Cur PV 6
Act PV 6
Max PE per PV 10240
VGDA 12
PE Size (Mbytes) 8
Total PE 11444
Alloc PE 8650
Free PE 2794
Total PVG 1
Total Spare PVs 0
Total Spare PVs in use 0

The bloated filesystem:

uteudc11[/opt/jde/oneworld/app/PrintQueue]# ls -l |wc
136891 1232012 11898612
uteudc11[/opt/jde/oneworld/app/PrintQueue]#

Any suggestions or ideas would be appreciated.
11 REPLIES 11
Geoff Wild
Honored Contributor

Re: Increasing LVM performance.

What was listed in the fsadm output:

Directory Fragmentation Report
Dirs Total Immed Immeds Dirs to Blocks to
Searched Blocks Dirs to Add Reduce Reduce
total 79 5087 39 0 5 3103


Were there any blocks to reduce?

If yes, then keep on defragging until down to (or close to) zero...

Rgds...Geoff
Proverbs 3:5,6 Trust in the Lord with all your heart and lean not on your own understanding; in all your ways acknowledge him, and he will make all your paths straight.
Simon Hargrave
Honored Contributor

Re: Increasing LVM performance.

130000 is a lot of files for one directory! Given that this directory appears to be for a print queue of some form, presumably they're only needed there during printing/spooling. Can they not be moved to an archive area afterwards, perhaps on a separate filesystem?

find /opt/jde/oneworld/app/PrintQueue -atime +7 -exec mv {} /archivearea

This would move files unaccessed in last 7 days to /archivearea. A cron job to do this would allow you to keep the files (if they are indeed needed), but at least maintain performance on the main filesystem.
RAC_1
Honored Contributor

Re: Increasing LVM performance.

No. of files in that directory. That is what the problem is.

Do you alaternate paths?? If so set it properly. This wont help much, but do it.

With the information that you have given seems that these files are print siles. If yes, are they required all the time?? If not move the ones which are not required.

Another thing would be setting up a stripped LV. You will have to remove LV, prepare one with striped across your disks in that lvol.

Anil
There is no substitute to HARDWORK
Rita C Workman
Honored Contributor

Re: Increasing LVM performance.

You post a lvdisplay of /dev/vgmcjde, but you seem to be pointing to the problem of your printqueue on /var.....

So let's address your printqueue, since you say you removed & replaced it and saw some improvement before.
/var is a very busy mountpoint...is does stuff and it is writing all the time ! It is a critical filesystem (fill up /var and you stop..). Remember /var/opt/* has logs writing to it; /var/adm/syslog more logs writing. Are you running MWA..even more /var/opt/perf/datafiles. See my point.

If you want to try and improve performance to known 'busy' directories, then set up seperate disk and create a lvol just for that mountpoint with the disks that are just for them. Like:
/var
/var/opt/spool
/var/opt/oneworld
/var/opt/perf
...and maybe you can think of others you could set up their own disk so they won't be struggling to get I/O attention from others.

Just a thought, HTH
Rita
Jean-Luc Oudart
Honored Contributor

Re: Increasing LVM performance.

Hi,

As mentioned already 130,000+ files si quite a lot for same directory.
According to the Filesystem name under /var this could be temporary files.

You give info for :
/opt/jde/oneworld/app/PrintQueue

and you ran fsadm command on :
var/opt/oneworld/PrintQueue
Are they the same fs or did I miss something ?

Regards,
Jean-Luc
fiat lux
Jean-Luc Oudart
Honored Contributor
Solution

Re: Increasing LVM performance.

Tom Bies
Occasional Advisor

Re: Increasing LVM performance.

I should have said I defragmented the following - /opt/jde/oneworld/app/PrintQueue NOT /var/opt/oneworld/PrintQueue. Sorry, I didn't have my coffee in me yet.

This filesystem resides on its own EMC fiberchannel disks. We are setup for dual paths.

/dev/vgmcjde/lvol4 32768000 23638232 8559220 73% /opt/jde/oneworld/app/PrintQueue

I will check with the application folks to see if moving older files to a alternate filesystem via a cronjob would not break the application. To my understanding, these are print jobs that may (or may not) need to be recalled at any given time.
Bill Hassell
Honored Contributor

Re: Increasing LVM performance.

/opt/jde/oneworld/app/PrintQueue is definitely NOT the place for temporary files like print jobs. /opt is the directory structure for an application but temp files shoul dbe configured in /var/opt/jde/oneworld/app/PrintQueue to follow the V.4 layout recommendations. I realize that some apps are not configurable, so you may not have that choice. But in either case, /opt/jde/oneworld/app/PrintQueue should be turned into a mountpoint. Using /opt in this way is killing other applications. This also allows for a much larger space for the PrintQueue without affecting /opt or /var. You can also assign this mountpoint to a specific LUN or set of LUNs.

As for performance, 130,000 files in a single directory is GUARANTEED to cause major performance issues if the directory is searched. Now that does NOT mean open and read a specific file which will run at full speed. It means when creating a new file or listing files (like ls, especially when using pattern matching such as ls a* to find all files that start with "a"). You will probably see massive numbers in sar -a 5 20 when performance is bad. This is inevitable when asking the system to search the directory structure. The only fix is an application rewrite: either keep a local index in the program of all files by name so it doesn't kill the system searching for filenames, or eliminate the 130,000 files and use a real database with just a few files.


Bill Hassell, sysadmin
Alzhy
Honored Contributor

Re: Increasing LVM performance.

Tune the inode cache of the filesystem containing those zillion or so little files.

man vxtunefs

If using JFS 3.5, there are a couple more tunables in relation to very large inode numbers (file entries..)
Hakuna Matata.