Socket or DIsk performance

[MohitAnchlia]

I have a java client application that reads files from a directory "A" and sends that data to C++ application on a socket connection. Java client application then moves these files to directory "B" directories. Directory "A" has subdirectories based on the day and time when those files were received. What I am seeing is that when we have 1M+ files the process really slows down. When I take the stack trace I see the following:

"0" prio=10 tid=00055ac0 nid=36 lwp_id=4707656 runnable [21240000..21240738]
at java.io.UnixFileSystem.rename0(Native Method)
at java.io.UnixFileSystem.rename(UnixFileSystem.java:318)
at java.io.File.renameTo(File.java:1212) at com.i.e.R.moveFile(Unknown Source)

It looks like it's taking time to move files. I am not sure why it would be slow in moving files if there are lot of files. I am also not sure if writing on socket is slow. How can I tell which resource is slowing this App. I tried to look at the sar data but I didn't know how to relate the device name with the file system. I've done basic analysis I am looking for something advanced that would tell me about the resources that are slowing this app.

[Hein van den Heuvel]

>> What I am seeing is that when we have 1M+ files the process really slows down.

Sure! Those directories have on on-disk structure which needs to be maintained. As they grow that's more work adn there will be less cache to go arounf.

Check out the tail end of:
http://docs.hp.com/en/5576/JFS_Tuning.pdf

>> It looks like it's taking time to move files.

May we assume the moves are just between directories onthe same underlying volume?

>> I am not sure why it would be slow in moving files if there are lot of files.

Because there is more data to trounce through?

>> I am also not sure if writing on socket is slow. How can I tell which resource is slowing this App.

Isolate and measure! Try 1000 moves outside application context on shell level

What else is the application doing beside a 'rename'? Could it be doing readdir's galore? Or doing a stat on all file before moving 1? Maybe it is blowing the inode cache?

You might want to try 'truss' or other system call trace tool

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[MohitAnchlia]

- I haven't looked at the JFS tunning , but I'll look at it. thanks
- Yes it's moving file within same volume
- So when file is moved does it slow down if that directory has lot of files ?
- Yes this application does read files from directory, how can I tell if it blowing the inode cache. Also what's the role of inode cache and how can I say if that's the problem
- How can I use truss and how to intrepret the output in this context ?
- If I find writing on socket is a problem then how can I tell if it's a network related issue. Could it be a network related issue if client and server are on the same box ?

[Michael Steele_2]

Its as easy (* for the problem file systems *) as looking for a disk bottleneck with 'sar -d'. Refer to monthly data (* if you have it *) in /var/adm/sa/sa##. And use :

sar -d -f /var/adm/sa/sa10 (* for 3/10 *)

-or-

sar -d 5 5 in a cron every fifteen minutes and outputed into a file. Look for disk entries where avwait is greater than avserv. Use 'pvdisplay' to id the file system on the disk.

avwait Average time (in milliseconds) that transfer requests waited idly on queue for the device;

avserv Average time (in milliseconds) to service each transfer request (includes seek, rotational latency, and data transfer times) for the device.

Look into fragmentation. If you have online JFS you should be defragging once a week.

fsadm -F vxfs -d -D -e -E /filesystem

Note: Once started don't interrupt or kill. If too long then read man page under number of passes and how to reduce the number of passes.

As for sockets, 'lsof' (* list of open files *) provides you with part of the information. You'll also neet 'netstat' to cound collisions, etc. Or, the GNU HP-UX version of the Sun command 'snoop'.

'lsof' is also a GNU command. Here's 'lsof'.

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/

As for a 'snoop' equivalient. look into 'tcpdump', tcptrace, and tcpflow which can be found here:

http://hpux.cs.utah.edu/hppd/auto/ia64-11.31-T.html

[MohitAnchlia]

I am trying to relate my file system to the devices that I get in sar output. How can I use pvdisplay or lvdisplay to list all the devices in sar. I tries pvdisplay and also pvdisplay but I don't get the information.

I was reading about "Directory Name Lookup Cache" on http://docs.hp.com/en/5576/JFS_Tuning.pdf. How can I verify if this is the problem ? And could this be increased ? Is it advisable to increase this cache ? Since client application reads the file name and then opens it and writes it to the socket I was thinking probably fast lookup from DNLC could help.

[MohitAnchlia]

Could somebody reply to my questions.

[Tim Nelson]

lvdisplay will show you the device files used for each logical volume.

e.g.
lvdisplay -v /dev/vgabc/lvol1

or the reverse.

pvdisplay /dev/dsk/cxtxdx will show you what lvols the device is used in.

[OldSchool]

pvdisplay /dev/dsk/cXtYdZ -or-
pvdisplay -v /dev/dsk/cXtYdZ

where X, Y and Z come from from the sar output.

However, I suggest that you start with:

"bdf /your/filesystem" ......which will tell you what logical volume the filesystem resides in.

Then do a lvdisplay of the logical volume...from above

"lvdisplay -v /dev/vgXXX/lvolXXX" where the logical volume is taken from the output of the prior "bdf" above. That should give you a list of the physical volumes used by the LV.

You can use the PV Names listed by the LV display to match to the output from SAR

[Michael Steele_2]

Sorry.

'pvdisplay -v /dev/dsk/c#t#d# | more'

This will list out several pages, you only need the first and second. The other pages will display PE (* physical extent *) data but isn't needed for this.

[MohitAnchlia]

When I do lvdisplay I get

$ /usr/sbin/lvdisplay /dev/vx/dsk/app1dg/app1vol02
lvdisplay: Illegal path "/dev/vx/dsk/app1dg".
lvdisplay: Cannot display logical volume "/dev/vx/dsk/app1dg/app1vol02".

I took "/dev/vx/dsk/app1dg/app1vol02" this from bdf.

Also, if someone can help me understand how to diagnose if DNLC is a problem. And if increasing this will help.