lsof to find worst offender of open files

[Tim Nelson]

I am investigating a recent issue with vx_iget inode table overflow messages and intend on upping my vx_ninode setting this weekend but wanted to investigate who was the worst offender of open inodes.

Looking for help to find a combination of switches in lsof that would help ID the worst offender or any other ideas as well.

HPUX 11.11

Thanks to all !!

[Dennis Handly]

I'm not sure lsof is set up to do that? You'll probably have to do some scripting on the output of all open files on a file system. Create tuples of PID and file, then sort by PID.

What would you do with that output? Figure out who you can kill/delay until you change it?

[Tim Nelson]

That is the idea. If I can releive the pressure until saturday's reboot it would help.

[A. Clay Stephenson]

Well, here's a script that should do it:

#!/usr/bin/sh

lsof | awk '{if (!($0 ~ /^COMMAND/)) {print $2, $1}}' | uniq -c | sort -rn

If my logic (and syntax) is correct this should print 1 line for each process listing the number of file descriptors, the PID, and the process name sorted in reverse order from the biggest file descriptor hog to the smallest.

I'm willing to bet that some of the biggest file descriptor hogs are also your most critical applications.

[Tim Nelson]

Thanks Clay.

I pulled the lsof into excel then created a pivot table to count the occurances.

About the same as you suggested.

Using the output to compare agaist some other systems not having issue to see if I can find some really large differences.

This is a case of pulling down vx_ninode too far to save some memory has caused an issue. 8000 descriptors was enough initially but we have since grown. I am setting vx_ninode now to NFILE to cover the contingency.

[A. Clay Stephenson]

I'm not familiar with the UNIX excel command and pivot tables. This must be some non-standard utilty that you have developed.

[Tim Nelson]

U2 Funny.

Thanks for the tips..