topic Re: lsof to find worst offender of open files in Operating System - HP-UX

lsof to find worst offender of open files

Tim Nelson — Wed, 11 Jul 2007 13:13:18 GMT

I am investigating a recent issue with vx_iget inode table overflow messages and intend on upping my vx_ninode setting this weekend but wanted to investigate who was the worst offender of open inodes.

Looking for help to find a combination of switches in lsof that would help ID the worst offender or any other ideas as well.

HPUX 11.11

Thanks to all !!

Re: lsof to find worst offender of open files

Dennis Handly — Wed, 11 Jul 2007 13:19:36 GMT

I'm not sure lsof is set up to do that? You'll probably have to do some scripting on the output of all open files on a file system. Create tuples of PID and file, then sort by PID.

What would you do with that output? Figure out who you can kill/delay until you change it?

Re: lsof to find worst offender of open files

Tim Nelson — Wed, 11 Jul 2007 13:26:54 GMT

That is the idea. If I can releive the pressure until saturday's reboot it would help.

Re: lsof to find worst offender of open files

A. Clay Stephenson — Wed, 11 Jul 2007 14:09:37 GMT

Well, here's a script that should do it:

#!/usr/bin/sh

lsof | awk '{if (!($0 ~ /^COMMAND/)) {print $2, $1}}' | uniq -c | sort -rn

If my logic (and syntax) is correct this should print 1 line for each process listing the number of file descriptors, the PID, and the process name sorted in reverse order from the biggest file descriptor hog to the smallest.

I'm willing to bet that some of the biggest file descriptor hogs are also your most critical applications.

Re: lsof to find worst offender of open files

Tim Nelson — Wed, 11 Jul 2007 14:16:04 GMT

Thanks Clay.

I pulled the lsof into excel then created a pivot table to count the occurances.

About the same as you suggested.

Using the output to compare agaist some other systems not having issue to see if I can find some really large differences.

This is a case of pulling down vx_ninode too far to save some memory has caused an issue. 8000 descriptors was enough initially but we have since grown. I am setting vx_ninode now to NFILE to cover the contingency.

Re: lsof to find worst offender of open files

A. Clay Stephenson — Wed, 11 Jul 2007 14:25:07 GMT

I'm not familiar with the UNIX excel command and pivot tables. This must be some non-standard utilty that you have developed.

Re: lsof to find worst offender of open files

Tim Nelson — Wed, 11 Jul 2007 14:35:25 GMT

U2 Funny.

Thanks for the tips..