find growing files

[arkie]

Hi,

One of the lvols in a production server is 99% full. How can I find the files which have recently grown inside that lvol

thanks in advance

[Jupinder Bedi]

see this doc , it is very helpful for find command with lot of options

http://www.scripterworld.com/2009/07/unix-find-command-with-examples-and.html

[arkie]

Hi Jupinder,

This is a very good link. But I am not getting the appropriate command for this scenario.

I would just like to ask what should be the approach when we find that some lvol of a data VG are getting used unusually fast. In last 3 days the %used has gone to 99% from 98%. If,

find .|xargs ls -l|sort -r -n -k 5,5|head -10

shows the 10 largest files

&&

find . -type f -mtime -3

gives the files that have been modified in tthe last 3 days,

but the results are contradicting:-
1) 10 largest files' timestamps are very old
2) files modified in the last three days are rrather not too big to be significant

[R.K. #]

Hi Sankar,


>>>>
1) 10 largest files' timestamps are very old
2) files modified in the last three days are rrather not too big to be significant
>>>>

Nothing wrong in this. The file modified in last 3 days might not be the largest one. Similarly the file that is the largest one , might not be modified recently.

[R.K. #]

Please tell us more about the logical volume.

What is this LV used for?

# bdf /lv
# du -sk /lv

Is there is any significant difference between above values?

[Dennis Handly]

>but the results are contradicting:

Not contradictory, just unfortunate. If there was some overlap, your job would be done.

If you can't find the growth in 2), you are out of luck for easy solutions.

As RK says, perhaps you have some open unlinked files that are growing? Any big files you have removed lately that were still open?

[arkie]

Hi R.K.,

# bdf /dev/vg04/lvol10
Filesystem kbytes used avail %used Mounted on
/dev/vg04/lvol10 3145728 3116920 28808 99% /oracle/N11/920_64

# cd /oracle/N11/920_64
# du -sk
2475320 .

Yes, looks to have significant difference
3116920 - 2475320

This lvol is used for housing SAP executables

Recently, there was only 1 /oracle/N11/920_64/network/log/listenser.log file, whose older contents were removed to shrink the file size from about 18M to 1.9M at present. This was done by the SAP admins as this activity had helped in the past

But this time, its different

[arkie]

Hi Dennis,

Yes exactly,

(2) is not showing any growth

Recently, there was only 1 /oracle/N11/920_64/network/log/listenser.log file, whose older contents were removed to shrink the file size from about 18M to 1.9M at present. This was done by the SAP admins as this activity had helped in the past

But this time, in the last 3 days usage has gone up to 99% from 98%.

[Dennis Handly]

>looks to have significant difference: 3116920 - 2475320

This is 650 Mb. You'll need to use lsof to find these unlinked files.

>in the last 3 days usage has gone up to 99% from 98%.

This is a trivial 30 Mb.

>whose older contents were removed to shrink the file size from about 18M to 1.9M

How was this "removal" done?

[arkie]

Hi Dennis,

I am halfway stuck here. I have to install the lsof utility and its a production server.

The file that was shrunk earlier was FTP'd to a window m/c, edited, part contents deleted and kept back, as vi was unable to open the file at source.

[Dennis Handly]

>I have to install the lsof utility and it's a production server.

You don't have to reboot.

>part contents deleted and kept back

Has the file changed at all since that was done?

[arkie]

The file changed from approx. 18Mb to current 1.95Mb.

Regarding lsof, where can I download the appropriate utility for HP-UX B.11.11. What can be the risks involved.

[R.K. #]

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.82/

As said, it do not require reboot, you can go ahead and install it.

[Dennis Handly]

>The file changed from approx. 18Mb to current 1.95Mb.

Has it been modified since you manually edited it?

[arkie]

No the file has not been modified in the last 4 days.

Also, when trying to install lsof, I am getting the following error:-

# swinstall -s /tmp/lsof/lsof-4.82-hppa-11.11.depot \*
WARNING: Cannot lock "/var/adm/sw/queue/number.lck" because another
command holds a conflicting lock. The process id of that
command is -1.
ERROR: The attempt to create the job failed. (Internal error)

ERROR: Command line parsing failed.

[R.K. #]

Hi..

>>> WARNING: Cannot lock "/var/adm/sw/queue/number.lck"

Another SD command is running that prevents the swinstall or swremove command from running. Wait for that command to finish and try again.

You can also check from "ps -ef" if any SD command is running.

[arkie]

Checked the /var/adm/sw/swagentd.log file and found following excerpts:-

...
ERROR: Cannot lock depot/root at "/" due to fcntl() error "ENOLCK".
If the soc is on a remote NFS file system, the NFS locking
facility is probably inactive or having problems. If it is on
a local file system, this indicates too many locks are already
in use and no more are available.
...

Is it likely because this is a node of a 2-node cluster

[Bill Hassell]

Looking for big files or looking for recent files doesn't work that well. You are looking for changes in disk space so start with the largest directories. Use this:

# du /oracle/N11/920_64 | sort -rn | head -20

Those top 20 directories are the most important. Should each one be that large (you need to know what the directory is supposed to contain)? If one of the directories seems too large then sort the files in that directory by size:

# cd /oracle/N11/920_64/BIG_directory
# ll | sort -rnk5 | head -20

This technique finds those pesky directories where all the files are less than 1 MB but there are (unexpectedly) thousands of them.

To watch this directory, run the du command above and see what is growing. Note that log files can be removed but the space won't change until the process that had the logfile open closes the file.