topic Re: Finding files to trim in Operating System - Linux

Finding files to trim

TheJuiceman — Tue, 17 Jan 2006 19:28:36 GMT

Hey gang,

/ on one of our systems is slowly growing. I have a two part question to ask the group....

1) Does anyone have a script that will look in / and find and list the largest files in / excluding all other mounted volumes? A script that would be portable or would require no editing as volumes are added/deleted would be preferred.

2) I was wondering if anyone had a script that would look for files to trim or delete on / or various other volumes (like /var or /opt). Is there a list of files or filetypes to look for that would qualify for occasional purging? This is for 11.11 systems.

Thanks for your help.

Re: Finding files to trim

James R. Ferguson — Tue, 17 Jan 2006 20:01:35 GMT

Hi:

The following script will list (and sum) the size of all directories except mounted ones:

# ls -il / | perl -nae 'if ($.>1 && $F[0]!=2 && $F[1]=~/^d/) {$v=`du -ks $F[9]`;print $v;@s=split(/\s+/,$v);$t+=$s[0]};END{print "$t\n"}'

This code relies on the fact that a directory that is a mountpoint will *always* have an inode=2. Remember, inode values are only unique within a filesystem.

Another way to list directory sizes is :

# du -xk /var|sort -k1nr|more

...This will restrict 'du' to reporting directory size in the order largest-to-smallest beneath a mountpoint.

The idea of both the above is to identify directories which have significant space.

As for looking for "large" files you could do something like:

# find /pathname -xdev -type f -size +10000c -exec ls -l {} \; | sort -k5nr

This would find all files in /pathname (without crossing mountpoints) whose size exceeds 10,000 characters. The output would be a stnadard 'ls' file listing sorted from largest to smallest by size.

As for trimming and/or deleting files in '/var/ and /'opt', the place to concentrate is '/var'. '/opt' should be static. See the manpages for 'hier(5)' for details.

That being said, files in:

/var/preserve --- are old 'vi' recovery files and if old can be deleted.

/var/adm/sw --- do *NOT* manage this manually. The *only* safe way to reclaim space is to use 'cleanup'. To do otherwise you run the risk of corrupting your Installed Product Database and you will not be able to patch your server again!

/var/adm/wtmp, /var/adm/btmp --- these are logs of successful and unsuccessful logins respectively. Trim them to zero by redirecting '/dev/null' to them or trim them with 'fwtmp' by converting them to ASCII to edit them and reconverting them to binary after they have been trimmed.

/var/adm/crash --- should only have unanalyzed crash dumps from high-priority machine checks. If you find something here that's old you can delete it unless you want HP Suppport to analyze it.

/var/adm/mail --- have your users cleanup their mail folders and this won't be a problem.

Regards!

...JRF...

Re: Finding files to trim

TheJuiceman — Tue, 17 Jan 2006 22:30:12 GMT

Hey James,

Wow...what a great response!!! That deserves 20 points!!!

I have another question along these lines...

I have a file that contains a list of directories. I want to get a sort these files by size so that the largest file is displayed first. Something like this...

find /{this is where I want to check the list of dirs in my file) -depth -type f -exec ll {} \; >> /tmp/file1

sort -r -n -k 5,5 /tmp/file1 > /tmp/file2

How can I write this so that it will go through my file and do the find on each of the entries in the file so I can then do the sort?

Thanks for the help.

Re: Finding files to trim

RAC_1 — Tue, 17 Jan 2006 22:36:09 GMT

There are some simple commands that will do the work. You need to be careful about trimming the files. you should understnad what are those files and what processes might be accessing them.

du -kx / | sort -n --> Will list dir sorted in ascending order

Once you note the dir that are using most of the space, run du on them again to see what is holding the space.

Re: Finding files to trim

TheJuiceman — Tue, 17 Jan 2006 23:16:22 GMT

Is there way to get a sort of just the files? Here is what I'm trying to do...

I have a file...FILE1 that contains the following...

entry1
entry2
entry3

I want to run a find on the entries in the file so I can collect and sort the output.

Also getting all the information that a ls -l gives would help tremendously.

Re: Finding files to trim

Arunvijai_4 — Tue, 17 Jan 2006 23:23:51 GMT

# du -skx /usr/* | sort -nr | head -10

(List the top 10 files)

# ls -lR | sort +4 -5nr | more

(List all files starting with biggest)

-Arun

Re: Finding files to trim

TheJuiceman — Wed, 18 Jan 2006 00:49:20 GMT

I REALLY REALLY need this to do the find on the entries in the file. How can I do that? Thanks.

Re: Finding files to trim

RAC_1 — Wed, 18 Jan 2006 01:00:12 GMT

for i in $(do
find ${i} -exec ll -d {} \; | sort -nrk5
done

Re: Finding files to trim

Devender Khatana — Wed, 18 Jan 2006 01:04:19 GMT

Hi,

#du -xk `cat /home/hp/file1` |sort -rnk1 |pg

Where /home/hp/file1 is the file and you are executing the command at any location. The contents of the file are with absolute path like

/var
/opt
/home

HTH,
Devender

Re: Finding files to trim

James R. Ferguson — Wed, 18 Jan 2006 07:46:06 GMT

Hi (again):

Perhaps this is what you are seeking (based upon your last post, above):

cat filelist.sh
#!/usr/bin/sh
while read LINE
do
find ${LINE} -xdev -type f | xargs ls -l
done < filelist | \
sort -k5nr
exit 0

...The 'filelist' is a simple file with directories of interest listed one per line. For example:

# cat filelist
/etc
/sbin
/stand

Note that you don't have to use any temporary files (only pipes) and that I have avoided the more expensive '-exec' with 'find' which would spawn a separate process for every 'ls' request. The 'xargs' bundles multiple requests and greatly reduces subprocess initiation. The 'find' traversing directories is brutal enough on any system!

Regards!

...JRF...
Regards!

...JRF...

Re: Finding files to trim

Devender Khatana — Wed, 18 Jan 2006 07:49:42 GMT

Hi,

Apart from this I always use the SAM to trim the log files. This I think is the best way to this, as it also shows you the existing size of all files in one windows itself.

HTH,
Devender

Re: Finding files to trim

Yogeeraj_1 — Wed, 18 Jan 2006 08:01:42 GMT

hi,

if you go in SAM, you will get a few system logfiles that you can trim using the same interface. However, the list exclude maybe larger log files that could be generated by application programs.

beware that there are some files that you cannot just delete! You must document yourself fully before doing so. E.g. Oracle Redolog files cannot be just deleted!!

hope this helps too!

kind regards
yogeeraj

Re: Finding files to trim

Ted Buis — Wed, 18 Jan 2006 10:58:31 GMT

It is very easy to add files to the list in SAM that can be monitored and trimmed. HP can know which application files you have that may need to be monitored.

Re: Finding files to trim

TheJuiceman — Wed, 18 Jan 2006 18:31:22 GMT

Thank you all for the help!!! I was able to get what I needed done!!!

Re: Finding files to trim

Arturo Galbiati — Thu, 19 Jan 2006 04:03:22 GMT

Hi,
cat your_file|xargs -i find . -name {} -exec ll {} \;

this will find the files contained into your_file.

HTH,
ART

Re: Finding files to trim

TheJuiceman — Thu, 05 Oct 2006 21:54:47 GMT

thanks