Q about huge filesystem handling

[Gordon_3]

Hi all,

I come across quite a standard headache problem here and wanna to listen any good suggestions on how to handle this. Our environment has deployed the NAS techonlogy here, which are "widely" NFS shared by big number of Unix hosts, and no doubt the file system size is growthing bigger & bigger. If I want to write a script to report those "ascii" file size, how should we tackle this, those cmd come up to my mind a.t.m. like "find / -name "log"" , file , ls -l etc etc, but considering the FS size, don't think it's feasible, any "advanced" function I can use to get those info?

[saravanan08 · Accepted solution]

for file system size

use command
# bdf

for file and directory size u can use
#du

for file size

for example if u need names of file size whose size is greater than 100 bytes
use
find . -size +100 -print

i think this might b useful for u

thank u

[Gordon_3]

Hi, yeah I also think of those cmd, but the tricky part here is I need to traverse the whole FS starting from "/" , ( maybe by find ), and then base on the list, then maybe "grep" those name with "log" and do a "ls -l" on that,however as the size we talk about here is very big, I guess > 100k of files and in terms of "TB" of size, so those normally Unix cmd like "find" etc may hit issue for

1. Run for a month or even year to finish :)
2. The internal buffer of the cmd will get full during execution of such huge list

[Steven Schweda]

> 2. The internal buffer of the cmd will get
> full during execution of such huge list

Avoiding problems like that is what "find" is
good for.

You might start by writing a script to do
what you need done, and testing it on
something smaller than the entire universe.
That should give you some idea how long it
would need to work on the whole thing. Then,
if it really is too slow, you can worry about
making it faster.

> [...] any "advanced" function I can use to
> get those info?

I don't know what an '"ascii" file size' is,
so it's not entirely clear to me what,
exactly, "those info" means.

[Gordon_3]

HI Steven,

Thx, for ascii file, what I refer to is like below

file < a file >
rc: ascii text

Coz my ultimate goal is to count the size of those ascii text files, and make an estimation on how many % can be saved by "gzip" it. ( Binary is excluded )

[James R. Ferguson]

Hi Gordon:

If I understand correctly, you want to examine every file under the root ('/') file system looking only for ASCII files (i.e. non-binary, "text" files) and report their names and character sizes.

If that is all you want, Perl does this effortlessly:

# perl -MFile::Find -e 'find(sub{printf "%10d %s\n",(stat($_))[7],$File::Find::name if -T $_},@ARGV)' /

Notice that I passed "/" as the argument to the script. You can specify any number of directories (e.g. "/var /usr /apps") or you can 'cd' to a directory and simply specify "." for the current directory argument.

Tranversing the entire filesystem ("/") isn't inexpensive, but if that's your requirement the above will perform about as well as you can expect without limitations.

Regards!

...JRF...

[Steven Schweda]

> Coz my ultimate goal is to count the size
> of those ascii text files, and make an
> estimation on how many % can be saved by
> "gzip" it.

The compression fraction you'll get from gzip
(or any other program) will depend on what's
in the file(s) to be compressed. Estimating
without actually compressing some typical
files may be quite inaccurate.

> ( Binary is excluded )

Why? Binary files are often quite
compressible. Again, actual tests are more
reliable than my guesses.

[OldSchool]

the downside is that, as this is disk that is NFS shared with a number of servers, you may have little idea of the impact this would have on other users or applications were you to simply look for every "ascii" file and start zipping them.

[James R. Ferguson]

Hi (again) Gordon:

For completeness and safety, I should have explictly limited your searches to regular files, although in most cases the test for "text" files would have returned that subset. This avoids any problems if you scan device files:

# perl -MFile::Find -e 'find(sub{printf "%10d %s\n",(stat($_))[7],$File::Find::name if -f $_ && -T _},@ARGV)' /path

Regards!

...JRF...

[Gordon_3]

HI James,

Thx for yr gr8 advice agai ( and thx all also ), btw I did try to run on 1 of the comparable small volumes, it almost iterate a day to finis it :)

However the difficult situation is the output is too long, and even using a Excel cannot open it ( too many listing as u can expect ), so if I want to further "limit my scope" to filer those not updated for 1/2 year by timestamp, how to tune yr Perl script ? Thx. ( sorry I am a novice on Perl ) :)

[Dennis Handly]

>filter out those not updated for 1/2 year by timestamp

For find(1), you can use: -mtime -183
This only does the ones modified in the last 1/2 year. +183 for the reverse.

[James R. Ferguson]

Hi (again) Gordon:

> filter out those not updated for 1/2 year by timestamp

By this, I assume that you want files that are older than about 183 days. Simply do:

# # perl -MFile::Find -e 'find(sub{printf "%10d %s\n",(stat($_))[7],$File::Find::name if -f $_ && -M _ > 183 && && -T _},@ARGV)' /path

The '-M' has units of days and represents the last modification timestamp ('mtime'). Fractional days are allowed and you can compare '>=', '>', '==', '=<' or '<' to a value.

By the way, as a professional forum, please use professional English and skip the chat-room, instant-message abbreviations.

Regards!

...JRF...

[James R. Ferguson]

Hi:

Oops! I have an extra "and". The last post should be:

# perl -MFile::Find -e 'find(sub{printf "%10d %s\n",(stat($_))[7],$File::Find::name if -f $_ && -M _ > 183 && -T _},@ARGV)' /path

Regards!

...JRF...

[Hein van den Heuvel]

>> However the difficult situation is the output is too long, and even using a Excel cannot open it ( too many listing as u can expect ),

So don't let a billy-boy tool do man's job!

Do you really want to know those file names?
WHY would you stick this stuff in excel if you hpux script can just properly report all you want to know right there.

Just change the perl script graciouisly offered and make it add up instead of report files.

Something along the lines of (untested)...

use File::Find;
use strict;
my ($files, $target_files,$bytes)
find ( sub{
$files++;
if ( -f $_ && -M _ > 183 && -T $_) {
$bytes+=(stat($_))[7];
$target_files++;
}
},shift);
printf "$bytes bytes in $target_files/$files files.\n"

With that in place it will be trivial generate an array of files and bytes based on age: over a year, 1/2 year - 1 year, less than 1/2 year.

It will also be easy enough to group those text files in for example '.log' and '.csv' and 'others' to be able to use better emperical compression data points.

Of course you want that perl find subroutin to become a proper, labeled, subroutine if you add any more complexity.

Good luck!
Hein.

[Bill Hassell]

Are you sure that the effort is worth the savings in disk space? Unless the ASCII files are in the millions and the files themselves are quite large (megabytes), the effort to constantly scan and compress a massive NFS filesystem will probably destroy the performance of the NAS product. AS you have already seen, the effort is tedious and slow...add the reading, computing and rewriting for millions of files and your NAS appliance will make floppies look speedy.

Or you can take the approach that every file stored is important and must never be removed, so just keeping adding more terabytes. It would certainly improve the performance of the system.

Personally, I would require each Unix system administrator to manage their disk space as if it were not shared. In Unix, ordinary users are restricted to their home directories and /tmp, /var/tmp to create files. Simple quotas resolve home, and the tmp directories are, well, temporary. Just slash and burn old files and oversize files. Until disk costs are measured in a few dollars per petabyte, managing disk space will still be the sysadmin's responsibility.