topic Re: more scripting help on calculations in Operating System - Linux

more scripting help on calculations

lawrenzo — Wed, 14 Jun 2006 04:38:13 GMT

Hi all,

I have a filesystem where the data is all over the place and no archiving or housekeeping has happened for some time. I have a definitive list of all files / data users etc.

#!/bin/ksh

# This script will check a file where find . -xargs |ls -ld |sort -rnk 5 > usrdump.lst
# is run and the definitive list has been created.

# Create list of unique users - you can store this in an array, or better in a temp file.

cat usrdump.lst | tail +1 |awk '{print $3}' |sort |uniq > usrlist.log

# for each user in list above print file size and add it up

for usrname in `awk '{print $1}' usrlist.log`
do
echo " checking user $usrname"
filesize_count=0
for filesize in `grep "$usrname " usrdump.lst |awk '{print $5}'`
do
filesize_count=$filesize_count+$filesize
done
echo "$usrname\ttotal\t$filesize"
done

the problem I have is that the usrdump file is over 120mb and the system runs out of memory:

--> ./filesystemcounter.sc
checking user cronlog
./filesystemcounter.sc[15]: 0403-029 There is not enough memory available now.

there is 16gb of memory in the system.

Anyone have a better solution to calculate the filesystem usage for all users accessing the FS?

Many Thanks again.

Re: more scripting help on calculations

Robert-Jan Goossens — Wed, 14 Jun 2006 04:48:35 GMT

Hi Lawrenzo,

How about someting like this.

# du -ks * /home | awk '{ print int($1/1024+.5)" MB"" "$2}'

Regards,
Robert-Jan

Re: more scripting help on calculations

Ninad_1 — Wed, 14 Jun 2006 05:03:08 GMT

Hi,

You can do

for i in `awk '{print $1}' usrlist.log`
do
echo "$i uses \c"
find ./ -type f -user $i | xargs ls -l | awk 'BEGIN {tot=0} {tot+=$5} END {print tot/1024,"KB"}'
done

Regards,
Ninad

Re: more scripting help on calculations

lawrenzo — Wed, 14 Jun 2006 05:04:58 GMT

Thanks robert-jan this gives some idication on file sizes however there has been no housekeepig policy for some time here and everything is a bit f a mess. Some users have no idea of files dumped in the area because it is done through an application.

I have the lsit of all files

# find . -xargs |ls -ld |sort -rnk 5 > usrdump.lst

however I need to then name and shame the users who have the most fs usage and I want to use the sript to prove this.

Thanks

Re: more scripting help on calculations

Enrico P. — Wed, 14 Jun 2006 05:20:27 GMT

Hi,

for USER in `awk '{print $1}' usrlist.log`

do
echo " checking user $USER"

echo "$USER \c"; find . -type f -user $USER -xdev|xargs ll|awk '{ x += $5 } END { print "total bytes: " x }'

done

Enrico

Re: more scripting help on calculations

lawrenzo — Wed, 14 Jun 2006 05:22:58 GMT

ok this is great stuff however the search function onthe actual server will significantly increase the load.

I was hoping I could interogate the usrdump.lst file to work out the sizes.

is that possible also?

TY

Re: more scripting help on calculations

Ninad_1 — Wed, 14 Jun 2006 05:45:56 GMT

I dont think it is a seperate load on your system compared to the find you will be running to get the userdump.lst file.
Any how try this

for user in `awk '{print $1}' usrlist.log`
do
echo "$user uses \c"
grep $user usrdump.lst | awk 'BEGIN {tot=0} {tot+=$5} END {print tot/1024,"KB"}'
done

Regards,
Ninad

Re: more scripting help on calculations

James R. Ferguson — Wed, 14 Jun 2006 07:11:41 GMT

Hi Lawrenzo:

If you want to quickly summarize the utilization (total characters) by user within any filesystem, I'd use Perl. Something like this should do:

# cat ./userutil
#!/usr/bin/perl
#@(#)userutil $ Summarize by user total file sizes - JRF $
use strict;
use warnings;
use File::Basename;
use File::Find;

my %util;
my ($uid, $size, $name);
my $path = @ARGV ? shift : ".";

sub wanted {
return unless -f;
($uid, $size) = ((stat($_))[4,7]);
return unless $uid > 10;
$util{$uid}+=$size;
}

File::Find::find(\&wanted, $path);
for $uid (keys %util) {
$name = getpwuid($uid);
$name = $uid unless defined $name;
printf "%8s %10d\n",$name,$util{$uid};
}
1;

Users whose 'uid' is less than ten (10) are skipped. This eliminates files owned by "root" and "bin", in particular.

Only files, not directories, are examined. Run the script passing a directory (filesystem), e.g. "/home":

# ./userutil /tmp
spfmweb 175
oper 5026
jrf 48

Regards!

...JRF...