Need advise on how to SUM a series of numbers in a file

[Charles Fear]

I have a script I am working on that

1. Looks at a directory and does an ls-AlogR
2. Parses the results and sorts by file extension and saves the output from each grep to a new file with only files containg that extention.
3. Parses those entries and strips out everything but the file size.

************************
4. Here is where I am stuck. Now I want it to take this file with nothing but numbers in it and tell me what thier total is.
************************

In case you are curious why I would do all this. We have a Network Appliance that our SRM tool HP OV SAM does not do a very good job of reporting on. I basically need to tell the business how much storage space is consumed by files of a certain type... {ie Unigraphics files, etc}

I have attached a copy of the script... Any sugesstions would be appreciated.

[James R. Ferguson]

Hi:

You can always use 'awk' to read and sum numbers in a file:

# awk 'BEGIN{N=0};{N+=$1};END{print N}' filein

This example assumes 'filein' contains the numbers to sum in the first column ($1) of each record.

Regards!

...JRF...

[A. Clay Stephenson]

There are many ways to do this, awk, Perl, or the shell (POSIX or Korn):

NUMFILE="/tmp/numbers.txt"
typeset -i TOT=0
cat ${NUMFILE} | while read X
do
TOT=$(( ${TOT} + X ))
done
echo "Total = ${TOT}"

Note: This will only work if you are summing integers; otherwise we need to use another approach:

Here's an awk exemple

create a small file, my.awk (or build it on the fly in your script)

BEGIN {
tot = 0
}
{
tot += ($1 + 0)
}
END {
printf("%8.2f\n",tot)
}

TOT=$(awk -f my.awk ${NUMFILE})
echo "Total = ${TOT}"

[John Meissner]

number=$(cat file | awk '{print $1}')
e=0
for i in $number
do
((e=e+$i))
done
unset number

echo "$e is the total ammount"

[Charles Fear]

I ran that and got back what looks to be a calculation error
---------Start------------
awk 'BEGIN{N=0};{N+=$1};END{print N}' mp3.list.size

1.47033e+07
----------end--------------

The file I used contains the following numbers..... I don't know much about awk.... I need to grab my sed/awk book and learn a little.

------------start----------------
2872154
5838848
318384
294624
666576
648000
746496
231552
358992
399600
638064
832032
266544
396144
195264
-------------end---------------

Thanks

[James R. Ferguson]

Hi Charles:

That's not a calculation error, but rather 'awk' representing the output in scientific notation.

1.47033e+07

...is 14,703,330

or 1.47033 times 10 raised to the seventh power.

If you prefer, use this:

# awk 'BEGIN{N=0};{N+=$1};END{printf "%-12d\n",N}'

You will see the result of your file is:

14,703,274

which is the accurate result we obtained in rounded format above. The 'printf' does the formatting we want (here up to 12-digis, left-justified).

Regards!

...JRF...

[Shannon Petry]

I just wanted to add that you can do this pretty much on the fly by using a slightly different method.

ls -la |awk '{ s += $9;printf(....)}'

With just one more variable in awk, you could print the size of the item, name, and running total. :P I like Awk.

I found however that C was much much faster at doing this. If I can find the code I wrote to do this about 6 years ago, I'll post it. It compiled just fine on the standard K&R compiler.

Since I moved 2 times, and the company was bought out just as many, I may not be able to find it though.

Regards,
Shannon

[Charles Fear]

Thanks for the many sugesstions. I have tried a few of them and I seem to be running into an upper limit on the calculations. Is anyone aware of a limit to the awk method mentioned in James Furgusons relpy. I get a several of them that seem to cap out at 2147483647

Any ideas

[Shannon Petry]

There should be no limit that I can think of, but here is a thought.

When you look at the size of the file, it is always in bytes. Why not to a fast conversion before doing the math?

I.E.

ls -l | awk '{ s = ($5/1024) ; t+= s ; printf(...)}'

This would get the number size down a bit, so maybe help some??

Regards,
Shannon

[Charles Fear]

Also to note I tried John's method as well and I got a number with a "-" in front of it for many of them. I don't know if that is an indicator of an upper limit as well or not.

[A. Clay Stephenson · Accepted solution]

That is the signed integer limit. Change the %12d to %12.0f in the printf and you should be ok because now it will do floating point math.

[James R. Ferguson]

Hi (again) Charles:

The decimal number 2147483647 is 0x7FFFFFFF which is the largest integer. If you change the 'printf' specification to floating point your large sums will work:

# awk 'BEGIN{N=0};{N+=$1};END{printf "%-12f\n",N}' filein

Regards!

...JRF...

[Charles Fear]

o.k.

This forum has worked out great. Thanks for all the input.

Now I have to break open the awk book and break down the command you have given me. It seems to have worked with replacing "%-12d" with "%-12.0f".

Now I just need it to print a variable such as "$name" on the same line an I am set to go. Again... thanks to all of you.

Once I clean the script up I will post it for others who may have a need.

[A. Clay Stephenson]

That's quite easy as well:

modify the printf to something like "%s %12.0f\n", myvar, N

awn when you invoke awk add the -v argument to define a variable. e.g.

awk -v "myvar=${name}" '{ ....... }'

[James R. Ferguson]

Hi Charles:

...just change the 'printf' statement to something like:

printf "Sum = %-12.0f\n",N

# awk 'BEGIN{N=0};{N+=$1};END{printf "Sum = %-12.0f\n",N}' filein

Regards!

...JRF...

[Charles Fear]

Here is the section of the script that I am having a little bit of trouble with?

----------start-----------------

for i in `cat $SOURCEdir/FILETYPE_list`
do
cat $OUTdir/$i.list | sed -e 's/ / /g' | sed -e 's/ / /g' | sed -e 's/ / /g' | sed -e 's/ /,/g' | cut -d"," -f3 > $OUTdir/$i.size

awk 'BEGIN{N=0};{N+=$1};END{printf "$i = %-12.0f\n",N}' $OUTdir/$i.size >> $OUTdir/srm.final

done

-------------end---------------------

What I want it to do is print out the variable of $i and then the file size on the same line like such:

mp3 = 123456789
pst = 234567891
etc....

What I am getting with this syntax is

$i = 123456789
$i = 234567891

If I change it so the $i is before the " I get several errors.

[Sridhar Bhaskarla]

Hi charles,

You are trying to pass a shell variable into awk. You will need to use "-v" switch of awk to do it.

awk -v "var=$i" 'awk 'BEGIN{N=0};{N+=$1};END{printf "%s = %-12.0f\n",var,N}' $OUTdir/$i.size

If you put var under "", then it will be evaluated as a constant instead a variable.

-Sri

[James R. Ferguson]

Hi (again) Charles:

You want to do this:

Instead of:

# awk 'BEGIN{N=0};{N+=$1};END{printf "$i = %-12.0f\n",N}' $OUTdir/$i.size >> $OUTdir/srm.final

...do:

# awk -v i=$i 'BEGIN{N=0};{N+=$1};END{printf "%s = %-12.0f\n",i,N}' $OUTdir/$i.size >> $OUTdir/srm.final

Regards!

...JRF...

[A. Clay Stephenson]

This is why I suggested that you build an awk script; it's much less messy looking than quoting everything --- especially if you are an awk novice. When you get more proficient then you can start quoting in-line awk.

If the awk input looks like this:
mp3 12345
pst 23456

Then in awk mp3 is $1, and the size is $2. Your modifications need to go in current the {N += $1} section

BEGIN{...}{printf("%-12.12s %12.0f\n",$1,($2 + 0)); N += ($2 + 0)}END{ ... }

NOTE: What was $1 is now $2.


The "%-12x" LEFT justifies while "%12x" RIGHT justifies. The other awkism that keeps awk working in everycase is the += ($2 + 0) idiom rather than simply += $2. Adding zero forces an unambigious numeric context rather than a possible string context.