topic Re: computing limitation error - long and double in Operating System - HP-UX

computing limitation error - long and double

deCG — Mon, 29 Sep 2008 21:13:58 GMT

I am writing a shell script to compute really long numbers. But due to limitation, it is giving out wrong numbers.

I tried 'double' and 'long' as variable type but none of them are valid type.

This is output.

2847744950/3234344960 = 5295336 (huh??)

Does anybody have any clue???

Thanks a bunch.

Yong

Re: computing limitation error - long and double

James R. Ferguson — Mon, 29 Sep 2008 21:22:28 GMT

Hi:

The shell limits you to 32-bit arithmetic. Use something like Perl or 'bc'. For example:

# perl -le 'print +(2**52)'
4503599627370496

# echo 2^52|bc
4503599627370496

Regards!

...JRF...

Re: computing limitation error - long and double

Dennis Handly — Tue, 30 Sep 2008 05:02:35 GMT

>JRF: The shell limits you to 32-bit arithmetic.

ksh was changed in 11.23 to use 64 bit ints. Unfortunately the posix shell wasn't changed.

You can use awk to do calculations in double.
And gdb to do calculations in long double.
(gdb) p (long double)2847744950/3234344960
$2 = 0.88047038433401983194767202568275215

Re: computing limitation error - long and double

deCG — Tue, 30 Sep 2008 06:02:52 GMT

You guys are great... I will get back to you when I try at work...

Thanks

Re: computing limitation error - long and double

deCG — Tue, 30 Sep 2008 06:03:33 GMT

.... ....

Re: computing limitation error - long and double

deCG — Wed, 01 Oct 2008 12:02:04 GMT

thanks. it works now...