awk arithmethic - calculation error Risc/UX11.11

[Tor-Arne Nostdal]

We discovered a calculation error in our SAP system. This happens only on Risc based machines while Itanium is ok (all HP-UX machines).
When running the small example below we get result:
s=9,9999999999999956E-01, x=2, y=3
on Risc machines (ux 11.11) while we get the correct answer on ia64 (ux 11.23):
s=0,0000000000000000E+00, x=3, y=3

ABAP (SAP native) program language
report zastest1 .
data: s type f.
data: t type f.
data: x type p.
data: y type p.
s = 1000.
t = 1000.
s = frac( log10( s ) ).
x = trunc( log10( t ) ).
y = log10( t ).
write: / 's= ', s.
write: / 'x= ', x.
write: / 'y= ', y.

NB! type p is an integer type

I would like to test this outside SAP and would need something similare to run as a shell/awk-script.

/Tor-Arne

[Hein van den Heuvel · Accepted solution]

Those results are the same, within the constraints of the datatype used.
Some (many!) numbers do NOT have an exact representation in floating point, there will be an error in the final bit.
The simplest, silly example of this is the classic 3*(1/3).
1/3 = 0.333333333...
3*(1/3) = 0.9999999999... or 1.0 ?

The exact hardware implementation of the floating point (ieee or native) defines how the cookie crumbles.

It is up to the application to select the right datatype and to appropriate rounding or truncation operations for that datatype.

Just round the log resulst to 10 digits or so and you'll see the same 'expected' result in both cases.

fwiw,
Hein.

[R. Allan Hicks]

I'm not sure how many signifcant digits are in SAP's floating point. One of the things that I noticed about machines from the old 16 bit days was that HP old 16 bit machines had less bits in the mantissa of the floating point number than the INTEL machines. So if the log10(10) to come out 2.999999999 would not be that suprising.

In the old FORTRAN days, floating point numbers with standard precision was only good to 6 digits.

For fun try the following example by hand....

Represent .1 as a floating point number.

So like the example of 1/3 you have a repeating fraction (this time a binary one). So as has been pointed out some numbers can't be represented in base 10 math and some numbers that you wouldn't normally suspect, cannot be represented in binary.

[Tor-Arne Nostdal]

Thank you both for the answer. It put me on the track, though it wasn't a direct answer to my question for an awk program where I could check the precision.

I found in SAP's documentation of "ABAP Numeric Types" a description of the danger of using floating point in such business transactions...

Here is some of their own brief statements of their datatypes:
1)
Rule: If you want to calculate "down to the last penny", you should use the type P.
P is a type with specified precision.
Type P is typically used for sizes, lengths, weights and sums of money.
2)
Use floating point arithmetic if you need a very large value range or you are making decimal calculations, but be aware of the following features of floating point arithmetic.
--- then there is a description who try to show what you already told me, though they explain it in a more complex way---.
3)
---And in there some warnings to the programmer---.
You should also note that multiplication using powers of 10 (positive or negative), is not an exact operation.
---and---
You should also note that multiplication using powers of 10 (positive or negative), is not an exact operation.

QED

From our business we got a report about a price error on approx. 200 Euro on 1 order...

We have created a temporary workaround until they come up with a fix.

The small program example was an extract of the original SAP std. program where we saw the error happened after debugging.
I'm still trying to convince SAP, among others by pointing to their own documentation.

That's part of life or what...

P.S.
I found that awk have the log-function and that it is natural logarithm. From this I can get to the base-10 representation.

/Tor-Arne