BR Forum

Why does the answer to this:

5.15-2.55-2.6

not come out to 0?

try it in your console and see what you get?

I'm asking for a friend.

-John

This is caused by the difference between decimal fraction and binary fraction values. IE-the binary values of the decimal numbers you picked don't EXACTLY equal the original decimal value. This is only a problem with some fractions. Whole numbers are exactly equal between decimal and binary. This is also why sometimes an apparent zero value will print as -0.00 or .00CR.

Gordon jump in if I have gotten it wrong.

Very interesting.

I do note that if its printed in any kind of formatted way, or if its tested in an if statement, then it still works correctly.
Gabriel

Yes John. You stated it correctly. All programming languages that work with floating point numbers have the same problem. By using scientific notation internally the binary fractions used in floating point math do not precisely match decimal fractions. Floating point arithmetic necessarily and specifically accepts the notion that results will be infinitesimally imprecise.

For that reason when adding a long column of numbers it is a good idea to round the result every so often. This will erase any accumulating low level inaccuracy.

In BR the number of decimal places where rounding occurs during output formatting and comparisons, defaults to 6 decimal places. However the RD setting can be changed. See: http://brwiki2.brulescorp.com/index.php?title=RD

You can better see the results using formated output: Returns: 0.000000000000000444

You can see that there are 15 0's after the decimal place.

Loop 1 million times:
Returns: 0.0000000004440892098500630000

Not surprisingly, moves the decimal 6 places

Add the Round Command (to 15 digits): Returns 0