5.15-2.55-2.6

General development discussion.

Moderators: Susan Smith, admin, Gabriel

Post Reply
John
Posts: 555
Joined: Sun Apr 26, 2009 8:27 am

5.15-2.55-2.6

Post by John »

Why does the answer to this:

5.15-2.55-2.6

not come out to 0?

try it in your console and see what you get?

I'm asking for a friend.

-John
John Bowman
John Carr
Posts: 3
Joined: Tue Jun 09, 2009 7:59 am

Re: 5.15-2.55-2.6

Post by John Carr »

This is caused by the difference between decimal fraction and binary fraction values. IE-the binary values of the decimal numbers you picked don't EXACTLY equal the original decimal value. This is only a problem with some fractions. Whole numbers are exactly equal between decimal and binary. This is also why sometimes an apparent zero value will print as -0.00 or .00CR.

Gordon jump in if I have gotten it wrong.
Gabriel
Posts: 412
Joined: Sun Aug 10, 2008 7:37 am
Location: Arlington, TX
Contact:

Re: 5.15-2.55-2.6

Post by Gabriel »

Very interesting.

I do note that if its printed in any kind of formatted way, or if its tested in an if statement, then it still works correctly.
Untitled.png
Untitled.png (2.25 KiB) Viewed 16166 times
Gabriel
gordon
Posts: 358
Joined: Fri Apr 24, 2009 6:02 pm

Re: 5.15-2.55-2.6

Post by gordon »

Yes John. You stated it correctly. All programming languages that work with floating point numbers have the same problem. By using scientific notation internally the binary fractions used in floating point math do not precisely match decimal fractions. Floating point arithmetic necessarily and specifically accepts the notion that results will be infinitesimally imprecise.

For that reason when adding a long column of numbers it is a good idea to round the result every so often. This will erase any accumulating low level inaccuracy.

In BR the number of decimal places where rounding occurs during output formatting and comparisons, defaults to 6 decimal places. However the RD setting can be changed. See: http://brwiki2.brulescorp.com/index.php?title=RD
GomezL
Posts: 258
Joined: Wed Apr 29, 2009 5:51 am
Contact:

Re: 5.15-2.55-2.6

Post by GomezL »

You can better see the results using formated output:

Code: Select all

pr using "form n 30.28":5.15-2.55-2.6
Returns: 0.000000000000000444

You can see that there are 15 0's after the decimal place.

Loop 1 million times:

Code: Select all

00001   PRINT Newpage
00010   FOR Loop=1 TO 1000000
00020     LET A+=(5.15-2.55-2.6)
00030   NEXT Loop
00040   PRINT USING "FORM N 30.28": A
Returns: 0.0000000004440892098500630000

Not surprisingly, moves the decimal 6 places

Add the Round Command (to 15 digits):

Code: Select all

00001   PRINT Newpage
00010   FOR Loop=1 TO 1000000
00020     LET A+=Round(5.15-2.55-2.6,15)
00030   NEXT Loop
00040   PRINT USING "FORM N 30.28": A
Returns 0
Post Reply