Rounding error with variables

[Dan Goldberg]

Hi all
I have looked up previous discussions on problems with round() but have not been able to solve this. I am getting unexpected results in my order entry program when calculating tax. Below is an example of the problem. Any suggestions will be much appreciated.
Dan


SET DECIMAL TO 3

MQTY = 3
MPRICE = 3.30
MGSTRATE = .05

MSUB = 0
MGST = 0
MMGST = 0
MMTEST = 0

MSUB = MQTY * MPRICE

MGST = MSUB * MGSTRATE

MMGST = ROUND(MGST,2)

MMTEST = ROUND(.495,2)


? MGST // variable = 0.495 before rounding
? MMGST // rounds to 0.49 but should be 0.50 ??
? MMTEST //rounds to 0.50 which is correct

[Mervyn Bick]

On Wed, 17 Sep 2008 22:27:49 +0200, Dan Goldberg

Post by Dan Goldberg
Hi all
I have looked up previous discussions on problems with round() but have
not been able to solve this. I am getting unexpected results in my
order entry program when calculating tax. Below is an example of the
problem. Any suggestions will be much appreciated.
Dan
SET DECIMAL TO 3
MQTY = 3
MPRICE = 3.30
MGSTRATE = .05
MSUB = 0
MGST = 0
MMGST = 0
MMTEST = 0
MSUB = MQTY * MPRICE
MGST = MSUB * MGSTRATE
MMGST = ROUND(MGST,2)

MMGST = ROUND((MGST*10000+5)/10000,2)

Post by Dan Goldberg
MMTEST = ROUND(.495,2)
? MGST // variable = 0.495 before rounding
? MMGST // rounds to 0.49 but should be 0.50 ??
? MMTEST //rounds to 0.50 which is correct

Mervyn

***** Start of example program
set decimals to 3
for MGST = .490 to .500 step .001
MMGST = round((MGST*10000+5)/10000,2)
? MGST,MMGST
next
set decimals to
****** End of example program *****

[Dan Goldberg]

Thanks Mervin this works great!! As for the why and how, the debate goes on.
Dan

Post by Mervyn Bick
On Wed, 17 Sep 2008 22:27:49 +0200, Dan Goldberg

Post by Dan Goldberg
Hi all
I have looked up previous discussions on problems with round() but have
not been able to solve this. I am getting unexpected results in my
order entry program when calculating tax. Below is an example of the
problem. Any suggestions will be much appreciated.
Dan
SET DECIMAL TO 3
MQTY = 3
MPRICE = 3.30
MGSTRATE = .05
MSUB = 0
MGST = 0
MMGST = 0
MMTEST = 0
MSUB = MQTY * MPRICE
MGST = MSUB * MGSTRATE
MMGST = ROUND(MGST,2)

MMGST = ROUND((MGST*10000+5)/10000,2)

Post by Dan Goldberg
MMTEST = ROUND(.495,2)
? MGST // variable = 0.495 before rounding
? MMGST // rounds to 0.49 but should be 0.50 ??
? MMTEST //rounds to 0.50 which is correct

Mervyn
***** Start of example program
set decimals to 3
for MGST = .490 to .500 step .001
MMGST = round((MGST*10000+5)/10000,2)
? MGST,MMGST
next
set decimals to
****** End of example program *****

[*Lysander*]

Post by Dan Goldberg
? MGST // variable = 0.495 before rounding
? MMGST // rounds to 0.49 but should be 0.50 ??
? MMTEST //rounds to 0.50 which is correct

a human brain might not understand too easily that this can be normal,
but I bet it is.

There is a rule, followed by nearly all programming languages, and all
processors, old and new, according to which sometims the "5" is rounded
down and sometimes up; depending on what the leading figures are.

see anything you can find on "rounding" and "IEEE 754".

According to this rule, the result would have to be rounded into the
direction of the nearest even integer, whenever a number is exactly
between the two nearest numbers.

I would understand that 0.495 is exactly between 0.49 and 0.50 and thus
would have to be rounded in direction of the nearest _EVEN_ integer (0)
and thus to 0.49 which is closer to 0 than 0.50 is.

Rest assured that I never can get used to it, because my teachers told
me differently decades ago :)
But I accept that there is such a rule in binary IT and try to live with
it; try to do the same :)

[Mervyn Bick]

Post by *Lysander*
According to this rule, the result would have to be rounded into the
direction of the nearest even integer, whenever a number is exactly
between the two nearest numbers.
I would understand that 0.495 is exactly between 0.49 and 0.50 and thus
would have to be rounded in direction of the nearest _EVEN_ integer (0)
and thus to 0.49 which is closer to 0 than 0.50 is.
Rest assured that I never can get used to it, because my teachers told
me differently decades ago :)
But I accept that there is such a rule in binary IT and try to live with
it; try to do the same :)

This is all very well for mathematicians but in the real world people,
including the tax man, all seem to expect what you, and I, were taught all
those long years ago. Round up if the next digit to the right is 5 or
greater. Fortunately it is easy enough to give people what they want even
if the programming language wants to do things the "pure maths" way.

Mervyn.

[bigMike]

If you SET DECI to 2 you'll get .50. Do you need deci = 3?

bigMike

Post by Dan Goldberg
Hi all
I have looked up previous discussions on problems with round() but have
not been able to solve this. I am getting unexpected results in my order
entry program when calculating tax. Below is an example of the problem.
Any suggestions will be much appreciated.
Dan
SET DECIMAL TO 3
MQTY = 3
MPRICE = 3.30
MGSTRATE = .05
MSUB = 0
MGST = 0
MMGST = 0
MMTEST = 0
MSUB = MQTY * MPRICE
MGST = MSUB * MGSTRATE
MMGST = ROUND(MGST,2)
MMTEST = ROUND(.495,2)
? MGST // variable = 0.495 before rounding
? MMGST // rounds to 0.49 but should be 0.50 ??
? MMTEST //rounds to 0.50 which is correct