I'm going bananas... 100 does not equal 100?

HCL Domino Domino Forum
1

Ok, I’m SURE I’m missing something VERY obvious here, the problem is: I’m not getting what…

Please help, I’m about to tear out most of my hair! :open_mouth:

I wrote this agent to test a problem a user was having:

Sub Initialize

On Error Goto PrintError



Dim s As New NotesSession

Dim mylog As NotesLog

Dim x As Double

Dim y As Double



Set mylog = s.createlog("WTF")

Call mylog.openagentlog



x = 4.08 + 18.84 + 18.84 + 18.84 + 18.84 + 20.56

y = 4.11 + 18.84 + 18.84 + 18.84 + 18.84 + 20.53



Call mylog.logaction("--")

If (x <> 100) Then

	Call mylog.logaction("X IS NOT 100, BUT " & Cstr(x))

Else

	Call mylog.logaction("X IS 100 !")

End If



If (y <> 100) Then

	Call mylog.logaction("Y IS NOT 100, BUT " & Cstr(y))

Else

	Call mylog.logaction("Y IS 100 !")

End If



Call mylog.logaction("--")

Call mylog.close



Exit Sub

PrintError:

Call mylog.logerror(Err(),Erl() & ": " & Error())

Exit Sub

End Sub

If you just read it like this, you would think the output would be for the log to report both instances (x & y) being 100. But what do I see?

Started running agent ‘wtf’ on 06/04/2005 10:56:52

06/04/2005 10:56:52: –

06/04/2005 10:56:52: X IS NOT 100, BUT 100

06/04/2005 10:56:52: Y IS 100 !

06/04/2005 10:56:52: –

Ran LotusScript code

Done running agent ‘wtf’ on 06/04/2005 10:56:52

“X IS NOT 100, BUT 100” ?! What the…

Even in the debugger I can clearly see X as a Double with 100 as value. Help! (I just feel I’m going to smack my head against the wall after hearing the explanation)

Thanks in advance for those pointing me in the right direction!

Chris.

2

Subject: I’m going bananas… 100 does not equal 100 ?

There are many posts out on the web about this sort of problem, but it comes down to the fact that floating point numbers can’t be represented exactly by binary computers. Different languages/computers/operating systems handle the details slightly differently, but it is not a bug in LotusScript or Formula language, but an inherent issue with floating point values. The answer is to round the number and check for that, or compare within a degree of precision. For example, here is a post about SAS which relates to the problem, and here is another about C/C++, and one about Excel 97. From that last link, there is this quote:

The second problem arises from the fact that a computer, any computer, cannot store most fractional numbers with total accuracy. Computers, in general, use the IEEE (Institute Of Electrical And Electronic Engineers) standard for floating point numbers. This standard provides a way to store fractional numbers in the limited space of an 8-byte number. Of course, for most numbers, some approximation must be made.

3

Subject: RE: I’m going bananas… 100 does not equal 100 ?

Thanks Ben, for your clear answer. I guess I’ll just have to find another way to compare the 2 (e.g. by making strings out of the two).

What puzzles me is that I never encountered this before, in all my years as a developer. Weird! :wink:

4

Subject: SOLUTION : Floating Point Calcuations

I guess thats the reason @FloatEq was introduced in R6.

@FloatEq

Example

Compares two numbers for equality within a confidence range.

Note This @function is new with Release 6.

Syntax

@FloatEq( number ; number ; confidenceRange )

Parameters

number

Number. Any numeric value.

confidenceRange

Number. Optional. The amount within which the numbers must be equal. Defaults to 0.0001.

Return value

flag

Boolean.

Returns 1 (True) if the difference of the numbers is less than or equal to the confidence range.

Returns 0 (False) if the difference of the numbers exceeds the confidence range.

Usage

@FloatEq is helpful in dealing with the inexactness of floating point operations.

Example

See Also

Performing arithmetic operations

cheers !!

Ashish

5

Subject: *Hey, you learn something new every day. Thanks for that tip!

6

Subject: I’m going bananas… 100 does not equal 100 ?

If you REALLY need to handle numbers with 2 decimals and exact values, multiply them by 100 and store them as integers.

As noted in other posts, computers cannot store exact values of fractional numbers.

e.g. It might be that (.18 *100) <> 18 This is not just in Notes, but any computer language.

7

Subject: I’m going bananas… 100 does not equal 100 ?

This is quite shocking, both Lotus Script & @Formulas give the same results.

If you notice that the x variable has decimal values with a “0” in it. If you change that to a Non Zero value the comparison seems fine.

Try replacing

x = 4.08 + 18.84 + 18.84 + 18.84 + 18.84 + 20.56

with

x = 4.18 + 18.84 + 18.84 + 18.84 + 18.84 + 20.46

Note that we have added .10 to the first value and reduced .10 from the last value.

But the mistery still remains and i am quite anxious to know whats happening.

cheers !!

Ashish

8

Subject: RE: I’m going bananas… 100 does not equal 100 ?

what i’ve learned in school:) about using floating point numbers: don’t make equal comparisions, use < or > instead.

but this usually happens with things like 1 / 3 x 3, which isn’t 1, but 0.999999…

i don’t know how notes handles them internally, but i guess the debugger is lying to you.

just use int(x)=100 instead…

Markus Seitz

markus.seitz@icodex.com

9

Subject: RE: I’m going bananas… 100 does not equal 100 ?

I can’t use Cint I’m afraid, because I ned to check whether the sum is exactly 100. Using Cint would make 100.01 also equal to 100! :frowning:

What does work is comparing them bitwise, but IMHO that still isn’t a very elegant solution. Shouldn’t this just work??

Thanks for the suggestions though.

10

Subject: RE: I’m going bananas… 100 does not equal 100 ?

forget it… read Ben’s post…