I discovered this little “feature” and thought I’d pass it along.
When you save a number in both a computed field and an editable field at the same time, the decimal precision will be different - even though each field has the same decimal precision field property setting.
I have a price field that is computed. Before returning the result, it also updates a computed freight field (hidden) and an editable freight field. If the price is changed, then I need to update the freight fields - but only if the editable one had not been overridden. If the user put in an override freight amount, then I’ll leave the freight as is. If the computed freight is still equal to the editable freight, then the formula will update both fields to the new freight amount.
This was not working; the freight would calc the first time, but would never reset with a price change. It wasn’t until I added a field to calc the difference between the two freight fields that I realized why. The comp frt field had more decimal precision than the editable field - so they were never the same.
I verified the field property settings for both freight fields and they’re both set to currency, 2 decimals. So, they looked the same, but they weren’t.
My solution was to round the freight to two decimals before setting the field values - and now I’m in business.
Subject: Gotcha - decimal precision different in comp vs editable fields
More generally, the issue is that the formula for a computed field – or the input translation for an editable field – “trumps” the properties you have set for the field. This includes not only the precision but even the datatype. For instance, if you have a Text field and the input translation is @ToNumber(@ThisValue), then assuming the value you enter can in fact be converted to a number, the “item” stored in the document has a numeric type – you can look at it in the document properties dialog from a view, and see that it’s type is number.
When a user enters a value in an editable field, on the other hand, this value is interpreted and converted based on the settings of the field.
Subject: RE: Gotcha - decimal precision different in comp vs editable fields
And, if I’m not mistaken, this is precisely the reason why @FloatEq was added to Formula Language – it allows you to set a “close enough” margin so that conversion errors could be ignored. Remember, Robert, that there is no exact binary representation for most fractional decimal numbers – there will always be a tiny error if the decimal number does not express a multiple of a power of two (.5, .25, .125, .0625, and so on).
Subject: RE: Gotcha - decimal precision different in comp vs editable fields
Something still does not seem right though. I have been using Notes since 4.0 was first released and I have not run into this issue until just recently. Has notes always behaved this way? Even acknowledging the issue with binary fractions, it seems like the math library should not be comparing at levels of precision that are not present in the operands. Even when using scientific notation, the precision errors surface. The precision of the operands should be taken into account. What Notes is doing is contrary to all the lessons about precision I learned in my math and physics classes.
I used the following example as a demo for my peers.
Sub Initialize
Dim d1 As Double
Dim d2 As Double
Dim d3 As Double
Dim d4 As Double
Dim v1 As Variant
Dim v2 As Variant
Dim v3 As Variant
Dim v4 As Variant
Dim v5 As Variant
d1=10.7
d2=10.9
d3=.200
d4=d2-d3
v1=d4-d1
If (d4 <> d1) Then
Print( "Precision Error")
End If
d1=107E-1
d2=109E-1
d3=200E-3
d4=d2-d3
v1=d4-d1
If (d4 <> d1) Then
Print( "Precision Error")
End If
v1=107E-1
v2=109E-1
v3=200E-3
v4=v2-v3
v5=v4-v1
If (v4 <> v1) Then
Print( "Precision Error")
End If
Subject: RE: Gotcha - decimal precision different in comp vs editable fields
In LotusScript, there are precisely two levels of floating-point precision: Single and Double. Equality testing checks all bits – any difference at any bit level is an inequality. This is true across all general-purpose computing languages, although the number of bits available to each variable type and the number of different types may vary from language to language. If you need to compare within a confidence margin, you need to round your values at your arbitrary precision depth. Some specialised languages on some platforms have similar facilities built in, but you can and should consider them oddities.
Calculators designed exclusively for decimal use often get around the problem by using binary-coded-decimal values internally, where each four-bit group represents a single decimal digit (0000 to 1001), but that is wasteful of resources and does not allow the same registers to be used for general-purpose (non-decimal) data.