cancel
Showing results for 
Search instead for 
Did you mean: 

Why does post builder come up with such "strange" numbers?

Esteemed Contributor
Esteemed Contributor

A link to this site was posted in the customization & programming forum, but as it comes up periodially in relation to posting, I thought I'd post it here as well.

 

If you've ever wondered why post builder comes up with a value of "0.75000000000000044" (instead of "0.75000000000000000" or (more interestingly) "1.96499999999999990" instead of "1.96500000000000000", you might want to read thru this web site:

 

http://floating-point-gui.de/

 

I'm sure there's a wiki out there somewhere as well...

 

Ken

Ken Akerboom Sr CAx Systems Engr, Moog, Inc.
Production: NX10.0.3.5 MP5 + patch/TC11.2
I'd rather be e-steemed than e-diseaseled


2 REPLIES

Re: Why does post builder come up with such "strange" numbers?

Valued Contributor
Valued Contributor
XKCD had me loling

Re: Why does post builder come up with such "strange" numbers?

Legend
Legend

In addition to thinking "why 0.75000000000000044", it's more important to apply the tcl procedures for floating numbers like EQ_is_zero and EQ_is_equal in Postbuilder, because any floating mathematics has a precision inside some tolerance.

Learn online





Solution Information