Showing results for 
Search instead for 
Do you mean 
Reply

Why does post builder come up with such "strange" numbers?

A link to this site was posted in the customization & programming forum, but as it comes up periodially in relation to posting, I thought I'd post it here as well.

 

If you've ever wondered why post builder comes up with a value of "0.75000000000000044" (instead of "0.75000000000000000" or (more interestingly) "1.96499999999999990" instead of "1.96500000000000000", you might want to read thru this web site:

 

http://floating-point-gui.de/

 

I'm sure there's a wiki out there somewhere as well...

 

Ken

Ken Akerboom Sr CAx Systems Engr, Moog, Inc.
Production: NX10.0.3.5 MP5 + patch/TC11.2
I'd rather be e-steemed than e-diseaseled


2 REPLIES

Re: Why does post builder come up with such "strange" numbers?

XKCD had me loling

Re: Why does post builder come up with such "strange" numbers?

In addition to thinking "why 0.75000000000000044", it's more important to apply the tcl procedures for floating numbers like EQ_is_zero and EQ_is_equal in Postbuilder, because any floating mathematics has a precision inside some tolerance.

Learn online





Solution Information