cancel
Showing results for 
Search instead for 
Did you mean: 

Why does post builder come up with such "strange" numbers?

Esteemed Contributor
Esteemed Contributor

A link to this site was posted in the customization & programming forum, but as it comes up periodially in relation to posting, I thought I'd post it here as well.

 

If you've ever wondered why post builder comes up with a value of "0.75000000000000044" (instead of "0.75000000000000000" or (more interestingly) "1.96499999999999990" instead of "1.96500000000000000", you might want to read thru this web site:

 

http://floating-point-gui.de/

 

I'm sure there's a wiki out there somewhere as well...

 

Ken

Ken Akerboom Sr CAx Systems Engr, Moog, Inc.
Production: NX10.0.3.5 MP5 + patch/TC11.2
I'd rather be e-steemed than e-diseaseled


4 REPLIES

Re: Why does post builder come up with such "strange" numbers?

Valued Contributor
Valued Contributor
XKCD had me loling

Re: Why does post builder come up with such "strange" numbers?

Legend
Legend

In addition to thinking "why 0.75000000000000044", it's more important to apply the tcl procedures for floating numbers like EQ_is_zero and EQ_is_equal in Postbuilder, because any floating mathematics has a precision inside some tolerance.

Re: Why does post builder come up with such "strange" numbers?

Valued Contributor
Valued Contributor

Hi FROBI,

 

I am trying to understand what EQ_is_equal does after I found it being used to compare two variables.  Does it choose a specific precision?  Is there a good source available to learn about this command?

 

Thank you,

Jesse

Re: Why does post builder come up with such "strange" numbers?

Solution Partner Phenom Solution Partner Phenom
Solution Partner Phenom

Precision which is used by EQ_is_equal is defined by mom_system_tolerance

As an alternative, you can also use EQ_is_equal_tol, where you can define what value of precision you want to use.

 

You can see source code of EQ_is_equal and others in this file:

(...)\MACH\resource\postprocessor\ugpost_base_math.tcl

 

proc EQ_is_zero [list s [list tol $mom_system_tolerance]] {
   expr { abs($s) <= $tol }
}

proc EQ_is_equal [list s t [list tol $mom_system_tolerance]] {
	expr { abs($s - $t) <= $tol }
}

proc EQ_is_ge [list s t [list tol $mom_system_tolerance]] {
	expr { $s > ($t - $tol) } 
}

proc EQ_is_gt [list s t [list tol $mom_system_tolerance]] {
	expr { $s > ($t + $tol) }
}

proc EQ_is_le [list s t [list tol $mom_system_tolerance]] {
	expr { $s < ($t + $tol) } 
}

proc EQ_is_lt [list s t [list tol $mom_system_tolerance]] {
	expr { $s < ($t - $tol) } 
}

proc EQ_is_zero_tol { s tol } {
   expr { abs($s) <= $tol }
}

proc EQ_is_equal_tol { s t tol } {
   expr { abs($s - $t) <= $tol }
}

 

Marek Pawlus, NCmatic

Production: NX 11.0.2
Development: C#, Tcl/Tk, CSE

Learn online





Solution Information