Tag Archives: floating point

Exercises in Emulation: Xbox 360’s FMA Instruction

Posted on March 20, 2019 by brucedawson

Years ago I worked in the Xbox 360 group at Microsoft. We were thinking about releasing a new console, and we thought it would be nice if that console could run the games of the previous console. Emulation is always … Continue reading →

Posted in Floating Point | Tagged floating point, fma, PowerPC, x86, Xbox 360 | 16 Comments

Sometimes Floating Point Math is Perfect

Posted on June 19, 2017 by brucedawson

I’ve written in the past about how to compare floating-point numbers for the common scenario where two results should be similar but may not be identical. In that scenario it is reasonable to use an AlmostEqual function for comparisons. But … Continue reading →

Posted in Floating Point | Tagged floating point | 16 Comments

There are Only Four Billion Floats–So Test Them All!

Posted on January 27, 2014 by brucedawson

A few months ago I saw a blog post touting fancy new SSE3 functions for implementing vector floor, ceil, and round functions. There was the inevitable proud proclaiming of impressive performance and correctness. However the ceil function gave the wrong … Continue reading →

Posted in Floating Point | Tagged floating point, floating point comparisons | 35 Comments

Float Precision Revisited: Nine Digit Float Portability

Posted on February 7, 2013 by brucedawson

Last year I pointed out that float variables can be converted to text and then back to the same binary value using printf(“%1.8e”). You can also use %.9g for more compact results. I also supplied a test program that used … Continue reading →

Posted in Floating Point | Tagged float, floating point, floating point precision, printf, scanf | 17 Comments

Doubles are not floats, so don’t compare them

Posted on June 26, 2012 by brucedawson

I’ve seen a few online discussions linking to my Comparing Floating Point Numbers page for misguided reasons and I wanted to discuss those reasons to help people understand why throwing epsilons at the problem without understanding the situation is a … Continue reading →

Posted in Floating Point, Programming | Tagged base conversion, epsilon, float, floating point, precision | 24 Comments

That’s Not Normal–the Performance of Odd Floats

Posted on May 20, 2012 by brucedawson

Denormals, NaNs, and infinities round out the set of standard floating-point values, and these important values can sometimes cause performance problems. The good news is, it’s getting better, and there are diagnostics you can use to watch for problems. In … Continue reading →

Posted in AltDevBlogADay, Floating Point, Performance, Programming | Tagged denormal, floating point, infinity, NaN, special numbers, specials | 21 Comments

Exceptional Floating Point

Posted on April 21, 2012 by brucedawson

Floating-point math has an answer for everything, but sometimes that’s not what you want. Sometimes instead of getting an answer to the question sqrt(-1.0) (it’s NaN) it’s better to know that your software is asking imaginary questions. The IEEE standard … Continue reading →

Posted in AltDevBlogADay, Floating Point, Programming | Tagged divide by zero, double, exceptions, float, floating point | 24 Comments

Floating-point complexities

Posted on April 5, 2012 by brucedawson

Binary floating-point math is complex and subtle. I’ve collected here a few of my favorite oddball facts about IEEE floating-point math, based on the articles so far in my floating-point series. The focus in this list is on float but … Continue reading →

Posted in AltDevBlogADay, Floating Point, Programming | Tagged comparing, denormal, digits, float, floating point, floating point precision, floating point tricks, precision, printing | 15 Comments

Intermediate Floating-Point Precision

Posted on March 21, 2012 by brucedawson

Riddle me this Batman: how much precision are these calculations evaluated at? If you answered ‘double’ and ‘float’ then you score one point for youthful idealism, but zero points for correctness. The correct answer, for zero-idealism points and forty two … Continue reading →

Posted in AltDevBlogADay, Floating Point, Programming, Visual Studio | Tagged double, float, floating point, floating point precision, intermediate precision, precision | 43 Comments

Float Precision–From Zero to 100+ Digits

Posted on March 8, 2012 by brucedawson

How much precision does a float have? It depends on the float, and it depends on what you mean by precision. Typical reasonable answers range from 6-9 decimal digits, but it turns out that you can make a case for … Continue reading →

Posted in AltDevBlogADay, Floating Point, Programming | Tagged double, float, floating point, floating point precision, precision, significant figures | 52 Comments