Tag Archives: accuracy

Intel Underestimates Error Bounds by 1.3 quintillion

Posted on October 9, 2014 by brucedawson

Intel’s manuals for their x86/x64 processor clearly state that the fsin instruction (calculating the trigonometric sine) has a maximum error, in round-to-nearest mode, of one unit in the last place. This is not true. It’s not even close. The worst-case … Continue reading →

Posted in Floating Point, Investigative Reporting, Programming | Tagged , , | 152 Comments