How to Avoid Rounding & Overflow Probs on Pentium Processors

This article was previously published under Q126455
This article has been archived. It is offered "as is" and will no longer be updated.
Intel Corporation has identified two minor bugs in the Pentium processorrelating to the conversion of floating point values to integer numbers whenusing the Floating Point Integer Store (FIST) instruction. These bugsinvolve the processing of exceptions for a narrow range of bounds forfloating point numbers and involve unexpected behavior in two roundingmodes for six specific operands. Programmers can easily work around thefirst bug by range checking prior to converting the values. Programmers canavoid the second bug by not overriding the default rounding modes.
More information
Different programming languages handle conversions from floating point tointeger in different ways. C, C++, and FORTRAN programs are most likelysafe; that is, they are not likely to be affected without explicitmodification of rounding modes by the programmer. Basic programs are morelikely to be affected by the bugs.

Application programs in general are not affected. Applications shouldinclude appropriate error checking and take appropriate actions to recoverfrom exceptions. To do this, applications need to do explicit rangechecking prior to conversion or implement explicit exception handling. Ifan application relies on explicit exception handling, the possibility ofencountering the bug exists. Applications written in languages that useexception handling will most likely use range checking, thereby allowingthe application to be compiled with a no-check option for performancereasons.

FIST Overview

The FIST instruction is used to convert floating point numbers to signedintegers. For example, in a C or FORTRAN program, when an integer variableis assigned a floating point value, that value must be truncated to aninteger before being stored. Other conversions from floating point tointeger are also possible; for example, Basic and Pascal round instead oftruncating the fractional part.

The FIST instruction operates in one of four rounding modes: chop (alsoknown as truncate), nearest (nearest or even), up, and down. The defaulttype conversions done by C and FORTRAN use the chop rounding mode, whereasBasic uses nearest. The processor indicates whether the input operand is aninteger by means of a precision exception (PE) flag. The PE flag is set totrue if the number to be converted was not already an integer; that is, ithad a fractional part. It is unchanged otherwise. The effect of rounding isshown in the rounding direction( C1) flag. If an input value is rounded toa number with a greater magnitude, the C1 flag is set; otherwise, it iscleared.

FIST converts floating point numbers to 16, 32, or 64-bit signed integers.Because the range of a floating point number is larger than any of theseformats, some floating point numbers cannot be converted to integers. Forexample, the largest 16-bit integer is 32767, so an attempt to convert32768.5 to a 16-bit integer results in an exception (in this case, theinvalid operation exception (IE)). The processing of an exception iscontrolled by the exception mask in the floating point control register. Ifan invalid operation exception mask is set, the exception is masked,otherwise, it is unmasked.

For example, applications running under Microsoft Windows version 3.1 beginwith all floating point exceptions masked. Different applications maychange the exception masks and provide their own methods for handlingexceptions. For example Microsoft Visual C++ sets the invalid operationexception to unmasked, while leaving the precision exception masked. Anapplication developed in C++ will see a pop-up window appear when a FISTinstruction attempts to convert a number that is out of bounds, unless theexception handling is changed.

Undetected Overflow

A bug has been identified by Intel in FIST instructions that convert afloating point number to either a 16- or 32-bit signed integer. The FISTPinstruction that stores to a 64-bit integer is not affected. The erroroccurs when a rounding mode is set to "nearest" or "up" for one limitedrange of floating point values in the out of bounds region. For this range,regardless of the exception mask, the value 0 is written to memory, the PEflag is set, the IE flag is not set, and no IE exception is raised.

In the figures below, floating point values from A and B can be convertedto integers. Values outside this range should raise an exception. This bugaffects values in the C and D range only.

16-Bit FIST
                                                 65535.5 -          -32768.5     0      32767.5            65536.0<-------------[--------|--------]-------------------|-------->   overflow       Normal Range        overflow              A                 B                   C32-Bit FIST                                                          4,294,967,295.5 -      -2,147,483,648.5     0      2,147,483,647.5         4,294,967,296.0<-------------[------------|------------]-----------------------|-------->   overflow           Normal Range             overflow              A                         B                       D				

For example, take a number outside the range of a 32-bit signed integer,such as 4,294,967,295.7. This is rounded mathematically to 4,294,967,296.Due to the bug, the Pentium processor does not raise an exception to thisnumber. It simply writes a 0 into memory and does not transfer control to ahandler or raise the IE flag.

Applications that Use Microsoft Visual Basic

Microsoft Visual Basic automatically promotes data types to handle mixedtype operations. This avoids the overflow condition for many cases. Forexample:
   IntX = SngY*IntZ   IntX = 65535.7*2				

However, implicit conversions may be affected. The following Visual Basicversion 3.0 code produces different results depending on the processor:
   Dim X As Integer   Dim Y As Single   Y = 65535.5   X = Y   MsgBox Cstr(X)				

The above code should result in a run-time error 6 "Overflow" message. On aPentium processor-based machine, this code results in a return of zero (0).A simple way to avoid this error is to include a range check in your code.For example:
   If (x)>=65535.5   Then Error 6				

The only programs affected would be those that explicitly trap error 6(overflow) and involve either explicit assignment of a floating-pointexpression to an Integer data type or a Long data type or explicitconversion of a floating point expression using CInt() or CLng().

Programs that employ explicit bounds checking to prevent overflowconditions should not be affected.

If bounds checking adversely affects performance in a loop, migrating thecode to a C language dynamic-link library (DLL) may be a better option.

Program Results

At the assembly level, the recommended workaround is the insertion of theFRNDINT (Floating Point Round to Integer) instruction immediately precedingthe FIST instruction. FRNDINT will correctly round the floating point valuebefore executing the FIST instruction (as opposed to FISTP); the correctionwill also need to preserve its input. Future versions of Visual Basic willinclude this fix.

NOTE: Any method that forces the processor to emulate floating pointinstructions will avoid the problem, at the cost of reduced performance andspeed.

Rounding Mode Errors

For six specific operands, the results of the FIST instructions are not asexpected. There are flag differences in all four rounding modes. There isalso the possibility of an incorrect number being stored in the "up" and"down" rounding modes. In the "nearest" and "chop" rounding modes, whichare most frequently used, the value stored to memory is correct. This bugaffects the 16-, 32-, and 64-bit variants of the instruction.

The following table shows the affected numbers, rounding modes, andexpected and actual values:
   Operand     | Rounding | Exp. | Actual | Exp. | Actual | Exp. | Actual(any one of)   |  mode    | mode |  mode  |  PE  |   PE   |  C1  |   C1-----------------------------------------------------------------------1/16 (0.0625)  |  nearest |  0   |   0    |   1  |  unch. |  0   |   01/8 (0.125)    |   chop   |  0   |   0    |   1  |  unch. |  0   |   03/16 (0.1875)  |   down   |  0   |   0    |   1  |  unch. |  0   |   0               |    up    |  1   |   0    |   1  |  unch. |  1   |   0-1/16 (-0.0625)|  nearest |  0   |   0    |   1  |  unch. |  0   |   0-1/8 (-0.125)  |   chop   |  0   |   0    |   1  |  unch. |  0   |   0-3/16 (-0.1875)|   down   | -1   |   0    |   1  |  unch. |  1   |   0               |    up    |  0   |   0    |   1  |  unch. |  0   |   0				

For six numbers (1/16, 1/8, 3/16, -1/16, -1/8, and -3/16) incorrect resultscan occur. In all rounding modes, the PE flag is not set consistently withthe rounding that occurred; it remains unchanged. For the three positivenumbers in the "up" rounding mode, FIST stores an incorrect value (0instead of 1); similarly, for the three negative numbers in the "down"rounding mode, FIST gives an incorrect result (0 instead of -1). In each ofthese cases, the C1 bit is also not set correctly. However, in the"nearest" and "chop" rounding modes, the value stored to memory is alwayscorrect, as is the C1 bit.

Program Results

Incorrect results from the second bug are only returned when the roundingmode is set to "up" or "down." Therefore, it should not affect anapplication unless the application explicitly changes the rounding modefrom the language default. If an application changes to "up" or "down"rounding mode, it can no longer rely on language-provided floating-point-value-to-integer-number conversion facilities and must provide its own by,for example, using FRNDINT.

The incorrect status information in the PE flag, affecting all roundingmodes, should be insignificant in most applications. If rounding occurs,the PE flag is already set. The PE exception is also typically masked. TheC1 bit is only used by an exception handler, so the incorrect values areinsignificant.


At the assembly level the recommended workaround for the second FIST bug isthe same for the first; inserting the FRNDINT instruction immediatelypreceding the FIST instruction.

Application programmers can avoid rounding errors in the second bug by notoverriding the default rounding modes.

NOTE: Any method that forces the processor to emulate floating pointinstructions will avoid both problems, at the cost of reduced performanceand speed.

Article ID: 126455 - Last Review: 10/26/2013 01:30:00 - Revision: 3.0

  • Microsoft Visual Basic 2.0 Standard Edition
  • Microsoft Visual Basic 3.0 Professional Edition
  • Microsoft Visual Basic 2.0 Professional Edition
  • kbnosurvey kbarchive KB126455