“Know how to solve every problem that has been solved.” “What I cannot create, I do not understand.” — Richard Feynman

Floating-point arithmetic

Numerical Methods

Floating-point arithmetic is fundamentally a finite-precision operation. Most things we call finite are concrete: a finite number of cars in a parking lot, a finite number of beads on an abacus. Finite precision is the more abstract kind — it means the numbers you can represent are a discrete lattice, with gaps between consecutive representable values. Think of an old-school ruler. A meter stick has marks every millimeter; you can read off 10 mm or 11 mm, but 10.5 mm doesn't exist on the ruler — the number between the marks isn't there. Floating-point is a stranger version of the same idea: the marks aren't evenly spaced. Near zero they're packed tight; near $1 0^{20}$ they're millions apart. That's why 1e20 + 1 returns 1e20 — the 1 you tried to add was smaller than the gap between two consecutive marks at that scale, and the computer rounded it away.

Zeno's paradox — walk halfway, then halfway again, then halfway again, forever — is dissolved in classical mathematics by noticing that the infinite halvings converge to a finite total. The equals sign in $\frac{1}{2} + \frac{1}{4} + \frac{1}{8} + \dots = 1$ does a lot of work, though: it declares the infinite process complete, even though no finite procedure has actually summed every term. Classical math papers over the infinity with that equals sign. On a computer, there is no such equals sign. After about 50 halvings in double precision, the remaining distance falls below the smallest representable number, the computer rounds it to zero, and you stop. Zeno is wrong in mathematics but right in floating-point — because floating-point has no completed-infinity to retreat to.

Addition isn't associative

Elementary school taught you that $(a + b) + c = a + (b + c)$ . You can group the additions however you like and the answer is the same. This is true for the real numbers. It is not true for the numbers a computer adds. Open a Python REPL:

>>> (1e20 + 1) - 1e20
0.0
>>> (1e20 - 1e20) + 1
1.0

Same three numbers, same operators, two different parenthesizations, two different answers. In the first, adding 1 to 1e20 runs into the ruler's gap-spacing at that scale — about 22,000 between consecutive marks. The 1 is below the precision and rounds away; the subsequent -1e20 leaves nothing. In the second, the two huge numbers cancel first, then 1 is added at a scale where 1 is well above precision.

This is the structural fact behind everything else on this page. Floating-point addition fails to be associative whenever the operands span very different magnitudes, and two algorithms that look mathematically equivalent can produce very different numerical results.

Sum a billion ones

Single-precision float has a 24-bit mantissa, so it can represent integers exactly up to $2^{24} = 16, 777, 216$ . Above that, the gap between consecutive representable integers is 2, then 4, then 8 — the spacing grows as the magnitude grows. Adding 1.0 to 16,777,216.0 rounds to the nearest representable value, and the nearest representable value is still 16,777,216.0:

import numpy as np
total = np.float32(2**24)        # 16,777,216.0
print(total + np.float32(1.0))
# 16777216.0   — the +1 disappeared

Now imagine summing 1.0 to itself a billion times in single precision. The running sum climbs cleanly up to $2^{24}$ . The next +1 does nothing, and so does the next, and so does the next. The remaining 983 million additions evaporate silently. The program runs to completion, produces no error, and returns 16,777,216 — off from the true answer of one billion by a factor of sixty.

You can't even store 0.1

Type 0.1 into a Python REPL. Looks fine. Now ask for it back at high precision:

>>> format(0.1, '.20f')
'0.10000000000000000555'

What you typed wasn't 0.1. It was the nearest double-precision number to 0.1, which is a hair bigger. The reason: 0.1 in binary is an infinitely repeating fraction (like 1/3 = 0.333... in decimal), and the double-precision representation has to truncate it somewhere. Whatever decimal value you typed, you got back the rounded binary version, which converts back to a slightly-different decimal.

The consequences cascade. Adding 0.1 three times doesn't give 0.3:

>>> 0.1 + 0.1 + 0.1
0.30000000000000004

Each 0.1 was slightly too big, and the errors compounded. This is why most finance code doesn't use floats at all — it uses decimal libraries (Python's decimal, Java's BigDecimal) where 0.1 is exactly 0.1 because the representation is base-10 rather than base-2.

IEEE 754 in two minutes

A double-precision floating-point number is 64 bits, laid out as:

| s | exponent (11 bits) | mantissa (52 bits) |

The value is $(- 1)^{s} \cdot (1. m) \cdot 2^{e - 1023}$ . The sign bit s flips positive or negative. The mantissa m, with an implicit leading 1, gives 53 bits of significand — about 15-17 decimal digits of precision. The exponent shifts the binary point.

The exponent shift is why the gaps between representable numbers grow with magnitude. Near 1.0 the spacing is $2^{- 52} \approx 2.2 \times 1 0^{- 16}$ . Near $1 0^{20}$ the spacing is $2^{- 52} \cdot 2^{66} \approx 22, 000$ . The mantissa has the same 52 bits of resolution at both scales, but those bits are scaled by the exponent. Add a number smaller than the local spacing and it disappears — the same fact behind every other failure on this page.

Single-precision float is the same idea with smaller fields: 1 sign bit, 8 exponent bits, 23 mantissa bits — about 7 decimal digits of precision, and the $2^{24}$ plateau from the "sum a billion ones" section.

When this bites you

Where this actually shows up on the rest of the site:

Long sums of mixed-magnitude numbers — the associativity failure from above. The fix is summation algorithms (Kahan, Neumaier) that recover the bits you'd otherwise lose. See stable summation.
Linear systems with a wide spread of singular values — small perturbations in the input amplify into large errors in the output. See condition number and stability.
Long-time numerical integration of ODEs — each step makes a small error; over millions of steps the errors accumulate. The energy of a "conservative" system drifts because the integrator isn't actually conservative once you account for floating-point.
Subtraction of nearby quantities — computing f(x + h) - f(x) for small h is the textbook example of catastrophic cancellation. Both terms have ~15 significant digits, but their difference might have only 3 or 4. The leading digits canceled and the rounding noise is now the dominant content.