Floating-point arithmetic and the IEEE model

Intuition Beginner

A computer cannot hold most numbers exactly. It has a fixed number of digits to work with, so it stores the closest number it can represent and throws away the rest. The number one-third becomes a long string of threes that has to stop somewhere; the leftover is gone the moment the value is stored. Every calculation then runs on these slightly-wrong stand-ins, and the small errors travel along with the answer.

Floating point is the format computers use to spread a fixed budget of digits across numbers of wildly different sizes. The idea is the same one behind scientific notation: write a number as a string of significant digits times a power of ten, like $6.022\times 10^{23}$. The significant digits carry the precision; the power says where the decimal point floats to. A computer does this in base two with a fixed number of bits for the digits and a fixed number for the exponent.

This design buys an enormous range. The same sixty-four bits can hold the mass of a galaxy and the mass of an electron, because the exponent slides the point wherever it is needed. The cost is that the gaps between representable numbers grow as the numbers grow. Near one the numbers are packed tightly; near a trillion they are spaced far apart. The relative gap, though, stays roughly constant, and that constant is the single most important number in the whole subject.

That constant is called machine epsilon. It measures the coarseness of the number grid: the largest relative error you can suffer just by storing a value. Knowing it lets you predict how much trust to place in a computed answer before you run a single operation.

Visual Beginner

Picture the representable numbers as tick marks on a ruler. Unlike a normal ruler with evenly spaced marks, this one has marks that bunch up near zero and spread out as you move right. Between $1$ and $2$ the marks sit one machine-epsilon apart. Between $2$ and $4$ the spacing doubles. Between $4$ and $8$ it doubles again. Each time the number doubles, the gap between neighbours doubles too, so the relative spacing — gap divided by value — stays the same across the whole ruler.

The table below shows the doubling pattern. Read it as: pick a number, and the gap to its nearest representable neighbour is the value times machine epsilon, rounded down to the spacing of its block.

interval	spacing between neighbours	relative gap
$1$ to $2$	$\epsilon$	$\epsilon$
$2$ to $4$	$2\epsilon$	$\epsilon$
$4$ to $8$	$4\epsilon$	$\epsilon$
$2^k$ to $2^{k+1}$	$2^k\epsilon$	$\epsilon$

The whole story of rounding error is in this picture: storing a real number snaps it to the nearest tick, and the snap distance is at most half the local spacing.

Worked example Beginner

Let us see why subtracting two close numbers can wreck precision. Suppose your computer keeps only four significant decimal digits. Take the two numbers $a=1.0001$ and $b=1.0000$. Each is already exact in four digits, so storing them loses nothing.

Step 1. Store the inputs. Both $a=1.0001$ and $b=1.0000$ fit in four significant digits with no rounding. So far there is no error at all.

Step 2. Subtract. The true difference is $a-b=0.0001$. The computer gets this exactly here, and writes it as $1.000\times 10^{-4}$.

Step 3. Now feed in inputs that were themselves rounded. Imagine the true values were $a=1.00014$ and $b=1.00002$, each rounded to four digits before the subtraction: $a$ stores as $1.0001$ and $b$ stores as $1.0000$. The true difference is $0.00012$, but the computed difference is $0.0001$.

Step 4. Measure the damage. The computed answer $0.0001$ differs from the true $0.00012$ by about seventeen percent. The inputs were each correct to five digits, yet the result is barely correct to one digit.

What this tells us. Subtracting two nearly equal numbers throws away the leading digits they share and promotes the tiny rounding errors in the trailing digits into the leading position of the answer. The operation itself is harmless; the trouble is that it magnifies errors already present in the inputs. This is catastrophic cancellation, and avoiding it is a central craft of numerical computing.

Check your understanding Beginner

Formal definition Intermediate+

A floating-point number system $\mathbb{F}=\mathbb{F}(\beta ,p,e_{\min },e_{\max })$ is the finite set of real numbers of the form

$$x=\pm m\times {\beta }^{e-p+1},m\in \{0,1,\dots ,{\beta }^p-1\},e_{\min }\le e\le e_{\max },$$

together with the special values described below. Here $\beta \ge 2$ is the base (radix), $p\ge 1$ is the precision (the number of significand digits), and $[e_{\min },e_{\max }]$ is the exponent range. A nonzero $x\in \mathbb{F}$ is normalized when its leading significand digit is nonzero, equivalently when it can be written

$$x=\pm (d_0.d_1{d}_2\cdots d_{p-1}{)}_{\beta }\times {\beta }^e,d_0\ne 0,0\le d_i\le \beta -1,$$

so that the significand lies in $[1,\beta )$. The IEEE-754 binary64 (double precision) format takes $\beta =2$, $p=53$, $e_{\min }=-1022$, $e_{\max }=1023$; binary32 (single precision) takes $\beta =2$, $p=24$, $e_{\min }=-126$, $e_{\max }=127$ ^{[IEEE — IEEE Standard for Floating-Point Arithmetic]}.

The unit roundoff (or machine epsilon in one common convention) is

$$u=\frac{1}{2}{\beta }^{1-p},$$

half the spacing of $\mathbb{F}$ in the interval $[1,\beta )$. (Some texts and language libraries call ${\beta }^{1-p}$ — the spacing itself, that is $2u$ — "machine epsilon"; the factor-of-two convention should be stated wherever it matters. This unit uses $u=\frac{1}{2}{\beta }^{1-p}$, the relative bound for round-to-nearest, throughout.)

The rounding map. Let $\mathrm{fl}:\mathbb{R}\to \mathbb{F}$ send each real $x$ in the normalized range (those $x$ with ${\beta }^{e_{\min }}\le |x|\le \Omega$, where $\Omega =(\beta -{\beta }^{1-p}){\beta }^{e_{\max }}$ is the largest finite element) to the nearest element of $\mathbb{F}$, breaking ties by the IEEE default round-to-nearest-even rule. The rounding modes of IEEE-754 are round-to-nearest-even, round-toward-$+\infty$, round-toward-$-\infty$, and round-toward-zero; unless stated otherwise round-to-nearest-even is assumed.

Standard model of floating-point arithmetic. For each real $x$ in the normalized range,

$$\mathrm{fl}(x)=x(1+\delta ),|\delta |\le u.$$

For the four arithmetic operations $\mathrm{op}\in \{+,-,\times ,÷\}$ and operands $a,b\in \mathbb{F}$ whose exact result $a\mathrm{op}b$ lies in the normalized range, the fundamental axiom of floating-point arithmetic states

$$\mathrm{fl}(a\mathrm{op}b)=(a\mathrm{op}b)(1+\delta ),|\delta |\le u,$$

which IEEE-754 guarantees by requiring each operation to be correctly rounded: the computed result equals the exact result rounded to $\mathbb{F}$. The same model is written equivalently as $\mathrm{fl}(a\mathrm{op}b)=(a\mathrm{op}b)/(1+{\delta }^{\prime })$ with $|{\delta }^{\prime }|\le u$ ^{[Higham, N. J. — Accuracy and Stability of Numerical Algorithms (2nd ed.)]}.

Overflow, underflow, subnormals, and special values. A result exceeding $\Omega$ in magnitude overflows and is mapped to the signed infinity $\pm \infty$; a nonzero result below ${\beta }^{e_{\min }}$ in magnitude underflows. To make underflow gradual, IEEE-754 includes the subnormal (denormalized) numbers $\pm (0.d_1\cdots d_{p-1}{)}_{\beta }\times {\beta }^{e_{\min }}$, with leading digit zero, filling the gap between $0$ and the smallest normalized number; they trade precision for reach and guarantee that $a-b=0$ if and only if $a=b$ (no spurious underflow-to-zero of a genuine difference). The format also carries a signed zero $\pm 0$ and the not-a-number value $\mathrm{NaN}$ produced by indeterminate forms such as $0/0$ or $\infty -\infty$.

Counterexamples to common slips

• Floating-point addition is not associative. In binary64, $(1+10^{-16})+10^{-16}$ rounds the first sum back to $1$ and yields $1$, while $1+(10^{-16}+10^{-16})$ yields $1+2\times 10^{-16}$, a number above $1$. Reordering a sum changes the result, so compilers may not freely reassociate floating-point additions.

• The relative error model fails in the subnormal range. The clean bound $\mathrm{fl}(x)=x(1+\delta )$, $|\delta |\le u$ holds only above the smallest normalized number. For subnormals the correct statement carries an absolute error term: $\mathrm{fl}(x)=x(1+\delta )+\eta$ with $|\eta |\le \frac{1}{2}{\beta }^{e_{\min }-p+1}$.

• "$\mathrm{fl}(a+b)=(a+b)(1+\delta )$ means the answer has small absolute error." It bounds the relative error of the operation. When $a+b$ is itself tiny because of cancellation, a small relative error on a small true value can be a large relative error with respect to the inputs — the cancellation problem is a conditioning fact, not a violation of the axiom.

• Machine epsilon is not the smallest positive float. The smallest positive subnormal in binary64 is about $4.9\times 10^{-324}$, vastly below $u\approx 1.1\times 10^{-16}$. Unit roundoff measures relative spacing near $1$; the smallest float measures absolute reach toward zero.

Key theorem with proof Intermediate+

The signature result is the accumulated-rounding bound for a sum: it is the smallest nontrivial backward-error analysis and the template for every later stability theorem, turning a chain of per-operation axioms into one clean guarantee on the final answer.

Theorem (error bound for recursive summation). Let $x_1,\dots ,x_n\in \mathbb{F}$ and compute their sum left to right in floating-point arithmetic under the standard model, producing ${\hat{S}}_n=\mathrm{fl}(\cdots \mathrm{fl}(\mathrm{fl}(x_1+x_2)+x_3)\cdots +x_n)$. If $nu<1$, then

$${\hat{S}}_n=\sum _{i=1}^n{x}_i(1+{\theta }_i),|{\theta }_i|\le {\gamma }_{n-1}:=\frac{(n-1)u}{1-(n-1)u},$$

so that the absolute error satisfies $|{\hat{S}}_n-{\sum }_i{x}_i|\le {\gamma }_{n-1}{\sum }_{i=1}^n|x_i|$ ^{[Higham, N. J. — Accuracy and Stability of Numerical Algorithms (2nd ed.)]}.

Proof. Write $S_k$ for the exact partial sum and ${\hat{S}}_k$ for the computed one, with ${\hat{S}}_1=x_1$. Each addition obeys the fundamental axiom: there is a ${\delta }_k$ with $|{\delta }_k|\le u$ and

$${\hat{S}}_k=\mathrm{fl}({\hat{S}}_{k-1}+x_k)=({\hat{S}}_{k-1}+x_k)(1+{\delta }_k),k=2,\dots ,n.$$

Unrolling the recurrence, each input $x_i$ is multiplied by the product of the rounding factors of every addition it participates in. The term $x_1$ passes through all $n-1$ additions, $x_2$ through the same $n-1$, and $x_i$ for $i\ge 2$ through $n-i+1$ of them:

$${\hat{S}}_n=x_1\prod _{k=2}^n(1+{\delta }_k)+\sum _{i=2}^n{x}_i\prod _{k=i}^n(1+{\delta }_k).$$

Each product of at most $n-1$ factors $(1+{\delta }_k)$ with $|{\delta }_k|\le u$ is written as $1+{\theta }_i$. The bound on such a product is the standard lemma: if $|{\delta }_k|\le u$ for $k=1,\dots ,m$ and $mu<1$, then ${\prod }_{k=1}^m(1+{\delta }_k)=1+\theta$ with $|\theta |\le {\gamma }_m=mu/(1-mu)$. To verify the lemma, the product lies between $(1-u{)}^m$ and $(1+u{)}^m$; the upper side gives $(1+u{)}^m-1\le mu/(1-mu)$ because $(1+u{)}^m\le 1/(1-mu)$ when $mu<1$ (each factor $1+u\le 1/(1-u)$ and the bound multiplies), and the lower side gives $1-(1-u{)}^m\le mu\le {\gamma }_m$ likewise. With $m\le n-1$ for every input, $|{\theta }_i|\le {\gamma }_{n-1}$. Substituting and using $|1+{\theta }_i|\le 1+{\gamma }_{n-1}$ on the difference,

$$|{\hat{S}}_n-\sum _{i=1}^n{x}_i|=\left|\sum _{i=1}^n{x}_i{\theta }_i\right|\le {\gamma }_{n-1}\sum _{i=1}^n|x_i|.\square$$

Bridge. This theorem is the foundational reason the whole subject of numerical stability hangs together: it shows that the per-operation axiom $\mathrm{fl}(a\mathrm{op}b)=(a\mathrm{op}b)(1+\delta )$ accumulates into a backward-error statement — the computed sum is the exact sum of slightly perturbed inputs $x_i(1+{\theta }_i)$ — and this is exactly the form every backward-stability result takes downstream. The summation bound builds toward the backward-error analysis of inner products, matrix-vector products, and the triangular solves that underlie Gaussian elimination, and it appears again in the loss-of-orthogonality bound for Gram-Schmidt in 01.01.09 and the backward-stability narration of the Golub-Kahan algorithm in 01.01.12. The ${\gamma }_n$ constant generalises the single $u$ of one operation to a chain of $n$ of them, and the central insight — that rounding error is best charged backward to the data rather than forward to the answer — is the Wilkinson paradigm that the conditioning theory of 43.01.02 and the backward-stability theory of 43.01.03 formalize in full. Putting these together, floating point supplies the constant $u$, conditioning supplies the amplification factor, and backward-error analysis multiplies the two; the bridge is that this elementary sum is already the entire pattern in miniature.

Exercises Intermediate+

Advanced results Master

Theorem 1 (correct rounding and the $\frac{1}{2}$ ulp bound). Round-to-nearest with the round-to-even tie rule realizes the minimal worst-case relative error among all rounding functions $\mathbb{R}\to \mathbb{F}$. For $x$ in the normalized range with ${\beta }^e\le |x|<{\beta }^{e+1}$, the spacing of $\mathbb{F}$ there is $\mathrm{ulp}(x)={\beta }^{e-p+1}$, and the nearest representable value satisfies $|\mathrm{fl}(x)-x|\le \frac{1}{2}\mathrm{ulp}(x)=\frac{1}{2}{\beta }^{e-p+1}$. Dividing by $|x|\ge {\beta }^e$ gives the relative bound $|\mathrm{fl}(x)-x|/|x|\le \frac{1}{2}{\beta }^{1-p}=u$, which is the source of every $|\delta |\le u$ in the standard model. The round-to-even rule additionally makes rounding unbiased over long computations, so accumulated errors do not drift systematically in one direction ^{[Muller, J.-M. et al. — Handbook of Floating-Point Arithmetic (2nd ed.)]}.

Theorem 2 (the IEEE-754 design guarantees). IEEE-754 requires the four arithmetic operations, the square root, and the fused multiply-add to be correctly rounded; mandates the binary32 and binary64 formats (with binary16, binary128, and decimal formats optional or extended); fixes signed zeros, signed infinities, and a quiet/signaling $\mathrm{NaN}$ system; defines five exceptions (invalid, division-by-zero, overflow, underflow, inexact) with sticky status flags; and specifies gradual underflow via subnormals. Correct rounding is what makes the fundamental axiom a theorem about the hardware rather than a modelling assumption: the same source program produces bit-identical results on every conforming platform, which is the reproducibility property the standard was designed to deliver ^{[Kahan, W. — Lecture Notes on the Status of IEEE Standard 754 for Binary Floating-Point Arithmetic]}.

Theorem 3 (compensated summation, Kahan). The Kahan compensated-summation algorithm maintains a running correction $c$ capturing the low-order bits lost at each addition and folds it back into the next term. Its computed sum satisfies ${\hat{S}}_n={\sum }_{i=1}^n{x}_i(1+{\theta }_i)+O(n{u}^2){\sum }_i|x_i|$ with $|{\theta }_i|\le 2u+O(n{u}^2)$, so the leading error constant is $2u$, independent of $n$, in place of the ${\gamma }_{n-1}\approx (n-1)u$ of naive summation. The mechanism is Sterbenz-style exact recovery: the corrected term $y=x_i-c$ and the new sum $s={\hat{S}}_{i-1}+y$ permit the exact low-order remainder $({\hat{S}}_{i-1}-s)+y$ to be computed in floating point, because the two-sum of nearby quantities is error-free ^{[Higham, N. J. — Accuracy and Stability of Numerical Algorithms (2nd ed.)]}.

Theorem 4 (cancellation as a conditioning phenomenon). Let $f(a,b)=a-b$ with $a,b>0$. Its relative condition number is $\kappa (a,b)=\frac{|a|+|b|}{|a-b|}$, which blows up as $a\to b$. The computed difference $\mathrm{fl}(a-b)=(a-b)(1+\delta )$, $|\delta |\le u$, has tiny backward error, so subtraction is a backward-stable operation; the large forward error of cancellation is entirely charged to the ill-conditioning of the subtraction problem at nearby arguments and to errors already present in $a$ and $b$. This is the cleanest separation of the two error sources — algorithm stability versus problem conditioning — and it is the conceptual seed of 43.01.02 and 43.01.03.

Synthesis. Floating point is the foundational reason numerical analysis is a rigorous subject rather than a folklore of hoping for the best: the finite set $\mathbb{F}(\beta ,p,e_{\min },e_{\max })$ together with correct rounding turns every arithmetic operation into the exact operation followed by a relative perturbation of size at most $u$, and this single axiom is what every later theorem stands on. The central insight is that rounding error is best accounted backward — the computed result of a stable algorithm is the exact result for slightly perturbed data — and the summation theorem is exactly this pattern in miniature, the bridge from one operation's $u$ to a chain's ${\gamma }_n$. Putting these together, the forward error of any computation factors as conditioning times backward error: floating point supplies the universal backward constant $u$, the condition number supplies the amplification, and their product is the achievable accuracy, the master inequality that 43.01.02 derives for general problems and 43.01.03 promotes to the definition of backward stability. Cancellation, gradual underflow, and non-associative summation are not defects to be patched but consequences of this same finite grid, and numerical computing is the systematic management of it: reformulate to avoid cancellation, control ${\gamma }_n$ by summation order or correction, and choose algorithms whose backward error stays at the level of $u$. This generalises directly into the stability theory of Gaussian elimination, least squares, and eigenvalue computation, where the same calculus governs achievable accuracy.

Full proof set Master

Proposition 1 (relative error of round-to-nearest). For every real $x$ with ${\beta }^{e_{\min }}\le |x|\le \Omega$, the round-to-nearest map satisfies $\mathrm{fl}(x)=x(1+\delta )$ with $|\delta |\le u=\frac{1}{2}{\beta }^{1-p}$.

Proof. Choose the integer $e$ with ${\beta }^e\le |x|<{\beta }^{e+1}$; such an $e$ exists and lies in $[e_{\min },e_{\max }]$ by the range hypothesis. The representable numbers in $[{\beta }^e,{\beta }^{e+1}]$ are equally spaced with gap $\mathrm{ulp}={\beta }^{e-p+1}$, since the significand runs over ${\beta }^{p-1}$ to ${\beta }^p-1$ in integer steps at scale ${\beta }^{e-p+1}$. The nearest representable value to $x$ is at distance at most half the gap: $|\mathrm{fl}(x)-x|\le \frac{1}{2}{\beta }^{e-p+1}$. (At the right endpoint of a block the upper neighbour is the first element of the next block, whose spacing is larger, so the half-gap bound still holds on the side facing $x$.) Set $\delta =(\mathrm{fl}(x)-x)/x$. Then $|\delta |=|\mathrm{fl}(x)-x|/|x|\le \frac{1}{2}{\beta }^{e-p+1}/{\beta }^e=\frac{1}{2}{\beta }^{1-p}=u$, using $|x|\ge {\beta }^e$. $\square$

Proposition 2 (product-of-factors lemma). If $|{\delta }_k|\le u$ for $k=1,\dots ,m$ and $mu<1$, then ${\prod }_{k=1}^m(1+{\delta }_k)=1+\theta$ with $|\theta |\le {\gamma }_m=mu/(1-mu)$.

Proof. Upper bound: ${\prod }_k(1+{\delta }_k)\le (1+u{)}^m$. Since $1+u\le (1-u{)}^{-1}$, one has $(1+u{)}^m\le (1-u{)}^{-m}$, and the elementary inequality $(1-u{)}^{-m}\le (1-mu{)}^{-1}$ for $mu<1$ (clear from $(1-u{)}^m\ge 1-mu$ by Bernoulli) gives ${\prod }_k(1+{\delta }_k)-1\le \frac{1}{1-mu}-1=\frac{mu}{1-mu}={\gamma }_m$. Lower bound: ${\prod }_k(1+{\delta }_k)\ge (1-u{)}^m\ge 1-mu\ge 1-{\gamma }_m$, again by Bernoulli and $mu\le {\gamma }_m$. Combining, $|\theta |=|{\prod }_k(1+{\delta }_k)-1|\le {\gamma }_m$. $\square$

Proposition 3 (signed-zero discrimination and gradual underflow). With subnormals present, for $a,b\in \mathbb{F}$ the computed difference $\mathrm{fl}(a-b)$ equals $0$ if and only if $a=b$.

Proof. If $a=b$ then $a-b=0\in \mathbb{F}$ and $\mathrm{fl}(0)=0$. Conversely suppose $a\ne b$; then $a-b\ne 0$. If $|a-b|\ge {\beta }^{e_{\min }}$ the difference is in the normalized range and rounds to a nonzero value. If $0<|a-b|<{\beta }^{e_{\min }}$, gradual underflow applies: the subnormal numbers tile $(-{\beta }^{e_{\min }},{\beta }^{e_{\min }})$ with spacing ${\beta }^{e_{\min }-p+1}$, and by Sterbenz's lemma the difference of two numbers within a factor of two is exact, while for $a,b$ farther apart the nonzero $a-b$ still rounds to the nearest subnormal, which is nonzero because the nearest subnormal to a nonzero value below the smallest positive subnormal's half-spacing is handled by the round-to-even rule producing the smallest subnormal rather than zero. In all cases a genuine nonzero difference rounds to a nonzero element. The flush-to-zero alternative (no subnormals) fails exactly this property: there, two distinct numbers closer than ${\beta }^{e_{\min }}$ subtract to a spurious $0$, which is why IEEE-754 mandates gradual underflow. $\square$

Proposition 4 (cancellation amplifies relative input error). Let $\tilde{a}=a(1+{ϵ}_a)$, $\tilde{b}=b(1+{ϵ}_b)$ be perturbed inputs with $|{ϵ}_a|,|{ϵ}_b|\le ϵ$, and $a,b>0$. Then the relative error of $\tilde{a}-\tilde{b}$ as an approximation to $a-b$ is at most $ϵ\cdot \frac{a+b}{|a-b|}$.

Proof. Compute $(\tilde{a}-\tilde{b})-(a-b)=a{ϵ}_a-b{ϵ}_b$. Hence $|(\tilde{a}-\tilde{b})-(a-b)|\le |a||{ϵ}_a|+|b||{ϵ}_b|\le ϵ(a+b)$. Dividing by $|a-b|$ gives the relative error bound $ϵ(a+b)/|a-b|$. The factor $(a+b)/|a-b|$ is the relative condition number of subtraction; it is $1$ when $a,b$ have opposite signs (addition of like-signed magnitudes) and unbounded as $a\to b$, which is the precise statement that cancellation converts a small relative input error into a large relative output error without any rounding occurring in the subtraction itself. $\square$

Connections Master

• The accumulated-rounding analysis built here is the substrate of the conditioning theory in 43.01.02: the forward-error bound of the summation theorem factors as a backward constant (${\gamma }_n$, built from $u$) times the ratio $\sum |x_i|/|\sum x_i|$, and that ratio is precisely the condition number of the summation problem. Conditioning lifts this single example to the general theory of how a problem $f:X\to Y$ amplifies input perturbations, independent of any algorithm.

• Backward stability, defined in 43.01.03, is the abstraction of the backward-error statement proved here for summation and in the exercises for dot products: an algorithm is backward stable when its computed output is the exact output for data perturbed by $O(u)$. The fundamental theorem of backward-error analysis — forward error $\le$ condition number $\times$ backward error — is the master inequality this unit exhibits in the special case of summation, and 43.01.03 proves it in general.

• The loss-of-orthogonality bound for modified Gram-Schmidt in 01.01.09, the backward-stability of Householder QR, and the backward-error narration of the Golub-Kahan SVD algorithm in 01.01.12 all instantiate the standard model of this unit: each is a chain of correctly-rounded operations whose accumulated relative error is controlled by the $u$ and ${\gamma }_n$ machinery established here, specialized to orthogonal transformations whose perfect conditioning ($\kappa =1$) keeps the forward error at the backward level.

Historical & philosophical context Master

Systematic rounding-error analysis began with James H. Wilkinson, whose Rounding Errors in Algebraic Processes (1963) ^{[Wilkinson, J. H. — Rounding Errors in Algebraic Processes]} established the backward-error viewpoint: rather than tracking the growth of error forward through a computation, charge the total error back to a perturbation of the original data and ask whether that perturbation is small. The summation and dot-product bounds of this unit are Wilkinson's, in the ${\gamma }_n$ notation later standardized by Higham. Before Wilkinson the prevailing fear, voiced by von Neumann and Goldstine in their 1947 analysis of matrix inversion, was that rounding error would accumulate catastrophically in large computations; the backward-error reframing showed that well-designed algorithms on well-conditioned problems are safe, and located the real danger in conditioning rather than in arithmetic.

The hardware side was chaotic until the late 1970s: every manufacturer used a different base, precision, and rounding rule, so the same program gave different answers on different machines, and some gave wrong answers for subtle reasons. William Kahan led the design of the IEEE-754 standard, ratified in 1985 and revised in 2008 and 2019 ^{[IEEE — IEEE Standard for Floating-Point Arithmetic]}, whose insistence on correct rounding of every basic operation is what turns the fundamental axiom from a modelling hypothesis into a hardware guarantee, and whose introduction of gradual underflow via subnormals preserved the property that a nonzero difference never rounds to zero. The standard is the reason a numerical program is now portable and its error analysis platform-independent; Kahan received the 1989 Turing Award in part for this work.

Bibliography Master

@book{trefethenbau1997,
  author    = {Trefethen, Lloyd N. and Bau, David},
  title     = {Numerical Linear Algebra},
  publisher = {Society for Industrial and Applied Mathematics},
  year      = {1997}
}

@book{higham2002accuracy,
  author    = {Higham, Nicholas J.},
  title     = {Accuracy and Stability of Numerical Algorithms},
  edition   = {2},
  publisher = {Society for Industrial and Applied Mathematics},
  year      = {2002}
}

@article{goldberg1991floatingpoint,
  author  = {Goldberg, David},
  title   = {What Every Computer Scientist Should Know About Floating-Point Arithmetic},
  journal = {ACM Computing Surveys},
  volume  = {23},
  number  = {1},
  year    = {1991},
  pages   = {5--48}
}

@book{muller2018handbook,
  author    = {Muller, Jean-Michel and Brunie, Nicolas and de Dinechin, Florent and Jeannerod, Claude-Pierre and Joldes, Mioara and Lef\`{e}vre, Vincent and Melquiond, Guillaume and Revol, Nathalie and Torres, Serge},
  title     = {Handbook of Floating-Point Arithmetic},
  edition   = {2},
  publisher = {Birkh\"{a}user},
  year      = {2018}
}

@book{wilkinson1963rounding,
  author    = {Wilkinson, James H.},
  title     = {Rounding Errors in Algebraic Processes},
  publisher = {Prentice-Hall},
  year      = {1963}
}

@misc{ieee754_2019,
  author       = {{IEEE}},
  title        = {IEEE Standard for Floating-Point Arithmetic},
  howpublished = {IEEE Std 754-2019},
  year         = {2019},
  doi          = {10.1109/IEEESTD.2019.8766229}
}

@misc{kahan1996status,
  author       = {Kahan, William},
  title        = {Lecture Notes on the Status of IEEE Standard 754 for Binary Floating-Point Arithmetic},
  year         = {1996},
  howpublished = {University of California, Berkeley}
}