Rayleigh quotient and the Courant-Fischer min-max characterisation of eigenvalues

Intuition Beginner

A symmetric matrix stretches space along a set of perpendicular axes, with a stretch factor for each axis. The spectral theorem 01.01.13 guarantees those axes and factors exist; the factors are the eigenvalues. A natural question is whether you can find the biggest and smallest stretch factors without first solving for every eigenvalue. The Rayleigh quotient answers yes, and it does so by a simple measurement you can take on any vector at all.

The Rayleigh quotient of a vector is the amount the matrix stretches that vector along its own direction, measured as a ratio. Push the vector through the matrix, see how much of the result points back along the original vector, and divide by the vector's own length-squared. For an eigenvector the answer is exactly its eigenvalue. For any other vector the answer is a blend of eigenvalues, and a blend can never exceed the largest ingredient or fall below the smallest.

So the largest eigenvalue is the highest value the Rayleigh quotient ever reaches, and the smallest eigenvalue is its lowest value. The middle eigenvalues are trickier: each one is the best you can do on some restricted family of directions, and the Courant-Fischer theorem says exactly which family. This turns every eigenvalue into the answer to an optimisation problem.

Visual Beginner

The picture shows a symmetric two-by-two matrix and its Rayleigh quotient evaluated all the way around the unit circle. As a unit vector sweeps from angle zero to a full turn, the Rayleigh quotient rises and falls between two values: it peaks at the direction of the top eigenvector, where it equals the larger eigenvalue, and it bottoms out at the perpendicular direction, where it equals the smaller eigenvalue.

Two facts are visible. The curve never goes above the larger eigenvalue or below the smaller one, so the eigenvalues are the ceiling and floor of the Rayleigh quotient. The peak and the valley occur at perpendicular directions, which are the two eigendirections of the symmetric matrix.

Worked example Beginner

Take the symmetric matrix

$$A=\left(\begin{array}{cc}3& 1\\ 1& 3\end{array}\right).$$

Step 1. Write the Rayleigh quotient for a unit vector $v=(\cos \theta ,\sin \theta )$. Since $v$ has length $1$, the denominator is $1$ and the quotient is just the top part, $R(v)=(Av)\cdot v$.

Step 2. Compute $Av$. Multiplying, $Av=(3\cos \theta +\sin \theta ,\cos \theta +3\sin \theta )$.

Step 3. Dot $Av$ with $v$. We get $R(v)=(3\cos \theta +\sin \theta )\cos \theta +(\cos \theta +3\sin \theta )\sin \theta =3{\cos }^2\theta +2\sin \theta \cos \theta +3{\sin }^2\theta$.

Step 4. Simplify. Using ${\cos }^2\theta +{\sin }^2\theta =1$ and $2\sin \theta \cos \theta =\sin (2\theta )$, this becomes $R(v)=3+\sin (2\theta )$.

Step 5. Read off the high and low values. The term $\sin (2\theta )$ ranges from $-1$ to $+1$, so $R(v)$ ranges from $2$ to $4$. The maximum $4$ happens when $\sin (2\theta )=1$, that is $\theta =45$ degrees, the direction $(1,1)/\sqrt{2}$. The minimum $2$ happens at $\theta =135$ degrees, the direction $(1,-1)/\sqrt{2}$.

Step 6. Check against the eigenvalues. The characteristic polynomial is $(t-3{)}^2-1=(t-2)(t-4)$, so the eigenvalues are $2$ and $4$. These match the floor and ceiling found above.

What this tells us: the largest eigenvalue $4$ is the maximum of the Rayleigh quotient and the smallest eigenvalue $2$ is its minimum, with both extremes reached along the eigendirections. You found the eigenvalues by maximising and minimising a single ratio, not by any extra factoring work.

Check your understanding Beginner

Formal definition Intermediate+

Let $(V,⟨\cdot ,\cdot ⟩)$ be a finite-dimensional inner-product space over $\mathbb{F}\in \{\mathbb{R},\mathbb{C}\}$ in the sense of 01.01.09, with norm $\Vert v\Vert =\sqrt{⟨v,v⟩}$ and the inner product linear in the first slot and conjugate-linear in the second. Throughout this unit $A:V\to V$ is self-adjoint, $A=A^{\ast }$, with adjoint as in 01.01.10. By the spectral theorem 01.01.13 the eigenvalues of $A$ are real; list them with multiplicity in decreasing order

$${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_n,n=\dim V,$$

and fix an orthonormal eigenbasis $u_1,\dots ,u_n$ with $A{u}_i={\lambda }_i{u}_i$. The largest eigenvalue ${\lambda }_1={\lambda }_{\max }$ and the smallest ${\lambda }_n={\lambda }_{\min }$.

Definition (Rayleigh quotient). For $v\in V$ with $v\ne 0$, the Rayleigh quotient of $A$ at $v$ is

$$R(v)=R_A(v)=\frac{⟨Av,v⟩}{⟨v,v⟩}.$$

Self-adjointness makes the numerator real: $⟨Av,v⟩=⟨v,Av⟩=\overline{⟨Av,v⟩}$, so $R(v)\in \mathbb{R}$ for all $v\ne 0$. The quotient is scale-invariant, $R(cv)=R(v)$ for every nonzero scalar $c$, so it is a function on the projective space of directions, equivalently on the unit sphere $\{v:\Vert v\Vert =1\}$, where it reduces to $R(v)=⟨Av,v⟩$.

Definition (Rayleigh-Ritz subspace value). For a subspace $S\subseteq V$ with $S\ne \{0\}$, write

$$\overline{R}(S)=\underset{0\ne v\in S}{\max }R(v),\underline{R}(S)=\underset{0\ne v\in S}{\min }R(v)$$

for the largest and smallest Rayleigh values over the directions inside $S$. Both extrema are attained because $R$ is continuous on the compact unit sphere of $S$. The eigenvalues will be obtained by optimising $\overline{R}$ and $\underline{R}$ over all subspaces of a fixed dimension.

Notation: $\mathrm{Gr}(k,V)$ denotes the set of $k$-dimensional subspaces of $V$; $S^{\perp }$ is the orthogonal complement of $S$; $\dim S$ is the dimension of $S$. The Kronecker delta ${\delta }_{ij}$ is $1$ when $i=j$ and $0$ otherwise. The notation $A⪰0$ means $A$ is positive semidefinite, that is $⟨Av,v⟩\ge 0$ for all $v$.

Counterexamples to common slips

• The reality $R(v)\in \mathbb{R}$ and the range $[{\lambda }_n,{\lambda }_1]$ both require self-adjointness. For the non-self-adjoint $A=\left(\begin{array}{cc}0& -1\\ 1& 0\end{array}\right)$ on ${\mathbb{C}}^2$, the numerator $⟨Av,v⟩$ is purely imaginary on most vectors, and the eigenvalues $\pm i$ are not real; the extremal characterisation simply does not apply.
• "The maximum of $R$ is attained at a unique direction" fails when ${\lambda }_1$ is a repeated eigenvalue. If ${\lambda }_1={\lambda }_2$, every unit vector in the two-dimensional top eigenspace attains the maximum, so the maximiser is a whole sphere of directions, not a point.
• Courant-Fischer requires the eigenvalues to be listed with multiplicity. Omitting repetitions shifts the index $k$ and breaks the formula; ${\lambda }_k$ in the min-max statement is the $k$-th eigenvalue counted with multiplicity, not the $k$-th distinct value.

Key theorem with proof Intermediate+

Theorem (Rayleigh-Ritz extremal characterisation; Shilov Ch. 10 §10.2 ^{[source pending]}; Horn-Johnson §4.2 ^{[source pending]}). Let $A = A^$beself-adjointonafinite-dimensionalinner-productspace$V$witheigenvalues$\lambda_1 \geq \cdots \geq \lambda_n$andorthonormaleigenbasis$u_1, \ldots, u_n$.Thenforevery$v \neq 0$,*

$${\lambda }_n\le R(v)\le {\lambda }_1,$$

the range of $R$ is exactly the closed interval $[{\lambda }_n,{\lambda }_1]$, and

$${\lambda }_1=\underset{v\ne 0}{\max }R(v),{\lambda }_n=\underset{v\ne 0}{\min }R(v),$$

with the maximum attained precisely on the ${\lambda }_1$-eigenspace and the minimum on the ${\lambda }_n$-eigenspace.

Proof. Expand an arbitrary $v\ne 0$ in the orthonormal eigenbasis: $v={\sum }_{i=1}^n{c}_i{u}_i$ with $c_i=⟨v,u_i⟩\in \mathbb{F}$. Orthonormality gives $⟨v,v⟩={\sum }_i|c_i{|}^2$, and since $A{u}_i={\lambda }_i{u}_i$,

$$⟨Av,v⟩=⟨\sum _i{\lambda }_i{c}_i{u}_i,\sum _j{c}_j{u}_j⟩=\sum _{i,j}{\lambda }_i{c}_i\overline{c_j}⟨u_i,u_j⟩=\sum _i{\lambda }_i|c_i{|}^2,$$

where orthonormality $⟨u_i,u_j⟩={\delta }_{ij}$ collapsed the double sum. Therefore

$$R(v)=\frac{{\sum }_i{\lambda }_i|c_i{|}^2}{{\sum }_i|c_i{|}^2}.$$

The right-hand side is a convex combination of the eigenvalues ${\lambda }_i$ with weights $w_i=|c_i{|}^2/{\sum }_j|c_j{|}^2\ge 0$ summing to $1$. A convex combination of real numbers lies between their minimum and maximum, so ${\lambda }_n\le R(v)\le {\lambda }_1$.

For attainment and the full range: setting $v=u_1$ puts all weight on ${\lambda }_1$, giving $R(u_1)={\lambda }_1$; setting $v=u_n$ gives $R(u_n)={\lambda }_n$. For an intermediate value $\mu \in [{\lambda }_n,{\lambda }_1]$, choose $t\in [0,1]$ with $\mu =t{\lambda }_1+(1-t){\lambda }_n$ and take $v=\sqrt{t}{u}_1+\sqrt{1-t}{u}_n$; then $R(v)=t{\lambda }_1+(1-t){\lambda }_n=\mu$. So the range is all of $[{\lambda }_n,{\lambda }_1]$.

Finally, suppose $R(v)={\lambda }_1$ for some $v={\sum }_i{c}_i{u}_i\ne 0$. Then ${\sum }_i{\lambda }_i|c_i{|}^2={\lambda }_1{\sum }_i|c_i{|}^2$, that is ${\sum }_i({\lambda }_1-{\lambda }_i)|c_i{|}^2=0$. Every term is non-negative because ${\lambda }_1\ge {\lambda }_i$, so each term vanishes: $c_i=0$ whenever ${\lambda }_i<{\lambda }_1$. Hence $v$ lies in the ${\lambda }_1$-eigenspace. The minimum case is symmetric, replacing ${\lambda }_1-{\lambda }_i\ge 0$ by ${\lambda }_i-{\lambda }_n\ge 0$. $\square$

Bridge. The Rayleigh-Ritz characterisation builds toward the full Courant-Fischer min-max theorem proved in the Advanced results, where the same convex-combination identity is restricted to subspaces to pin down every intermediate eigenvalue, and it appears again in 01.01.12 (singular value decomposition), where applying it to $A^{\ast }A$ identifies the largest singular value as ${\sigma }_1={\max }_{\Vert v\Vert =1}\Vert Av\Vert$. The mechanism is the diagonalisation $R(v)={\sum }_i{w}_i{\lambda }_i$ supplied by the spectral theorem 01.01.13: once the self-adjoint operator is written in its orthonormal eigenbasis, the Rayleigh quotient becomes a weighted average of eigenvalues, and weighted averages are governed entirely by the extreme ingredients. The foundational identity is $⟨Av,v⟩={\sum }_i{\lambda }_i|⟨v,u_i⟩{|}^2$, the spectral expansion of the quadratic form attached to $A$. Putting these together, the extremal characterisation removes the need to compute a characteristic polynomial to find the dominant eigenvalue, justifies the power-iteration and Rayleigh-quotient-iteration algorithms of numerical linear algebra, and is the finite-dimensional shadow of the variational principle that estimates ground-state energies in quantum mechanics, which appears again in 12.07.03 (the Rayleigh-Ritz method).

Exercises Intermediate+

Lean formalization Intermediate+

Mathlib carries the Rayleigh quotient and the supremum/infimum eigenvalue facts for self-adjoint operators, but the full Courant-Fischer min-max equality, Cauchy interlacing, and Weyl's inequalities are not present as single named theorems. The readable view of the intended statements:

import Mathlib.Analysis.InnerProductSpace.Rayleigh
import Mathlib.Analysis.InnerProductSpace.Spectrum

variable {𝕜 V : Type*} [RCLike 𝕜]
  [NormedAddCommGroup V] [InnerProductSpace 𝕜 V] [FiniteDimensional 𝕜 V]

/-- The supremum of the Rayleigh quotient over the unit sphere is an
eigenvalue (the top eigenvalue). Mathlib:
`LinearMap.IsSymmetric.hasEigenvalue_iSup`. -/
example (T : V →ₗ[𝕜] V) (hT : T.IsSymmetric) (hV : 0 < Module.rank 𝕜 V) :
    Module.End.HasEigenvalue T
      (⨆ x : {v : V // v ≠ 0}, RCLike.re (T.reApplyInnerSelf x / ‖(x : V)‖ ^ 2)) :=
  hT.hasEigenvalue_iSup hV

/-- Courant-Fischer max-min characterisation of the k-th eigenvalue.
NOT a single named Mathlib theorem. -/
theorem courant_fischer_maxmin
    (T : V →ₗ[𝕜] V) (hT : T.IsSymmetric) (k : ℕ)
    (lam : ℕ → ℝ) (hsort : Antitone lam) /- eigenvalues, decreasing -/ :
    lam k = sSup { r : ℝ | ∃ S : Submodule 𝕜 V, Module.finrank 𝕜 S = k + 1 ∧
        r = sInf { R : ℝ | ∃ v : V, v ∈ S ∧ v ≠ 0 ∧
          R = RCLike.re (inner (T v) v) / ‖v‖ ^ 2 } } :=
  sorry  -- S_k test subspace for ≥; dimension-count intersection for ≤

/-- Cauchy interlacing: eigenvalues of a compression to a codim-1
subspace interlace those of T. NOT a single named Mathlib theorem. -/
theorem cauchy_interlace
    (T : V →ₗ[𝕜] V) (hT : T.IsSymmetric) (W : Submodule 𝕜 V)
    (hW : Module.finrank 𝕜 W + 1 = Module.finrank 𝕜 V)
    (lamT lamW : ℕ → ℝ) (k : ℕ) :
    lamT (k + 1) ≤ lamW k ∧ lamW k ≤ lamT k :=
  sorry  -- apply courant_fischer to T and to the compression P_W T P_W

The proof gap to a clean contribution: (i) order the eigenvalues with multiplicity as an antitone Fin n → ℝ and connect to Module.End.eigenvalues; (ii) prove courant_fischer_maxmin from the spectral orthonormal basis using the S_k = span(u_0, ..., u_k) test subspace and the Submodule.finrank_sup_add_finrank_inf_le dimension-count for the intersection; (iii) derive cauchy_interlace and weyl_inequality as corollaries. Each is reachable from Mathlib.Analysis.InnerProductSpace.Rayleigh and the finite-dimensional spectral theorem, but no current named lemma presents the min-max double characterisation or the interlacing inequalities in packaged form.

Advanced results Master

Theorem (Courant-Fischer min-max; Fischer 1905 ^{[source pending]}; Courant 1920 ^{[source pending]}; Horn-Johnson §4.2 ^{[source pending]}). Let $A = A^$on$V$witheigenvalues$\lambda_1 \geq \cdots \geq \lambda_n$listedwithmultiplicity.Foreach$k \in {1, \ldots, n}$,*

$${\lambda }_k=\underset{\begin{array}{c}S\subseteq V\\ \dim S=k\end{array}}{\max }\underset{0\ne v\in S}{\min }R(v)=\underset{\begin{array}{c}S\subseteq V\\ \dim S=n-k+1\end{array}}{\min }\underset{0\ne v\in S}{\max }R(v).$$

The first equality recovers ${\lambda }_1$ as a plain maximum (the case $k=1$) and ${\lambda }_n$ as a plain minimum (the case $k=n$ of the second form), so the Rayleigh-Ritz characterisation is the boundary case of the general formula. The decisive feature is that the characterisation of ${\lambda }_k$ refers to no eigenvector other than through the optimisation over $k$-dimensional subspaces; this is what makes the formula usable for operators whose eigenvectors are unavailable, and it is the form that survives passage to the infinite-dimensional discrete spectrum. The full proof is in the proof set below.

Theorem (Cauchy interlacing; Cauchy 1829 ^{[source pending]}). Let $A = A^$onan$n$-dimensionalspacehaveeigenvalues$\lambda_1 \geq \cdots \geq \lambda_n$,andlet$B$bethecompression$B = P A|W$of$A$toan$(n-1)$-dimensionalsubspace$W$(equivalently,aprincipal$(n-1) \times (n-1)$submatrixinanorthonormalbasisadaptedto$W$),witheigenvalues$\mu_1 \geq \cdots \geq \mu{n-1}$. Then*

$${\lambda }_1\ge {\mu }_1\ge {\lambda }_2\ge {\mu }_2\ge \cdots \ge {\mu }_{n-1}\ge {\lambda }_n.$$

More generally, deleting $m$ rows and the corresponding columns produces eigenvalues ${\mu }_j$ with ${\lambda }_{j+m}\le {\mu }_j\le {\lambda }_j$.

Interlacing is the workhorse behind the symmetric eigenvalue problem: the eigenvalues of a leading principal submatrix sandwich those of the full matrix, which justifies bisection methods using Sturm sequences and explains why adding a row and column to a symmetric matrix can shift each eigenvalue by at most one slot.

Theorem (Weyl's inequalities; Weyl 1912 ^{[source pending]}). Let $A,B$ be self-adjoint on $V$ with eigenvalues listed decreasingly. For indices $i,j$ with $i+j-1\le n$,

$${\lambda }_{i+j-1}(A+B)\le {\lambda }_i(A)+{\lambda }_j(B),$$

and dually ${\lambda }_{i+j-n}(A+B)\ge {\lambda }_i(A)+{\lambda }_j(B)$ when $i+j-n\ge 1$. In particular ${\lambda }_k(A)+{\lambda }_n(B)\le {\lambda }_k(A+B)\le {\lambda }_k(A)+{\lambda }_1(B)$, so each eigenvalue is Lipschitz in the operator: $|{\lambda }_k(A+B)-{\lambda }_k(A)|\le \Vert B\Vert$.

Weyl's inequalities are the eigenvalue-perturbation theory that controls how the spectrum moves when a self-adjoint operator is perturbed by another. The special case $B⪰0$ recovers the monotonicity ${\lambda }_k(A)\le {\lambda }_k(A+B)$, and the Lipschitz bound is the spectral stability statement underpinning numerical eigenvalue computation and the continuity of the spectrum in perturbation theory.

Theorem (spectral gap and the second eigenvalue). With ${\lambda }_1\ge {\lambda }_2\ge \cdots$, the second eigenvalue admits the constrained form

$${\lambda }_2=\underset{0\ne v\perp u_1}{\max }R(v),$$

the maximum of the Rayleigh quotient over the orthogonal complement of the top eigenvector. The gap ${\lambda }_1-{\lambda }_2$ measures how sharply the maximiser is isolated: for a positive operator normalised to ${\lambda }_1=1$, the gap $1-{\lambda }_2$ controls the geometric convergence rate of power iteration and, in the graph-Laplacian setting, the algebraic connectivity and mixing time.

The deflation form ${\lambda }_{k+1}={\max }_{v\perp u_1,\dots ,u_k}R(v)$ is the recursive version of the Rayleigh-Ritz characterisation: having found the top $k$ eigenvectors, the next eigenvalue is the maximum of $R$ on what remains. Courant-Fischer is the eigenvector-free upgrade that removes the dependence on $u_1,\dots ,u_k$.

Theorem (min-max for semibounded self-adjoint operators; Courant-Fischer-Weyl in the Hilbert-space form; Reed-Simon Vol. IV §XIII.1 ^{[source pending]}). Let $A$ be a self-adjoint operator on a Hilbert space $H$, bounded below, with $\Sigma =\inf {\sigma }_{\mathrm{ess}}(A)$ the bottom of the essential spectrum. Define

$${\mu }_k(A)=\underset{\begin{array}{c}S\subseteq D(A)\\ \dim S=k\end{array}}{\min }\underset{\begin{array}{c}v\in S\\ \Vert v\Vert =1\end{array}}{\max }⟨Av,v⟩.$$

Then for each $k$ either ${\mu }_k(A)<\Sigma$ and ${\mu }_k(A)$ is the $k$-th eigenvalue below the essential spectrum (counted with multiplicity), or ${\mu }_k(A)=\Sigma$ and there are at most $k-1$ eigenvalues below $\Sigma$.

This is the theorem that powers the variational estimation of bound-state energies. The finite sum over eigenvalues becomes an optimisation over finite-dimensional trial subspaces of the domain, and the Rayleigh-Ritz method computes upper bounds for the discrete eigenvalues below the continuous spectrum. The Sturm-Liouville operator $-\frac{d}{dx}(p\frac{d}{dx})+q$ on a bounded interval and the Dirichlet Laplacian on a bounded domain are the canonical instances, with the min-max principle giving Courant's nodal-domain bound and Weyl's eigenvalue-counting asymptotics.

Synthesis. The Rayleigh quotient converts the algebraic eigenvalue problem for a self-adjoint operator into a problem of optimisation, and the spectral theorem 01.01.13 is what makes the conversion exact: in the orthonormal eigenbasis $R(v)$ is a weighted average of eigenvalues, so its extremes are the extreme eigenvalues and its critical points are the eigenvectors. The Courant-Fischer min-max formula completes the picture by characterising every eigenvalue, not only the top and bottom, through optimisation over subspaces of fixed dimension, and it does so without naming a single eigenvector. From that one formula the interlacing of Cauchy and the perturbation inequalities of Weyl follow as corollaries by applying the same max-min to a compression or to a sum, and the monotonicity ${\lambda }_k(A)\le {\lambda }_k(A+B)$ for $B⪰0$ is immediate from the pointwise inequality of Rayleigh quotients. Putting these together, the gradient computation $\nabla R(v)\propto Av-R(v)v$ identifies critical points with eigenvectors and drives Rayleigh-quotient iteration, the second-eigenvalue gap controls the convergence of power iteration and the connectivity of graphs, and the eigenvector-free min-max form passes intact to the discrete spectrum of a semibounded self-adjoint operator, where it becomes the variational principle of 12.07.03 and the eigenvalue-asymptotics machinery of Weyl and Courant for the differential operators of mathematical physics. The Rayleigh quotient, the min-max theorem, interlacing, and the perturbation inequalities are four readings of the single fact that the quadratic form $⟨Av,v⟩$ on the unit sphere has its eigenvalues as critical values.

Full proof set Master

Proposition (Courant-Fischer min-max, both forms). Let $A = A^$on$V$haveeigenvalues$\lambda_1 \geq \cdots \geq \lambda_n$withorthonormaleigenbasis$u_1, \ldots, u_n$.Foreach$k$,* $$ \lambda_k = \max_{\dim S = k}\ \min_{0 \neq v \in S} R(v) = \min_{\dim S = n-k+1}\ \max_{0 \neq v \in S} R(v). $$

Proof. Max-min form, lower bound. Take $S_k=\mathrm{span}(u_1,\dots ,u_k)$, which has dimension $k$. For $0\ne v={\sum }_{i=1}^k{c}_i{u}_i\in S_k$, $$ R(v) = \frac{\sum_{i=1}^k \lambda_i |c_i|^2}{\sum_{i=1}^k |c_i|^2} \geq \lambda_k, $$ because each ${\lambda }_i\ge {\lambda }_k$ for $i\le k$, with equality at $v=u_k$. Hence ${\min }_{0\ne v\in S_k}R(v)={\lambda }_k$, so the outer maximum is at least ${\lambda }_k$.

Max-min form, upper bound. Let $S$ be any $k$-dimensional subspace and set $W=\mathrm{span}(u_k,\dots ,u_n)$, of dimension $n-k+1$. By the subspace dimension formula $\dim (S\cap W)=\dim S+\dim W-\dim (S+W)\ge k+(n-k+1)-n=1$, there is a nonzero $v\in S\cap W$. Writing $v={\sum }_{i=k}^n{c}_i{u}_i$, $$ R(v) = \frac{\sum_{i=k}^n \lambda_i |c_i|^2}{\sum_{i=k}^n |c_i|^2} \leq \lambda_k, $$ since each ${\lambda }_i\le {\lambda }_k$ for $i\ge k$. Thus ${\min }_{0\ne v\in S}R(v)\le R(v)\le {\lambda }_k$ for every such $S$, so the outer maximum is at most ${\lambda }_k$. The two bounds give the max-min equality.

Min-max form. Apply the max-min form to $-A$, whose eigenvalues are $-{\lambda }_n\ge \cdots \ge -{\lambda }_1$ in decreasing order, so the $k$-th eigenvalue of $-A$ is $-{\lambda }_{n-k+1}$. The Rayleigh quotient of $-A$ is $-R$, and $\min (-R)=-\max (R)$, $\max (-R)=-\min (R)$. The max-min statement for $-A$ at index $k$ reads $-{\lambda }_{n-k+1}={\max }_{\dim S=k}{\min }_{v\in S}(-R(v))=-{\min }_{\dim S=k}{\max }_{v\in S}R(v)$. Re-indexing $k\mapsto n-k+1$ (so $\dim S=n-k+1$) gives ${\lambda }_k={\min }_{\dim S=n-k+1}{\max }_{0\ne v\in S}R(v)$. $\square$

Proposition (Cauchy interlacing). Let $A = A^$onan$n$-dimensional$V$witheigenvalues$\lambda_1 \geq \cdots \geq \lambda_n$,let$W \subseteq V$havedimension$n - 1$,andlet$B = P_W A|W$bethecompression(with$P_W$theorthogonalprojectiononto$W$),aself-adjointoperatoron$W$witheigenvalues$\mu_1 \geq \cdots \geq \mu{n-1}$.Then$\lambda_{k+1} \leq \mu_k \leq \lambda_k$for$1 \leq k \leq n-1$.*

Proof. For $v\in W$, $⟨Bv,v⟩=⟨P_{W}Av,v⟩=⟨Av,P_{W}v⟩=⟨Av,v⟩$ since $P_{W}v=v$ and $P_W=P_W^{\ast }$. So the Rayleigh quotient of $B$ on $W$ agrees with that of $A$ restricted to $W$: $R_B(v)=R_A(v)$ for $0\ne v\in W$.

Upper bound ${\mu }_k\le {\lambda }_k$. By Courant-Fischer for $B$ on the $(n-1)$-dimensional $W$, ${\mu }_k={\max }_{\begin{array}{c}S\subseteq W\\ \dim S=k\end{array}}{\min }_{0\ne v\in S}{R}_A(v)$. Every $k$-dimensional $S\subseteq W$ is also a $k$-dimensional subspace of $V$, so this maximum is over a subfamily of the subspaces appearing in the Courant-Fischer max-min for ${\lambda }_k$. A maximum over a smaller family is no larger, so ${\mu }_k\le {\lambda }_k$.

Lower bound ${\lambda }_{k+1}\le {\mu }_k$. Use the min-max form. For $A$, ${\lambda }_{k+1}={\min }_{\dim T=n-k}{\max }_{0\ne v\in T}{R}_A(v)$. For $B$ on $W$, ${\mu }_k={\min }_{\begin{array}{c}T\subseteq W\\ \dim T=(n-1)-k+1=n-k\end{array}}{\max }_{0\ne v\in T}{R}_A(v)$. The subspaces $T\subseteq W$ of dimension $n-k$ form a subfamily of all dimension-$(n-k)$ subspaces of $V$, and a minimum over a smaller family is no smaller, so ${\lambda }_{k+1}\le {\mu }_k$. Combining, ${\lambda }_{k+1}\le {\mu }_k\le {\lambda }_k$. The general $m$-deletion bound ${\lambda }_{k+m}\le {\mu }_k\le {\lambda }_k$ follows by iterating the codimension-one step $m$ times. $\square$

Proposition (Weyl's inequality ${\lambda }_{i+j-1}(A+B)\le {\lambda }_i(A)+{\lambda }_j(B)$). For self-adjoint $A,B$ on $V$ and indices with $i+j-1\le n$, the stated inequality holds.

Proof. Let $u_1,\dots ,u_n$ be an orthonormal eigenbasis for $A$ and $w_1,\dots ,w_n$ one for $B$. Set $$ U = \mathrm{span}(u_i, u_{i+1}, \ldots, u_n), \quad \dim U = n - i + 1, \qquad W = \mathrm{span}(w_j, \ldots, w_n), \quad \dim W = n - j + 1. $$ On $U$, every unit vector has $R_A\le {\lambda }_i(A)$ (it is a combination of eigenvectors with eigenvalues $\le {\lambda }_i$); on $W$, $R_B\le {\lambda }_j(B)$. The intersection $U\cap W$ has dimension at least $(n-i+1)+(n-j+1)-n=n-i-j+2$. For any unit $v\in U\cap W$, $$ R_{A+B}(v) = \langle (A+B)v, v\rangle = R_A(v) + R_B(v) \leq \lambda_i(A) + \lambda_j(B). $$ Now apply the min-max form to $A+B$. Set $m=i+j-1$, so a subspace of dimension $n-m+1=n-i-j+2$ exists inside $U\cap W$. Then $$ \lambda_m(A+B) = \min_{\dim S = n-m+1}\ \max_{0\neq v\in S} R_{A+B}(v) \leq \max_{0\neq v\in U\cap W} R_{A+B}(v) \leq \lambda_i(A) + \lambda_j(B), $$ where the first inequality chooses $S$ to be an $(n-m+1)$-dimensional subspace of $U\cap W$. This is the claim ${\lambda }_{i+j-1}(A+B)\le {\lambda }_i(A)+{\lambda }_j(B)$. The dual lower inequality follows by applying this to $-A,-B$ and re-indexing. $\square$

Proposition (Rayleigh-quotient iteration converges cubically near a simple eigenvalue). Let $A = A^$haveasimpleeigenvalue$\lambda$withuniteigenvector$u$.Theiteration$v_{m+1} = (A - R(v_m) I)^{-1} v_m$followedbynormalisation,startingfrom$v_0$closeenoughto$u$,satisfies$\sin\angle(v_{m+1}, u) = O\big(\sin^3\angle(v_m, u)\big)$.*

Proof. Write $v_m=(\cos {\theta }_m)u+(\sin {\theta }_m)z_m$ with $z_m\perp u$, $\Vert z_m\Vert =1$, where ${\theta }_m=\angle (v_m,u)$. The Rayleigh quotient satisfies $R(v_m)=\lambda {\cos }^2{\theta }_m+⟨A{z}_m,z_m⟩{\sin }^2{\theta }_m=\lambda +(R_z-\lambda ){\sin }^2{\theta }_m$, with $R_z=⟨A{z}_m,z_m⟩$ bounded; so $R(v_m)-\lambda =O({\sin }^2{\theta }_m)$. Let $A^{\prime }=A{|}_{u^{\perp }}$ act on the invariant complement $u^{\perp }$, with spectral gap $\gamma =\mathrm{dist}(\lambda ,\sigma (A^{\prime }))>0$ by simplicity. Applying $(A-R(v_m)I{)}^{-1}$ scales the $u$-component by $(\lambda -R(v_m){)}^{-1}$ and the $z_m$-component by a factor of size at most $(\gamma -|R(v_m)-\lambda |{)}^{-1}=O(1)$. Hence $$ \tan\theta_{m+1} = \frac{|\text{component} \perp u|}{|\text{component along } u|} \leq \frac{O(1)\sin\theta_m}{|\lambda - R(v_m)|^{-1}\cos\theta_m} = O\big(|R(v_m) - \lambda|\big)\tan\theta_m = O(\sin^2\theta_m)\tan\theta_m. $$ Since $\tan {\theta }_m=O(\sin {\theta }_m)$ for small angles, $\tan {\theta }_{m+1}=O({\sin }^3{\theta }_m)$, giving the cubic local convergence. The self-adjointness is used twice: to make $u^{\perp }$ an $A$-invariant orthogonal complement, and to make $R(v_m)-\lambda$ second order in ${\theta }_m$ rather than first order. $\square$

Connections Master

The Rayleigh quotient is the variational face of the spectral theorem 01.01.13: where the spectral theorem produces the eigenvalues by diagonalisation, the extremal characterisation produces them by optimisation, and the equivalence of the two is the identity $⟨Av,v⟩={\sum }_i{\lambda }_i|⟨v,u_i⟩{|}^2$. The self-adjoint hypothesis is exactly the condition from 01.01.10 that makes $⟨Av,v⟩$ real and the eigenvalues real, without which the quotient has no order structure to extremise.

The min-max characterisation specialises to the singular value decomposition 01.01.12 through $A^{\ast }A$: the singular values are ${\sigma }_k=\sqrt{{\lambda }_k(A^{\ast }A)}$, so Courant-Fischer applied to the positive self-adjoint $A^{\ast }A$ gives ${\sigma }_k={\min }_{\dim S=n-k+1}{\max }_{0\ne v\in S}\Vert Av\Vert /\Vert v\Vert$, the variational characterisation of singular values that underlies the Eckart-Young low-rank approximation theorem.

The interlacing and perturbation inequalities feed the quadratic-form theory of 01.01.15: Sylvester's law of inertia classifies symmetric matrices by the signs of their eigenvalues, and Weyl's monotonicity plus Cauchy interlacing govern how those signs persist under congruence, compression, and perturbation, which is the algebraic content of the inertia of a bordered or perturbed form.

The eigenvector-free min-max form is the bridge to the variational method in quantum mechanics 12.07.03: the Rayleigh-Ritz estimate of a ground-state energy is the statement $E_0\le R_H(\Psi )$ for any trial state $\Psi$, the $k=1$ Rayleigh case for the Hamiltonian $H$, and the min-max over trial subspaces produces upper bounds for the excited-state energies (MacDonald's theorem) exactly as the finite-dimensional Courant-Fischer formula predicts.

Historical & philosophical context Master

The Rayleigh quotient began in mechanics, not in linear algebra. In The Theory of Sound (1877) Lord Rayleigh studied the small oscillations of a system about equilibrium and observed that the ratio of the potential energy to the kinetic energy of a trial configuration is a stationary estimate of the squared natural frequencies, accurate to second order when the trial mode is close to a true normal mode ^{[Rayleigh 1877]}. The quotient $⟨Av,v⟩/⟨v,v⟩$ is the abstraction of that energy ratio, with $A$ the stiffness operator and the denominator the mass form. Ritz turned the stationarity observation into an algorithm in 1909, expanding the trial configuration in a finite basis and reducing the variational problem to a generalised matrix eigenvalue problem, which is why the finite-basis variational method carries his name ^{[Ritz 1909]}.

The min-max formula appeared first in Fischer's 1905 study of real quadratic forms, where the $k$-th eigenvalue is given as a max-min over $k$-dimensional subspaces ^{[Fischer 1905]}. Weyl, in his 1912 work on the asymptotic distribution of the eigenvalues of the Laplacian on a membrane, used the same variational characterisation to prove both the perturbation inequalities now bearing his name and the leading-order eigenvalue-counting law, establishing that the count of eigenvalues below a threshold grows like the volume of the domain times a universal constant ^{[Weyl 1912]}. Courant extended the principle to general self-adjoint differential operators in 1920 and connected it to the nodal domains of eigenfunctions, giving the geometric form of the theorem that propagated through Methoden der mathematischen Physik into the standard toolkit of mathematical physics ^{[Courant 1920]}. The finite-dimensional Cauchy interlacing relation, which the modern proof derives as a corollary of min-max, predates all of this: Cauchy established it in 1829 while analysing the secular equation of planetary perturbation theory ^{[Cauchy 1829]}.

Bibliography Master

@book{Rayleigh1877,
  author    = {Rayleigh, John William Strutt, Baron},
  title     = {The Theory of Sound, Volume I},
  publisher = {Macmillan},
  address   = {London},
  year      = {1877}
}

@article{Fischer1905,
  author  = {Fischer, Ernst},
  title   = {{\"U}ber quadratische Formen mit reellen Koeffizienten},
  journal = {Monatshefte f{\"u}r Mathematik und Physik},
  volume  = {16},
  year    = {1905},
  pages   = {234--249}
}

@article{Ritz1909,
  author  = {Ritz, Walther},
  title   = {{\"U}ber eine neue Methode zur L{\"o}sung gewisser Variationsprobleme der mathematischen Physik},
  journal = {Journal f{\"u}r die reine und angewandte Mathematik},
  volume  = {135},
  year    = {1909},
  pages   = {1--61}
}

@article{Weyl1912,
  author  = {Weyl, Hermann},
  title   = {Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung)},
  journal = {Mathematische Annalen},
  volume  = {71},
  year    = {1912},
  pages   = {441--479}
}

@article{Courant1920,
  author  = {Courant, Richard},
  title   = {{\"U}ber die Eigenwerte bei den Differentialgleichungen der mathematischen Physik},
  journal = {Mathematische Zeitschrift},
  volume  = {7},
  year    = {1920},
  pages   = {1--57}
}

@article{Cauchy1829,
  author  = {Cauchy, Augustin-Louis},
  title   = {Sur l'{\'e}quation {\`a} l'aide de laquelle on d{\'e}termine les in{\'e}galit{\'e}s s{\'e}culaires des mouvements des plan{\`e}tes},
  journal = {Exercices de Math{\'e}matiques},
  volume  = {4},
  year    = {1829},
  pages   = {140--160}
}

@book{HornJohnson2013,
  author    = {Horn, Roger A. and Johnson, Charles R.},
  title     = {Matrix Analysis},
  edition   = {2nd},
  publisher = {Cambridge University Press},
  year      = {2013}
}

@book{Parlett1998,
  author    = {Parlett, Beresford N.},
  title     = {The Symmetric Eigenvalue Problem},
  publisher = {SIAM},
  series    = {Classics in Applied Mathematics},
  year      = {1998}
}

@book{ReedSimon1978,
  author    = {Reed, Michael and Simon, Barry},
  title     = {Methods of Modern Mathematical Physics, Vol. IV: Analysis of Operators},
  publisher = {Academic Press},
  address   = {New York},
  year      = {1978}
}

@book{Shilov1977,
  author    = {Shilov, Georgi E.},
  title     = {Linear Algebra},
  publisher = {Dover Publications},
  address   = {New York},
  year      = {1977},
  note      = {Translation of the 1971 Russian edition, transl. R. A. Silverman}
}