Skip to content
NumeriveMATHUniversity mathematics

TOPIC 3 / 3

Eigenvalues & Orthogonal Projection

Useful preparation: Span, Basis, Rank & Nullity

For a long equation, swipe sideways inside its formula box to see every term.

Meaning & notation

Most vectors change direction under a matrix. Eigenvectors identify exceptional directions on which the whole transformation acts like multiplication by one number. Projection is a complementary idea: retain the component inside a chosen subspace and discard the perpendicular component. A projection therefore acts with eigenvalue one on its target and zero on the perpendicular directions.

Eigenpair
For a square matrix A, a nonzero vector v and scalar lambda form an eigenpair when Av=lambda v. The scalar may be zero even though v cannot be zero.
Diagonalization
A diagonalization factors A as P times D times the inverse of P, using a basis of eigenvectors in the columns of P and matching eigenvalues in the diagonal matrix D.
Orthogonal projection
The orthogonal projection of b onto a subspace is the vector p in that subspace for which b−p\displaystyle b - p is perpendicular to every vector in the subspace.

Conditions before calculation

  • Eigenvalues require a square matrix. Real matrices can have nonreal eigenvalues, so specify whether calculations are over the real or complex numbers.
  • An n by n matrix is diagonalizable exactly when it has n independent eigenvectors. Distinct eigenvalues guarantee independence; repeated eigenvalues alone do not settle the question.
  • For least squares, the normal equations always hold at a minimizer. The inverse of A transpose A exists only when A has full column rank; real symmetric matrices admit an orthonormal eigenbasis.

Why the method works

1. Find preserved directions

A nonzero solution of the homogeneous eigenvector equation exists exactly when its coefficient matrix is singular.

(A−λI)v=0,det⁡(A−λI)=0(A-\lambda I)v=0,\qquad \det(A-\lambda I)=0

2. Check multiplicities

For each eigenvalue, solve its nullspace problem. Count independent eigenvectors, not just roots of the characteristic polynomial, before using a diagonalization.

3. Minimize perpendicular error

Moving within the column space cannot shorten a perpendicular residual. Algebraically, its dot product with each column vanishes, giving the normal equations.

AT(b−Ax^)=0A^{\mathsf T}(b-A\widehat x)=0

Worked examples

EXAMPLE 1

Diagonalize the matrix with rows (3,1) and (0,2).

  1. The triangular determinant gives two distinct roots.

    det⁡(A−λI)=(3−λ)(2−λ)\det(A-\lambda I)=(3-\lambda)(2-\lambda)

  2. Solving each nullspace gives directions (1,0) and (-1,1). Direct multiplication verifies eigenvalues 3 and 2.

  3. Use those independent vectors as columns; their determinant is 1.

    P=(1−101),D=(3002),A=PDP−1P=\begin{pmatrix}1&-1\\0&1\end{pmatrix},\quad D=\begin{pmatrix}3&0\\0&2\end{pmatrix},\quad A=PDP^{-1}

Conclusion: The two eigenvectors form a basis, so powers of A reduce to powers of 3 and 2.

EXAMPLE 2

Project b=(3,1)\displaystyle b = \left(3, 1\right) onto the line spanned by a=(1,1)\displaystyle a = \left(1, 1\right).

  1. Every candidate on the line has form ca. Require its residual to be perpendicular to a.

    aT(b−ca)=0a^{\mathsf T}(b-ca)=0

  2. Solve the scalar equation using the squared length of a.

    c=aTbaTa=42=2c=\frac{a^{\mathsf T}b}{a^{\mathsf T}a}=\frac42=2

  3. The projection is (2,2), with residual (1,-1). Their direction test succeeds.

    (1,1)⋅(1,−1)=0(1,1)\mathbin{\cdot}(1,-1)=0

Conclusion: The closest point is (2,2); the least-squares error has squared length 2.

Pitfalls & counterexamples

Tempting claim: Every square matrix is diagonalizable.

Correction: Enough independent eigenvectors are required.

Test the claim: The matrix with rows (1,1) and (0,1) has only the eigenvalue 1 and a one-dimensional eigenspace.

Check your understanding

Can zero be an eigenvalue of an invertible matrix?

Write a reason before opening the explanation. This reflection is not graded or saved.

Compare your reasoning

No.

A zero eigenvalue would give a nonzero v with Av=0, contradicting invertibility.

Explore an interactive illustration
Projection leaves a perpendicular residual

Project b onto the line spanned by a=(1,1)\displaystyle a = \left(1, 1\right). This illustrates a one-column least-squares problem.

Follow the perpendicular projection
0%
b

Moving point: (3, 3)

c=a⋅ba⋅a=3+32=3\displaystyle c = \frac{a \cdot b}{a \cdot a} = \frac{3 + 3}{2} = 3. Projection=(3,3); residual=(0,0). Its dot product with (1,1) is 0, confirming perpendicularity.

Connect this idea

Eigenvalues govern modes in linear differential equations. Orthogonal projection produces least-squares regression and separates explained components from residuals.

Practise this topic →Review concept flashcards

Reference reading

Use these texts to study the underlying definitions and theorems in greater depth. The explanations and examples above are written for Numerive.

Numerive