Linear Algebra: Vector Spaces, Matrices, and Linear Transformations
Introduction
Linear algebra is the study of vector spaces and linear transformations between them. It is simultaneously one of the most abstract areas of mathematics — dealing with structures defined by axioms — and one of the most applicable, with uses ranging from quantum mechanics to machine learning to computer graphics. This duality makes linear algebra unique among mathematical subjects.
The subject emerged from the study of systems of linear equations. The Babylonians solved linear systems geometrically. Gauss developed systematic elimination methods. Cayley, Grassmann, and Peano in the nineteenth century recognized that the essential structure was independent of any particular coordinate system, giving birth to modern linear algebra.
Vector Spaces
A vector space over a field F is a set V together with two operations: vector addition and scalar multiplication. The elements of V are called vectors, and the elements of F are called scalars. The operations must satisfy eight axioms: associativity and commutativity of addition, existence of an additive identity and additive inverses, and four compatibility conditions for scalar multiplication.
Fundamental Examples
The prototypical vector space is Fⁿ, the set of n-tuples of elements from F. Function spaces form infinite-dimensional vector spaces: the set of all continuous functions on [0, 1] is a vector space over R. Polynomial spaces, sequence spaces, and matrix spaces all provide important examples.
The concept of a vector space is explored further in abstract algebra, where modules generalize vector spaces by allowing scalars from a ring instead of a field.
Linear Transformations
A linear transformation T: V → W between vector spaces over the same field satisfies T(u + v) = T(u) + T(v) and T(cv) = cT(v). Linear transformations preserve the vector space structure — they are the natural maps between vector spaces.
Kernel and Image
The kernel of a linear transformation is the set of vectors mapped to zero. The image is the set of all outputs. The Rank-Nullity Theorem states that the dimension of the kernel plus the dimension of the image equals the dimension of the domain. This theorem is the first instance of a general phenomenon: invariants of maps often decompose the dimension of the source.
The space of all linear transformations from V to W is itself a vector space. This observation connects linear algebra to functional analysis, where the focus shifts to infinite-dimensional spaces and continuity becomes essential.
Matrices and Determinants
A matrix is a rectangular array of scalars representing a linear transformation with respect to chosen bases. Matrix multiplication corresponds to composition of linear transformations. The entries of the matrix depend on the choice of basis, but the underlying linear transformation is basis-independent.
Determinants
The determinant of a square matrix measures how the corresponding linear transformation scales volumes. A transformation is invertible exactly when its determinant is nonzero. The determinant is multiplicative: det(AB) = det(A)det(B).
Determinants have a geometric interpretation: in two dimensions, the determinant gives the area of the parallelogram spanned by the column vectors. In three dimensions, it gives the volume of the parallelepiped. In higher dimensions, it gives the signed n-dimensional volume.
Eigenvalues and Eigenvectors
An eigenvector of a linear transformation is a nonzero vector that is scaled by the transformation: T(v) = λv. The scalar λ is the eigenvalue. Eigenvectors capture the directions in which the transformation acts most simply — by mere stretching or compression.
Diagonalization
A linear transformation is diagonalizable if there exists a basis of eigenvectors. In this basis, the matrix representation is diagonal, with eigenvalues on the diagonal. Diagonalization simplifies computations enormously: powers of the transformation, exponentials, and functions of the transformation all become easy to compute.
The spectral theorem states that every symmetric real matrix is diagonalizable by an orthogonal matrix. This theorem underlies principal component analysis in statistics, the analysis of vibrations in mechanical systems, and the quantum mechanical description of observables.
Inner Product Spaces
An inner product is a generalization of the dot product. It allows the definition of length, angle, and orthogonality in abstract vector spaces. The Cauchy-Schwarz inequality bounds the inner product by the product of lengths: |⟨u, v⟩| ≤ ‖u‖‖v‖.
Orthogonality and Projections
Gram-Schmidt orthogonalization produces an orthonormal basis from any basis. Orthogonal projections onto subspaces solve the least squares problem: find the vector in a subspace closest to a given vector. The least squares solution minimizes the sum of squared errors and is the standard method for fitting models to data.
Fourier series provide a beautiful example of orthogonal projections in infinite-dimensional spaces. The Fourier coefficients are the projections of a function onto the orthonormal basis of sine and cosine functions. This connection links linear algebra to real analysis and approximation theory.
Multilinear Algebra
Multilinear algebra extends linear algebra to functions of several vector variables that are linear in each argument. Tensors are the fundamental objects of multilinear algebra, generalizing vectors and matrices to higher ranks. The tensor product of vector spaces provides a universal way to combine vector spaces.
The exterior algebra, built from the tensor product with the relation v ∧ v = 0, captures the theory of determinants and oriented volumes. Wedge products of vectors represent oriented parallelepipeds, and the exterior derivative unifies gradient, curl, and divergence in differential geometry.
Symmetric and Alternating Tensors
Symmetric tensors correspond to polynomials and encode symmetric multilinear functions. Alternating tensors correspond to determinants and encode volume elements. The decomposition of tensors into symmetric and alternating parts is the foundation of invariant theory.
The representation theory of the symmetric group governs the decomposition of tensor powers. The Schur-Weyl duality describes how the actions of GL(V) and the symmetric group on tensor powers commute, providing a deep connection between linear algebra and group theory. These ideas are essential in quantum mechanics, where the symmetrization postulate determines the possible states of identical particles.
Applications in Modern Mathematics
Linear algebra is indispensable throughout mathematics. The theory of group representations studies groups through their actions on vector spaces. Algebraic geometry studies solutions to polynomial equations using the linear algebra of vector bundles. Differential geometry uses the tangent space — a vector space attached to each point of a manifold.
Canonical Forms
Canonical forms classify linear transformations up to similarity. The Jordan canonical form represents a linear transformation in a basis that makes the transformation as close to diagonal as possible. Each Jordan block corresponds to a generalized eigenvector chain.
The rational canonical form uses the invariant factors of the characteristic polynomial and works over any field. The companion matrix of a polynomial provides a canonical representative for each similarity class. These forms are essential for understanding the structure of linear transformations.
Applications to Differential Equations
Systems of linear differential equations are solved using matrix exponentials and canonical forms. The exponential of a matrix in Jordan form can be computed explicitly. The stability of solutions is determined by the eigenvalues of the coefficient matrix.
The Putzer algorithm provides an efficient method for computing matrix exponentials without computing Jordan forms. The connection between linear algebra and differential equations extends to the theory of Lie groups, where the matrix exponential maps the Lie algebra to the Lie group.
Numerical Linear Algebra
Numerical linear algebra develops algorithms for solving linear algebra problems on computers. Gaussian elimination with partial pivoting solves linear systems in O(n³) time. The LU decomposition factors a matrix into lower and upper triangular matrices.
Iterative methods like the conjugate gradient method solve large sparse systems that cannot be handled by direct methods. The conditioning of a problem measures its sensitivity to perturbations in the data. The condition number of a matrix bounds the worst-case error in solving a linear system.
Singular Value Decomposition
The singular value decomposition (SVD) writes any matrix as a product UΣVᵀ where U and V are orthogonal and Σ is diagonal with nonnegative entries called singular values. The SVD reveals the rank of the matrix and provides the best low-rank approximation in the least squares sense.
The SVD is the most important matrix factorization for data analysis. Principal component analysis (PCA) computes the SVD of the data matrix. The SVD is also used for image compression, recommendation systems, and latent semantic analysis.
What is the difference between a vector space and a module? Vector spaces have scalars from a field, where every nonzero element is invertible. Modules have scalars from a ring, where inverses may not exist.
Why are eigenvalues important? Eigenvalues reveal the fundamental behavior of a linear transformation — which directions are stretched, compressed, or left unchanged. They determine stability in differential equations and the energy levels in quantum mechanics.
What does the determinant tell us? The determinant tells whether a transformation is invertible, how volumes change, and whether orientation is preserved.
How is linear algebra used in data science? Dimensionality reduction, recommendation systems, and neural networks all rely on linear algebra. Principal component analysis uses eigendecomposition.
Abstract Algebra Guide — Functional Analysis — Real Analysis Guide