Introduction
Consider the regression problem The least squares estimation of under the constraint generally involves solving the Lagrangian. Like many optimisation problems in linear models, a purely linear algebraic proof involving projections exists. Such projective proofs are typically more enlightening and less tedious. Seber and Lee (2003) contains such a projective proof, but the proof is far from enlightening and if anything, even more tedious than the Lagrangian method. I present a non-standard proof that is not only less tedious but the use of projections gives way easily to the sum of squared identities used in -test.
Inner products induced by a matrix
In the euclidiean inner product space, the orthogonal projector into the column space of a full-column-rank matrix is given by . This form obviously does not carry over to different inner products.
In the euclidean inner product space, the column space of a matrix is orthogonal to its row space. A similar result exists for this inner product.
Projections into affine subspaces
Standard treatments of linear algebra generally only talk about projections into linear subspaces. Whenever a projection into an affine subspace is required, the affine subspace is converted into a linear subspace by an appropriate shift in the origin. However, treating affine projections as bona fide projections in their own right can be more instructive.
It can be easily verified that projections into affine subspaces are unique as well. This leads to an affine projector operator . The following is also easy to verify.
Affine projectionslike linear projectionsminimise norm of the difference.
If a linear subspace is included in a larger one, iteratively projecting first, into the larger one, and then into the smaller one is equivalent to directly projecting into the smaller one. We have a similar result for affine subspaces.
Proof of the estimator
Now, let’s go back to regression problem. Let be a left-inverse of , and be a particular solution to the constraint. Then, since if and only if , We thus want the solution to the problem Owing to the preceding theorem, we can first project into , and then into . The first projection is just the least squares projection , where is the least squares coefficient. The second projection is the minimisation problem but since , this is equivalent to This is attained at the projection of into under the inner product induced by , or equivalently plus the projection of into . By the lemmas in the discussion of inner product, the projector into is the projector into the orthogonal complement of which is We thus have the final solution where it has been used that .
Decomposition of sum of squares
Let be the least squares projection of , and the projection of into the constrained affine subspace We have easily from the Pythagorean theorem that since is orthogonal to and is in . The projection approach shows clearly that minimises over the constrained space. This identity is usually written as and the -statistic is given by where is the number of columns in (including the column of s), and the number of rows in (i.e., the number of linear constraints). The numerator is just where and are the projectors into the and . We thus have the -statistic as