91Ó°ÊÓ

Least-squares solutions serve as a cornerstone of linear regression, especially valuable when dealing with over-determined systems where there are more equations (data points) than unknowns. This approach aims to minimize the residuals, which are the differences between the observed values and the values predicted by the model.

When the number of predictors, denoted as p, is larger than the number of observations, N, you have a situation where p ≫ N. These conditions can result in more than one solution to the least-squares problem, leading to the conundrum of determining which one is the 'best'. The exercise highlights that due to the rank-nullity theorem, the existence of non-zero vectors in the null space of the design matrix X implies infinite solutions with zero residuals.

Another interesting aspect of least-squares solutions is highlighted when singular value decomposition or ridge regression is applied, altering the solution's properties, such as uniqueness or stability, which is elucidated in the subsequent sections.

SVD is a powerful factorization technique in linear algebra, commonly used in signal processing and statistics. In the context of linear regression, SVD decomposes the original design matrix X into three components: a matrix U holding the left singular vectors, a diagonal matrix D with singular values, and the transpose of a matrix V that contains the right singular vectors. These components are crucial for understanding the geometry and stability of the solutions to the regression problem.

The singular vectors encapsulate important properties such as the directions of maximum variance. The singular values, found along the diagonal of D, are pivotal in determining the sensitivity of the output to the input and can be instrumental when addressing issues like multicollinearity or the presence of near-zero singular values that can cause numerical problems in calculations. By restructuring the regression in terms of U, D, and V, we gather insights into the nature and behavior of the regression equations, facilitating an enhanced grasp of the least-squares solutions.

Ridge regression is a variant of linear regression that incorporates a regularization parameter, λ, to impose a penalty on the size of coefficients. This technique combats overfitting by shrinking the coefficients, thereby making the model less sensitive to the individual data points.

When the design matrix X has a greater number of variables than observations, it is prone to instability and overfitting. Ridge regression ameliorates this by introducing bias into the estimates through λ, which helps manage multicollinearity (when independent variables are highly correlated) by adding a degree of bias to the regression estimates.

The exercise distinctly shows how the estimate for β changes with different values of λ. When the regularization parameter is set to zero, the method simplifies back to least-squares, as captured in the formula provided for βλ. However, nonzero values of λ alter the solution, offering a method to balance between the bias and variance, an essential concept in the bias-variance tradeoff, leading to more robust models in practical scenarios.