91Ó°ÊÓ

In the context of least squares regression, the objective function is the mathematical expression we aim to minimize to find the best-fitting line for a set of data points. The formula used is \( f(m, b) = \sum_{i=1}^{n} (y_i - mx_i - b)^2 \). Here,

• \(m\) represents the slope of the line,
• \(b\) is the y-intercept,
• \((x_i, y_i)\) are the given data points,
• The expression \((y_i - mx_i - b)^2 \) represents the square of the distance from each point to the line.

By minimizing this function, we ensure that the line will have the smallest possible sum of squared vertical distances from the data points. This objective is fundamental in creating a line that closely fits the data, reducing the total error between the observed and predicted values.

Once the objective function is defined, the next step is to find its minimum by using calculus. This involves taking the partial derivatives of the function with respect to its parameters, \(m\) and \(b\). A partial derivative measures how the function changes as one of the variables changes, while keeping others constant. For our function,

• The partial derivative with respect to \(m\) is \( \frac{\partial f}{\partial m} = -2 \sum (y_i - mx_i - b)x_i = 0 \).
• The partial derivative with respect to \(b\) is \( \frac{\partial f}{\partial b} = -2 \sum (y_i - mx_i - b) = 0 \).

By setting these derivatives equal to zero, we find the critical points where the function's slope is zero, indicating potential minima or maxima. In the case of least squares, these points represent the optimal values for \(m\) and \(b\) that minimize the function.

After taking partial derivatives and equating them to zero, we derive a system of linear equations. These equations are:

• \( \sum x_i y_i = m \sum x_i^2 + b \sum x_i \)
• \( \sum y_i = m \sum x_i + nb \)

This system arises from substituting the conditions from the partial derivatives. Solving this system allows us to find the exact values of \(m\) and \(b\).
The equations can be solved using algebraic methods such as substitution. Alternatively, they can be resolved using a more streamlined mathematical approach known as the matrix method. These solutions give us the critical values for the least squares fitting line.

The matrix method is a powerful tool for solving systems of equations, especially in linear regression problems like least squares. We express the system of linear equations in a compact form using matrices:\[\begin{pmatrix} \sum x_i^2 & \sum x_i \\sum x_i & n \end{pmatrix} \begin{pmatrix} m \b \end{pmatrix} = \begin{pmatrix} \sum x_i y_i \\sum y_i \end{pmatrix}\]In this setup:

• The first matrix is called the coefficient matrix.
• The second is the vector of unknowns \(m\) and \(b\).
• The third is the result vector from our equations.

We solve for \(m\) and \(b\) by inverting the coefficient matrix and multiplying both sides of the equation. This method provides an efficient way to find solutions, especially when dealing with large sets of data. Moreover, it better illustrates the linear algebra principles underlying the least squares regression process.