91Ó°ÊÓ

Linear regression is a statistical method used to create a straight-line relationship between two variables. The goal is to fit a line, called the regression line, that best predicts the dependent variable based on the independent variable.

In simple linear regression, the model can be written as:
i.e., fit a line such a way that i calculated a slope and intercept to minimize error comparing the predicted and actual residue.i.e. $$Y = \beta_{0} + \beta_{1}X_{1} $$

Where:

• - **\(Y\)** is the dependent variable
• - **\(X\)** is the independent variable
• - **\(\beta_{0}\)** is the y-intercept
• - **\(\beta_{1}\)** is the slope of the line

Ìý
In regular regression, the residues calculated, the weight assigned to each data point, used to calculate the coefficients \(\beta_{0}\) and \(\beta_{1}\).

Unlike simple linear regression, weighted least squares (WLS) assigns different weights to each data point. This approach is helpful when the variability of the error changes across different values of the independent variable.

Think of it as giving more importance to some points than others. The objective function to minimize in WLS is:
$$Q (\beta_{0},\beta_{1} ) = \sum_{i=1}^{n} w_{i} (Y_{i}-\beta_{0}- \beta_{1} X_{i} ) ^{2}$$
where
- **\(w_{i}\)** is the weight for data point \(i\).

The benefit of using weights is that it can handle heteroscedasticity (non-constant variance of errors). This means that if certain data points are noisier than others, we can adjust the regression to account for this variability using the appropriate weights.

Normal equations arise from the process of minimizing the weighted least squares objective function. By taking partial derivatives of this objective function with respect to coefficients \(\beta_{0}\) and \(\beta_{1}\) and setting them to zero, we can derive the formulas to solve for our coefficients.

The partial derivatives of the simplified WLS objective function with weights included:
$$Q (\beta_{0},\beta_{1} ) = \sum_{i=1}^{n} \frac{1}{k X_{i} } (Y_{i} - \beta_{0} - \beta_{1} X_{i} ) $$ \(^{2}\)

are given by:
ForÌý \(\beta_{0}\):
$$\sum_{i=1} ^{n} \frac{ \partial Q}{ \partial \beta_{0}} = \sum_{i=1}^{n} \frac{ -(Y_{i} - \beta_{0} - \beta_{1} \ X_{i}}{X_{i}} = 0$$

For \(\beta_{1}\):
$$\sum_{i=1}^{n} \frac{\partial Q }{\partial \beta_{1}} = \sum_{i=1}^{n} \frac{ -(Y_{i} - \beta_{0} - \beta_{1} X_{i} ) X_{i} }{X_{i} } = 0$$

By solving these equations, we can find the optimal values of \(\beta_{0}\) and \(\beta_{1}\) that minimize the WLS objective function.

In the given exercise, the variance of the errors is proportional to the independent variable, i.e., \(\sigma_{i}^{2} = k X_{i}\). This means that the variance increases or decreases in direct proportion to the values of \(X_{i}\).

This situation can occur in real-world data where variability isn’t constant. For instance:

• Higher income may lead to higher variability in spending
• ÌýIncreased time spent studying may lead to more variability in scores

By incorporating this variance proportionality into our weights, we get:

$$w_{i} = \frac{1}{ \sigma_{i} ^{2} } = \frac{1 }{k X_{i}}$$

meaning the weights assign more importance to data points where the variance is lower.
Consequently, when we use these weights in the WLS objective function, it helps handle and correct the effect that changing variance would have on the estimates of our regression coefficients. This is another way of ensuring our model is robust and accurate, even when facing real-world data complexities.