Regression Models

Overview

This chapter provides a rigorous examination of regression models, a fundamental class of supervised learning algorithms. Our primary objective is to elucidate the methods by which we can model the relationship between a dependent (or target) variable and one or more independent (or predictor) variables. Regression analysis is central to the field of data science, enabling us to make quantitative predictions about future outcomes based on observed data. A thorough understanding of these models is indispensable for success in the GATE examination, where questions frequently assess the ability to interpret, apply, and evaluate predictive models.

We shall commence our study with Simple Linear Regression, which establishes the foundational principles by modeling the linear relationship between a single predictor and a target variable. From this groundwork, we will extend the framework to Multiple Linear Regression, a more powerful and practical technique that accommodates several predictor variables simultaneously. In doing so, we will also confront the challenges inherent in higher-dimensional models, such as overfitting and multicollinearity. To address these issues, the chapter culminates with an introduction to Ridge Regression, a regularized linear model designed to improve model stability and predictive accuracy in the presence of correlated features.

---

Chapter Contents

| # | Topic | What You'll Learn |
|---|-------|-------------------|
| 1 | Simple Linear Regression | Modeling relationships with a single predictor. |
| 2 | Multiple Linear Regression | Extending the model to multiple predictors. |
| 3 | Ridge Regression | Regularization to prevent model overfitting. |

---

Learning Objectives

---

We now turn our attention to Simple Linear Regression...

Part 1: Simple Linear Regression

Introduction

Simple Linear Regression (SLR) is a foundational supervised learning algorithm used to model the relationship between two continuous variables. It seeks to establish a linear relationship between a single independent variable, often termed the predictor or feature (denoted by $x$), and a single dependent variable, known as the response or target (denoted by $y$). The fundamental objective is to find the "best-fit" straight line that describes how the response variable changes as the predictor variable changes.

This straight line, or regression line, can then be used for prediction. Given a new value of the predictor variable $x$, we can use the model to estimate the corresponding value of the response variable $y$. In the context of the GATE examination, a thorough understanding of the underlying principles of SLR, particularly the method of least squares and the derivation of model parameters, is essential for solving numerical problems efficiently and accurately.

---

---

Key Concepts

1. The Linear Model and Residuals

The core of simple linear regression is the equation of a straight line. For a given dataset of $n$ pairs of observations $(x_1, y_1)$, $(x_2, y_2)$, $\dots$, $(x_n, y_n)$, we want to find the specific line that best represents this data.

The predicted value for the $i$-th observation $x_i$ is given by:

The difference between the actual observed value $y_i$ and the value predicted by our model $\hat{y}_i$ is called the residual or error, denoted by $e_i$.

The residuals represent the "unexplained" variation. A good model will have small residuals. The following diagram illustrates these concepts visually.

x (Predictor)
y (Response)

$\hat{y} = w_0 + w_1x$

(x_i, y_i)



(x_i, $\hat{y}_i$)



Residual $e_i$

2. The Principle of Least Squares

To find the "best-fit" line, we need a criterion for what "best" means. The most common method is the principle of least squares. This principle states that the best-fitting line is the one that minimizes the sum of the squared residuals.

We define a loss function, $L(w_0, w_1)$, as the Sum of Squared Errors (SSE), also known as the Residual Sum of Squares (RSS).

Our objective is to find the values of the parameters $w_0$ and $w_1$ that minimize this loss function. This is an optimization problem that can be solved using calculus.

3. Derivation of Model Parameters

To find the minimum of the loss function $L(w_0, w_1)$, we take the partial derivatives with respect to $w_0$ and $w_1$ and set them to zero. This gives us a system of two linear equations known as the normal equations.

Derivation for $w_0$ and $w_1$

Step 1: Define the loss function.

Step 2: Compute the partial derivative with respect to $w_0$ and set it to zero.

Dividing by $n$, we get $\bar{y} - w_0 - w_1\bar{x} = 0$, which gives the formula for $w_0$:

This result shows that the least-squares regression line always passes through the point of means, $(\bar{x}, \bar{y})$.

Step 3: Compute the partial derivative with respect to $w_1$ and set it to zero.

Step 4: Substitute the expression for $w_0$ from Step 2 into the equation from Step 3.

Since $\bar{x} = \frac{\sum x_i}{n}$, we can write $\sum x_i = n\bar{x}$. Substituting this gives the final formula for $w_1$.

This formula can be expressed in a more common form related to covariance and variance.

---

4. Special Case: Regression Through the Origin

Occasionally, a problem may specify that the line must pass through the origin. This implies that the intercept $w_0$ is fixed at 0. The model simplifies to $y = wx$. This was the case in a previous GATE question.

The objective is now to find the optimal slope $w$ that minimizes the SSE for this simpler model.

Step 1: Define the loss function with $w_0=0$.

Step 2: Compute the derivative with respect to $w$ and set it to zero.

Step 3: Solve for $w$.

Worked Example:

Problem: Given the data points $\{(1, 3), (2, 4), (3, 8)\}$, fit a model of the form $y = wx$ using linear least-squares regression. Find the optimal value of $w$.

Solution:

Step 1: Identify the required sums from the formula $w = \frac{\sum x_i y_i}{\sum x_i^2}$. We need to calculate $\sum x_i y_i$ and $\sum x_i^2$. We can construct a table for clarity.

| $x_i$ | $y_i$ | $x_i y_i$ | $x_i^2$ |
| :---: | :---: | :-------: | :-----: |
| 1 | 3 | 3 | 1 |
| 2 | 4 | 8 | 4 |
| 3 | 8 | 24 | 9 |
| Sum | | 35 | 14 |

Step 2: Calculate the sums.

Step 3: Apply the formula for $w$.

Step 4: Compute the final value.

Answer: The optimal value of $w$ is $2.5$.

---

Problem-Solving Strategies

---

---

Common Mistakes

---

Practice Questions

:::question type="NAT" question="A simple linear regression model of the form $y = wx$ is fitted to the data points $\{(1, 2), (2, 5), (-3, -6)\}$. The optimal value of $w$, determined by the method of least squares, is ______. (Round off to two decimal places)" answer="2.14" hint="Use the formula for regression through the origin. You will need to calculate

and ." solution="
Step 1: The model is $y = wx$. The formula for the optimal slope is

Step 2: Calculate the sums from the data $\{(1, 2), (2, 5), (-3, -6)\}$.

Step 3: Substitute the sums into the formula.

Step 4: Compute the final value and round to two decimal places.

Result:
Rounding to two decimal places, the value is $2.14$.
Answer: $\boxed{2.14}$
"
:::

:::question type="MCQ" question="A researcher fits a simple linear regression model $y = w_0 + w_1x$ to study the relationship between hours of study ( $x$ ) and exam score ( $y$ ). The resulting equation is $\hat{y} = 40 + 5x$. How should the slope parameter $w_1 = 5$ be interpreted?" options=["For every 5 hours of study, the exam score increases by 1 point.","The minimum exam score is 40.","For each additional hour of study, the exam score is predicted to increase by 5 points.","A student who does not study is predicted to score 5 points."] answer="For each additional hour of study, the exam score is predicted to increase by 5 points." hint="The slope represents the change in the dependent variable for a one-unit change in the independent variable." solution="
The slope $w_1$ in a simple linear regression model represents the average change in the response variable $y$ for a one-unit increase in the predictor variable $x$.

In the equation $\hat{y} = 40 + 5x$:

• The predictor $x$ is 'hours of study'.


• The response $y$ is 'exam score'.


• The slope $w_1$ is 5.

Therefore, a slope of 5 means that for each additional hour of study (a one-unit increase in $x$), the predicted exam score ( $\hat{y}$ ) increases by 5 points. Option C correctly states this interpretation.

• Option A is incorrect; it reverses the relationship.


• Option B refers to the intercept, not the minimum possible score.


• Option D is incorrect; a student who does not study ( $x=0$ ) is predicted to score 40 points (the intercept).

Answer: $\boxed{\text{For each additional hour of study, the exam score is predicted to increase by 5 points.}}$
"
:::

:::question type="NAT" question="For the dataset $\{(0, 2), (2, 6), (5, 7)\}$, a regression line of the form $y = w_0 + w_1x$ is fitted. The value of the slope parameter $w_1$ is ______. (Round off to two decimal places)" answer="0.95" hint="Use the formula

A tabular calculation is recommended." solution="
Step 1: We need to find the slope $w_1$. We compute the necessary sums for the dataset $\{(0, 2), (2, 6), (5, 7)\}$, where $n=3$.

| $x_i$ | $y_i$ | $x_i y_i$ | $x_i^2$ |
| :---: | :---: | :-------: | :-----: |
| 0 | 2 | 0 | 0 |
| 2 | 6 | 12 | 4 |
| 5 | 7 | 35 | 25 |
| $\sum x_i=7$ | $\sum y_i=15$ | $\sum x_i y_i=47$ | $\sum x_i^2=29$ |

Step 2: From the table, we have:

Step 3: Apply the formula for $w_1$.

Step 4: Simplify the expression.

Step 5: Compute the final value and round.

Result:
Rounding to two decimal places, the value is $0.95$.
Answer: $\boxed{0.95}$
"
:::

:::question type="MSQ" question="Which of the following statements are always true for a simple linear regression model $\hat{y} = w_0 + w_1x$ fitted using the ordinary least squares (OLS) method on a dataset with at least two distinct points?" options=["The sum of the residuals, $\sum_{i=1}^{n} (y_i - \hat{y}_i)$, is equal to zero.","The regression line passes through the point of means, $(\bar{x}, \bar{y})$.","The value of the intercept $w_0$ must be positive.","The sum of the squared residuals is maximized."] answer="The sum of the residuals, $\sum_{i=1}^{n} (y_i - \hat{y}_i)$, is equal to zero.,The regression line passes through the point of means, $(\bar{x}, \bar{y})$." hint="Recall the normal equations derived from minimizing the sum of squared errors." solution="
Let us evaluate each statement based on the derivation of the OLS parameters.

• Statement A: The first normal equation, derived by taking the partial derivative of the SSE with respect to $w_0$ and setting it to zero, is

Since $\hat{y}_i = w_0 + w_1x_i$, this equation is equivalent to Thus, the sum of the residuals is always zero. This statement is correct.

• Statement B: From the first normal equation

if we divide by $n$, we get Rearranging gives This equation shows that the point ( $\bar{x}, \bar{y}$ ) satisfies the regression line equation. Therefore, the regression line always passes through the point of means. This statement is correct.

• Statement C: The intercept

can be positive, negative, or zero depending on the data. For example, if the line has a positive slope and passes through the origin of the means, $w_0$ could be negative if the means are positive. There is no constraint that it must be positive. This statement is incorrect.

• Statement D: The principle of ordinary least squares is to minimize, not maximize, the sum of the squared residuals. This statement is incorrect.

Therefore, the only statements that are always true are A and B. Answer: $\boxed{\text{The sum of the residuals, } \sum_{i=1}^{n} (y_i - \hat{y}_i)\text{, is equal to zero.,The regression line passes through the point of means, } (\bar{x}, \bar{y})\text{.}}$ " :::

---

Summary

---

---

What's Next?

---

---

Part 2: Multiple Linear Regression

Introduction

In our study of regression models, we often begin with the case of a single predictor variable, known as simple linear regression. While this provides a foundational understanding of the relationship between two variables, real-world phenomena are rarely so straightforward. The value of a dependent variable is typically influenced by a confluence of factors. Multiple Linear Regression extends the principles of simple linear regression to model the relationship between a single dependent variable and two or more independent (or predictor) variables.

This powerful technique allows us to build more realistic and explanatory models by accounting for the simultaneous influence of several factors. For instance, a student's exam score is not merely a function of hours studied; it may also depend on prior academic performance, attendance, and quality of sleep. By incorporating these multiple predictors, we can construct a more nuanced and accurate model. Our focus will be on understanding the mathematical formulation of the model, the interpretation of its parameters, and its fundamental assumptions.

---

---

Key Concepts

1. The Regression Equation

The core of multiple linear regression is its governing equation. Unlike simple linear regression, which describes a line, the model for multiple linear regression describes a hyperplane in a multi-dimensional space. For a given observation $i$, the model is expressed as a linear combination of the predictor variables.

Let us consider a dataset with $n$ observations and $p$ predictor variables. The relationship for the $i$-th observation is given by:

Here, $y_i$ is the value of the dependent variable for the $i$-th observation, $x_{ij}$ is the value of the $j$-th predictor for the $i$-th observation, $\beta_0$ is the intercept, $\beta_j$ (for $j=1, \dots, p$) are the regression coefficients for each predictor, and $\epsilon_i$ is the random error term for the $i$-th observation.

The model can be expressed more compactly using matrix notation, which is standard in both theoretical and computational contexts. Let $\mathbf{y}$ be the vector of observed outcomes, $\mathbf{X}$ be the design matrix (which includes a leading column of ones for the intercept), $\boldsymbol{\beta}$ be the vector of coefficients, and $\boldsymbol{\epsilon}$ be the vector of errors. The model is then:

The primary objective is to estimate the coefficient vector $\boldsymbol{\beta}$ that minimizes the sum of squared errors, a method known as Ordinary Least Squares (OLS).

2. Interpretation of Coefficients

A crucial aspect of multiple linear regression is the correct interpretation of the regression coefficients, $\hat{\beta}_j$. Each coefficient represents the estimated change in the dependent variable for a one-unit change in the corresponding predictor variable, while holding all other predictor variables constant. This principle is often referred to as ceteris paribus, a Latin phrase meaning "other things being equal."

For a coefficient $\hat{\beta}_j$, its interpretation is:
"A one-unit increase in $X_j$ is associated with an average change of $\hat{\beta}_j$ units in $y$, assuming all other predictors ($X_k$ for $k \neq j$) in the model remain constant."

This conditional interpretation is fundamental and distinguishes multiple regression from running several simple linear regressions. The value of a coefficient for a particular predictor depends on which other predictors are also included in the model.

Worked Example:

Problem: A real estate analyst develops a model to predict house prices. The fitted model is:

where `Price` is in dollars, `SqFt` is the square footage of the house, and `Age` is the age of the house in years. Predict the price of a 1500 sq. ft. house that is 10 years old. Also, interpret the coefficient for the `Age` variable.

Solution:

Step 1: Identify the given values and the model equation.
The model is $\hat{y} = 50000 + 150 X_1 - 2000 X_2$.
We are given $X_1 = \text{SqFt} = 1500$ and $X_2 = \text{Age} = 10$.

Step 2: Substitute the given values into the model equation to predict the price.

Step 3: Perform the calculations.

Step 4: Compute the final predicted value.

Answer: \boxed{\text{\$255,000}}

Interpretation of the coefficient for `Age`: The coefficient $\hat{\beta}_{\text{Age}} = -2000$. This means that for a given square footage, each additional year of age is associated with a decrease of \$2000 in the predicted price of the house, on average.

---

Problem-Solving Strategies

When faced with multiple linear regression problems in an exam, the task often involves interpreting a given model output or using a fitted equation for prediction.

---

Common Mistakes

A solid understanding of multiple linear regression requires avoiding common pitfalls related to coefficient interpretation and causality.

---

Practice Questions

:::question type="NAT" question="A researcher models the fuel efficiency (in MPG) of a car based on its weight (in kg) and engine displacement (in liters). The fitted regression equation is:

What is the predicted MPG for a car that weighs 1500 kg and has an engine displacement of 2.0 liters?" answer="30.9" hint="Substitute the given values for Weight and Displacement directly into the equation." solution="
Step 1: Write down the given regression equation.

Step 2: Substitute the given values: Weight = 1500 and Displacement = 2.0.

Step 3: Calculate the individual terms.

Step 4: Compute the final value.

Answer: \boxed{30.9}
"
:::

:::question type="MCQ" question="In a multiple linear regression model,

what is the correct interpretation of the coefficient $\hat{\beta}_1$?" options=["The average change in $\hat{y}$ for a one-unit change in $X_1$.","The average change in $\hat{y}$ for a one-unit change in $X_1$, holding $X_2$ constant.","The change in $\hat{y}$ when $X_1$ is 1 and $X_2$ is 0.","The correlation between $X_1$ and $\hat{y}$."] answer="The average change in $\hat{y}$ for a one-unit change in $X_1$, holding $X_2$ constant." hint="The key to interpreting coefficients in multiple regression is the 'ceteris paribus' condition." solution="The coefficient $\hat{\beta}_j$ in a multiple regression model represents the expected change in the dependent variable for a one-unit increase in the predictor $X_j$, under the condition that all other predictors included in the model are held constant. Therefore, the correct interpretation for $\hat{\beta}_1$ is its effect on $\hat{y}$ while controlling for the effect of $X_2$.
Answer: \boxed{The average change in $\hat{y}$ for a one-unit change in $X_1$, holding $X_2$ constant.}"
:::

:::question type="NAT" question="Consider the regression model for predicting employee performance score (from 0 to 100):

According to this model, holding training hours constant, how much is the performance score expected to increase for an employee who gains 4 years of experience?" answer="10" hint="The coefficient for YearsExp gives the change per year. Multiply this by the total number of years." solution="
Step 1: Identify the relevant coefficient.
The coefficient for `YearsExp` is $\hat{\beta}_1 = 2.5$. This means for each one-year increase in experience, the score is expected to increase by 2.5 points, holding `TrainingHours` constant.

Step 2: Calculate the total change for 4 years of experience.

Step 3: Compute the final result.

Answer: \boxed{10}
"
:::

:::question type="MSQ" question="Which of the following statements about multiple linear regression are correct?" options=["The model assumes a linear relationship between each independent variable and the dependent variable.","The dependent variable must be a categorical variable.","The term 'multiple' refers to having more than one dependent variable.","The value of a regression coefficient for a predictor can change if another predictor is added to or removed from the model."] answer="The model assumes a linear relationship between each independent variable and the dependent variable.,The value of a regression coefficient for a predictor can change if another predictor is added to or removed from the model." hint="Consider the fundamental assumptions of linear regression and the conditional nature of its coefficients." solution="

• 'The model assumes a linear relationship between each independent variable and the dependent variable.' This is a core assumption of the model. The relationship between the set of predictors and the outcome is modeled as a linear combination. This statement is correct.


• 'The dependent variable must be a categorical variable.' This is incorrect. For linear regression, the dependent variable must be continuous. For categorical dependent variables, models like logistic regression are used.


• 'The term 'multiple' refers to having more than one dependent variable.' This is incorrect. The term 'multiple' refers to having multiple independent (predictor) variables. Models with multiple dependent variables are known as multivariate regression.


• 'The value of a regression coefficient for a predictor can change if another predictor is added to or removed from the model.' This is correct. The coefficients are estimated while controlling for the other variables in the model. If the set of control variables changes, the estimated coefficient for a given predictor will likely change as well, due to potential correlations between the predictors.

Answer: \boxed{The model assumes a linear relationship between each independent variable and the dependent variable.,The value of a regression coefficient for a predictor can change if another predictor is added to or removed from the model.}"
:::

---

Summary

---

What's Next?

---

---

---

Part 3: Ridge Regression

Introduction

In the study of linear models, our primary objective is often to find the set of coefficients that minimizes the sum of squared errors between predicted and actual values. This method, known as Ordinary Least Squares (OLS), provides excellent, unbiased estimates when its assumptions are met. However, in practical scenarios, we frequently encounter issues such as multicollinearity—where predictor variables are highly correlated—and overfitting, particularly when the number of predictors is large. These problems can lead to large, unstable coefficient estimates with high variance, which generalize poorly to unseen data.

To address these limitations, we introduce regularization techniques. Ridge Regression is one of the most fundamental and widely used regularization methods. It extends standard linear regression by introducing a penalty term to the objective function. This penalty, known as L2 regularization, constrains the magnitude of the model's coefficients. By doing so, Ridge Regression intentionally introduces a small amount of bias into the estimates to achieve a significant reduction in variance, thereby improving the model's overall predictive performance and stability.

---

Key Concepts

1. The L2 Regularization Penalty

The core innovation of Ridge Regression is the addition of the shrinkage penalty, $\lambda \sum_{j=1}^{p} \beta_j^2$. Let us dissect its function. The first component of the objective function, the RSS, seeks to make the model fit the training data as closely as possible. The second component, the L2 penalty, seeks to keep the magnitudes of the coefficients small. The model must therefore find a balance between these two competing goals.

We observe that the penalty term does not include the intercept term, $\beta_0$. This is because the intercept represents the mean prediction when all predictors are zero, and penalizing it would make the model dependent on the origin of the response variable $y$. The summation is over the $p$ predictor coefficients. By penalizing the sum of their squared values, Ridge Regression discourages large coefficients, effectively "shrinking" them towards zero.

This shrinkage is particularly effective in the presence of multicollinearity. When predictors are highly correlated, OLS estimates can become very large and unstable, with small changes in the data leading to large swings in the coefficients. Ridge Regression stabilizes these estimates by pulling them towards zero, making the model more robust.

2. The Regularization Hyperparameter ($\lambda$)

The hyperparameter $\lambda$ (lambda) controls the strength of the L2 penalty and is a critical component of the model. Its value dictates the trade-off between the model's fit to the data (bias) and the magnitude of its coefficients (variance).

• When $\lambda = 0$: The penalty term vanishes, and the Ridge Regression objective function becomes identical to the OLS objective function. The resulting coefficient estimates will be the same as those from Ordinary Least Squares.
• When $\lambda \to \infty$: The penalty for non-zero coefficients becomes overwhelmingly large. To minimize the objective function, the model is forced to make all coefficients approach zero. This results in a model that predicts the mean of the response variable for all inputs, a state of high bias and low variance.
• For $0 < \lambda < \infty$: The model balances fitting the data and shrinking the coefficients. The choice of an optimal $\lambda$ is crucial and is typically determined using cross-validation techniques.

The effect of $\lambda$ on the coefficients is illustrated below. As $\lambda$ increases, the coefficients are continuously shrunk towards zero but do not become exactly zero (unless they were already zero).

λ (Lambda)
Coefficient Value
0
→ ∞


β₁

β₂

β₃

0
Ridge Coefficient Paths

3. Closed-Form Solution

Similar to OLS, Ridge Regression has a closed-form solution for its coefficients. This is a significant advantage, as it allows for direct computation without iterative optimization methods. The solution is expressed in matrix form.

The term $(X^T X + \lambda I)$ is guaranteed to be invertible as long as $\lambda > 0$, even if $X^T X$ is singular (which occurs in cases of perfect multicollinearity). This is a key reason why Ridge Regression is more stable than OLS.

---

Problem-Solving Strategies

---

Common Mistakes

---

Practice Questions

:::question type="MCQ" question="In the context of Ridge Regression, what is the primary effect of increasing the regularization parameter $\lambda$ from a small positive value to a very large value?" options=["The model's variance increases, and its bias decreases.","The model's variance decreases, and its bias increases.","Both the model's bias and variance increase.","The model's coefficients are scaled up, away from zero."] answer="The model's variance decreases, and its bias increases." hint="Recall the bias-variance trade-off. A stronger penalty (larger λ) simplifies the model." solution="Increasing $\lambda$ increases the penalty on the magnitude of the coefficients. This forces the coefficients to shrink towards zero. A simpler model with smaller coefficients has lower variance but is less flexible, leading to higher bias. Therefore, as $\lambda$ increases, variance decreases and bias increases."
:::

:::question type="NAT" question="Consider a dataset with a standardized feature matrix $X$ and response vector $y$. Let $X^T X = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$ and $X^T y = \begin{pmatrix} 5 \\ 2 \end{pmatrix}$. For a Ridge Regression model with $\lambda = 2$, what is the value of the first coefficient, $\hat{\beta}_1$?" answer="1.2" hint="Use the closed-form solution $\hat{\boldsymbol{\beta}}_{\operatorname{ridge}} = (\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^T \mathbf{y}$. You will need to compute the inverse of a 2x2 matrix." solution="
Step 1: Set up the equation for the Ridge coefficients.
The formula is $\hat{\boldsymbol{\beta}}_{\operatorname{ridge}} = (\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^T \mathbf{y}$.

Step 2: Calculate the term $(\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})$.
We are given $\lambda = 2$ and $\mathbf{X}^T \mathbf{X} = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$. The identity matrix $\mathbf{I}$ is $\begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix}$.

Step 3: Compute the inverse of $(\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})$.
For a $2 \times 2$ matrix $A = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$, the inverse is $A^{-1} = \frac{1}{ad-bc} \begin{pmatrix} d & -b \\ -c & a \end{pmatrix}$.
Here, $a=4, b=1, c=1, d=4$. The determinant is $(4)(4) - (1)(1) = 16 - 1 = 15$.

Step 4: Calculate the final coefficient vector $\hat{\boldsymbol{\beta}}_{\operatorname{ridge}}$.

The question asks for the first coefficient, $\hat{\beta}_1$.

Answer: \boxed{1.2} ::: :::question type="MSQ" question="Which of the following statements about Ridge Regression are true?" options=["It can be used to mitigate the problem of multicollinearity.","It performs feature selection by setting some coefficients to exactly zero.","The solution for Ridge coefficients is typically found using iterative optimization methods.","As the regularization parameter $\lambda$ approaches infinity, the coefficients approach zero."] answer="It can be used to mitigate the problem of multicollinearity.,As the regularization parameter λ approaches infinity, the coefficients approach zero." hint="Consider the core purpose of Ridge Regression and the mathematical properties of the L2 penalty." solution="

• Option A is correct. Ridge Regression is specifically designed to handle multicollinearity by penalizing large coefficients, which are a common symptom of highly correlated predictors. This stabilizes the model.
• Option B is incorrect. This describes Lasso (L1) regression. The L2 penalty in Ridge Regression shrinks coefficients towards zero but does not set them to exactly zero unless $\lambda$ is infinite.
• Option C is incorrect. Ridge Regression has a closed-form analytical solution, $\hat{\boldsymbol{\beta}}_{\operatorname{ridge}} = (\mathbf{X}^T \mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^T \mathbf{y}$, so iterative methods are not required.
• Option D is correct. As $\lambda$ becomes infinitely large, the penalty term dominates the loss function. To minimize the loss, the model must shrink the coefficients to be infinitesimally close to zero.

Answer: \boxed{It can be used to mitigate the problem of multicollinearity.,As the regularization parameter $\lambda$ approaches infinity, the coefficients approach zero.} " :::

---

Summary

---

What's Next?

---

Chapter Summary

---

Chapter Review Questions

:::question type="MCQ" question="Consider a multiple linear regression model built using Ordinary Least Squares (OLS). A new predictor variable is added that is highly correlated with one of the existing predictors. Which of the following statements most accurately describes the likely consequence for the OLS model and a corresponding Ridge Regression model?" options=["The OLS coefficient estimates may become unstable, while the Ridge Regression estimates will remain relatively stable.","Both OLS and Ridge Regression coefficient estimates will become highly unstable.","The model's coefficient of determination ($R^2$) will necessarily decrease for the OLS model.","The OLS estimates will remain stable, but the Ridge Regression estimates will be shrunk aggressively towards zero."] answer="A" hint="Think about the effect of multicollinearity on the $(\mathbf{X}^T\mathbf{X})$ matrix and how the Ridge Regression formula counteracts this effect." solution="The introduction of a highly correlated predictor induces multicollinearity.

Impact on OLS: In OLS, the coefficients are calculated using $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$. Multicollinearity makes the matrix $\mathbf{X}^T\mathbf{X}$ nearly singular, causing its inverse $(\mathbf{X}^T\mathbf{X})^{-1}$ to be numerically unstable. This results in large standard errors and highly sensitive (unstable) coefficient estimates.

Impact on Ridge Regression: The Ridge Regression formula is $\hat{\boldsymbol{\beta}}_{\operatorname{ridge}} = (\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$. The term $\lambda\mathbf{I}$ (where $\lambda > 0$) is added to $\mathbf{X}^T\mathbf{X}$ before inversion. This ensures that the matrix $(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})$ is always invertible and well-conditioned. Consequently, the coefficient estimates remain stable even in the presence of multicollinearity.

Therefore, the OLS estimates become unstable, while Ridge Regression provides a more stable solution.
Answer: \boxed{A}
"
:::

:::question type="NAT" question="For a simple linear regression model $y = \beta_0 + \beta_1 x$, the following summary statistics have been computed from a dataset of $n=20$ observations:
$\sum_{i=1}^{20} x_i = 100$, $\sum_{i=1}^{20} y_i = 300$, $\sum_{i=1}^{20} x_i y_i = 1800$, and $\sum_{i=1}^{20} x_i^2 = 700$.
Calculate the value of the slope coefficient, $\hat{\beta}_1$, estimated using Ordinary Least Squares." answer="1.5" hint="Recall the computational formula for the OLS slope estimator $\hat{\beta}_1$ that uses sums of observations." solution="The formula for the Ordinary Least Squares (OLS) estimator of the slope coefficient, $\hat{\beta}_1$, is given by:

We are given the following values:

• $n = 20$


• $\sum x_i = 100$


• $\sum y_i = 300$


• $\sum x_i y_i = 1800$


• $\sum x_i^2 = 700$

Now, we substitute these values into the formula.

Numerator:

Denominator:

Calculation of $\hat{\beta}_1$:

Thus, the estimated slope coefficient is 1.5.
Answer: \boxed{1.5}
"
:::

:::question type="MCQ" question="Which of the following statements correctly describes the bias-variance trade-off in Ridge Regression as the regularization parameter $\lambda$ is increased from zero?" options=["Bias decreases and variance increases.","Bias increases and variance decreases.","Both bias and variance increase.","Both bias and variance decrease."] answer="B" hint="Consider how increasing the penalty on the magnitude of the coefficients affects the model's flexibility and its sensitivity to the training data." solution="The regularization parameter $\lambda$ in Ridge Regression controls the penalty on the size of the coefficients.

• When $\lambda = 0$, Ridge Regression is identical to OLS. Assuming the true model is linear, OLS is an unbiased estimator, but it can have high variance, especially with multicollinearity or a large number of predictors.

• As we increase $\lambda$ from zero, we impose a greater penalty on large coefficients. This forces the coefficients to shrink towards zero. This shrinkage introduces bias into the model because the coefficients are now likely to be smaller than the true population values.

• However, by constraining the coefficients, we make the model less sensitive to the specific training data. A small change in the training set will lead to a smaller change in the estimated coefficients compared to OLS. This means the model's variance decreases.

Therefore, increasing $\lambda$ increases the model's bias while decreasing its variance. The goal of tuning $\lambda$ is to find a sweet spot that minimizes the total error (e.g., MSE), which is a function of both bias and variance. Answer: \boxed{B} " :::

:::question type="NAT" question="In a multiple linear regression problem with two predictors, the relevant matrices after centering the data are given as:

Calculate the first coefficient, $\hat{\beta}_1$, for a Ridge Regression model with a regularization parameter $\lambda = 5$. Provide the answer rounded to one decimal place." answer="0.6" hint="Use the formula $\hat{\boldsymbol{\beta}}_{\operatorname{ridge}} = (\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$ and solve for the coefficient vector." solution="The solution for the Ridge Regression coefficient vector is given by the formula $\hat{\boldsymbol{\beta}}_{\operatorname{ridge}} = (\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$.

Step 1: Compute the matrix $(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})$
Given $\lambda = 5$, we have:

Step 2: Compute the inverse of this matrix
For a general $2 \times 2$ matrix $\begin{pmatrix} a & b \\ c & d \end{pmatrix}$, the inverse is $\frac{1}{ad-bc} \begin{pmatrix} d & -b \\ -c & a \end{pmatrix}$.

• The determinant is $ad-bc = (25)(25) - (10)(10) = 625 - 100 = 525$.


• The inverse is therefore:

Step 3: Multiply the inverse by $\mathbf{X}^T\mathbf{y}$ to find the coefficients

Step 4: Extract the value of $\hat{\beta}_1$ and round
The question asks for the first coefficient, $\hat{\beta}_1$:

Rounding to one decimal place, the answer is 0.6.
Answer: \boxed{0.6}
"
:::

---

What's Next?