Simple Linear Regression: Concepts, Computation, and Interpretation

Chapter 11: Simple Regression

Chapter Goals

• Explain the simple linear regression model

• Obtain and interpret the simple linear regression equation for a set of data

• Describe R2 as a measure of explanatory power of the regression model

• Understand the assumptions behind regression analysis

• Explain measures of variation and determine whether the independent variable is significant

• Calculate and interpret confidence intervals for the regression coefficients

• Use a regression equation for prediction

• Form forecast intervals around an estimated Y value for a given X

• Use graphical analysis to recognize potential problems in regression analysis

• Explain the correlation coefficient and perform a hypothesis test for zero population correlation

Section 11.1 Overview of Linear Models

Simple Linear Regression Model

The simple linear regression model describes the relationship between two variables using a straight line. The general form is:

• Equation: $Y = \beta_0 + \beta_1 X$

• Y: Dependent variable (the outcome we wish to explain)

• X: Independent variable (the predictor or explanatory variable)

• \beta_0: Y-intercept (value of Y when X = 0)

• \beta_1: Slope (change in Y for a one-unit change in X)

Least Squares Regression

Estimates for the coefficients are found using the least squares regression technique, which minimizes the sum of squared errors between observed and predicted values.

• Sample regression equation: $\hat{y} = b_0 + b_1 x$

• Slope estimator: $b_1 = \frac{\text{Cov}(x, y)}{s_x^2} = r \frac{s_y}{s_x}$

• Intercept estimator: $b_0 = \bar{y} - b_1 \bar{x}$

Introduction to Regression Analysis

• Purpose:

  • Predict the value of a dependent variable based on the value of at least one independent variable

  • Explain the impact of changes in an independent variable on the dependent variable

• Dependent variable: Also called the endogenous variable

• Independent variable: Also called the exogenous variable

Section 11.2 Linear Regression Model

Population Regression Equation

• The relationship between X and Y is described by a linear function:

$y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$

• $\beta_0$ and $\beta_1$ are population coefficients

• $\varepsilon_i$ is a random error term

Regression Model Components

• Linear component: $\beta_0 + \beta_1 x_i$

• Random error component: $\varepsilon_i$

Assumptions of Linear Regression

• The true relationship is linear: Y is a linear function of X plus random error

• Error terms $\varepsilon_i$ are independent of X values

• Error terms are random variables with mean 0 and constant variance $\sigma^2$ (homoscedasticity):

  • $E[\varepsilon_i] = 0$

  • $E[\varepsilon_i^2] = \sigma^2$ for all $i$

  • $E[\varepsilon_i \varepsilon_j] = 0$ for all $i \neq j$ (no autocorrelation)

Section 11.3 Least Squares Coefficient Estimators

Finding the Best-Fit Line

• Coefficients $b_0$ and $b_1$ are chosen to minimize the sum of squared errors (SSE):

$\text{min SSE} = \min \sum_{i=1}^n e_i^2 = \min \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \min \sum_{i=1}^n [y_i - (b_0 + b_1 x_i)]^2$

• Slope estimator: $b_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{\text{Cov}(x, y)}{s_x^2} = r \frac{s_y}{s_x}$

• Intercept estimator: $b_0 = \bar{y} - b_1 \bar{x}$

• The regression line always passes through the means $(\bar{x}, \bar{y})$

Computation Using Software

• Hand calculations are tedious; statistical software (e.g., Excel) is commonly used

Interpretation of Coefficients

• Intercept ($b_0$): Estimated average value of Y when X = 0 (if X = 0 is within the observed range)

• Slope ($b_1$): Estimated change in average Y for a one-unit increase in X

Simple Linear Regression Example

Application: House Price and Size

• Dependent variable (Y): House price in $1000s

• Independent variable (X): Square feet

• Sample data (10 houses):

House Price in $1000s (Y)	Square Feet (X)
245	1400
312	1600
279	1700
308	1875
199	1100
319	1550
405	2350
324	2450
319	1425
255	1700

• Scatter plot and regression line can be generated using Excel

Regression Equation from Excel Output

• Estimated regression equation: $\text{house price} = 98.24833 + 0.10977 \times \text{(square feet)}$

• Interpretation of $b_0$: $98,248.33 is the portion of house price not explained by square feet (within observed range)

• Interpretation of $b_1$: For each additional square foot, house price increases by $0.10977 \times 1000 = $109.77 on average

Section 11.4 Explanatory Power of a Linear Regression Equation

Measures of Variation

• Total Sum of Squares (SST): $SST = \sum (y_i - \bar{y})^2$

• Regression Sum of Squares (SSR): $SSR = \sum (\hat{y}_i - \bar{y})^2$

• Error Sum of Squares (SSE): $SSE = \sum (y_i - \hat{y}_i)^2$

• Relationship: $SST = SSR + SSE$

Analysis of Variance (ANOVA)

• SST: Variation of Y values around their mean

• SSR: Explained variation due to X

• SSE: Unexplained variation (random error)

Coefficient of Determination ($R^2$)

• Proportion of total variation in Y explained by X

• $R^2 = \frac{SSR}{SST}$

• Ranges from 0 to 1

• Interpretation: Higher $R^2$ indicates a better fit

Examples of $r^2$ Values

• $r^2 = 1$: Perfect linear relationship

• $0 < r^2 < 1$: Weaker linear relationship

• $r^2 = 0$: No linear relationship

Excel Output Example

• In the house price example, $R^2 = 0.58082$

• Interpretation: 58.08% of the variation in house prices is explained by variation in square feet

Relationship Between Correlation and $R^2$

• For simple regression: $R^2 = r^2$

Estimation of Model Error Variance

• Estimator for variance of model error:

$\hat{\sigma}^2 = s_e^2 = \frac{\sum_{i=1}^n e_i^2}{n-2} = \frac{SSE}{n-2}$

• $s_e = \sqrt{s_e^2}$ is the standard error of the estimate

• Division by $n-2$ (not $n-1$) because two parameters are estimated ($b_0$ and $b_1$)

Summary

• Simple linear regression models the relationship between two variables using a straight line

• Coefficients are estimated using least squares, minimizing the sum of squared errors

• Interpretation of coefficients provides insight into the relationship between variables

• Measures of variation and $R^2$ assess the explanatory power of the model

• Statistical software is commonly used for computation