Study Notes: Simple Linear Regression, Inference, and Multiple Regression

Ch. 12: Simple Linear Regression

Introduction to Simple Linear Regression

Simple linear regression is a statistical method used to model the relationship between a single explanatory variable and a response variable by fitting a linear equation to observed data.

• Explanatory variable (predictor): The variable used to explain or predict changes in the response variable.

• Response variable: The outcome variable whose variation is being studied.

• Least squares regression method: Finds the line that minimizes the sum of squared differences between observed and predicted values.

Regression Equation:

The regression line is given by:

$y = b_0 + b_1 x$

• Intercept ($b_0$): The predicted value of y when x = 0.

• Slope ($b_1$): The change in y for a one-unit increase in x.

Correlation ($r$): Measures the strength and direction of the linear relationship between two variables. $-1 \leq r \leq 1$.

Coefficient of Determination ($R^2$): Represents the proportion of variance in the response variable explained by the explanatory variable.

$R^2 = r^2$

• Residuals: The differences between observed and predicted values ($e = y - \hat{y}$).

Example: If the regression equation is $\hat{y} = 2 + 3x$, then for $x = 4$, the predicted $y$ is $\hat{y} = 2 + 3 \times 4 = 14$.

Ch. 12: Inference for Simple Linear Regression

Statistical Inference in Regression

Inference in regression involves making conclusions about the population regression parameters based on sample data.

• Hypothesis Test for Slope: Tests whether the explanatory variable is significantly associated with the response variable.

Null Hypothesis ($H_0$): $\beta_1 = 0$ (no association)

Alternative Hypothesis ($H_a$): $\beta_1 \neq 0$ (association exists)

Test Statistic:

$t = \frac{b_1 - 0}{SE_{b_1}}$

• Where $SE_{b_1}$ is the standard error of the slope estimate.

Confidence Interval for Slope:

$b_1 \pm t^* SE_{b_1}$

• Where $t^*$ is the critical value from the t-distribution.

Assumptions:

• Linearity

• Independence

• Normality of residuals

• Equal variance (homoscedasticity)

Example: Testing if the slope is significantly different from zero to determine if the predictor variable has a meaningful effect on the response.

Ch. 13: Multiple Regression

Introduction to Multiple Regression

Multiple regression extends simple linear regression by modeling the relationship between a response variable and two or more explanatory variables.

• Model: $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \ldots + \beta_k x_k + \epsilon$

• Response variable: The outcome being predicted.

• Explanatory variables: The predictors used to explain variation in the response.

ANOVA for Regression: Used to test the overall significance of the regression model.

Individual Variable Testing: Each coefficient is tested to determine if it significantly contributes to the model.

Assumptions:

• Linearity

• Independence

• Normality of residuals

• Equal variance

Model Selection: Variables can be added or removed based on their statistical significance and contribution to $R^2$.

Quadratic Regression: Includes squared terms to model curvature in the relationship.

Interpretation of Coefficients: Each coefficient represents the expected change in the response variable for a one-unit change in the predictor, holding other variables constant.

Example: Predicting house prices using multiple features such as size, number of bedrooms, and age of the house.

Additional info: These notes summarize key concepts and formulas for regression analysis, including both simple and multiple regression, as well as inference procedures. The content is suitable for exam preparation in a college-level statistics course.