Least Squares Regression and Correlation: Study Notes

Least Squares Regression and Correlation

Introduction to Least Squares Regression

Least squares regression is a statistical method used to model the linear relationship between an explanatory variable (x) and a response variable (y). It is foundational in statistics for describing, explaining, and predicting how one variable changes in relation to another.

• Regression Line: A regression line summarizes the relationship between x and y and allows prediction of y for given values of x.

• Explanatory vs. Response Variable: In regression, it is essential to specify which variable is explanatory (independent) and which is response (dependent).

Correlation and Its Calculation

Correlation measures the strength and direction of the linear relationship between two variables. The correlation coefficient (r) ranges from -1 to 1.

• Positive Correlation: As x increases, y tends to increase.

• Negative Correlation: As x increases, y tends to decrease.

• Practice Example: Adding an outlier to a dataset can change the value of r, often reducing the strength of the correlation.

Correlation Calculation Table

i	xi	yi	(xi - x̄)	(yi - ȳ)	product
1	2	5	-1	+1.17	-1.28
2	3	2	0	-0.83	0.00
3	4	4	+1	+0.65	+0.65
Additional info: Table used to compute correlation coefficient by summing products and dividing by standard deviations.

Equation of a Straight Line

The general equation for a straight line in the (x, y)-plane is:

• Equation: $y = b_0 + b_1 x$

• y-intercept (b0): The value of y when x = 0.

• Slope (b1): The change in y for a one-unit increase in x.

Least Squares Regression Line

The least-squares regression line is the line that minimizes the sum of the squared vertical distances (errors or residuals) between the observed values and the predicted values.

• Mathematical Criterion: The line minimizes the quantity:

$\sum_{i=1}^n (y_i - (b_0 + b_1 x_i))^2$

• Predicted Value: $\hat{y}$ denotes the predicted value of y for a given x.

• Residual: The difference between the observed value and the predicted value: $\text{Residual} = y - \hat{y}$

Finding the Least Squares Regression Line

To find the least-squares regression line, use the following formulas:

• Slope: $b_1 = r \times \frac{s_y}{s_x}$

• Intercept: $b_0 = \bar{y} - b_1 \bar{x}$

Where:

• r = correlation coefficient

• $s_x$ = standard deviation of x

• $s_y$ = standard deviation of y

• $\bar{x}$ = mean of x

• $\bar{y}$ = mean of y

Worked Example: Golf Club Speed vs Distance

Given data for club-head speed and distance, we can fit a regression line and make predictions.

Club-head Speed (mph)	Distance (yards)	(x, y)
100	257	(100, 257)
102	264	(102, 264)
103	274	(103, 274)
105	275	(105, 275)
99	258	(99, 258)
101	266	(101, 266)
115	279	(115, 279)
109	239	(109, 239)

• Step 1: Select two points, e.g., (99, 258) and (105, 275).

• Step 2: Calculate the slope:

$m = \frac{275 - 258}{105 - 99} = \frac{17}{6} = 2.8333$

• Step 3: Use the point-slope formula:

$y - 258 = 2.8333(x - 99)$ $y = 2.8333x - 22.4967$

• Prediction: For $x = 104$:

$y = 2.8333 \times 104 - 22.4967 = 272.2$ yards

Least Squares Regression Line from Summary Statistics

• Given $s_x = 2$, $s_y = 4$, $r = 0.5$, $\bar{x} = 5$, $\bar{y} = 7$:

$b_1 = 0.5 \times \frac{4}{2} = 1$ $b_0 = 7 - 1 \times 5 = 2$ Regression equation: $\hat{y} = 2 + 1x$

Interpreting Slope and Intercept

• Slope: Indicates the average change in y for a one-unit increase in x. Interpretation should be probabilistic, not deterministic.

• Intercept: Represents the predicted value of y when x = 0. Only interpret if x = 0 is reasonable and observed in the data.

Scope and Caution in Regression

• Scope of the Model: Predictions should only be made within the range of observed x values. Extrapolation beyond this range is unreliable.

• Example: Do not use the regression line to predict distance for club-head speeds much higher than those observed (e.g., 140 mph when the highest observed is 105 mph).

Switching Explanatory and Response Variables

Switching the roles of explanatory and response variables does not change the correlation coefficient (r), but it does change the regression line. The least-squares regression line minimizes vertical distances (errors in y), while the reverse regression minimizes horizontal distances (errors in x).

• Example: Predicting velocity for a given distance vs. predicting distance for a given velocity yields different regression lines.

Comparison of Linear Approximations

Different methods of fitting a line (e.g., using two points vs. least squares) can yield different regression equations. The least-squares method is preferred for minimizing overall error.

• Example: Comparing $y = 3.1661x - 55.7966$ (least squares) and $y = 2.8333x - 22.4967$ (two-point method) on the same scatterplot.

Summary Table: Key Concepts in Least Squares Regression

Concept	Definition	Formula
Regression Line	Line summarizing relationship between x and y	$y = b_0 + b_1 x$
Slope (b1)	Change in y per unit change in x	$b_1 = r \frac{s_y}{s_x}$
Intercept (b0)	Value of y when x = 0	$b_0 = \bar{y} - b_1 \bar{x}$
Residual	Difference between observed and predicted y	$y - \hat{y}$
Sum of Squared Residuals	Total squared error minimized by regression	$\sum (y_i - \hat{y}_i)^2$

Additional info: These notes expand on the lecture slides and images, providing full definitions, formulas, and context for least squares regression and correlation, suitable for college-level statistics students.