Least Squares Regression and Correlation: Study Notes

Least Squares Regression

Introduction to Regression

Regression analysis is a statistical method used to examine the relationship between an explanatory variable (x) and a response variable (y). When these variables appear to have a linear association, a regression line can summarize their relationship and be used for prediction.

• Regression line: Describes how y changes with x.

• Prediction: Allows estimation of y for a given x.

• Explanatory vs. Response: In regression, it is essential to specify which variable is explanatory and which is response.

Equation of a Straight Line

The general equation for a straight line in the (x,y)-plane is:

• Equation: $y = b_0 + b_1 x$

• y-intercept ($b_0$): The value of y when x = 0.

• Slope ($b_1$): The change in y for a one-unit increase in x; units are (units of y)/(units of x).

Interpretation: An increase of 1 unit in x produces an increase of $b_1$ units in y.

Least-Squares Regression Line

The least-squares regression line is the line that minimizes the sum of the squared vertical distances (errors or residuals) between the observed values and the predicted values.

• Equation: $y = b_0 + b_1 x$

• Predicted value ($\hat{y}$): For any x, $\hat{y}$ is the value on the regression line, not necessarily an observed value.

• Residual: The difference between the observed value and the predicted value: $\text{Residual} = y - \hat{y}$

Minimization criterion: The least-squares regression line minimizes:

$\sum_{i=1}^n (y_i - (b_0 + b_1 x_i))^2$

Finding the Least-Squares Regression Line

To find the best-fitting line, we use summary statistics and the correlation coefficient:

• Slope: $b_1 = r \times \frac{s_y}{s_x}$

• Intercept: $b_0 = \bar{y} - b_1 \bar{x}$

• Where $r$ is the correlation coefficient, $s_x$ and $s_y$ are the standard deviations of x and y, and $\bar{x}$, $\bar{y}$ are their means.

Worked Example: Golf Club Speed vs Distance

Given data for club-head speed and distance traveled by a golf ball:

Club-head Speed (mph)	Distance (yards)
100	257
102	264
103	274
105	275
99	258
101	266
115	279
109	289

Step 1: Select two points, e.g., (99, 258) and (105, 275).

Step 2: Calculate the slope using the point-slope formula:

$m = \frac{275 - 258}{105 - 99} = \frac{17}{6} = 2.8333$

Step 3: Find the equation of the line:

$y - 258 = 2.8333(x - 99)$ $y = 2.8333x - 22.4967$

Step 4: Predict distance for club-head speed of 104 mph:

$y = 2.8333 \times 104 - 22.4967 = 272.2$ yards

Least-Squares Regression Line Using All Data

Using all data and summary statistics:

• $b_1 = 3.1661$

• $b_0 = -55.7966$

• Regression equation: $\hat{y} = 3.1661x - 55.7966$

Prediction for x = 103:

$\hat{y} = 3.1661 \times 103 - 55.7966 = 270.3$ yards

Residual calculation:

Observed value: 274 yards Residual: $274 - 270.3 = 3.7$ yards (above average)

Correlation and Regression

Correlation measures the strength and direction of a linear relationship between two variables. Regression quantifies the relationship and allows prediction.

• Correlation coefficient ($r$): Ranges from -1 to 1.

• Effect of adding points: Adding an outlier can change the value of $r$ significantly.

Switching Explanatory and Response Variables

If we exchange explanatory and response variables, the correlation coefficient $r$ remains the same, but the regression line changes. The least-squares regression line minimizes vertical distances; switching variables would minimize horizontal distances.

• Solid line: Predicts y for a given x.

• Dashed line: Predicts x for a given y.

Interpretation of Slope and Intercept

Slope: Indicates the average change in the response variable for a one-unit increase in the explanatory variable. Interpretation is probabilistic, not deterministic.

• If club-head speed increases by 1 mph, the expected distance increases by 3.1661 yards.

Intercept: Represents the predicted value of y when x = 0. Only interpret if x = 0 is reasonable and observed in the data.

Scope of the Model

Predictions should only be made within the range of observed values for the explanatory variable. Extrapolation outside this range is unreliable.

• Do not use the regression line to predict for x values much larger or smaller than those observed.

Comparison of Linear Approximations

Different methods of fitting a line (using two points vs. least-squares regression) can yield different equations. Least-squares regression is preferred for minimizing overall error.

Practice: Calculating Regression Line from Summary Statistics

Given $s_x = 2$, $s_y = 4$, $r = 0.5$, $\bar{x} = 5$, $\bar{y} = 7$:

• $b_1 = 0.5 \times \frac{4}{2} = 1$

• $b_0 = 7 - 1 \times 5 = 2$

• Regression equation: $\hat{y} = 2 + 1x$

Recap: Calculation of Correlation

Correlation is calculated using the formula:

$r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{(n-1)s_x s_y}$

i	x_i	y_i	(x_i - \bar{x})	(y_i - \bar{y})	product
1	2	5	-1	+1.17	-1.28
2	3	2	0	-0.83	0
3	4	4	+1	+0.65	+0.65
Additional info: Table used for step-by-step calculation of $r$.

Key Points Summary

• Least-squares regression line minimizes the sum of squared residuals.

• Regression equation allows prediction of response variable from explanatory variable.

• Interpretation of slope and intercept must consider the context and scope of the data.

• Correlation quantifies strength and direction of linear relationship.

• Extrapolation outside observed data range is not recommended.