Regression, Correlation, and Probability in Statistics

Regression and Correlation

Introduction to Regression and Correlation

Regression and correlation are fundamental concepts in statistics used to describe and analyze the relationship between two quantitative variables. Regression focuses on predicting the value of one variable based on another, while correlation measures the strength and direction of their linear relationship.

• Regression Line: A straight line that best fits the data points on a scatterplot, used for prediction.

• Correlation Coefficient (r): A numerical measure of the strength and direction of a linear relationship between two variables. Values range from -1 (perfect negative) to +1 (perfect positive).

• Extrapolation: Using the regression line to make predictions beyond the range of observed data. This can be unreliable as the relationship may not hold outside the observed range.

• Regression Toward the Mean: The phenomenon where extreme values tend to be closer to the average on subsequent measurements.

Example: If you know a person's height in inches, you can predict their height in centimeters using a regression line. If all points fall perfectly on the line, the correlation is 1, and the regression perfectly predicts the outcome.

Influential Points

Some data points, known as influential points, have a significant effect on the regression line. Their presence or absence can greatly change the results of the analysis.

• Influential Points: Observations that, if removed, would noticeably change the regression line or correlation coefficient.

Coefficient of Determination ($R^2$)

The coefficient of determination, denoted as $R^2$, measures the proportion of the variance in the dependent variable that is predictable from the independent variable using the regression line.

• Definition: $R^2$ is the square of the correlation coefficient ($r$).

• Interpretation: $R^2$ represents the percentage of variation in the response variable explained by the regression model.

• Formula:

$R^2 = r^2$

• Example: If $r = 0.8$, then $R^2 = 0.64$, meaning 64% of the variation in the response variable is explained by the model.

Application: In a dataset where height in inches and centimeters are perfectly correlated, $R^2 = 1$, indicating all variation is explained by the regression line.

Scatterplots and Linear Relationships

Scatterplots visually display the relationship between two quantitative variables. The strength and direction of the relationship can be assessed by how closely the points follow a straight line.

• Strong Linear Relationship: Points are close to the regression line; high $|r|$ and high $R^2$.

• Weak Linear Relationship: Points are widely scattered; low $|r|$ and low $R^2$.

Example: Comparing height versus weight and waist size versus weight, the variable with points more tightly clustered around the regression line (waist size) has a higher $R^2$ and is a better predictor.

Choosing the Best Predictor

When multiple explanatory variables are available, the one with the highest coefficient of determination ($R^2$) is generally the best predictor for the response variable.

• Example: If waist size has a higher $R^2$ with weight than height does, waist size is a better predictor of weight.

Probability Concepts

Basic Probability Rules

Probability quantifies the likelihood of an event occurring. The probability of an event ranges from 0 (impossible) to 1 (certain).

• Complement Rule: The probability that an event does not occur is $1$ minus the probability that it does occur.

• Formula:

$P( ext{not } A) = 1 - P(A)$

• Equally Likely Outcomes: When all outcomes have the same chance of occurring, probability is calculated as:

$P(A) = \frac{\text{Number of favorable outcomes}}{\text{Total number of outcomes}}$

Mutually Exclusive Events

Two events are mutually exclusive if they cannot both occur at the same time. In a Venn diagram, mutually exclusive events have no overlap.

• Example: Drawing a card that is both a heart and a club is impossible; these events are mutually exclusive.

• Addition Rule for Mutually Exclusive Events:

$P(A \text{ or } B) = P(A) + P(B)$

Non-Mutually Exclusive Events

Events that can occur together are not mutually exclusive. The probability that at least one occurs is:

$P(A \text{ or } B) = P(A) + P(B) - P(A \text{ and } B)$

• Example: The probability that a person is married or has a college degree includes those who are both married and have a college degree.

Probability Tables and Venn Diagrams

Tables and Venn diagrams are useful for organizing and visualizing probabilities, especially when dealing with overlapping categories.

	Married	Not Married	Total
College Degree	n1	n2	n1 + n2
No College Degree	n3	n4	n3 + n4
Total	n1 + n3	n2 + n4	N

Additional info: The table above is a generic contingency table for two categorical variables (e.g., marital status and education level). Probabilities can be calculated by dividing cell counts by the total N.

Summary Table: Key Terms and Concepts

Term	Definition	Example/Application
Regression Line	Best-fit line for predicting one variable from another	Predicting height in cm from height in inches
Correlation Coefficient ($r$)	Measures strength and direction of linear relationship	$r = 1$ for perfect positive correlation
Coefficient of Determination ($R^2$)	Proportion of variance explained by regression	$R^2 = 0.64$ means 64% of variation explained
Mutually Exclusive Events	Events that cannot occur together	Drawing a heart or a club in a single card draw
Complement	Probability of event not occurring	$P( ext{not } A) = 1 - P(A)$