MA 113: Midterm 1 & 2 Study Guide – Regression, Probability, Discrete & Continuous Distributions, Sampling, and Confidence Intervals

Regression

Introduction to Regression and the Coefficient of Determination

Regression analysis is a statistical method used to examine the relationship between two or more variables. The coefficient of determination, denoted as R2, quantifies the proportion of variance in the dependent variable that is predictable from the independent variable(s).

• Coefficient of Determination (R2): Measures the fraction of variance explained by the regression model. For linear least squares regression, $R^2 = r^2$ where $r$ is the correlation coefficient.

• Residual: The difference between an observed value and its predicted value: $y - \hat{y}$.

• Residual Plot: A graphical tool to assess the fit of a regression model and check for non-linearity or non-constant error variance.

• Outlier: An observation with a large residual.

• Influential Observation: A data point that significantly affects the regression line.

Formulas:

• Residual (Unexplained deviation): $y - \hat{y}$

• Explained deviation: $\hat{y} - \bar{y}$

• Total deviation: $y - \bar{y}$

• Unexplained variation: $\sum (y_i - \hat{y}_i)^2$

• Explained variation: $\sum (\hat{y}_i - \bar{y})^2$

• Total variation: $\sum (y_i - \bar{y})^2$

• $R^2 = \dfrac{\text{explained variation}}{\text{total variation}} = 1 - \dfrac{\text{unexplained variation}}{\text{total variation}}$

Example: If a regression model explains 80% of the variance in exam scores based on study hours, then $R^2 = 0.80$.

Probability

Rules of Probability

Probability quantifies the likelihood of events occurring in a random experiment. There are several approaches and rules for calculating probabilities.

• Sample Space (S): The set of all possible outcomes.

• Event: A subset of the sample space.

• Empirical Probability: Based on observed data; $P(E) = \text{relative frequency of } E$.

• Classical Probability: For equally likely outcomes; $P(E) = \dfrac{N(E)}{N(S)}$.

• Probability Model: Assigns probabilities to all outcomes in the sample space such that $0 \leq P(E) \leq 1$ and $\sum P(e_i) = 1$.

• Unusual Event: Probability less than 0.05.

Formulas:

• Empirical: $P(E) = \text{relative frequency of } E$

• Classical: $P(E) = \dfrac{N(E)}{N(S)}$

• $P(e_1) + P(e_2) + \cdots + P(e_n) = 1$

• $0 \leq P(E) \leq 1$

Example: If 3 out of 10 students prefer online classes, the empirical probability is $P(E) = 0.3$.

Addition Rules and Venn Diagrams

The addition rules help calculate the probability of the union of events. Venn diagrams visually represent these relationships.

• Disjoint (Mutually Exclusive) Events: Events that cannot occur together.

• General Addition Rule: For any events E and F, $P(E \text{ or } F) = P(E) + P(F) - P(E \text{ and } F)$.

• Complement Rule: $P(E^c) = 1 - P(E)$

Formulas:

• If E and F are disjoint: $P(E \text{ or } F) = P(E) + P(F)$

• General: $P(E \text{ or } F) = P(E) + P(F) - P(E \text{ and } F)$

• Complement: $P(E^c) = 1 - P(E)$

Example: If $P(A) = 0.3$, $P(B) = 0.4$, and $P(A \text{ and } B) = 0.1$, then $P(A \text{ or } B) = 0.3 + 0.4 - 0.1 = 0.6$.

Independence

Two events are independent if the occurrence of one does not affect the probability of the other.

• Independent Events: $P(A \text{ and } B) = P(A)P(B)$

• Dependent Events: Events that are not independent.

Example: Tossing two coins: the result of one does not affect the other.

Discrete Probability Distributions

Measures of Central Tendency

A discrete random variable takes on countable values, each with an associated probability. The mean (expected value) and standard deviation summarize its distribution.

• Random Variable: A variable whose value is a numerical outcome of a random phenomenon.

• Discrete Random Variable: Takes on countable values.

• Probability Mass Function (pmf): $P(x) = P(X = x)$

• Expected Value (Mean): $\mu_X = \sum [x \cdot P(x)]$

• Standard Deviation: $\sigma_X = \sqrt{\sum [(x - \mu_X)^2 \cdot P(x)]}$

• Law of Large Numbers: As sample size increases, the sample mean and standard deviation approach the population values.

Example: Rolling a fair die: $X$ can be 1-6, each with $P(x) = 1/6$.

Binomial Distribution

The binomial distribution models the number of successes in a fixed number of independent trials, each with the same probability of success.

• Binomial Experiment: Fixed number of independent trials, each with two outcomes (success/failure).

• Binomial Random Variable: Counts the number of successes in $n$ trials.

• Combination Notation: $nCk = \dfrac{n!}{k!(n-k)!}$

• Binomial Probability: $P(x) = nCx \cdot p^x (1-p)^{n-x}$

• Mean: $\mu_X = np$

• Standard Deviation: $\sigma_X = \sqrt{np(1-p)}$

Example: Flipping a coin 5 times, probability of 3 heads: $P(3) = 5C3 \cdot (0.5)^3 (0.5)^2$

Continuous Random Variables

Continuous Distributions and the Normal Curve

Continuous random variables can take any value in an interval. Their probabilities are described by a probability density function (pdf), and the area under the curve represents probability.

• Probability Density Function (pdf): A function $f(x)$ such that $P(a \leq X \leq b) = \int_a^b f(x) dx$

• Uniform Distribution: All intervals of the same length are equally probable.

• Normal Distribution: Bell-shaped, symmetric about the mean $\mu$, with standard deviation $\sigma$.

• Inflection Point: Points where the curve changes concavity, located at $\mu \pm \sigma$.

Properties of a pdf:

• $f(x) \geq 0$ for all $x$

• $\int_{-\infty}^{\infty} f(x) dx = 1$

Example: For a uniform distribution on [0,1], $f(x) = 1$ for $0 \leq x \leq 1$.

Calculating Probabilities with the Normal Distribution

The standard normal distribution has mean 0 and standard deviation 1. Z-scores standardize normal variables for probability calculations using tables.

• Z-score: $Z = \dfrac{X - \mu}{\sigma}$

• Standard Normal Distribution: $N(0,1)$

• Percentiles: The value below which a given percentage of observations fall.

Formulas:

• $Z = \dfrac{X - \mu}{\sigma}$

• $X = \mu + Z\sigma$

Example: If $X \sim N(100, 15)$, the Z-score for $X = 130$ is $Z = (130-100)/15 = 2$.

Sampling Distributions

Distribution of the Sample Mean

The sampling distribution of the sample mean describes the distribution of means from all possible samples of a given size from a population. The Central Limit Theorem states that, for large samples, this distribution is approximately normal.

• Sample Mean ($\bar{x}$): The average of a sample.

• Population Mean ($\mu$): The average of the population.

• Standard Error ($\sigma_{\bar{x}}$): The standard deviation of the sample mean.

• Central Limit Theorem: For $n \geq 30$, the sampling distribution of $\bar{x}$ is approximately normal, regardless of the population's distribution.

Formulas:

• $\mu_{\bar{x}} = \mu$

• $\sigma_{\bar{x}} = \dfrac{\sigma}{\sqrt{n}}$

• $z_{\bar{x}} = \dfrac{\bar{x} - \mu}{\sigma/\sqrt{n}}$

Example: If $\mu = 50$, $\sigma = 10$, $n = 25$, then $\sigma_{\bar{x}} = 10/5 = 2$.

Distribution of the Sample Proportion

The sample proportion $\hat{p}$ estimates the population proportion $p$. For large samples, its distribution is approximately normal.

• Sample Proportion ($\hat{p}$): $\hat{p} = \dfrac{x}{n}$, where $x$ is the number of successes in $n$ trials.

• Mean: $\mu_{\hat{p}} = p$

• Standard Deviation: $\sigma_{\hat{p}} = \sqrt{\dfrac{p(1-p)}{n}}$

• Z-score: $z_{\hat{p}} = \dfrac{\hat{p} - p}{\sqrt{p(1-p)/n}}$

Example: If $p = 0.6$, $n = 100$, then $\sigma_{\hat{p}} = \sqrt{0.6 \times 0.4 / 100} = 0.049$.

Confidence Intervals

Estimating Population Proportions

A confidence interval provides a range of plausible values for a population parameter, based on sample data and a specified confidence level.

• Point Estimate: The sample statistic used to estimate a population parameter (e.g., $\hat{p}$ for $p$).

• Confidence Interval: $\hat{p} \pm E$, where $E$ is the margin of error.

• Margin of Error: $E = z_{\alpha/2} \sqrt{\dfrac{\hat{p}(1-\hat{p})}{n}}$

• Critical Value ($z_{\alpha/2}$): The Z-score corresponding to the desired confidence level.

• Sample Size Calculation: $n = \dfrac{p(1-p)}{\left(z_{\alpha/2}/E\right)^2}$ (if $p$ is known), or $n = \dfrac{0.25}{\left(z_{\alpha/2}/E\right)^2}$ (if $p$ is unknown).

Formulas:

• Margin of Error: $E = z_{\alpha/2} \sqrt{\dfrac{\hat{p}(1-\hat{p})}{n}}$

• Confidence Interval: $\hat{p} - E$ to $\hat{p} + E$

• Sample Size: $n = p(1-p) \left(\dfrac{z_{\alpha/2}}{E}\right)^2$ or $n = 0.25 \left(\dfrac{z_{\alpha/2}}{E}\right)^2$

Example: For $\hat{p} = 0.5$, $n = 100$, $z_{0.025} = 1.96$, $E = 1.96 \times \sqrt{0.5 \times 0.5 / 100} = 0.098$; CI is (0.402, 0.598).

Estimating Population Means

When estimating a population mean, the Student's t-distribution is used if the population standard deviation is unknown and the sample size is small.

• Point Estimate: The sample mean $\bar{x}$.

• t-Distribution: Used instead of the normal distribution when $\sigma$ is unknown.

• Degrees of Freedom: $n-1$

• Margin of Error: $E = t_{\alpha/2} \dfrac{s}{\sqrt{n}}$

• Confidence Interval: $\bar{x} \pm E$

Formulas:

• Margin of Error: $E = t_{\alpha/2} \dfrac{s}{\sqrt{n}}$

• Confidence Interval: $\bar{x} - E$ to $\bar{x} + E$

Example: For $\bar{x} = 20$, $s = 4$, $n = 16$, $t_{0.025, 15} = 2.131$, $E = 2.131 \times 4/4 = 2.131$; CI is (17.869, 22.131).

Additional info: The t-distribution is more robust for small samples, especially when the population is normal or nearly normal. For large samples ($n \geq 30$), the t-distribution approximates the normal distribution.