Comprehensive Review of Statistics and Regression Methods

Review on Statistics

Summation Operator and Properties

The summation operator is a fundamental notation in statistics, used to denote the sum of a sequence of numbers. Several properties simplify calculations involving sums.

• Summation Operator: $\sum_{i=1}^n x_i = x_1 + x_2 + \cdots + x_n$

• Property 1 (Constant): $\sum_{i=1}^n c = nc$

• Property 2 (Constant Multiple): $\sum_{i=1}^n c x_i = c \sum_{i=1}^n x_i$

• Property 3 (Linearity): $\sum_{i=1}^n (a x_i + b y_i) = a \sum_{i=1}^n x_i + b \sum_{i=1}^n y_i$

Sample Statistics

Sample statistics are used to summarize and describe data from a sample.

• Sample Average (Mean): $\bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$

• Sum of Deviations: $\sum_{i=1}^n (x_i - \bar{x}) = 0$

• Sum of Squared Deviations: $\sum_{i=1}^n (x_i - \bar{x})^2 = \sum_{i=1}^n x_i^2 - n \bar{x}^2$

• Sum of Multiplied Deviations: $\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) = \sum_{i=1}^n x_i y_i - n \bar{x}\bar{y}$

Sample Variance and Covariance

• Sample Variance: $s_{xx} = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2$

• Sample Covariance: $s_{xy} = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})$

Statistics and Econometrics

• Statistics: The study of methods to draw useful information from data, including descriptive statistics (summarizing data) and inferential statistics (estimators, tests, confidence intervals).

• Econometrics: Application of statistical methods to economic data to extract meaningful information. Microeconometrics focuses on individual or firm-level data, while macroeconometrics deals with aggregate data.

Random Experiments and Random Variables

• Random Experiment: An experiment whose outcome cannot be predicted with certainty, but all possible outcomes can be described. The experiment can be repeated under the same conditions (e.g., tossing a coin).

• Random Variable (RV): A variable whose value is determined by a random experiment. It is a function mapping outcomes to real numbers.

• Discrete RV: Takes at most countably infinite values.

• Continuous RV: Takes values in an uncountably infinite set.

Probability Distributions

• Probability Distribution: Assigns probabilities to each possible value of a random variable.

• For discrete RVs: List of probabilities for each value.

• For continuous RVs: Probability density function (PDF); probability at a single point is zero.

	Discrete RV	Continuous RV
Range	Countable	Uncountable
Description	pmf $p(x)$	pdf $f(x)$
Probability at a point	Has mass	No mass
Expectation	$\sum_x x p(x)$	$\int x f(x) dx$

Population Mean and Variance

• Population Mean: $E(X) = \sum_x x P(x)$

• Laws of Expectation:

  • $E(c) = c$

  • $E(X + c) = E(X) + c$

  • $E(cX) = cE(X)$

• Population Variance: $V(X) = E[(X - \mu)^2]$

• Shortcut: $V(X) = E(X^2) - \mu^2$

• Laws of Variance:

  • $V(c) = 0$

  • $V(X + c) = V(X)$

  • $V(cX) = c^2 V(X)$

Covariance and Correlation

• Population Covariance: $cov(X, Y) = E[(X - \mu_X)(Y - \mu_Y)]$

• Shortcut: $cov(X, Y) = E(XY) - \mu_X \mu_Y$

• Laws of Covariance:

  • $cov(X, Y) = cov(Y, X)$

  • $cov(a_1 X + b_1, a_2 Y + b_2) = a_1 a_2 cov(X, Y)$

  • If $X$ and $Y$ are independent, $cov(X, Y) = 0$ (converse not always true)

  • If $E(X) = 0$ or $E(Y) = 0$, then $cov(X, Y) = E(XY)$

• Population Coefficient of Correlation:

  • $\rho = \frac{\sigma_{xy}}{\sigma_x \sigma_y} = \frac{cov(X, Y)}{\sqrt{V(X)}\sqrt{V(Y)}}$

Laws Regarding the Sum of Two Random Variables

• $E(X + Y) = E(X) + E(Y)$

• $V(X + Y) = V(X) + V(Y) + 2cov(X, Y)$

Conditional Expectation

The conditional expectation of $Y$ given $X = x$ is the expected value of $Y$ in the subpopulation where $X = x$.

• Notation: $E(Y|X = x) = \sum_y y P(y|x)$

• Conditional Probability: $P(y|x) = \frac{P(Y = y \& X = x)}{P(X = x)}$

Laws of Conditional Expectation

• For any function $c(X)$: $E(c(X)|X) = c(X)$

• $E(a(X)Y + b(X)|X) = a(X)E(Y|X) + b(X)$

• If $X$ and $Y$ are independent: $E(Y|X) = E(Y)$

• Law of Iterated Expectation: $E(Y) = E(E(Y|X))$

• More generally: $E(Y|X) = E(E(Y|X, Z)|X)$

• If $E(Y|X) = E(Y)$, then $cov(X, Y) = 0$; any function of $X$ is uncorrelated with $Y$.

Review on Linear Regression

Introduction to Regression

Regression analysis studies the conditional mean function of a response variable given explanatory variables. It is widely used in economics to investigate causal relationships and to focus on the mean response of the dependent variable.

• Regression: Analysis of $E(Y|X)$, the expected value of $Y$ given $X$.

• Application: Investigate how changes in $X$ affect $Y$, holding other variables constant.

Linear Multiple Regression Model

• Model: $Y_i = \beta_0 + \beta_1 X_{1i} + \beta_2 X_{2i} + \cdots + \beta_k X_{ki} + u_i$ for $i = 1, \ldots, n$

• Assumptions:

  • Random sampling: $(Y_i, X_{1i}, \ldots, X_{ki})$ are i.i.d.

  • Conditional mean: $E(u_i|X_{1i}, \ldots, X_{ki}) = 0$

  • Nonzero finite fourth moments (no large outliers)

  • No perfect multicollinearity (no exact linear relationship among regressors)

• Partial Effect: $\beta_k$ measures the effect of $X_k$ on $E(Y)$, holding other variables constant.

• Homoskedasticity: If $V(u_i|X_{1i}, \ldots, X_{ki})$ does not depend on $X$, errors are homoskedastic; otherwise, heteroskedastic.

Ordinary Least Squares (OLS) Estimator

• OLS Estimator: $\left(\hat{\beta}_0, \ldots, \hat{\beta}_k\right) = \arg\min_{b_0, \ldots, b_k} \sum_{i=1}^n (Y_i - b_0 - b_1 X_{1i} - \cdots - b_k X_{ki})^2$

• The OLS estimator finds the linear combination of regressors that minimizes the sum of squared residuals.

Measures of Fit

• Standard Error of the Regression (SER): Measures the spread of $Y$ around the regression line.

  • $SER = \sqrt{\hat{s}_u^2}$

  • $\hat{s}_u^2 = \frac{\sum_{i=1}^n \hat{u}_i^2}{n - k - 1}$

  • $\hat{u}_i = Y_i - \hat{Y}_i$

  • $\hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_{1i} + \cdots + \hat{\beta}_k X_{ki}$

• R-squared ($R^2$): Fraction of variation in $Y$ explained by regressors.

  • $R^2 = \frac{ESS}{TSS} = 1 - \frac{SSR}{TSS}$

  • $ESS = \sum_{i=1}^n (\hat{Y}_i - \bar{Y})^2$

  • $TSS = \sum_{i=1}^n (Y_i - \bar{Y})^2$

• Adjusted $R^2$: Adjusts $R^2$ for the number of regressors.

  • $\bar{R}^2 = 1 - \frac{n-1}{n-k-1} \frac{SSR}{TSS}$

  • Can be negative; penalizes adding unnecessary regressors.

Large Sample Distribution of OLS Estimator

• Under standard assumptions, as $n$ increases, the OLS estimators $\hat{\beta}_0, \ldots, \hat{\beta}_k$ are approximately jointly normally distributed.

• Each $\hat{\beta}_k$ is approximately $N(\beta_k, \sigma^2_{\hat{\beta}_k})$.

Hypothesis Testing in Regression

• Testing a Single Coefficient: $H_0: \beta_j = \beta_{j,0}$ vs. $\beta_j \neq \beta_{j,0}$

• t-statistic: $t = \frac{\hat{\beta}_j - \beta_{j,0}}{SE(\hat{\beta}_j)}$

• Standard Error: $SE(\hat{\beta}_j) = \sigma_{\hat{\beta}_j}$

• p-value: $p = 2\Phi(-|t|)$, where $\Phi$ is the normal CDF

• Reject $H_0$ if $p < 0.05$ or $|t| > 1.96$ (for 5% significance level)

Confidence Intervals and Joint Hypotheses

• 95% Confidence Interval for $\beta_j$: $[\hat{\beta}_j - 1.96 SE(\hat{\beta}_j), \hat{\beta}_j + 1.96 SE(\hat{\beta}_j)]$

• Joint Hypothesis: Use robust F-tests for multiple coefficients.

Consistency and Asymptotic Normality

• Consistency: An estimator is consistent if it converges in probability to the true parameter value as the sample size increases.

• Asymptotic Normality: As sample size grows, the distribution of the estimator approaches a normal distribution, allowing for inference using normal-based confidence intervals and hypothesis tests.