Basic Estimation Techniques in Regression Analysis

Basic Estimation Techniques in Regression Analysis

Learning Outcomes

This chapter introduces foundational concepts in regression analysis, focusing on estimation techniques for both simple and multiple regression models. Students will learn to set up regression equations, estimate parameters, assess statistical significance, and interpret model fit using various statistical measures.

• Set up and interpret simple linear regression equations.

• Estimate intercept and slope parameters using the method of least squares.

• Determine statistical significance using t tests and p values.

• Evaluate model fit using R2 and F tests.

• Set up and interpret multiple regression models.

• Estimate parameters for quadratic and log-linear regression models.

Basic Estimation

Parameters and Parameter Estimation

Parameters are the coefficients in an equation that define the exact mathematical relationship among variables. Parameter estimation is the process of finding numerical values for these coefficients based on sample data.

• Parameter: A constant in a model that quantifies the relationship between variables.

• Parameter estimation: The process of using sample data to estimate the values of parameters.

Regression Analysis

Key Concepts

Regression analysis is a statistical technique used to estimate the parameters of an equation and test for statistical significance.

• Dependent variable (Y): The variable whose variation is to be explained.

• Explanatory (independent) variables (X): Variables believed to influence the dependent variable.

Simple Linear Regression

Regression Equation

A simple linear regression relates the dependent variable Y to one independent variable X.

• Equation: $Y = a + bX$

• Intercept parameter (a): Value of Y when X is zero.

• Slope parameter (b): Change in Y for a one-unit change in X. $b = \Delta Y / \Delta X$

Hypothetical Regression Model

The regression line represents the average or expected value of Y for each level of X. The true relationship is unknown and must be estimated from sample data.

• Random error term: Captures effects of unpredictable factors not included as explanatory variables.

Data Types in Regression

Types of Data

• Time series: Data collected over time for a specific firm.

• Cross-sectional: Data collected from multiple firms or industries at a single point in time.

• Scatter diagram: Graphical representation of sample data points.

Fitting a Regression Line

Population vs. Sample Regression Line

• Population regression line: $Y = a + bX$ (true relationship).

• Sample regression line: $\hat{Y} = \hat{a} + \hat{b}X$ (estimated from sample data).

• Predicted value of Y ($\hat{Y}$): Obtained by substituting X into the sample regression equation.

Table: Sales and Advertising Expenditures for Seven Travel Agencies

Firm	Sales	Advertising expenditure
A	$15,000	$2,000
B	$30,000	$2,000
C	$30,000	$5,000
D	$25,000	$3,000
E	$55,000	$9,000
F	$45,000	$8,000
G	$60,000	$7,000

Method of Least Squares

Least Squares Estimation

The method of least squares estimates regression parameters by minimizing the sum of squared distances (residuals) from each data point to the regression line.

• Estimator: Formula used to compute parameter estimates.

• Residual: Difference between actual and predicted values: $e_i = Y_i - \hat{Y}_i$

Regression Output Interpretation

Key Statistics in Regression Output

• R-squared ($R^2$): Proportion of total variation in Y explained by the model. Range: $0 \leq R^2 \leq 1$

• F-ratio: Tests overall significance of the model. Large F suggests a better fit.

• p-value: Probability of observing a test statistic as extreme as the one obtained, assuming the null hypothesis is true. Small p-value (<0.05) indicates statistical significance.

• Standard error: Measures the average distance of estimated coefficients from the true population value. Smaller values indicate more precise estimates.

• t-ratio (t-statistic): Used to test if a coefficient is significantly different from zero. $t = \frac{\hat{b}}{S_b}$

• Multiple R: Correlation coefficient between observed and predicted values. Range: 0 to 1.

• Adjusted R2: Adjusts R2 for the number of explanatory variables, penalizing for unnecessary variables.

Table: Example Regression Output (Generic Style)

Variable	Parameter Estimate	Standard Error	T-Ratio	P-Value
Intercept	11573.0	7150.83	1.62	0.1665
A	4.97191	1.23154	4.04	0.0100

Statistical Significance and Hypothesis Testing

Level of Confidence and Significance

• Level of significance: Probability of finding a parameter estimate statistically different from zero when it is actually zero.

• Type I error: Incorrectly finding statistical significance.

• Level of confidence: Probability of correctly failing to reject the true hypothesis. $\text{Level of confidence} = 1 - \text{level of significance}$

t Test

• Purpose: Test the hypothesis that a parameter equals zero ($b = 0$).

• Formula: $t = \frac{\hat{b}}{S_b}$

• Degrees of freedom: $n - k$ (number of observations minus number of parameters estimated).

• Critical value: Value that t statistic must exceed to reject the null hypothesis.

Using p Values

• Interpretation: Treat parameter estimates as statistically significant only if p values are less than the chosen significance level.

Coefficient of Determination ($R^2$)

$R^2$ measures the fraction of total variation in Y explained by the regression equation. High $R^2$ indicates strong correlation but does not prove causality.

Multiple Regression

Model Structure

• Uses more than one explanatory variable to explain variation in the dependent variable.

• Each coefficient measures the change in Y for a one-unit change in its variable, holding others constant.

Nonlinear Regression Models

Quadratic Regression

• Model: $Y = a + bX + cX^2$

• Used when scatter plot is U-shaped or n-shaped.

• Linear transformation: $Z = X^2$

Log-Linear Regression

• Model: $Y = aX^bZ^c$

• Transform by taking natural logarithms: $\ln Y = (\ln a) + b(\ln X) + c(\ln Z)$

• b and c are elasticities (percentage change in Y per percentage change in X or Z).

Summary

• Simple linear regression relates Y to X, providing the expected value of Y for a given X.

• Parameter estimates are chosen to best fit the sample data.

• Statistical significance is assessed using t tests and p values.

• High $R^2$ indicates strong correlation and good model fit.

• Multiple regression and nonlinear models (quadratic, log-linear) extend these concepts to more complex relationships.

Table: Impact of Random Effects on January Sales

Firm	Advertising expenditure	Actual sales	Expected sales	Random effect
Tampa Travel Agency	$3,000	$30,000	$25,000	$5,000
Buccaneer Travel Service	$3,000	$21,000	$25,000	−$4,000
Happy Getaway Tours	$3,000	$25,000	$25,000	0

Additional info: These notes cover core topics in regression analysis, including estimation, hypothesis testing, and model evaluation, which are essential for statistics students.