Hypothesis Testing: Proportions, Type I & II Errors, and Power

Testing Proportions & Type Errors

The Monty Hall Problem: Probability and Simulation

The Monty Hall problem is a classic probability puzzle that illustrates the importance of strategy in decision-making under uncertainty. In this scenario, a contestant chooses one of three doors, behind one of which is a prize. After the initial choice, one non-winning door is revealed, and the contestant must decide whether to stick with their original choice or switch to the remaining unopened door.

• Probability of Winning by Staying: The probability of winning if the contestant stays with their first choice is $P(\text{Winning} \mid \text{Stay}) = \frac{1}{3} \approx 0.33 = 33\%$.

• Probability of Winning by Switching: The probability of winning if the contestant switches is $P(\text{Winning} \mid \text{Switch}) = \frac{2}{3} \approx 0.67 = 67\%$.

• Simulation: To empirically estimate these probabilities, students can simulate the game multiple times and record the outcomes.

Example: If a contestant initially picks Door 1 and the host reveals Door 2 as a non-winner, switching to Door 3 increases the chance of winning to 2/3.

Hypothesis Testing for Proportions

Hypothesis testing allows us to evaluate claims about population proportions using sample data. In the Monty Hall context, we can test the claim that the probability of winning by switching is 50%.

1. Establish Hypotheses:

   • Null Hypothesis ($H_0$): $p = 0.5$ (probability of winning by switching is 50%)

   • Alternative Hypothesis ($H_1$): $p \neq 0.5$

2. Significance Level ($\alpha$): The threshold for "reasonable doubt" is typically set at $\alpha = 0.05$ (5%).

3. Test Statistic: For proportions, the test statistic is calculated as: $z = \frac{\hat{p} - p}{\sqrt{\frac{p(1-p)}{n}}}$ where $\hat{p}$ is the sample proportion, $p$ is the hypothesized proportion, and $n$ is the sample size.

4. P-Value: The p-value is the probability of observing a test statistic as extreme as, or more extreme than, the observed value under the null hypothesis.

5. Decision: If the p-value is less than $\alpha$, reject $H_0$; otherwise, fail to reject $H_0$.

6. Conclusion: State the result in terms of the original claim.

Example: If the observed proportion of wins by switching is significantly greater than 0.5, we reject the null hypothesis and conclude there is evidence that switching increases the probability of winning.

Type I and Type II Errors

Definitions and Consequences

When conducting hypothesis tests, two types of errors can occur:

• Type I Error ($\alpha$): Rejecting the null hypothesis when it is actually true. This is also known as a "false positive." The probability of a Type I error is denoted by $\alpha$.

• Type II Error ($\beta$): Failing to reject the null hypothesis when it is actually false. This is also known as a "false negative." The probability of a Type II error is denoted by $\beta$.

Example (Courtroom Analogy):

• Null Hypothesis ($H_0$): The defendant is innocent.

• Alternative Hypothesis ($H_1$): The defendant is guilty.

• Type I Error: Convicting an innocent person ($\alpha$).

• Type II Error: Failing to convict a guilty person ($\beta$).

Adjusting the significance level ($\alpha$) affects the probabilities of these errors:

• Raising the level of "reasonable doubt" (decreasing $\alpha$) reduces Type I errors but increases Type II errors.

• Lowering the level of "reasonable doubt" (increasing $\alpha$) increases Type I errors but reduces Type II errors.

• To decrease both error probabilities, a larger sample size (more data) is required.

Which Error is Worse? In many contexts, a Type I error is considered more serious, so $\alpha$ is set to be as small as possible.

Preliminary Conclusion	Null hypothesis is true	Null hypothesis is false
Reject $H_0$	Type I error: Reject a true $H_0$ $P(\text{Type I error}) = \alpha$	Correct decision
Fail to reject $H_0$	Correct decision	Type II error: Fail to reject a false $H_0$ $P(\text{Type II error}) = \beta$

Power of a Hypothesis Test

Definition and Importance

The power of a hypothesis test is the probability of correctly rejecting a false null hypothesis. It is calculated as $1 - \beta$, where $\beta$ is the probability of a Type II error. High power means the test is effective at detecting an effect when one exists.

• Formula: $\text{Power} = 1 - \beta$

• Factors Affecting Power: Sample size, significance level ($\alpha$), and the true value of the population parameter.

Example: If a test has a power of 0.90, there is a 90% chance of detecting a true effect.

Visualizing the Normal Distribution

Many hypothesis tests rely on the normal distribution, especially when sample sizes are large. The area under the normal curve represents probabilities, and critical values are determined by the chosen significance level.

Summary Table: Type I and Type II Errors

The following table summarizes the possible outcomes of hypothesis testing:

Additional info: Analyzing the power of a hypothesis test is crucial for researchers to ensure their tests are sensitive enough to detect meaningful effects, but detailed calculations are beyond the scope of this introductory course.