Multiple Comparisons: Bonferoni Test: Videos & Practice Problems

Post hoc tests, like the Bonferroni method, identify which means differ after rejecting the null hypothesis in an ANOVA. This involves pairwise comparisons using a t-score calculated as $t=\frac{ȳ}{}$₁ - ȳ₂MSE ( 1n₁ + 1n₂ ), where MSE is the mean square error from ANOVA. Adjusted p-values account for multiple comparisons by multiplying by the number of pairs, controlling Type I error. This process clarifies which group means significantly differ, enhancing interpretation of Analysis of Variance results in experimental design and hypothesis testing.

The Bonferroni Test

When conducting a one-way ANOVA test, rejecting the null hypothesis indicates that at least one group mean differs from the others. However, this result does not specify which means are different. To identify the specific pairs of means that differ, a follow-up procedure called a post hoc test is used. One common post hoc test is the Bonferroni test, which compares pairs of means individually while controlling for the increased risk of Type I error due to multiple comparisons.

The Bonferroni test involves breaking down the groups into all possible pairs and performing a series of two-sample t-tests. For example, if there are three groups (such as grades 10, 11, and 12), the pairs tested would be (10 vs. 11), (11 vs. 12), and (10 vs. 12). Each pair is tested with the null hypothesis that the two means are equal, and the alternative hypothesis that they are not equal, making it a two-tailed test.

Key values needed for the Bonferroni test come from the ANOVA output, including the Mean Square Error (MSE), which represents the variance within groups. This MSE is used as the estimate of variance in the t-test calculations. Additionally, the total sample size (N), the number of groups (k), and the degrees of freedom for error (df = N - k) are essential for determining the test statistics and p-values.

The test statistic for each pair is calculated using the formula:

\[t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{MSE \left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}\]>

where \(\bar{x}_1\) and \(\bar{x}_2\) are the sample means of the two groups, \(n_1\) and \(n_2\) are their respective sample sizes, and MSE is the mean square error from the ANOVA.

After calculating the t-value, the corresponding p-value is found using the t-distribution with the appropriate degrees of freedom. Since the test is two-tailed, the p-value is doubled. However, because multiple pairwise comparisons increase the chance of falsely detecting a difference (Type I error), the Bonferroni correction adjusts the p-values by multiplying them by the number of comparisons (pairs). Alternatively, the significance level \(\alpha\) can be divided by the number of pairs, but both methods yield the same decision criterion.

For example, with three groups, there are three pairs, so each p-value is multiplied by 3. If the adjusted p-value is less than the original significance level (commonly 0.05), the null hypothesis for that pair is rejected, indicating a significant difference between those two means.

In practice, this method can be tedious due to multiple calculations, but it provides a rigorous way to pinpoint which specific group means differ after an overall ANOVA indicates a difference exists. The Bonferroni test is especially useful when sample sizes are equal, simplifying the calculations, but it remains applicable with unequal sample sizes as well.

A researcher is comparing mean cholesterol levels across 4 diet plans (A, B, C, D) in a One-Way ANOVA test. If $H_0$ was rejected and the researcher were to use a Bonferroni Test, how many pairs of comparisons would they do?

• A. $12$
• B. $6$
• C. $4$
• D. $16$

Answer: (B)

For which of the following scenarios would it be most appropriate to run a Bonferroni Test to see which mean(s) are significantly different from the rest?

• A.
• B.
• C.

Answer: (C)

The Bonferroni Test Example 1

A cereal brand conducted a taste test to compare three different types of cereal based on customer ratings from 1 to 10. An initial analysis using ANOVA (Analysis of Variance) rejected the null hypothesis, indicating that not all mean taste ratings are equal. This rejection allows for further pairwise comparisons using the Bonferroni test to identify which specific cereals differ in taste appeal.

In this scenario, the focus is on comparing cereal A and cereal C. The total sample size (N) is 18, with three groups (k = 3), each having six participants. The degrees of freedom for error is calculated as \(df = N - k = 18 - 3 = 15\). The Mean Square Error (MSE), a key value from the ANOVA output representing the variance within groups, is 0.944.

The hypotheses for the pairwise comparison are set as follows: the null hypothesis (\(H_0\)) assumes the means of cereal A and cereal C are equal (\(\mu_A = \mu_C\)), while the alternative hypothesis (\(H_a\)) states they are not equal (\(\mu_A \neq \mu_C\)), making this a two-tailed test.

The test statistic for this comparison is calculated using the formula:

\[t = \frac{\bar{x}_A - \bar{x}_C}{\sqrt{MSE \left(\frac{1}{n_A} + \frac{1}{n_C}\right)}}\]

where \(\bar{x}_A = 7.67\) and \(\bar{x}_C = 8.0\) are the sample means, and \(n_A = n_C = 6\) are the sample sizes. Plugging in the values:

\[t = \frac{7.67 - 8.0}{\sqrt{0.944 \left(\frac{1}{6} + \frac{1}{6}\right)}} = \frac{-0.33}{\sqrt{0.944 \times \frac{2}{6}}} = -0.588\]

To find the p-value, the two-tailed probability associated with this t-score and 15 degrees of freedom is calculated. Using statistical software or a graphing calculator, the p-value before adjustment is approximately 0.565.

Since multiple pairwise comparisons are being made, the Bonferroni correction adjusts the p-value to control the family-wise error rate. With three comparisons, the adjusted p-value is:

\[p_{\text{adjusted}} = p \times 3 = 0.565 \times 3 = 1.695\]

Because p-values cannot exceed 1, this is capped at 1. Since the adjusted p-value is much greater than the significance level \(\alpha = 0.05\), we fail to reject the null hypothesis for the comparison between cereal A and cereal C. This means there is no statistically significant difference in taste ratings between these two cereals.

Additional comparisons show very low p-values for cereal A versus B and cereal B versus C, indicating significant differences in those pairs. This suggests that cereal B's mean taste rating is significantly different from both A and C, while A and C are similar. Such pairwise comparisons following ANOVA help identify which specific groups differ, providing clearer insights into consumer preferences.

Here’s what students ask on this topic:

What is the purpose of the Bonferroni test in multiple comparisons after ANOVA?

The Bonferroni test is a post hoc method used after rejecting the null hypothesis in an ANOVA test. Its main purpose is to identify which specific group means differ from each other. Since ANOVA only tells us that at least one mean is different, the Bonferroni test breaks down the problem into pairwise comparisons between group means. It adjusts for the increased risk of Type I error (false positives) that occurs when multiple comparisons are made by multiplying the p-values by the number of pairs tested. This correction ensures that the overall significance level is maintained, helping us confidently determine which pairs of means are significantly different.

How do you calculate the test statistic for the Bonferroni test?

In the Bonferroni test, the test statistic is a t-score calculated for each pair of means. The formula is:

$t=\frac{\mathrm{mean}}{}$₁−mean₂MSE⋅(1n₁+1n₂)

Here, $\mathrm{MSE}$ is the mean square error from the ANOVA, and $n$₁ and $n$₂ are the sample sizes of the two groups being compared. This t-score measures how far apart the two means are relative to the variability within groups.

Why do we adjust p-values in the Bonferroni test, and how is this adjustment done?

When performing multiple pairwise comparisons, the chance of incorrectly rejecting a true null hypothesis (Type I error) increases. The Bonferroni test adjusts for this by controlling the overall error rate. This adjustment is done by multiplying each individual p-value by the number of comparisons (pairs) made. Alternatively, the significance level (alpha) can be divided by the number of pairs. Both methods ensure that the probability of making one or more Type I errors across all comparisons remains at the desired alpha level, typically 0.05. This correction makes the test more conservative, reducing false positives.

How do you determine the number of pairs for the Bonferroni correction in multiple comparisons?

The number of pairs in the Bonferroni correction corresponds to the number of unique pairwise comparisons between groups. If you have $k$ groups, the number of pairs is calculated using combinations:

$(kchoose2)=\frac{k}{\cdot }\frac{1}{2}$

For example, with 3 groups, the number of pairs is 3, calculated as (3 × 2) / 2 = 3. This number is used to adjust p-values or alpha in the Bonferroni test.

What are the steps to perform a Bonferroni test after rejecting the null hypothesis in ANOVA?

To perform a Bonferroni test after rejecting the null hypothesis in ANOVA, follow these steps:

1. Confirm that the ANOVA null hypothesis was rejected, indicating at least one mean differs.
2. Obtain key values from the ANOVA output: mean square error (MSE), total sample size (N), number of groups (k), and degrees of freedom.
3. List all unique pairs of group means to compare.
4. For each pair, set up hypotheses: $H0:\mu$₁=μ₂ and $H$_a:μ₁≠μ₂.
5. Calculate the t-statistic for each pair using the formula involving MSE and sample sizes.
6. Find the p-value for each t-statistic using the degrees of freedom.
7. Adjust each p-value by multiplying by the number of pairs (Bonferroni correction).
8. Compare adjusted p-values to alpha (e.g., 0.05) to decide whether to reject each null hypothesis.

This process identifies which specific group means differ significantly.

How does the Bonferroni test control the Type I error rate in multiple comparisons?

The Bonferroni test controls the Type I error rate by adjusting for the increased chance of false positives when making multiple comparisons. When testing multiple pairs, the probability of incorrectly rejecting at least one true null hypothesis increases. The Bonferroni correction addresses this by dividing the overall significance level (alpha) by the number of comparisons or equivalently multiplying each p-value by the number of comparisons. This adjustment ensures that the family-wise error rate—the probability of making one or more Type I errors across all tests—does not exceed the chosen alpha level, typically 0.05. As a result, the test maintains the integrity of statistical conclusions despite multiple testing.