Which of the following is not true about $p$-values in hypothesis testing?

• A. A $p$-value represents the probability of obtaining a test statistic as extreme as, or more extreme than, the observed value assuming the null hypothesis is true.
• B. A small $p$-value indicates strong evidence against the null hypothesis.
• C. If the $p$-value is less than the significance level, we reject the null hypothesis.
• D. A $p$-value tells us the probability that the null hypothesis is true.

Answer: (D)

Verified step by step guidance

1. Step 1: Understand the definition of a p-value. A p-value is the probability of obtaining a test statistic at least as extreme as the one observed, assuming the null hypothesis is true. This means it measures how compatible the data is with the null hypothesis.
2. Step 2: Recognize that a small p-value indicates strong evidence against the null hypothesis because it suggests that the observed data is unlikely under the null hypothesis.
3. Step 3: Recall the decision rule in hypothesis testing: if the p-value is less than the chosen significance level (\$\alpha\$), we reject the null hypothesis, indicating that the data provides sufficient evidence against it.
4. Step 4: Identify the incorrect statement: a p-value does NOT tell us the probability that the null hypothesis is true. Instead, it assumes the null hypothesis is true and evaluates the probability of the observed data under that assumption.
5. Step 5: Summarize that the misconception is interpreting the p-value as the probability of the null hypothesis being true, which is not correct in frequentist hypothesis testing.