Evaluating Statistical Studies: Guidelines for Critical Analysis

Section 1.4: Should You Believe a Statistical Study?

Objectives

• Critically evaluate a statistical study

• Decide whether the results are meaningful

Statistical studies must be interpreted with caution. Understanding the eight guidelines for evaluating studies helps ensure that conclusions are valid and reliable.

The Eight Guidelines for Evaluating Statistical Studies

Guideline 1: Get a Big Picture View of the Study

Before analyzing details, understand the overall purpose and design of the study. Ask:

• What is the goal of the study?

• Who is the population under study?

• Is the study observational or experimental?

• Are the results generalizable?

Example: A study on the effectiveness of microscopes should clarify its goal, population, and whether it is observational or experimental. If the study only includes men, results should only apply to men.

Additional info: Validity depends on appropriate population and study design. Peer review is important for credibility.

Guideline 2: Consider the Source

Evaluate who conducted the study and their potential biases. Reliable studies are often peer-reviewed and conducted by reputable organizations.

• Is the source credible?

• Is there potential for bias?

Example: Studies funded by organizations with vested interests (e.g., tobacco industry) may be biased.

Additional info: Peer review helps ensure objectivity.

Guideline 3: Look for Problems with the Sample

The sample should be representative of the population. Watch for selection bias and self-selection.

• Was the sample randomly selected?

• Is the sample size adequate?

• Are there issues with participation?

Example: If only 13% of people participate, results may be biased. Large planets are easier to detect, so they might be discovered first in astronomical studies.

Additional info: Simple random samples (SRS) are more representative.

Guideline 4: Look for Problems in Defining or Measuring the Variable of Interest

Variables must be clearly defined and measurable. Ambiguity leads to unreliable results.

• Are variables well-defined?

• Are measurements accurate?

Example: If the number of illegal drugs entering a country is unknown, the variable is not measurable.

Additional info: Poorly defined variables undermine study validity.

Guideline 5: Beware of Confounding Variables

Confounding variables are factors not accounted for that may influence results.

• Are all relevant variables considered?

• Could other factors explain the results?

Example: In studies of radon and cancer, smoking and air pollution are confounding variables.

Additional info: Confounding variables must be controlled for accurate conclusions.

Guideline 6: Consider the Setting and Wording in Surveys

The environment and phrasing of survey questions can influence responses.

• Is the survey setting neutral?

• Are questions unbiased?

Example: Surveys about sensitive topics may yield different results depending on privacy and wording.

Additional info: Biased questions and settings can skew survey results.

Guideline 7: Check That Results Are Presented Fairly

Results should be presented accurately, without misleading graphs or statistics.

• Are results interpreted correctly?

• Is data presentation honest?

Example: Below national average does not necessarily mean below grade level.

Additional info: Misleading presentations can distort findings.

Guideline 8: Stand Back and Consider the Conclusions

Evaluate whether the study's conclusions are meaningful and practically significant.

• Do the results have practical significance?

• Are the conclusions justified?

Example: A weight loss of one-half pound may not be practically significant for the average person.

Additional info: Statistical significance does not always imply practical importance.

Summary Table: Guidelines for Evaluating Statistical Studies

Guideline	Key Question	Example
Big Picture View	What is the study's goal and population?	Study only includes men; results apply to men
Source	Is the source credible?	Tobacco industry-funded study
Sample	Is the sample representative?	Low participation rate
Variable Definition	Are variables well-defined?	Unknown number of illegal drugs
Confounding Variables	Are other factors considered?	Smoking, air pollution
Survey Setting & Wording	Is the survey unbiased?	Biased question wording
Fair Presentation	Are results presented honestly?	Misleading graphs
Conclusions	Are results practically significant?	Minimal weight loss