Experiments and Observational Studies: Principles and Practice

Experiments and Observational Studies

Introduction

This chapter explores the distinction between observational studies and experiments, the principles of experimental design, and the importance of randomization, control, replication, and blocking in drawing valid statistical conclusions.

Observational Studies

Definition and Types

• Observational Study: Researchers observe subjects without assigning choices or treatments. They simply record naturally occurring behaviors or outcomes.

• Retrospective Study: Subjects are identified, and data about their past behaviors or exposures are collected.

• Prospective Study: Subjects are identified in advance and followed into the future as events unfold.

Key Point: Observational studies are useful for discovering trends and possible relationships but cannot establish causality.

• Example: Studying the relationship between music education and grades by observing students who already participate in music programs.

Experiments

Definition and Structure

• Experiment: A study design where researchers manipulate factor levels to create treatments, randomly assign subjects to these treatments, and compare responses.

• Factor: An explanatory variable manipulated by the experimenter.

• Response Variable: The outcome measured for each experimental unit.

• Experimental Units: The individuals or objects on which the experiment is performed (called subjects or participants if human).

• Levels: The specific values chosen for a factor.

• Treatment: A combination of factor levels assigned to an experimental unit.

The Four Principles of Experimental Design

1. Control

• Control sources of variation other than the factors being tested by making conditions as similar as possible for all treatment groups.

2. Randomize

• Randomly assign subjects to treatments to equalize the effects of unknown or uncontrollable sources of variation.

• Randomization does not eliminate variation but spreads it across treatment groups, reducing bias.

3. Replicate

• Apply each treatment to multiple subjects to ensure results are not due to chance or anecdote.

• Replication can also mean repeating the experiment in different settings or with different groups.

4. Blocking

• Group similar experimental units together into blocks and randomize within each block to reduce variability due to differences among blocks.

• Blocking is analogous to stratification in surveys.

Diagram of a Blocked Experiment

The following diagram illustrates a blocked experimental design, where subjects are grouped by hand (right or left), then randomly assigned to treatments (red or nude polish), and the outcome (polish chipped) is compared:

Statistical Significance

Definition

• Statistically Significant: A difference in treatments is statistically significant if it is unlikely to have occurred by random chance alone.

Experiments vs. Sample Surveys

• Sample Surveys: Aim to estimate population parameters; samples must be representative of the population.

• Experiments: Aim to assess treatment effects; experimental units may not be randomly drawn from the population.

Control Treatments and Blinding

Control Treatments

• Control Treatment: A baseline or status quo treatment for comparison, applied to the control group.

Blinding

• Blinding prevents bias by keeping subjects and/or evaluators unaware of which treatment was assigned.

• Single-Blind: Either subjects or evaluators are blinded.

• Double-Blind: Both subjects and evaluators are blinded.

Placebos

• Placebo: A fake treatment that mimics the real treatment, used to blind subjects.

• Placebo Effect: When subjects respond to the placebo as if it were the actual treatment.

• Placebo controls are essential for effective blinding and valid comparisons.

Blocking and Matched Pairs

• Blocking groups similar experimental units to reduce variability.

• Randomization occurs within blocks (randomized block design).

• In observational studies, matching pairs subjects with similar characteristics to reduce variability, similar to blocking in experiments.

Adding More Factors

• Including multiple factors in an experiment allows for the study of interactions and increases efficiency.

• Example: In a nail polish study, factors could include color, finish, number of coats, and brand.

Confounding and Lurking Variables

Confounding

• Occurs when the levels of one factor are associated with another, making it impossible to separate their effects.

Lurking Variables

• A lurking variable is an unmeasured variable that influences both the explanatory and response variables, creating a spurious association.

• Randomization helps neutralize the effects of lurking variables in experiments.

What Can Go Wrong?

• Confounding can threaten the validity of conclusions.

• Randomization and careful design help mitigate confounding.

• Pilot studies and recording additional information can improve experimental design.

Summary of Key Concepts

• Observational studies can identify associations but not causality.

• Experiments use randomization, control, replication, and blocking to establish cause-and-effect relationships.

• Blinding and placebos reduce bias.

• Confounding and lurking variables must be considered and controlled for valid inference.