Experiments and Observational Studies: Principles and Practice

Experiments and Observational Studies

Introduction

This chapter explores the fundamental differences between observational studies and experiments, focusing on how each approach is used in statistics to draw conclusions about relationships and causality. Key principles of experimental design, including control, randomization, replication, and blocking, are discussed in detail.

Observational Studies

Definition and Types

• Observational Study: Researchers observe subjects without assigning choices or treatments. They simply record naturally occurring behaviors or outcomes.

• Retrospective Study: Subjects are identified, and data about their past behaviors or exposures are collected.

• Prospective Study: Subjects are identified in advance and followed over time as events unfold.

Key Point: Observational studies are valuable for discovering trends and possible relationships but cannot establish causality.

Experiments

Definition and Structure

• Experiment: A study design where researchers manipulate factor levels to create treatments, randomly assign subjects to these treatments, and compare responses across groups.

• Factor: An explanatory variable manipulated by the experimenter.

• Response Variable: The outcome measured in the experiment.

• Experimental Units: The individuals or objects on which the experiment is performed. When humans are involved, they are called subjects or participants.

• Levels: The specific values chosen for a factor.

• Treatment: A combination of factor levels assigned to an experimental unit.

The Four Principles of Experimental Design

1. Control: Make conditions as similar as possible for all treatment groups to control sources of variation other than the factors being tested.

2. Randomize: Assign subjects to treatments at random to equalize the effects of unknown or uncontrollable sources of variation and reduce bias.

3. Replicate: Apply each treatment to multiple subjects to ensure results are not due to chance. Replication can also involve repeating the experiment in different settings or with different groups.

4. Block: Group similar experimental units together and randomize within these blocks to remove variability due to differences among the blocks. Blocking is analogous to stratification in surveys.

Blinding and Placebos

Blinding

• Blinding: Concealing the treatment assignment from individuals who could influence or evaluate the results to avoid bias.

• Single-Blind: Either the subjects or the evaluators are blinded.

• Double-Blind: Both the subjects and the evaluators are blinded.

Placebos

• Placebo: A fake treatment that mimics the real treatment, used to blind subjects from knowing whether they are receiving the treatment or not.

• Placebo Effect: When subjects experience changes in the response variable simply because they believe they are receiving a treatment.

• Best Practice: The best experiments are randomized, comparative, double-blind, and placebo-controlled.

Blocking and Experimental Design

Blocking in Practice

When groups of experimental units are similar, blocking can help isolate variability due to differences between blocks, making treatment effects clearer. Randomization occurs within each block, leading to a randomized block design.

Example: In a nail polish experiment, fingers are blocked by hand (right or left), then randomly assigned to different treatments (e.g., red or nude polish), and the outcome (polish chipped) is compared within each block.

Matching in Observational Studies

• Subjects may be paired or matched based on similarities to reduce variability, similar to blocking in experiments.

Adding More Factors

Multifactor Experiments

• Including multiple factors in an experiment allows for the examination of interactions between factors and increases efficiency.

• Example: In a nail polish study, factors could include type of finish, number of coats, base coat presence, and brand.

Confounding and Lurking Variables

Definitions

• Confounding: When the levels of one factor are associated with the levels of another, making it impossible to separate their effects on the response variable.

• Lurking Variable: A variable not included in the study that creates an association between two other variables, potentially misleading conclusions about causality.

Random assignment in experiments helps neutralize the effects of lurking variables, but confounding can still occur if variables are associated with both the treatment and the response.

Statistical Significance

Assessing Differences

• Statistically Significant: A difference is statistically significant if it is unlikely to have occurred by chance alone, based on the randomization process.

Experiments vs. Sample Surveys

Comparison

• Sample Surveys: Aim to estimate population parameters; samples must be representative of the population.

• Experiments: Aim to assess treatment effects; experimental units may not be randomly drawn from the population.

Control Treatments and Control Groups

Definitions

• Control Treatment: A baseline or "business as usual" measurement for comparison with other treatments.

• Control Group: The group of experimental units receiving the control treatment.

What Can Go Wrong?

Common Issues and Solutions

• Confounding can threaten the validity of conclusions; use randomization and blocking to minimize its effects.

• Record additional information and consider running a pilot experiment before a full-scale study.

• Report any unavoidable confounding in your results.

Summary of Key Concepts

• Observational studies can identify associations but not causality.

• Experiments, through randomization and control, can establish cause-and-effect relationships.

• The four principles of experimental design are control, randomize, replicate, and block.

• Blinding and placebos are essential for reducing bias.

• Confounding and lurking variables must be considered and addressed in study design.