Designing Experiments and Observational Studies

Observational Studies vs. Experiments

In statistics, it is crucial to distinguish between observational studies and experiments. Both are methods for collecting data, but they differ in how the data is obtained and the conclusions that can be drawn.

• Observational Study: Researchers observe individuals and measure variables of interest without assigning treatments or influencing responses. There is no randomization of treatments.

• Experiment: Researchers deliberately impose treatments (specific experimental conditions) on individuals, often using random assignment, to observe their responses.

Types of Observational Studies:

• Retrospective Study: Uses past data collected from subjects.

• Prospective Study: Follows subjects into the future, collecting data as events unfold.

Key Terms:

• Experimental Units: The individuals on which the experiment is performed. If humans, they are called subjects.

• Explanatory Variable (Factor): The variable whose levels are controlled by the experimenter. There may be multiple factors, each with one or more levels.

• Treatment: A specific combination of factor levels applied to experimental units.

• Placebo: A 'fake' treatment used as a control to measure the effect of the actual treatment.

• Response Variable: The outcome measured at the end of the experiment.

Example: An experiment investigates the effects of ad length (30s vs. 90s) and repetition (1, 3, or 5 times) on recall, attitude, and purchase intention for an iPhone ad. Each treatment (combination of ad length and repetition) is assigned to 12 subjects.

• Factors: Ad length (2 levels), Repetition (3 levels)

• Number of Treatments: 2 × 3 = 6

• Total Subjects Needed: 6 treatments × 12 subjects = 72 subjects

Confounding Variable: An outside variable that affects the response variable but is not the experimental factor being studied. In observational studies, these are called lurking variables.

Blocking: Controlling confounding variables by limiting the experiment to a homogeneous group. This reduces variation but may limit generalizability.

Example: A botanist tests if coffee grounds increase rose production by randomly assigning 30 rosebushes to two groups (with or without coffee grounds) and measuring the number of roses after three months.

• Treatments: Coffee grounds added vs. not added

• Experimental Units: Rosebushes

• Response Variable: Number of roses produced

• Random Assignment: Randomly assign 15 rosebushes to each treatment group

• Possible Confounding Variables: Sunlight, water, soil quality, disease

Identifying Study Designs

When analyzing a research report, follow these steps:

• If observational study:

  • Was it retrospective or prospective?

  • Who were the subjects and how were they selected?

  • What was the parameter of interest?

  • What is the scope of the conclusions?

• If experiment:

  • Who were the subjects?

  • What were the factors and their levels?

  • How many treatments were there?

  • What was the response variable measured?

Principles of Experimental Design

Blinding and the Placebo Effect

The placebo effect occurs when subjects receiving a placebo improve simply because they believe they are receiving the real treatment. To reduce bias, experiments often use blinding:

• Single-Blind: Only the subjects or the experimenters (not both) know which treatment is being administered.

• Double-Blind: Neither the subjects nor the experimenters know which treatment is being administered.

The primary purpose of blinding is to reduce hidden bias due to human expectations.

The Three Principles of a Well-Designed Experiment

• Control: Use a control group (no treatment or placebo) as a baseline for comparison. This helps determine if changes in the response variable are due to the treatment rather than confounding variables.

• Randomization: Assign experimental units to treatments using chance. This reduces bias and allows results to be generalized to the population.

• Replication: Apply each treatment to multiple experimental units. Replication reduces the impact of random variation.

Scope and Conclusion of Experiments

• Results can only be generalized to populations similar to the sample studied.

• No inference can be made about other populations or from observational studies.

Comparative Designs and Statistical Significance

Comparative designs ensure that differences in the response variable are due to the treatment, providing evidence for a cause-and-effect relationship. An effect is statistically significant if it is too large to attribute to chance alone.

Example Applications

• Energy Drink Study: Randomly assign students to receive either an energy drink or a placebo, then measure concentration scores. Double-blind design and replication ensure validity.

• Music and Endurance Study: Randomly assign cyclists to listen to music or not during exercise, then measure distance traveled. Double-blind design and replication help confirm statistical significance.

Summary Table: Observational Study vs. Experiment

Key Formulas

• Number of Treatments (for two factors):

Additional info: These notes expand on the provided material by including definitions, examples, and a summary table for clarity and completeness.