Lecture 2: Collecting Data – Foundations of Statistical Sampling and Study Design

Collecting Data

Introduction

Collecting data is the foundational step in any statistical investigation. The quality and reliability of statistical conclusions depend critically on how data are gathered. This section explores the main sources of data, the distinction between observational studies and experiments, and the principles of sampling.

Sources of Data

Anecdotal Evidence

Anecdotal evidence refers to data based on individual experiences or observations that are often striking or memorable. While emotionally persuasive, anecdotal evidence is scientifically weak because it lacks systematic collection and may not be representative.

• Definition: Data based on isolated cases or stories.

• Example: Stories of 'supermothers' lifting cars in emergencies.

• Limitation: Not reliable for scientific inference due to lack of control and representativeness.

Observational Studies

Observational studies involve measuring variables through surveys or censuses without interfering with the subjects. These studies are useful for describing and exploring relationships but are limited in establishing causation.

• Definition: Measurement of variables without manipulation.

• Example: Surveying health of male crickets that successfully mated.

Experiments

Experiments involve actively influencing some responses by assigning treatments to subjects. This design allows researchers to infer causation by controlling explanatory variables.

• Definition: Researcher manipulates explanatory variables and observes response.

• Example: Infecting male crickets with parasites to observe female mate choice.

Observational Studies vs. Experiments

Key Differences

• Observational Study: Measures response variable without influencing explanatory variables.

• Experiment: Researcher assigns groups and manipulates explanatory variables to observe effects.

Confounding and Lurking Variables

Confounding occurs when two variables' effects on a response cannot be distinguished. Lurking variables are unmeasured factors that may influence the outcome.

• Example: Studying ice cream sales (X) and drowning deaths (Y); temperature is a confounding variable.

• Cell Phones and Brain Cancer: Observational studies may confound cell phone use with age, occupation, and residence.

Limitations of Observational Studies

• Cannot definitively establish causation due to confounding.

• Example: Childcare study found associations but could not rule out other factors (e.g., working parents, siblings).

Parameter vs Statistic

Definitions

• Population: Entire group of individuals under study.

• Sample: Subset of the population selected for analysis.

• Parameter: Numerical summary of a population (e.g., population mean $\mu$).

• Statistic: Numerical summary based on a sample (e.g., sample mean $\bar{x}$).

Sampling Metaphor

Representativeness

Sampling is compared to tasting soup: a spoonful (sample) should represent the whole pot (population). If the soup is not stirred, the sample may not be representative.

• Exploratory Analysis: Tasting a spoonful is like analyzing a sample.

• Inference: Generalizing from the sample to the population.

Example: Battery Manufacturer

A manufacturer inspects 24 batteries daily to estimate the defect rate. The population is all batteries produced; the sample is the 24 inspected each day. The method of sample selection affects the reliability of the estimate.

Obtaining Good Samples

Randomness in Sampling

• Statistical methods rely on random sampling for valid inference.

• If data are not collected randomly, estimates and error calculations may be unreliable.

• Common random sampling techniques: simple random, stratified, and cluster sampling.

Sampling Strategies for Observational Studies

Types of Sampling

• Simple Random Sampling (SRS): Every set of n individuals has an equal chance of selection.

• Stratified Sampling: Population divided into strata (groups with similar characteristics); SRS taken from each stratum.

• Cluster/Multistage Sampling: Population divided into clusters; clusters are randomly selected, and all or some individuals within clusters are sampled.

Sampling Bias Example: 1936 Election

The Literary Digest Poll

• Polled 10 million Americans; 2.4 million responses.

• Predicted Landon would win; Roosevelt actually won with 62%.

• Sample was biased: surveyed magazine readers, automobile owners, and telephone users—groups with higher incomes, not representative of the general population.

Large Samples Are Preferable, But...

• Large sample size does not guarantee accuracy if the sample is biased.

• Representativeness is more important than size.

• Metaphor: A well-stirred soup can be tested with a small spoon; a poorly stirred soup cannot be tested accurately, regardless of spoon size.

Simple Random Sample

Definition and Practice

• Definition: SRS of size n: every set of n individuals has an equal chance of selection.

• Concept: Names in a hat, random digit tables, computers, or physical random number generators.

Stratified Sampling

Process

• Divide population into strata based on characteristics (e.g., age, occupation, living situation).

• Take SRS from each stratum and combine to form the sample.

• Adjust sample sizes to reflect population shares.

Examples of Stratification

Occupation	Age	Living Situation
Professional Clerical Blue-collar Service	under 20 20-30 31-40 41-50 over 50	Homeowner Renter Homeless Other

Cluster and Multistage Sampling

Cluster Sampling

• Population divided into clusters (e.g., ZIP codes, counties).

• Randomly select clusters; sample all individuals in selected clusters.

• Clusters are not necessarily homogeneous.

Multistage Sampling

• Randomly select clusters, then randomly sample individuals within those clusters.

• Used for large populations where SRS is impractical.

Sampling Cautions: What Can Go Wrong

• Undercoverage: Some groups are left out or underrepresented (e.g., homeless, mobile-phone users).

• Nonresponse: Sampled individuals cannot be contacted or refuse to respond.

• Response Bias: Respondents may answer to please the interviewer or hide behaviors.

• Poor Wording/Framing: Ambiguous or leading questions can distort results.

Pitfalls – Types of Sampling Bias

• Non-response: Low response rates can make the sample unrepresentative.

• Voluntary Response: Samples consisting of volunteers may be biased toward strong opinions.

• Convenience Sample: Easily accessible individuals are more likely to be included, risking bias.

Importance of Wording of Questions

Survey Design

• Question wording can influence responses and introduce bias.

• Example: High-school survey on substance access; responses vary based on phrasing.

• Example: "Is US spending too much on assistance to the poor?" vs. "Is US spending too much on welfare?" yields different results.

Summary Table: Sampling Methods

Method	Description	Advantages	Disadvantages
Simple Random Sampling	Every individual has equal chance of selection	Unbiased, easy to analyze	May miss subgroups by chance
Stratified Sampling	Population divided into strata; SRS from each	Ensures representation of subgroups	Requires knowledge of strata
Cluster Sampling	Population divided into clusters; clusters randomly selected	Economical for large populations	Clusters may not be representative
Multistage Sampling	Randomly select clusters, then sample within clusters	Flexible, practical for large populations	Complex design, potential for bias

Key Formulas

• Population Mean: $\mu = \frac{1}{N} \sum_{i=1}^{N} x_i$

• Sample Mean: $\bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i$

Additional info: These notes expand on the original slides by providing definitions, examples, and structured tables for sampling methods, as well as key formulas for population and sample means.