Foundations of Probability Theory: Sample Spaces, Events, and Probability Laws

Introduction to Probability and Statistics

Overview

This section introduces the fundamental concepts of probability and statistics, focusing on the construction and interpretation of sample spaces, events, and probability laws. These concepts form the basis for all further study in probability theory and statistical analysis.

• Probability is the mathematical study of randomness and uncertainty.

• Statistics involves the collection, analysis, interpretation, and presentation of data.

Set Operations and Sample Spaces

Set Operations - Review

Sets and their operations are foundational to probability theory, as sample spaces and events are defined as sets.

• Sample Space (Ω): The set of all possible outcomes of a random experiment.

• Power Set (2Ω): The set of all subsets of Ω, representing all possible events.

• Operations:

  • Union (A ∪ B): Outcomes in A or B.

  • Intersection (A ∩ B): Outcomes in both A and B.

  • Complement (Ac): Outcomes not in A.

  • Difference (A - B): Outcomes in A but not in B.

• Properties: Commutativity, Associativity, Distributivity, De Morgan's Laws.

Setting Up a Probability Space

Steps to Construct a Probability Space

To compute probabilities, a formal structure called a probability space is required.

1. Define the random experiment and list all possible outcomes (Ω).

2. Identify all possible events (subsets of Ω, i.e., 2Ω).

3. Specify the probability law (assigns probabilities to each event).

4. Calculate the probability of the desired events.

Example: Rolling a 6-faced die. Event: 'Getting 4'.

Properties of Outcomes

Characteristics of Outcomes

• Mutually Exclusive: No outcome can occur simultaneously with another.

• Collectively Exhaustive: All possible outcomes are included.

• Granularity: The level of detail in defining outcomes should be justified by experimental evidence.

Example: Considering whether wearing a watch affects the probability of heads in a coin toss.

Ockham’s Razor and Model Selection

Principle and Application

• Ockham’s Razor: Prefer simpler models that adequately explain the data.

• Akaike’s Information Criterion (AIC): A statistical method for model selection, balancing goodness of fit and model complexity.

Formula: $AIC = -2l + 2K$ where $l$ is the log-likelihood and $K$ is the number of parameters.

Axioms of Probability Theory

Kolmogorov’s Axioms (Finite Sample Spaces)

• I. Field of Sets: The set of events forms a field (closed under union, intersection, difference).

• II. Contains Ω: The field contains the sample space.

• III. Non-Negativity: $P[A] ext{ for any event } A ext{ is } ext{non-negative}$.

• IV. Normalization: $P[ ext{Ω}] = 1$.

• V. Additivity: If $A$ and $B$ are disjoint, $P[A ext{ or } B] = P[A] + P[B]$.

Probability Space

• Definition: $(Ω, ext{F}, P)$ where Ω is the sample space, F is the field of events, and P is the probability law.

• Probability Mass Function (PMF): In discrete spaces, $P[w]$ for each $w ext{ in } Ω$.

Discrete Uniform Probability Law

Definition and Example

• Equally Likely Outcomes: All outcomes have the same probability.

• Probability of an Event: Proportional to the number of outcomes in the event.

Example: Rolling a 4-faced die twice. Probability of event $E_1 = ext{{second roll gives 1}}$.

Axioms for Infinite and Continuous Sample Spaces

Sigma-Additivity (σ-Additivity) Axiom

• For countably infinite disjoint events $A_n$: $P[igcup_{n=1}^{ ext{∞}} A_n] = ext{Σ}_{n=1}^{ ext{∞}} P[A_n]$

• Purpose: Ensures probability theory excludes counterintuitive cases.

Continuous Sample Spaces and Borel σ-fields

Events in Continuous Spaces

• Borel σ-field: Collection of sets (intervals, areas, volumes) closed under countable unions and complements.

• Probability Law: For $A$ in the Borel σ-field, $P(A)$ is typically defined by area, length, or volume.

Continuous Uniform Probability Law

Definition and Example

• Random experiment: Dart thrown at a unit interval or square.

• Probability of event: Proportional to the length (interval) or area (square) of the event.

Example: Probability that $X > 0.6$ when $X$ is uniformly distributed on $[0,1]$ is $0.4$.

Summary Table: Types of Probability Spaces

Type	Sample Space (Ω)	Event Set (F)	Probability Law (P)
Discrete & Finite	Finite set	Power set $2^{Ω}$	Non-negativity, normalization, additivity
Discrete & Infinite	Countable infinite set	Power set $2^{Ω}$	Non-negativity, normalization, σ-additivity
Continuous	Uncountable set (e.g., interval, area)	Borel σ-field	Non-negativity, normalization, σ-additivity

Summary of Constructing a Probability Space

• Define the random experiment and sample space (Ω).

• Ensure outcomes are mutually exclusive, collectively exhaustive, and have appropriate granularity.

• Identify the set of all possible events (2Ω for countable Ω, Borel(Ω) for uncountable Ω).

• Define a probability law (P).

• Calculate the probability of desired events.

Axioms of Probability (Short List)

• $P[A] ext{≥} 0$ (Non-negativity)

• $P[Ω] = 1$ (Normalization)

• If $A$ and $B$ are disjoint, $P[A ext{ or } B] = P[A] + P[B]$ (Additivity)

• If $A_n$ are disjoint and $A = igcup_{n=1}^{ ext{∞}} A_n$, then $P[A] = ext{Σ}_{n=1}^{ ext{∞}} P[A_n]$ (σ-Additivity for infinite sample spaces)

Additional info: These notes provide foundational material for further study in probability, including discrete and continuous probability distributions, and set the stage for statistical inference.