Probability Fundamentals (Based on Freund's Mathematical Statistics)

Probability Fundamentals

A. Basic Concepts

Probability theory is the mathematical study of random phenomena and uncertainty. It provides a framework for quantifying the likelihood of events in experiments with uncertain outcomes.

• Experiment: Any procedure with an uncertain outcome. Example: Tossing a die.

• Sample Space (S): The set of all possible outcomes. Example: $S = \{1,2,3,4,5,6\}$ for a die toss.

• Event (E): Any subset of the sample space. Example: $E = \{2,4,6\}$ (even numbers on a die).

B. Set Operations

Set operations are used to describe relationships between events in probability.

• Union (A ∪ B): Outcomes in A or B.

• Intersection (A ∩ B): Outcomes in both A and B.

• Complement (Ac): Outcomes not in A.

• Mutually Exclusive: $A \cap B = \emptyset$ (no outcomes in common).

C. Definition of Probability

Probability assigns a numerical value to the likelihood of events, following certain axioms (Kolmogorov's axioms):

• 1. $P(S) = 1$ (the probability of the sample space is 1).

• 2. $P(E) \geq 0$ for all events $E$.

• 3. If $A_1, A_2, \ldots$ are disjoint, then: $P(\bigcup_i A_i) = \sum_i P(A_i)$

D. Rules of Probability

Several rules help calculate probabilities for combined or related events.

• Complement Rule: $P(A^c) = 1 - P(A)$

• Addition Rule: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$

• If A and B are disjoint: $P(A \cup B) = P(A) + P(B)$

E. Counting Methods

Counting methods are essential for determining the number of possible outcomes in probability problems.

• Multiplication Principle: If one event can occur in $m$ ways and another in $n$ ways, both together can occur in $m \times n$ ways.

• Permutations: Number of ways to arrange $n$ objects: $P(n, r) = \frac{n!}{(n-r)!}$

• Combinations: Number of ways to choose $r$ objects from $n$ without regard to order: $C(n, r) = \frac{n!}{r!(n-r)!}$

F. Conditional Probability & Independence

Conditional probability measures the likelihood of an event given that another event has occurred. Independence describes events that do not affect each other's probabilities.

• Conditional Probability: $P(A|B) = \frac{P(A \cap B)}{P(B)}$, provided $P(B) > 0$

• Independence: Events A and B are independent if $P(A \cap B) = P(A)P(B)$

G. Total Probability & Bayes' Theorem

The total probability rule and Bayes' theorem are used to compute probabilities when events are partitioned or when updating beliefs based on new information.

• Total Probability: If $B_1, B_2, \ldots, B_n$ partition $S$, then $P(A) = \sum_i P(A|B_i)P(B_i)$

• Bayes' Theorem: $P(B|A) = \frac{P(A|B)P(B)}{P(A)}$