Law of Total Probability and Bayes’ Rule: Probability, Partitions, and Applications

Law of Total Probability and Bayes’ Rule

Introduction

This study guide covers the Law of Total Probability and Bayes’ Rule, two fundamental concepts in probability theory. These tools are essential for analyzing complex probability scenarios, especially when dealing with conditional probabilities and partitions of sample spaces. Applications include medical testing, weather prediction, and more.

Recap: Foundational Concepts

Tree Diagrams

Tree diagrams are graphical representations used to map out all possible outcomes of a probabilistic experiment, especially when the experiment involves multiple stages.

• Stages: Each branch represents a possible outcome at each stage.

• Probabilities: Probabilities are assigned to each branch, and the probability of a particular outcome is the product of the probabilities along its path.

Example: Flipping a coin to choose an urn, then drawing a colored ball from that urn. The tree diagram helps visualize all possible outcomes and their probabilities.

Independence

Independent events are events where the occurrence of one does not affect the probability of the other.

• Formally, events A and B are independent if $P(A \cap B) = P(A)P(B)$.

Conditional Probability

Conditional probability is the probability of an event occurring given that another event has already occurred.

• Defined as $P(A|B) = \frac{P(A \cap B)}{P(B)}$, provided $P(B) > 0$.

Probability Tree Diagrams: Application

Drawing Balls from Urns

Suppose two urns each contain six balls of various colors. A fair coin is flipped to select an urn, and then a ball is drawn at random from that urn.

• Sample Space: All possible combinations of coin outcome and ball color.

• Tree Diagram: First level: coin flip (Head or Tail). Second level: possible ball colors from the selected urn.

Example Tree Structure:

• Head (H): 1/2 probability

  • Red (R): 2/6

  • White (W): 4/6

• Tail (T): 1/2 probability

  • Red (R): 5/6

  • Blue (B): 1/6

To find the probability of drawing a particular color, multiply along the branches and sum over all relevant paths.

Partitions of the Sample Space

Definition and Properties

A partition of a sample space S is a collection of events $A_1, A_2, ..., A_k$ such that:

• Each $A_i \neq \emptyset$ (no empty sets)

• Events are pairwise disjoint: $A_i \cap A_j = \emptyset$ for $i \neq j$

• Their union covers the entire sample space: $A_1 \cup A_2 \cup ... \cup A_k = S$

Partitions are essential for applying the Law of Total Probability and Bayes’ Rule.

Law of Total Probability

Statement and Formula

The Law of Total Probability allows us to compute the probability of an event by considering all possible ways that event can occur, via a partition of the sample space.

• For two events $A_1$ and $A_2$ forming a partition:

$P(E) = P(E \cap A_1) + P(E \cap A_2)$

Using the definition of conditional probability:

$P(E) = P(A_1) \cdot P(E|A_1) + P(A_2) \cdot P(E|A_2)$

• Generalized to k events:

$P(E) = \sum_{i=1}^k P(A_i) \cdot P(E|A_i)$

Example: Weather and Attendance

Suppose the probability of a day being Sunny, Cloudy, or Rainy is 40%, 30%, and 30%, respectively. The probability a student is late is 20% on Sunny days, 30% on Cloudy days, and 50% on Rainy days. What is the probability a student is late?

• Let S = Sunny, C = Cloudy, R = Rainy, and L = Late.

• Apply the law:

$P(L) = P(S)P(L|S) + P(C)P(L|C) + P(R)P(L|R)$

$= 0.4 \times 0.2 + 0.3 \times 0.3 + 0.3 \times 0.5 = 0.08 + 0.09 + 0.15 = 0.32$

Interpretation: The probability a student is late is 32%.

Bayes’ Rule

Motivation and Statement

Bayes’ Rule allows us to "reverse" conditional probabilities, i.e., to find $P(A|B)$ when we know $P(B|A)$ and the probabilities of A and B.

• Given events A and B with $P(B) > 0$:

$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$

For a partition $A_1, ..., A_k$ of the sample space:

$P(A_i|E) = \frac{P(E|A_i)P(A_i)}{\sum_{j=1}^k P(E|A_j)P(A_j)}$

Derivation Using the Law of Total Probability

• By the law of total probability:

$P(R) = P(R|H)P(H) + P(R|T)P(T)$

• By the multiplication rule:

$P(H \cap R) = P(R|H)P(H)$

• Therefore:

$P(H|R) = \frac{P(R|H)P(H)}{P(R|H)P(H) + P(R|T)P(T)}$

Example: Medical Testing

A test is used to detect a rare disease:

• Prevalence: 1 out of 20 people has the disease ($P(D) = 0.05$)

• Sensitivity: If diseased, test is positive 4 out of 5 times ($P(Pos|D) = 0.8$)

• False positive rate: If not diseased, test is positive 1 out of 10 times ($P(Pos|\neg D) = 0.1$)

Given a positive test, what is the probability the person actually has the disease?

$P(D|Pos) = \frac{P(Pos|D)P(D)}{P(Pos|D)P(D) + P(Pos|\neg D)P(\neg D)}$

$= \frac{0.8 \times 0.05}{0.8 \times 0.05 + 0.1 \times 0.95} = \frac{0.04}{0.04 + 0.095} = \frac{0.04}{0.135} \approx 0.296$

Interpretation: Even with a positive test, the probability of actually having the disease is about 29.6%.

Example: Weather and Bayes’ Rule

Given a student is late, what is the probability it was Rainy?

• From earlier: $P(R) = 0.3$, $P(L|R) = 0.5$, $P(L) = 0.32$

$P(R|L) = \frac{P(L|R)P(R)}{P(L)} = \frac{0.5 \times 0.3}{0.32} = \frac{0.15}{0.32} \approx 0.469$

Interpretation: If a student is late, there is about a 46.9% chance it was a rainy day.

Summary Table: Law of Total Probability vs. Bayes’ Rule

Concept	Formula	Purpose
Law of Total Probability	$P(E) = \sum_{i=1}^k P(A_i)P(E|A_i)$	Finds the probability of an event by considering all ways it can occur via a partition.
Bayes’ Rule	$P(A_i|E) = \frac{P(E|A_i)P(A_i)}{\sum_{j=1}^k P(E|A_j)P(A_j)}$	Reverses conditional probability; finds the probability of a cause given an observed effect.

Key Takeaways

• Tree diagrams are powerful tools for visualizing multi-stage probability experiments.

• Partitions of the sample space are essential for applying the Law of Total Probability and Bayes’ Rule.

• The Law of Total Probability helps compute the probability of an event by summing over all possible scenarios.

• Bayes’ Rule allows us to "flip" conditional probabilities, crucial in fields like medical testing and diagnostics.