Conditional Probability, Independence, Law of Total Probability, and Bayes’ Rule

Conditional Probability, Independence, Law of Total Probability, and Bayes’ Rule

Recap and Overview

This lecture covers foundational concepts in probability, focusing on conditional probability, independence, the law of total probability, and Bayes’ Rule. These topics are essential for understanding how to compute probabilities in complex situations, especially when events are not independent or when information is revealed sequentially.

• Independent and dependent events

• Multiplication rule for independent events

• Computing "at least" probabilities

• Conditional Probability

• General Multiplication Rule

Conditional Probability

Definition and Formula

Conditional probability quantifies the probability of an event A occurring given that another event B has occurred. It is denoted as P(A|B) and is defined as:

$P(A|B) = \frac{P(A \cap B)}{P(B)}$

• P(A|B): Probability of A given B.

• P(A ∩ B): Probability that both A and B occur.

• P(B): Probability that B occurs (must be > 0).

Example: If A and B are events in a sample space Ω, the conditional probability focuses on the part of A that overlaps with B, normalized by the probability of B.

General Multiplication Rule and Independence

General Multiplication Rule

The probability that two events E and F both occur is given by:

$P(E \text{ and } F) = P(E) \cdot P(F|E)$

• This rule applies whether or not E and F are independent.

Independence of Events

Two events E and F are independent if the occurrence of one does not affect the probability of the other. Formally, E and F are independent if:

• $P(E \text{ and } F) = P(E)P(F)$

• or $P(E|F) = P(E)$

• or $P(F|E) = P(F)$

Example: Tossing two fair coins; the outcome of one does not affect the other.

Multiplicative Rule: Example

Defective Circuits Problem

Problem: Out of 100 circuits, 5 are defective. A manager randomly selects 2 circuits. If both work, the shipment is accepted; otherwise, it is rejected. What is the probability that at least one defective circuit is found and the shipment is rejected?

Approach 1: Tree Diagram

• Construct a tree diagram to enumerate all possible outcomes (D = defective, G = good).

• Calculate the probability along each path by multiplying the probabilities at each stage.

Calculation:

• $P(GD) = 0.048$

• $P(DG) = 0.048$

• $P(DD) = 0.002$

• $P(\text{at least 1 defective}) = 0.048 + 0.048 + 0.002 = 0.098$

Conclusion: The probability the shipment is rejected is 0.098.

Approach 2: Complement Rule

• Compute the probability that both circuits are not defective, then use the complement rule.

$P(\text{at least 1 defective}) = 1 - P(\text{none defective})$

$= 1 - \left(\frac{95}{100}\right) \left(\frac{94}{99}\right) = 1 - 0.902 = 0.098$

Conclusion: Both approaches yield the same result.

General Rule for Sampling and Independence

When sampling without replacement from a large population, events can be treated as independent if the sample size is less than 5% of the population size. This is a useful rule of thumb in practical applications.

Independence: Examples

Example 1: Rolling Two Dice

Let A = {first die is even}, B = {sum is 7}. Are these events independent?

1. Compute $P(A)$

2. Compute $P(B)$

3. Compute $P(A \text{ and } B)$

Compare $P(A \text{ and } B)$ to $P(A)P(B)$ to determine independence.

Example 2: Rolling Two Dice

Let A = {first die shows a 6}, B = {sum of dice > 8}.

1. What is $P(B)$?

2. What is $P(B|A)$?

3. Are A and B independent?

Probability Tree Diagrams

Drawing Balls from Urns

Suppose there are two urns, each with six balls (various colors). A fair coin is flipped: heads means draw from urn 1, tails from urn 2. What is the sample space and the probability of drawing each color?

• Draw a tree diagram: first branch is coin flip, second branch is ball color from the selected urn.

Sample Tree Structure:

• Head (H): Red (R), White (W)

• Tails (T): Red (R), Blue (B)

Probabilities are calculated by multiplying along the branches.

Table: Probabilities for Each Color

Color	Probability
White (W)	$P(W) = \frac{1}{2} \times \frac{4}{6} = \frac{1}{3}$
Blue (B)	$P(B) = \frac{1}{2} \times \frac{1}{6} = \frac{1}{12}$
Red (R)	$P(R) = \frac{1}{2} \times \frac{2}{6} + \frac{1}{2} \times \frac{5}{6} = \frac{1}{6} + \frac{5}{12} = \frac{7}{12}$

Additional info: Probabilities inferred from the tree diagram structure and typical urn problems.

Partitions

Definition

A partition of a sample space S is a collection of events $A_1, A_2, ..., A_k$ such that:

• Each $A_i \neq \emptyset$

• $A_i \cap A_j = \emptyset$ for $i \neq j$ (pairwise disjoint)

• $A_1 \cup A_2 \cup ... \cup A_k = S$ (their union is the full sample space)

Partitions are useful for breaking down complex probability problems into manageable cases.

Law of Total Probability

Statement and Formula

The Law of Total Probability allows us to compute the probability of an event E by considering all the ways E can occur across a partition of the sample space:

$P(E) = P(A_1)P(E|A_1) + P(A_2)P(E|A_2) + ... + P(A_k)P(E|A_k)$

• Each $A_i$ is a part of the partition of the sample space.

• This law is especially useful when the probability of E depends on which $A_i$ occurs.

Bayes’ Theorem

Statement and Application

Bayes’ Theorem provides a way to "invert" conditional probabilities, allowing us to compute $P(A|B)$ from $P(B|A)$ and the marginal probabilities of A and B.

$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$

For a partition $A_1, ..., A_k$ of the sample space, Bayes’ Theorem generalizes to:

$P(A_i|B) = \frac{P(B|A_i)P(A_i)}{\sum_{j=1}^k P(B|A_j)P(A_j)}$

• Numerator: Probability of B given $A_i$ times the probability of $A_i$.

• Denominator: Total probability of B across all partition events.

Example Application: In the urn problem, if a red ball is drawn, Bayes’ Theorem can be used to find the probability that the coin flip was heads.

Summary Table: Key Probability Rules

Rule	Formula	When to Use
Conditional Probability	$P(A|B) = \frac{P(A \cap B)}{P(B)}$	Probability of A given B
Multiplication Rule	$P(E \text{ and } F) = P(E)P(F|E)$	Probability of both E and F
Independence	$P(E \text{ and } F) = P(E)P(F)$	When E and F are independent
Law of Total Probability	$P(E) = \sum_{i=1}^k P(A_i)P(E|A_i)$	When E can occur via several disjoint cases
Bayes’ Theorem	$P(A_i|B) = \frac{P(B|A_i)P(A_i)}{\sum_{j=1}^k P(B|A_j)P(A_j)}$	"Inverting" conditional probabilities

Conclusion

Understanding conditional probability, independence, the law of total probability, and Bayes’ Rule is crucial for solving complex probability problems. These concepts allow statisticians to model real-world situations where information is revealed in stages or where events are not independent.