Jointly Distributed Random Variables, Expectation, and Properties: Study Notes

Jointly Distributed Random Variables

Marginal and Joint Distributions

When dealing with two or more random variables, their joint distribution describes the probability structure of all variables together. The joint probability mass function (for discrete variables) or joint probability density function (for continuous variables) provides the probability or density for all combinations of values.

• Marginal Distribution: The probability distribution of a subset of the variables, obtained by summing (discrete) or integrating (continuous) the joint distribution over the other variables.

• Example (Continuous): If $f(x, y)$ is the joint density of $X$ and $Y$, then the marginal density of $X$ is $f_X(x) = \int_{-\infty}^{\infty} f(x, y) dy$.

• Example (Discrete): If $p(i, j)$ is the joint mass function, then $P\{X = i\} = \sum_j p(i, j)$.

Independence

Random variables $X$ and $Y$ are independent if the joint distribution factors into the product of their marginals:

• For all sets $A$ and $B$, $P\{X \in A, Y \in B\} = P\{X \in A\} P\{Y \in B\}$.

• For continuous variables: $f(x, y) = f_X(x) f_Y(y)$.

• For discrete variables: $p(i, j) = p_X(i) p_Y(j)$.

Conditional Distributions

• Discrete: $P\{X = x | Y = y\} = \frac{p(x, y)}{p_Y(y)}$

• Continuous: $f_{X|Y}(x|y) = \frac{f(x, y)}{f_Y(y)}$

Order Statistics

The order statistics of a sample are the sorted values $X_{(1)} \leq X_{(2)} \leq \cdots \leq X_{(n)}$.

• If $X_1, ..., X_n$ are independent and identically distributed (i.i.d.) with density $f(x)$, the joint density of the order statistics is:

$f(x_1, ..., x_n) = n! f(x_1) \cdots f(x_n), \quad x_1 < x_2 < \cdots < x_n$

Exchangeable Random Variables

A sequence $X_1, ..., X_n$ is exchangeable if the joint distribution is invariant under permutations:

• $P\{X_{i_1} \leq x_1, ..., X_{i_n} \leq x_n\} = P\{X_1 \leq x_1, ..., X_n \leq x_n\}$ for any permutation $(i_1, ..., i_n)$.

• For discrete variables, $p(x_1, ..., x_n)$ is a symmetric function of $(x_1, ..., x_n)$.

Example: Drawing balls without replacement from an urn, the indicator variables for drawing a special ball are exchangeable.

Key Properties and Examples

• Sum of Independent Exponential Random Variables: The sum $X_1 + ... + X_n$ of $n$ independent exponential($\lambda$) random variables is a gamma random variable with parameters $n$ and $\lambda$.

• Polya's Urn Model: In this model, the sequence of draws is exchangeable, and the probability that the $i$th ball is red is always $n/(n+m)$, regardless of $i$.

• Order Statistics of Uniform Variables: The spacings between order statistics of i.i.d. uniform(0,1) random variables are exchangeable.

Expectation and Its Properties

Definition of Expectation

• Discrete: $E[X] = \sum_x x p(x)$

• Continuous: $E[X] = \int_{-\infty}^{\infty} x f(x) dx$

• If $P\{a \leq X \leq b\} = 1$, then $a \leq E[X] \leq b$.

Expectation of Sums

• Linearity: $E[X + Y] = E[X] + E[Y]$ (holds for any random variables with finite expectations).

• For $n$ random variables: $E[X_1 + ... + X_n] = E[X_1] + ... + E[X_n]$.

• Sample Mean: For i.i.d. $X_1, ..., X_n$ with mean $\mu$, $E[\bar{X}] = \mu$ where $\bar{X} = (X_1 + ... + X_n)/n$.

Indicator Variables and Counting

• Let $I_A$ be the indicator of event $A$ ($I_A = 1$ if $A$ occurs, $0$otherwise). Then$E[I_A] = P(A)$.

• For $X = I_{A_1} + ... + I_{A_n}$, $E[X] = P(A_1) + ... + P(A_n)$.

• Boole's Inequality: $P(\cup_{i=1}^n A_i) \leq \sum_{i=1}^n P(A_i)$.

Expectation of a Random Sum

• If $N$ is a random variable (number of terms), and $X_1, X_2, ...$ are i.i.d. and independent of $N$, then:

$E\left[ \sum_{i=1}^N X_i \right] = E[N] E[X_1]$

Conditional Expectation

• Definition (Discrete): $E[X|Y = y] = \sum_x x P\{X = x | Y = y\}$

• Definition (Continuous): $E[X|Y = y] = \int_{-\infty}^{\infty} x f_{X|Y}(x|y) dx$

• Law of Total Expectation: $E[X] = E[E[X|Y]]$

• Law of Total Probability: $P(E) = E[P(E|Y)]$

Conditional Variance

• Definition: $Var(X|Y) = E[(X - E[X|Y])^2 | Y]$

• Law of Total Variance: $Var(X) = E[Var(X|Y)] + Var(E[X|Y])$

Prediction and Best Predictors

• The function $g(X)$ that minimizes $E[(Y - g(X))^2]$ is $g(X) = E[Y|X]$.

• The best linear predictor of $Y$ given $X$ is $a + bX$, where:

$b = \frac{Cov(X, Y)}{Var(X)}$, $a = E[Y] - b E[X]$

Covariance, Variance of Sums, and Correlation

Covariance

• Definition: $Cov(X, Y) = E[(X - E[X])(Y - E[Y])] = E[XY] - E[X]E[Y]$

• If $X$ and $Y$ are independent, $Cov(X, Y) = 0$ (but the converse is not always true).

• Properties:

  • $Cov(X, X) = Var(X)$

  • $Cov(aX, Y) = a Cov(X, Y)$

  • $Cov(X + Y, Z) = Cov(X, Z) + Cov(Y, Z)$

Variance of Sums

• $Var(X_1 + ... + X_n) = \sum_{i=1}^n Var(X_i) + 2 \sum_{i

• If $X_i$ are independent, $Var(X_1 + ... + X_n) = \sum_{i=1}^n Var(X_i)$

Correlation

• Definition: $\rho(X, Y) = \frac{Cov(X, Y)}{\sqrt{Var(X) Var(Y)}}$

• $-1 \leq \rho(X, Y) \leq 1$

• $\rho(X, Y) = 0$ means $X$ and $Y$ are uncorrelated (but not necessarily independent).

Moment Generating Functions (MGFs)

Definition and Properties

• Definition: $M(t) = E[e^{tX}]$

• For discrete $X$: $M(t) = \sum_x e^{tx} p(x)$

• For continuous $X$: $M(t) = \int_{-\infty}^{\infty} e^{tx} f(x) dx$

• Moments: The $n$th moment is $E[X^n] = M^{(n)}(0)$ (the $n$th derivative at $t=0$).

• MGF of Sums: If $X$ and $Y$ are independent, $M_{X+Y}(t) = M_X(t) M_Y(t)$

• Uniqueness: The MGF (if it exists in a neighborhood of $t=0$) uniquely determines the distribution.

Common MGFs

• Binomial($n, p$): $M(t) = (p e^t + 1 - p)^n$

• Poisson($\lambda$): $M(t) = \exp\{ \lambda (e^t - 1) \}$

• Exponential($\lambda$): $M(t) = \frac{\lambda}{\lambda - t}$ for $t < \lambda$

• Normal($\mu, \sigma^2$): $M(t) = \exp\{ \mu t + \frac{1}{2} \sigma^2 t^2 \}$

Multivariate Normal Distribution

Definition and Properties

• A vector $(X_1, ..., X_m)$ is multivariate normal if each $X_i$ is a linear combination of independent standard normal variables plus a constant.

• The joint MGF is: $M(t_1, ..., t_m) = \exp\left( \sum_{i=1}^m t_i \mu_i + \frac{1}{2} \sum_{i=1}^m \sum_{j=1}^m t_i t_j Cov(X_i, X_j) \right)$

• The joint distribution is completely determined by the means and covariances.

• Sum of Independent Normals: The sum of independent normal random variables is normal, with mean and variance equal to the sum of the means and variances, respectively.

Sample Mean and Variance (Normal Case)

• If $X_1, ..., X_n$ are i.i.d. normal($\mu, \sigma^2$):

  • The sample mean $\bar{X}$ and sample variance $S^2$ are independent.

  • $\bar{X} \sim N(\mu, \sigma^2/n)$

  • $(n-1)S^2/\sigma^2$ has a chi-squared distribution with $n-1$ degrees of freedom.

Summary Table: Key Formulas and Properties

Concept	Formula / Property
Marginal (Continuous)	$f_X(x) = \int_{-\infty}^{\infty} f(x, y) dy$
Marginal (Discrete)	$P\{X = i\} = \sum_j p(i, j)$
Independence	$f(x, y) = f_X(x) f_Y(y)$ or $p(i, j) = p_X(i) p_Y(j)$
Conditional Expectation	$E[X|Y = y] = \int x f_{X|Y}(x|y) dx$
Law of Total Expectation	$E[X] = E[E[X|Y]]$
Covariance	$Cov(X, Y) = E[XY] - E[X]E[Y]$
Variance of Sum	$Var(\sum X_i) = \sum Var(X_i) + 2 \sum_{i
Correlation	$\rho(X, Y) = \frac{Cov(X, Y)}{\sqrt{Var(X) Var(Y)}}$
MGF (Sum of Independent)	$M_{X+Y}(t) = M_X(t) M_Y(t)$
Best Linear Predictor	$a + bX$, $b = \frac{Cov(X, Y)}{Var(X)}$, $a = E[Y] - b E[X]$

Additional info:

• Many of the examples and exercises in the source material illustrate the application of these concepts to real-world problems, such as the coupon collector's problem, random walks, and the analysis of algorithms (e.g., quick-sort).

• For more advanced or theoretical results (e.g., Stieltjes integrals, general definition of expectation), the notes provide a foundation for understanding expectation beyond the discrete and continuous cases.