A probability distribution describes the possible values of a random variable and how likely these values are.
We first consider one random variable. We then study joint and conditional distributions of multiple variables.
The distribution of Y assigns probabilities to events involving Y.
The cumulative distribution function (CDF) F_Y fully characterizes this distribution.
Cumulative Distribution Function (CDF)
The CDF of a random variable Y is F_Y(a) := P(Y \leq a), \quad a \in \mathbb R.
A CDF is nondecreasing and right-continuous. It also satisfies \lim_{a \to -\infty}F_Y(a)=0 \quad \text{and} \quad \lim_{a \to +\infty}F_Y(a)=1.
For a<b, interval probabilities can be obtained from the CDF: P(a<Y\leq b)=F_Y(b)-F_Y(a).
For a more detailed introduction to probability theory, see my tutorial at probability.svenotto.com.
Consider the coin toss binary random variable Y = \begin{cases} 1 & \text{if the outcome is heads,} \\ 0 & \text{if the outcome is tails.} \end{cases}
The CDF for a fair coin is F_Y(a) = \begin{cases} 0 & a < 0, \\ 0.5 & 0 \leq a < 1, \\ 1 & a \geq 1. \end{cases}
A discrete random variable takes values in a finite or countable set.
Its CDF has a jump at every value with positive probability.
Let F_Y(a^-) denote the left limit of F_Y at a. The point probability is the size of the jump: P(Y=a)=F_Y(a)-F_Y(a^-).
Probability Mass Function (PMF)
The probability mass function (PMF) of a discrete random variable Y is \pi_Y(a):=P(Y=a), \quad a\in\mathbb R.
The PMF of the coin variable is \pi_Y(a)=\begin{cases} 0.5 & \text{if } a\in\{0,1\}, \\ 0 & \text{otherwise.} \end{cases}
Suppose you conduct a survey and ask a randomly selected person about their years of education.
The possible answers are Y\in\{10,12,14,16,18,21\}.
The education variable is discrete. It may have the following CDF and PMF:
Here, the PMF is \pi_Y(a)=\begin{cases} 0.06 & \text{if } a=10, \\ 0.43 & \text{if } a=12, \\ 0.16 & \text{if } a=14, \\ 0.08 & \text{if } a=16, \\ 0.24 & \text{if } a=18, \\ 0.03 & \text{if } a=21, \\ 0 & \text{otherwise.} \end{cases}
The support \mathcal Y is the set of values that Y takes with positive probability: \mathcal Y=\{a\in\mathbb R:\pi_Y(a)>0\}.
For the education variable, \mathcal Y=\{10,12,14,16,18,21\} and \sum_{a\in\mathcal Y}\pi_Y(a)=1.
A continuous random variable has a CDF without jumps. Therefore, P(Y=a)=F_Y(a)-F_Y(a^-)=0 for every a.
Probability Density Function (PDF)
If the CDF F_Y is continuously differentiable, its PDF is obtained by differentiation: f_Y(a)=\frac{d}{da}F_Y(a).
More generally, a nonnegative function f_Y is a PDF of Y if, for every a, F_Y(a)=\int_{-\infty}^{a}f_Y(u)\,du.
Then f_Y integrates to one.
A PDF may therefore exist even if the CDF is not differentiable at every point.
Suppose you ask a randomly selected employed person about their hourly wage in EUR.
We model wage as a continuous random variable with nonnegative values.
The CDF and PDF of wage may have the following form:
For a<b, P(a<Y\leq b)=F_Y(b)-F_Y(a)=\int_a^b f_Y(u)\,du.
The PDF value f_Y(a) is not a probability. Probabilities are areas under the density over intervals.
Because point probabilities are zero, strict and non-strict interval endpoints give the same probability.
For two random variables Y and Z, we distinguish:
The joint CDF of Y and Z is F_{Y,Z}(a,b)=P(Y\leq a,Z\leq b).
The marginal CDFs are obtained from the joint CDF: \begin{align*} F_Y(a)&=\lim_{b\to\infty}F_{Y,Z}(a,b), \\ F_Z(b)&=\lim_{a\to\infty}F_{Y,Z}(a,b). \end{align*}
When both variables are discrete, the joint PMF and marginal PMFs satisfy \begin{align*} \pi_{Y,Z}(a,b)&=P(Y=a,Z=b), \\ \pi_Y(a)&=\sum_{v\in\mathcal Z}\pi_{Y,Z}(a,v), \\ \pi_Z(b)&=\sum_{u\in\mathcal Y}\pi_{Y,Z}(u,b). \end{align*}
If (Y,Z) has a joint density f_{Y,Z}, then F_{Y,Z}(a,b) =\int_{-\infty}^{a}\int_{-\infty}^{b}f_{Y,Z}(u,v)\,dv\,du.
The corresponding marginal densities are \begin{align*} f_Y(a)&=\int_{-\infty}^{\infty}f_{Y,Z}(a,v)\,dv, \\ f_Z(b)&=\int_{-\infty}^{\infty}f_{Y,Z}(u,b)\,du. \end{align*}
The joint density can be visualized as a surface over pairs of values.
Integrating the surface over experience gives the marginal density of wage. Integrating over wage gives the marginal density of experience.
The marginal curves accumulate the surface along an axis. They are not slices at a fixed value.
The conditional CDF of Y given Z=b is F_{Y\mid Z=b}(a)=P(Y\leq a\mid Z=b).
It answers the question: If we know that Z=b, what is the distribution of Y?
If Z is discrete and P(Z=b)>0, the usual conditional-probability rule gives F_{Y\mid Z=b}(a) =\frac{P(Y\leq a,Z=b)}{P(Z=b)}.
If both variables are discrete, the corresponding conditional PMF is \pi_{Y\mid Z=b}(a)=\frac{\pi_{Y,Z}(a,b)}{\pi_Z(b)}.
If a joint density exists and f_Z(b)>0, the conditional density is f_{Y\mid Z=b}(a)=\frac{f_{Y,Z}(a,b)}{f_Z(b)}.
For a continuous Z, the event \{Z=b\} has probability zero. Therefore, the ordinary probability ratio cannot be used.
The density ratio therefore plays the role of the ordinary conditional-probability ratio.
Let Y denote wage and Z denote years of education.
The conditional CDF F_{Y\mid Z=12}(a) describes the distribution of wages among individuals with 12 years of education.
The plots show that the wage distribution varies with education.
At the threshold a=20, P(Y\leq20\mid Z=12)>P(Y\leq20\mid Z=18).
Individuals with 12 years of education are more likely to have a wage no greater than 20 than individuals with 18 years of education.
This difference shows that wage and education are dependent random variables.
We can also condition on a random vector \boldsymbol X=(X_1,\ldots,X_k)': F_{Y\mid\boldsymbol X=\boldsymbol x}(a) =P(Y\leq a\mid\boldsymbol X=\boldsymbol x).
In the regression chapters, Y denotes the outcome and \boldsymbol X contains the regressors.
A marginal distribution is a weighted average of conditional distributions.
If Z is discrete, F_Y(a)=\sum_{b\in\mathcal Z}F_{Y\mid Z=b}(a)\pi_Z(b).
If Z has a density, F_Y(a)=\int_{-\infty}^{\infty}F_{Y\mid Z=b}(a)f_Z(b)\,db.
These formulas are versions of the law of total probability.
The distribution of a coin flip should not depend on the marital status of the person tossing the coin.
In contrast, the wage distribution generally varies with education.
Independence
The random variables Y and Z are independent if and only if F_{Y,Z}(a,b)=F_Y(a)F_Z(b) for all a and b.
The definition has an equivalent and intuitive formulation: F_{Y\mid Z=b}(a)=F_Y(a) for all a and all b outside a set B with P(Z\in B)=0.
In other words, knowing the value of Z does not change the distribution of Y.
The conditional distribution of Y is the same as its marginal distribution.
If Z is discrete, this means every b with P(Z=b)>0.
The conditional CDF of the coin flip is therefore the same for married and unmarried individuals.
If both Y and Z are discrete, independence is also equivalent to \pi_{Y,Z}(a,b)=\pi_Y(a)\pi_Z(b) for all a and b.
When a joint density exists, independence is equivalent to f_Y(a)f_Z(b) being a joint density of (Y,Z). Thus, we can write f_{Y,Z}(a,b)=f_Y(a)f_Z(b).
For every b with f_Z(b)>0, the corresponding conditional density is f_{Y\mid Z=b}(a) =\frac{f_{Y,Z}(a,b)}{f_Z(b)} =f_Y(a).
This gives the same conditional-CDF interpretation.
The expected value summarizes the center of a distribution.
More generally, we can take the expected value of a function h(Y).
An interval (a,b] receives the probability weight F_Y(b)-F_Y(a)=P(a<Y\leq b).
If Y is discrete and the interval contains only one possible value u, this weight is the jump \pi_Y(u)=P(Y=u).
If Y has a density, the probability weights can be written as dF_Y(u)=f_Y(u)\,du.
The Stieltjes integral uses dF_Y(u) to cover both cases in one definition.
Expected Value
The expected value of h(Y) is E[h(Y)]=\int_{-\infty}^{\infty}h(u)\,dF_Y(u), provided that \int_{-\infty}^{\infty}|h(u)|\,dF_Y(u)<\infty.
If Y is discrete, E[h(Y)]=\sum_{u\in\mathcal Y}h(u)\pi_Y(u).
If Y has a density, E[h(Y)]=\int_{-\infty}^{\infty}h(u)f_Y(u)\,du.
For more details, see Expected Value: General Case.
Taking h(u)=u gives the expected value E[Y], also called the population mean.
For a 0-1 binary random variable, E[Y]=P(Y=1).
For the education distribution above, \begin{align*} E[Y] &=10(0.06)+12(0.43)+14(0.16) \\ &\quad+16(0.08)+18(0.24)+21(0.03) \\ &=14.23. \end{align*}
Expectation is linear. For constants a,b,c, E[a+bY+cZ]=a+bE[Y]+cE[Z].
This rule does not require Y and Z to be independent.
Expectations of random vectors and matrices are taken component by component.
The conditional expectation is the mean of a conditional distribution.
Suppose that E[|Y|]<\infty.
For values \boldsymbol x at which the conditional distribution is defined, let m(\boldsymbol x) :=E[Y\mid\boldsymbol X=\boldsymbol x] =\int_{-\infty}^{\infty}y\,dF_{Y\mid\boldsymbol X=\boldsymbol x}(y).
The function m(\boldsymbol x) is called the conditional expectation function (CEF).
If Y is discrete, E[Y\mid\boldsymbol X=\boldsymbol x] =\sum_{y\in\mathcal Y}y\pi_{Y\mid\boldsymbol X=\boldsymbol x}(y).
If Y has a conditional density, E[Y\mid\boldsymbol X=\boldsymbol x] =\int_{-\infty}^{\infty}y f_{Y\mid\boldsymbol X=\boldsymbol x}(y)\,dy.
It is important to distinguish between:
For wage Y and education Z, the CEF assigns a conditional mean wage to each education level: m(b)=E[Y\mid Z=b].
The following rules hold whenever the expectations exist.
Conditional expectation is linear: E[aA+bB\mid\boldsymbol X] =aE[A\mid\boldsymbol X]+bE[B\mid\boldsymbol X].
Let r(\boldsymbol X) be a function of \boldsymbol X.
Once we condition on \boldsymbol X, the value of r(\boldsymbol X) is known. Therefore, E[r(\boldsymbol X)\mid\boldsymbol X]=r(\boldsymbol X).
The conditioning theorem (CT), also called the factorization rule, states E[r(\boldsymbol X)Y\mid\boldsymbol X] =r(\boldsymbol X)E[Y\mid\boldsymbol X].
Thus, a function of \boldsymbol X can be taken outside the conditional expectation.
The law of iterated expectations (LIE) states E[E[Y\mid\boldsymbol X]]=E[Y].
We can first calculate the mean within groups defined by \boldsymbol X, and then average these conditional means.
If Y and \boldsymbol X are independent, then E[Y\mid\boldsymbol X]=E[Y].
The converse is not generally true. The equality E[Y\mid\boldsymbol X]=E[Y] means that Y is mean independent of \boldsymbol X, which is weaker than independence.
The CEF residual is U:=Y-m(\boldsymbol X).
This gives the exact decomposition Y=m(\boldsymbol X)+U.
By construction, the CEF residual has conditional mean zero: \begin{align*} E[U\mid\boldsymbol X] &=E[Y-m(\boldsymbol X)\mid\boldsymbol X] \\ &=E[Y\mid\boldsymbol X]-m(\boldsymbol X)=0. \end{align*}
Together, the LIE and CT then imply \begin{align*} E[r(\boldsymbol X)U] &=E\!\left[E[r(\boldsymbol X)U\mid\boldsymbol X]\right] \\ &=E[r(\boldsymbol X)E[U\mid\boldsymbol X]]=0. \end{align*}
The CEF m(\boldsymbol x) describes how the conditional population mean of Y varies with \boldsymbol x.
Regression models specify or approximate this function. Marginal effects describe how m(\boldsymbol x) changes when a regressor changes.
So far, F_Y, \pi_Y, f_Y, and the CEF have described a hypothetical population.
In practice, the population distribution is usually unknown. We observe a sample Y_1,\ldots,Y_n.
For a discrete variable, relative frequencies provide a sample counterpart of the population PMF: \widehat\pi_Y(a)=\frac{n(a)}{n}, where n(a) is the number of observations equal to a.
For a continuous variable, a histogram divides the range of values into fixed intervals called bins.
Each bar summarizes the observations within one bin.
A kernel density estimate (KDE) can be understood as a smooth moving-window version of a histogram.
Instead of using fixed bins, it moves across the possible values. At each value, it measures the local concentration of observations.
Nearby observations receive more weight than distant observations. The weighting rule is called a kernel.
The bandwidth controls how wide this neighborhood is. A larger bandwidth produces a smoother density curve.
Let’s use the CPS data to estimate region probabilities and visualize the wage density:
cps = read.csv("cps.csv")
n = length(cps$region)
pmf = table(cps$region)/n
par(mfrow = c(1, 2))
barplot(pmf,
names.arg = c("Northeast", "Midwest", "South", "West"),
ylab = "Relative frequency",
cex.names = 0.75)
hist(cps$wage,
breaks = 100,
probability = TRUE,
xlim = c(0, 80),
xlab = "Hourly wage",
ylab = "Density",
main = "")
lines(density(cps$wage, from = 0, to = 80), lwd = 2)
legend("topright", "KDE", lwd = 2, bty = "n")
The bar plot shows relative frequencies for the four regions.
In a density-scaled histogram, each bar’s area equals the relative frequency in its bin.
Across all bins, including wages above the displayed limit of 80, the areas sum to one.