6.4 Limiting Distributions

Sections 6.1 through 6.3 developed methods for approximating smooth functions (exponentials, logarithms) and factorials. In this chapter we will use those tools to derive the distributions for two essential probability models. Both can be derived as limits of the familiar Binomial distribution from Section 2.2.

Recall that $X \sim \text{Binomial}(n,p)$ if $X$ is the total number of successes in $n$ independent, identical binary trials with success probability $p$. Then:

Open the distribution plotter linked below to experiment with the Binomial PMF.

Poisson Distributions¶

We express the statement, $X$ is drawn from a Poisson distribution with parameter $\lambda$:

The Poisson distribution is normalized since, by the Taylor series expansion of the exponential:

Experiment with the Poisson distribution below. Try setting $\lambda = 5$.

You should notice that the result looks roughly Binomial with a mode near $\lambda = 5$. We will recover the Poisson as a limit of the Binomial distribution.

To find the appropriate limit, open the Binomial explorer again.

Now, attempt the following.

1. Gradually increase $n$. Start with $n = 10$, then work upwards.

2. As you increase $n$, decrease $p$. Keep $p \approx 5/n$. This will keep the peak of the Binomial PMF near $x = 5$ even as $n$ increases. We need to keep the peak near $x = 5$ since the Poisson PMF with $\lambda = 5$ was peaked near $x = 5$.

You should see that, as $n$ increases, your PMF looks closer and closer to the PMF for $\text{Poisson}(5)$.

Repeat this experiment for $\lambda = 2$ and $\lambda = 10$. Each time track the Binomial PMF as you increase $n$, while decreasing $p$. You should see that, if you keep $p = \lambda/n$, then, in each case, the sequence of Binomial PMF’s will approach the Poisson PMF.

... as a Binomial Limit¶

This experiment illustrates a limiting relationship between the Poisson and Binomial distributions. This relationship is sometimes called the Law of Small Numbers:

Before proving this law, it’s worth examining the limit statement.

1. Holding $n \times p(n) = \lambda$ is equivalent to keeping $\mathbb{E}[X] = n \times p(n)$ fixed at $\lambda$ as $n$ increases. So, this is a limit where, as the number of trials increases, the chance of success per trial decreases, so that the expected total number of successes remains constant.

2. Like any limiting statement, the Law of Small Numbers is most useful as an approximation. It guarantees that, when $n$ is large, and $p$ is small, then $X \sim \text{Binomial}(n,p)$ is approximately Poisson distributed. This result is useful since the Poisson distribution is easier to work with than the Binomial.

The Law of Small Numbers is often invoked when we ask about the total number of successes in a large number of trials that each rarely succeed. The “small” in small numbers references the idea that, if $p$ is small, then $\mathbb{E}[X] = n p$ is much smaller than $n$. We could have as well named this limiting relationship the “Law of Rare Counts.”

When do limits of this kind occur in practice?

... from Exponential Waiting Times¶

We’ve actually already seen situation where the limit involved in the Law of Small Numbers is sensible.

Recall the random incidents model from Section 6.2:

Suppose that, instead of asking for the time between successive incidents, we ask for the total number of incidents that occur between times 0 and time $t$. Let $X$ denote the total number of incidents that occur. Then:

In other words:

In the setting described above, the limit that appears in the law of small numbers is sensible. The expected value of $X(n)$ should converge to something sensible as $n$ diverges since the number of intervals in the partition, $n$, was an arbitrary number introduced to help analyze $X = \lim_{n \rightarrow \infty} X(n)$.

Proof of the Law of Small Numbers¶

To establish the law of small numbers, we need to show that the Binomial PMF converges to the Poisson PMF in the limit as $n$ diverges, provides $p(n)$ behaves like $\lambda/n$. The support of the Binomial converges to the support of the Poisson since $\{0,1,2,...,n\}$ approaches $\{0,1,2,...,\infty\}$ as $n$ diverges.

Substituting $p(n) = \lambda/n$ into the Binomial PMF gives:

One term is ready for a limit. By the limiting expression for the exponential:

This is the normalizing constant of the Poisson distribution.

Next, we need to work out the limit of:

To find this limit, we will expand it as a product. Let:

denote the ratio of successive values of the PMF. Then, the PMF at $x$ can be expanded:

This form is convenient since:

which converges to the normalizing constant $e^{-\lambda}$ we derived before.

Moreover, each ratio $r$ is simple:

So, if $p = \lambda/n$, then:

Now, if $n$ diverges, $n$ will dominate $- y + 1$, so the first term simplifies to $n/y$, and $n$ will dominate $-\lambda$, so the second term will simplify to $\lambda/n$. Therefore:

So, the PMF at $x$, in the limit of infinite $n$, is:

The right hand side is the Poisson PMF! $\square$

Normal Distributions¶

We express the statement, $X$ is drawn from a standard normal distribution with parameter:

You can experiment with the normal distribution using the code cell below. Notice that, changing the mean parameter translates the density, while changing the standard deviation dilates it.

The normal distribution is an important model since it is achieved as the limit of many other distributions. In particular, if we draw a set of $n$ independent, identical samples from a distribution with finite variance, and compute their sample average, then the sample average will be approximately normally distributed for large $n$.

In this chapter, we’ll recover the formula for the standard normal density:

from a limit of Binomial random variables.

... as a Binomial Limit¶

First, run the code cell below to visualize a Binomial PMF.

This time, keep the success probability, $p$, fixed, and increase $n$. Start with $p = 0.5$ and $n = 4$, then gradually increase $n$. You should see that, even for relatively small $n$, the Binomial PMF approaches a bell curve shape. If you repeat this experiment for $p \neq 0.5$ you’ll see the same result, though the bell curve will start out skewed.

So, consider the Binomial PMF with $p = 0.5$:

So, when $p = 1/2$, the Binomial PMF is proportional to the choose coefficient as a function of $x$:

This means that our analysis of the Binomial PMF will serve two ends at once. First, we will see that, as $n$ diverges, the Binomial PMF produces a normal curve. Second, by showing that the Binomial PMF approaches a normal curve, we will also develop a normal approximation for binomial coefficients.

Before starting, we will have to fix a basic discrepancy between our two models.

If $Z \sim \text{Normal}(0,1)$ then $\mathbb{E}[Z] = 0$ and $\text{Var}[Z] = 1$.

In contrast, if $X_n \sim \text{Binomial}(n,p)$ then $\mathbb{E}[X_n] = n p$ and $\text{Var}[X_n] = n p (1 - p)$. So, $X_n$ cannot converge to $Z$ as $n$ diverges, since the expected value of a Binomial distribution is proportional to $n$ and is nonzero when $p \neq 0$. Worse, its standard deviation grows at rate $\mathcal{O}(\sqrt{n})$.

To fix this issue, we will show that a standardized version of $X$ approaches $Z$. To standardize a random variable, subtract off its mean and divide by its standard deviation (see Section 4.3).

So, let:

We’ve added the subscript “sub $n$” to $Z$ to indicate that $Z_n$ is a standardized Binomial random variable on $n$ trials.

When $p = 0.5$:

In terms of $Z_n$:

Now, the PMF of $Z_n$ is:

Then, using the formula for the Binomial PMF:

To simplify, first expand the Binomial coefficient as a ratio of factorials:

Next, apply Stirling’s approximation to approximate each term:

Substituting each term for its approximation, then cancelling like terms, gives:

Therefore:

When $n$ is large, $n - z^2$ will be dominated by $n$. Therefore, we can make the approximation:

The standard normal random variable, $Z$, is continuously distributed, so is parameterized by a density. Each $Z_n$ is a discrete random variable. To recover a density from a probability, we need to divide out by the length of an interval.

In this case we can construct a density from $Z_n$ by replacing $Z_n$ with a random variable $W_n$, where $W_n|Z_n = z \sim \text{Uniform}(z - \Delta z_n/2, z + \Delta z_n/2)$ where $\Delta z_n$ is the gap between successive possible values of $Z_n$. Since $Z_n = \frac{1}{\sqrt{n}}(2 X_n - n)$, and $X_n$ are integer valued, $\Delta z_n = \frac{2}{\sqrt{n}}$.

Then, work with the density function of $W_n$:

This procedure is the same as:

1. Representing the PMF of $Z_n$ with a bar plot. The widths of the bars is $\Delta z_n$.

2. Scaling the height of the bars by their widths so that their area returns the PMF value. This returns the density function for $W_n$.

Integrating over the density function of $W_n$, with bounds equal to the endpoints of the bars, will sum over the PMF of $Z_n$. So, all probability questions we could ask about $Z_n$ could be answered by integrating over the density function of $W_n$.

Now that we’ve handled the normalizing constants, focus on the functional form:

To take the limit as $n$ goes to infinity, express each term in the form used for the limiting definition of the exponential:

Therefore:

So:

The expression on the left is a PDF since $\Delta z_n$ converges to zero as $n$ diverges. The expression on the right is the standard normal density function. Therefore:

It follows that: