Chapter Overview¶
Where We’re Going¶
In this chapter, we will answer the question, “In what sense are sample averages reliable estimators for unknown expectations?” In particular, “Are sample averages consistent estimators for unknown expectation?” If so, then we can guarantee that sample averages will converge to expectations in the limit of infinitely many sample draws.
This question is a natural stopping point for the course. It ties together ideas we’ve touched on over the whole semester. In particular, answering this question will justify the estimators we proposed in Section 12. It will also justify two of the fundamental ideas we’ve taken for granted throughout the course:
Expected values equal the value a long run average will approach
Chances equal long run frequencies.
The first statement is practically important for most applied uses of probability in economics, finance, statistics, data science, and machine learning. If long run averages did not become more stable as we collected more data, then we would lose one of the key tools needed to reduce uncertainty by collecting more observations.
The second assertion is a special case of the former, since frequencies are sample averages of indicators. It is absolutely essential. It is the statement that assigns a concrete meaning to chance. Without the second statement, chances would not be measurable under any hypothetical measurement process and could only reflect subjective evaluations of uncertainty.
We will:
Find the variance of sums and sample averages (see Section 13.1).
Expand the variance of a sum in terms of the variance of each term in the sum, and the covariance between the terms.
Show that the variance of a sample average depends on the association between the samples averaged, is more variable if the samples are positively associated.
Show that, if the samples are sufficiently uncorrelated, then the variance in the sample average decreases as the number of samples increases.
Show that, in the special case when the samples are independent, the variance in the sample average on samples is proportional to .
Prove that sample averages of sufficiently uncorrelated samples are consistent estimators for unknown expectations in the sense that the mean square error in the sample average converges to zero as the number of samples increases.
Use the variance in a random variable to introduce a bound on the probability that it differs significantly from its expectation (see Section 13.2).
We will prove Markov’s inequality, then use it to derive Chebyshev’s inequality
Combine our tools to prove the weak law of large numbers, which says that, the probability that a sample average differs from its expectation by more than a fixed threshold, converges to zero, in the limit of infinitely many samples, no matter the threshold (see Section 13.3).
We will use this result to justify the interpretation of an expectation as a long run sample average, and
To justify the definition of chance as long run frequency.
Show how to use Chebyshev’s inequality to
Study the asymptotic distribution of sample averages (see Section 13.3).
We will observe the Central Limit Theorem (CLT), which states that, long run sample averages are approximately normally distributed.
Prove the CLT in some special cases by applying the convolution formula to solve explicitly for the distribution of a long run sample average. We will focus on averages of Bernoulli random variables and exponential random variables.