7.1 Integrals of Sums and Products

Sums¶

Suppose that $f(x) = a g(x) + b h(x)$ for some constants $a$ and $b$, and some factors $g(x)$ and $h(x)$. What is $\sum_{\text{all } x} f(x)$? What about $\int_{\text{all } x} f(x) dx$?

Integrals and sums are linear operations. That means that, like the expectation, the integral of a sum is the sum of the integrals, the integral of a linear function is the linear function of the integral, and integrals of linear combinations are linear combinations of integrals. The same rules apply for summations. In all cases, the area under the sum of two curves is the sum of the area under each curve.

Notice the similarity to linearity of expectation. Expectations are linear since they are weighted averages, and averages are sums/integrals.

Products¶

Anytime we go to evaluate an expectation we need to compute a sum, or integral, of a product of two functions (typically, the function of $X$ whose expectation we are interested in, and the distribution function of $X$).

So, suppose $f(x) = g(x) \times h(x)$. What is $\sum_{\text{all } x} f(x)$? What about $\int_{\text{all }x} f(x) dx$?

We’ll state the general rule for integrals, provide some examples, then will illustrate an analogous rule for sums.

Integration by Parts¶

Integration by parts swaps which factor is differentiated. It replaces $h'$ with $h$ inside the integral, and replaces $g$ with $g'$. In essence, we are moving the derivative from $h$ to $g$. To move the derivative we will need to know $h$, which is the anti-derivative of $h'$. In probability $h'$ is often a PDF, so $h$ is often a CDF.

To apply integration by parts, always apply the same steps. These are:

1. Write $f$ as a product of two functions: $f(x) = g(x) h'(x)$. Look for a pair of factors where:

   • $g'(x)$ is a simpler function than $g(x)$

   • $h'(x)$ is easy to integrate to recover $h(x)$. If we can’t find the anti-derivative, then we won’t be able to integrate by parts.

2. Find $g'(x) = \frac{d}{dx} g(x)$.

3. Find $h(x) = \int^x h'(s) ds$.

4. Try to evaluate $\int_{a}^b g'(x) h(x) dx$.

5. Combine terms.

Tail Integrals¶

Any time we ask for the expectation of a continuous, nonnegative random variable we need to evaluate the integral $\int_{x = 0}^{b} x \text{PDF}(x) dx$ where $b$ is the upper endpoint of the support. In this case integration by parts produces an alternate formula for the expectation using the area under the survival function.

Setting $g(x) = x$ gives $g'(x) = 1$. Setting $h'(x) = \text{PDF}(x)$ sets $h(x) = \text{CDF}(x)$. Then, integrating by parts:

This formula holds for any $b$, so holds for any nonnegative random variable. We can extend the upper bound of the integral to infinity since $\text{Pr}(X > x) = 0$ if $x$ is above the upper bound of the support.

For example, if $X \sim \text{Exponential}(\lambda)$ then $\text{Pr}(X > x) = e^{-\lambda x}$ so $\mathbb{E}[X] = \int_{x = 0}^{\infty} e^{-\lambda x} = \frac{1}{\lambda}.$

Summation by Parts¶

Most integral rules match a rule for sums. The integration by parts strategy can also be used for sums of products.

Consider a sum of the form

for $n$ possibly infinite.

The discrete analog to a derivative is a difference. Let:

Then, just as derivatives are inverted by integrals, differences are inverted by running sums:

So, we can write our original sum:

To recover the discrete analog to integration by parts, reverse the order of the sum. First, notice that the sum runs over all pairs $x, y$ such that $1 \leq y \leq x \leq n$. We can write this sum:

Since $g'(y)$ does not depend on $x$:

The sum, $\sum_{x = y}^n h(x)$ is analogous to an anti-derivative. To make the analogy explicit, let $H(y) = \sum_{x = 1}^{y-1} h(x)$. Then $\sum_{x = y}^n h(x) = H(n+1) - H(y)$ so:

Finally, since $\sum_{y=1}^n g'(x) = g(n) - g(0)$:

The summation by parts formula is perfectly analogous to the integration by parts formula. You’ll prove integration by parts on your homework using the same steps, but replacing differencing with derivatives, and summation with integration.

Tail Sums¶

Summation by parts is most often used to find the expectation of count valued random variables.

Suppose that $X \in \{0,1,2,3,...,n\}$. Then:

where $n$ is possibly infinite.

Then, let:

Then:

and:

It follows that:

The tail sum formula is often proved using a different algebraic argument. Consider the sum:

For concision, let $p(x) = \text{PMF}(x)$. Then the sum has the form:

We can organize this sum:

Summing down the columns instead of across the rows returns the tail sum formula:

This proof is less general, but more transparent.