9.2 Gradients

In Secion 9.1 we suggested that organizing all the partial derivatives of $f$ into a vector might be useful. In this section, we will show that storing all the partial derivatives in a vector is a very good idea. This vector is called a gradient.

Gradients¶

In other words, the gradient of a surface at a point is the ordered collection of all partial derivatives of the surface at the point.

The animation below illustrates the gradient of the example surface introduced in Sections 8.2 and 9.1. The gradient is built by first finding the slope of the $x$ and $y$ cross-sections. These are the partial derivatives. Storing the partial derivatives in a vector produced the gradient. The gradient vector is shown in white, with side-lengths equal to the partial derivatives. The animation finishes by appending the level set passing through the point where the gradient is evaluated.

This animation reveals a pair of compelling geometric facts about the gradient.

1. When we view the surface as a heatmap, the gradient points uphill. In fact, it points in the direction of steepest ascent.

2. The gradient is perpendicular to the level set passing through the point where it is evaluated. The white arrow representing the gradient is perpendicular to the white contour at the end of the animation shown above.

These two facts suggest a new interpretation of the arrows you’ve appended to contour plots to indicate the uphill direction. Those arrows were parallel to the gradient!

We’ll dive into the geometric interpretation in a moment. For now, let’s run some examples.

Examples¶

Here are the three examples from Section 9.2:

1. $f(x,y) = x + 7 y - 3$ find $\nabla f(x,y)$.

2. $f(x,y) = x^2 \times (1 + y^3)$ find $\nabla f(x,y)$.

2. $f(x,y) = \log(x - y)$ find $\nabla f(x,y)$.

Vector-Valued Functions and Vector-Fields¶

The animation below shows how the gradient changes as the point where it is evaluated changes. Varying the point where the gradient is evaluated changes its length and direction.

Since the gradient accepts all inputs $x$, and, for every input $x$, returns a vector with the same number of elements as $x$, the gradient of a surface defines a vector field.

Vector fields are easiest to picture in two-dimensions. In two-dimensions, a vector field can be pictured as a collection of arrows, one for each location in the plane. You can imagine these as the arrows pointing uphill on a surface, or as the velocity of some flowing medium. Not all vector fields are the gradient of some surface, but the gradient of any surface is a vector field.

Run the code cell below to visualize a surface, it’s level sets, and the gradient vector field. The blue arrows on the $x,y$ plane are the gradients. The solid red arrow is the gradient at the input location $[x_0,y_0]$ The red curve is the level-set passing through $[x_0,y_0]$. Try varying the input point where the gradient is evaluated by changing $x_0$ and $y_0$.

Linearization via Gradients¶

In Section 9.1 we saw that the tangent plane to a surface $f$ at some input $[x_*,y_*]$ is produced by its linear approximation:

Click “Show Tangent Plane” in the demo above to visualize the tangent plane.

We can write the linearization more concisely using the gradient:

Note the analogy to linearization in one-dimension. In one-dimension:

The $d$-dimensional formula replaces $f'(x_*)$ with the gradient, $\nabla f(x_*)$, and uses an inner product to collate the linear approximations across all $d$ dimensions.

The animation below illustrates this process for our example surface. It shows the tangent plane (white) to the surface about an input point. The tangent plane is the plane containing the linear approximation to the $x$ and $y$ cross-sections passing through the input point (orange and green).

Directional Derivatives¶

The partial derivatives of a surface each evaluate the slope of the surface when only one input varies. Varying only one input moves along a line in the input space. For partial derivatives, these lines are parallel to a single coordinate axis.

We can generalize this idea:

We can work out directional derivatives directly, or by using gradients to linearize $f$ about $x$. Let’s try both approaches and see that they give the same answer. We’ll work with the example function:

the input point $[x,y] = [1,-3]$ and the direction $v = [-5,6]$.

Directly¶

First, set up the derivative.

Then, evaluate:

At $[x,y] = [-1,3]$:

Using Gradients¶

We can rearrange the same calculation to find a more efficient approach. Consider the first equality after performing the derivative. Each term was either multiplied by -5 or by 6. These numbers didn’t show up by accident. They were the entries of $v = [-5,6]$. Tracing their source, they appeared each time we applied the chain rule, and either took a derivative of $x - 5 t$ or $y + 6 t$ with respect to $t$. So, we can rewrite the directional derivative:

So, this directional derivative of $f$ with respect to $v$ equals an inner product between some vector-valued function of the input point, and the direction $v$.

Inspect the terms in the vector-valued function of $f$. They look a lot like partial derivatives. Indeed:

So, the vector-valued function is the gradient of $f$!

It follows that:

The equation, $\partial_v f(x,y) = \nabla f(x,y) \cdot v$ is too nice to be a coincidence. This equation holds for all directional derivatives where the gradient exists:

Here’s the calculation of the directional derivative using gradients:

1. $\nabla f(x,y) = [2 x y + 3, x^2]$ so $\nabla f(1,-3) = [-6 + 3,1] = [-3,1]$.

2. $v = [-5,6]$

3. $\partial_v f(1,-3) = [-3,1] \cdot [-5,6] = 15 + 6 = 21$.

This approach is more organized and almost always faster. It will also help us prove the two geometric observations we made at the start of this chapter.

It also provides a general chain rule for surfaces. If $x(t) = [x_1(t),x_2(t),...,x_d(t)]$ is a vector-valued function of some scalar variable $t$, then we can think about $x(t)$ as a point moving along a path, whose position on the path depends on the time $t$. Then, at any instant, the velocity of the point is $v(t) = \frac{d}{dt} x(t) = [\frac{d}{dt} x_1(t), \frac{d}{dt} x_2(t),..., \frac{d}{dt} x_d(t)]$. Then, using the directional derivative:

Gradient Geometry¶

The gradient is a vector, so, like all vectors, it has a length (magnitude) and a direction. We’ve observed two geometric facts about the direction of the gradient vector:

1. The gradient points in the direction of steepest ascent.

2. The gradient is perpendicular to the level set of $f$ about the point where it is evaluated.

Let’s prove these facts using the equation:

Along the way, we’ll work out an interpretation for the magnitude of the gradient. In all cases, we’ll work with the geometric interpretation of the inner-product introduced in Section 8.1:

where $\theta$ is the angle between the gradient, and the direction $v$.

Steepest Ascent¶

First, let’s find the direction of steepest ascent. To find the direction of steepest ascent at $x$, we should look for a direction $v$, that maximizes $\partial_v f(x)$.

Here we have a slight problem. Since

we can make the directional derivative along any direction arbitrarily large by picking $\|v\|$ arbitrarily large. If $f(x)$ increases along a direction $v$, then we can always make the slope along $v$ larger by increasing $v$.

The last sentence may have seemed strange. It seems strange since it is reveals an inconsistency between the definition we used to derive the chain rule for surfaces, and the geometry we expect a directional derivative to encode. The directional derivative really should be independent of length. Here’s an improved definition:

Using this definition, all vectors pointing in the same direction will produce the same directional derivative, no matter their length, since they all share the same unit vector. This definition is consistent with the definition we used before, provided we restricted our input directions to unit vectors.

In either case, we can now write the directional derivative more simply:

This definition is much better. Now the directional derivative depends only on the gradient vector, which controls the linearization of $f$, and the direction of $v$. It is independent of the magnitude of $v$.

Now, to find the direction of steepest ascent at $x$, we want to maximize $\partial_v f(x) = \|\nabla f(x) \| \cos(\theta)$. Since $x$ and $f$ are fixed, we cannot change the gradient term. So, the only term to maximize is the cosine term. Remember, the angle $\theta$ depends on our choice of $v$.

To maximize $\cos(\theta)$ our only option is $\theta = 0$. If $\theta = 0$, then the direction $v$ and the gradient $\nabla f(x)$ are parallel. So, the best choice for $v$ is $v \propto \nabla f(x)$!

Our analysis above also provides a nice interpretation for the magnitude of the gradient vector. If $v \propto \nabla f(x)$, then $v$ and $\nabla f(x)$ are parallel, so $\theta = 0$. If $\theta = 0$ then $\cos(\theta) = 1$, so:

Therefore:

These two facts make the gradient a wonderfully interpretible object. At every $x$, $\nabla f(x)$ is a vector pointing in the direction of steepest ascent, with magnitude equal to the fastest possible rate of ascent. The longer the gradient vector, the steeper the surface. The shorter the gradient vector, the shallower the surface.

Level Sets and Gradients¶

To show that the gradient, $\nabla f(x)$, is perpendicular to the level set of $f$ at $x$, we will again use the equation:

Along a level set, the surface stays at a constant height. So, if $v$ points along a level set, then the directional derivative of $f$ along $v$ must be zero. You can think about this statement topographically. If you walk along a path that stays at the same elevation, then your elevation will never change, so the slope of the ground along the path must equal zero.

Therefore, if $v$ is parallel to a level set, then:

If $\nabla f(x)$ is a nonzero vector, then $\|\nabla f(x)\| \neq 0$, so $\cos(\theta) = 0.$ As in Section 8.1, the cosine of the angle between two nonzero vectors can only equal zero if they are perpendicular to one another. Therefore:

This rule is incredibly useful when we want to visualize gradients. Given a contour plot, the gradient is easy to draw. Just add a vector field perpendicular to the level sets!

Run the code cell below to check that the gradient vector field is, as shown above, always perpendicular to the contours of each contour plot.