Background: review of differentiable functions of several variables

3.2. Background: review of differentiable functions of several variables#

We review the differential calculus of several variables. We highlight a few key results that will play an important role: the Chain Rule and the Mean Value Theorem.

3.2.1. Gradient#

Recall the definition of the gradient.

DEFINITION (Gradient) Let $f : D \to R$ where $D \subseteq R^{d}$ and let $x_{0} \in D$ be an interior point of $D$ . Assume $f$ is continuously differentiable at $x_{0}$ . The (column) vector

\nabla f (x_{0}) = (\frac{\partial f (x_{0})}{\partial x_{1}}, \dots, \frac{\partial f (x_{0})}{\partial x_{d}})

is called the gradient of $f$ at $x_{0}$ . $♮$

Note that the gradient is itself a function of $x$ . In fact, unlike $f$ , it is a vector-valued function.

Gradient as a function (with help from ChatGPT; adapted from (Source))

EXAMPLE: Consider the affine function

f (x) = q^{T} x + r

where $x = (x_{1}, \dots, x_{d})^{T}, q = (q_{1}, \dots, q_{d})^{T} \in R^{d}$ . The partial derivatives of the linear term are given by

\frac{\partial}{\partial x_{i}} [q^{T} x] = \frac{\partial}{\partial x_{i}} [\sum_{j = 1}^{d} q_{j} x_{j}] = \frac{\partial}{\partial x_{i}} [q_{i} x_{i}] = q_{i} .

So the gradient of $f$ is

\nabla f (x) = q .

$⊲$

EXAMPLE: Consider the quadratic function

f (x) = \frac{1}{2} x^{T} P x + q^{T} x + r .

where $x = (x_{1}, \dots, x_{d})^{T}, q = (q_{1}, \dots, q_{d})^{T} \in R^{d}$ and $P \in R^{d \times d}$ . The partial derivatives of the quadratic term are given by

\begin{aligned} \frac{\partial}{\partial x_{i}} [x^{T} P x] & = \frac{\partial}{\partial x_{i}} [\sum_{j, k = 1}^{d} P_{j k} x_{j} x_{k}] \\ = \frac{\partial}{\partial x_{i}} [P_{i i} x_{i}^{2} + \sum_{j = 1, j \neq i}^{d} P_{j i} x_{j} x_{i} + \sum_{k = 1, k \neq i}^{d} P_{i k} x_{i} x_{k}], \end{aligned}

where we used that all terms not including $x_{i}$ have partial derivative $0$ .

This last expression is

\begin{aligned} = 2 P_{i i} x_{i} + \sum_{j = 1, j \neq i}^{d} P_{j i} x_{j} + \sum_{k = 1, k \neq i}^{d} P_{i k} x_{k} \\ = \sum_{j = 1}^{d} [P^{T}]_{i j} x_{j} + \sum_{k = 1}^{d} [P]_{i k} x_{k} \\ = ([P + P^{T}] x)_{i} . \end{aligned}

So the gradient of $f$ is

\nabla f (x) = \frac{1}{2} [P + P^{T}] x + q .

If $P$ is symmetric, this further simplifies to $\nabla f (x) = P x + q$ . $⊲$

It will be useful to compute the derivative of a function $f (x)$ of several variables along a parametric curve $g (t) = (g_{1} (t), \dots, g_{d} (t))^{T} \in R^{d}$ for $t$ in some closed interval of $R$ . The following result is a special case of an important fact. We will use the following notation $g^{'} (t) = (g_{1}^{'} (t), \dots, g_{m}^{'} (t))$ , where $g_{i}^{'}$ is the derivative of $g_{i}$ . We say that $g (t)$ is continuously differentiable at $t = t_{0}$ if each of its component is.

EXAMPLE: (Parametric Line) The straight line between $x_{0} = (x_{0, 1}, \dots, x_{0, d})^{T}$ and $x_{1} = (x_{1, 1}, \dots, x_{1, d})^{T}$ in $R^{d}$ can be parametrized as

g (t) = x_{0} + t (x_{1} - x_{0}),

where $t$ goes from $0$ (at which $g (0) = x_{0}$ ) to $1$ (at which $g (1) = x_{1}$ ).

Then

g_{i}^{'} (t) = \frac{d}{d t} [x_{0, i} + t (x_{1, i} - x_{0, i})] = x_{1, i} - x_{0, i},

so that

g^{'} (t) = x_{1} - x_{0} .

$⊲$

Recall the Chain Rule in the single-variable case. Quoting Wikipedia:

The simplest form of the chain rule is for real-valued functions of one real variable. It states that if $g$ is a function that is differentiable at a point $c$ (i.e. the derivative $g^{'} (c)$ exists) and f is a function that is differentiable at $g (c)$ , then the composite function $f \circ g$ is differentiable at $c$ , and the derivative is $(f \circ g)^{'} (c) = f^{'} (g (c)) \cdot g^{'} (c)$ .

Here is a straightforward generalization of the Chain Rule.

THEOREM (Chain Rule) Let $f : D_{1} \to R$ , where $D_{1} \subseteq R$ , and let $g : D_{2} \to R$ , where $D_{2} \subseteq R^{d}$ . Assume that $f$ is continuously differentiable at $g (x_{0})$ , an interior point of $D_{1}$ , and that $g$ is continuously differentiable at $x_{0}$ , an interior point of $D_{2}$ . Then

\nabla (f \circ g) (x_{0}) = f^{'} (g (x_{0})) \nabla g (x_{0}) .

$♯$

Proof: We apply the Chain Rule for functions of one variable to the partial derivatives. For all $i$ ,

\frac{\partial}{\partial x_{i}} f (g (x_{0})) = f^{'} (g (x_{0})) \frac{\partial}{\partial x_{i}} g (x_{0}) .

Collecting the partial derivatives in a vector gives the claim. $◻$

Here is a different generalization of the Chain Rule. Again the composition $f \circ g$ denotes the function $f \circ g (t) = f (g (t))$ .

THEOREM (Chain Rule) Let $f : D_{1} \to R$ , where $D_{1} \subseteq R^{d}$ , and let $g : D_{2} \to R^{d}$ , where $D_{2} \subseteq R$ . Assume that $f$ is continuously differentiable at $g (t_{0})$ , an interior point of $D_{1}$ , and that $g$ is continuously differentiable at $t_{0}$ , an interior point of $D_{2}$ . Then

(f \circ g)^{'} (t_{0}) = \nabla f (g (t_{0}))^{T} g^{'} (t_{0}) .

$♯$

Proof: To simplify the notation, suppose that $f$ is a real-valued function of $x = (x_{1}, \dots, x_{d})$ whose components are themselves functions of $t \in R$ . Assume $f$ is continuously differentiable at $x (t)$ . To compute the total derivative $\frac{d f (t)}{d t}$ , let $Δ x_{k} = x_{k} (t + Δ t) - x_{k} (t)$ , $x_{k} = x_{k} (t)$ and

Δ f = f (x_{1} + Δ x_{1}, \dots, x_{d} + Δ x_{d}) - f (x_{1}, \dots, x_{d}) .

We seek to compute the limit $lim_{Δ t \to 0} \frac{Δ f}{Δ t}$ . To relate this limit to partial derivatives of $f$ , we re-write $Δ f$ as a telescoping sum where each term involves variation of a single variable $x_{k}$ . That is,

\begin{aligned} Δ f = & [f (x_{1} + Δ x_{1}, \dots, x_{d} + Δ x_{d}) - f (x_{1}, x_{2} + Δ x_{2}, \dots, x_{d} + Δ x_{d})] \\ + [f (x_{1}, x_{2} + Δ x_{2}, \dots, x_{d} + Δ x_{d}) - f (x_{1}, x_{2}, x_{3} + Δ x_{3}, \dots, x_{d} + Δ x_{d})] \\ + \dots + [f (x_{1}, \dots, x_{d - 1}, x_{d} + Δ x_{d}) - f (x_{1}, \dots, x_{d})] . \end{aligned}

Applying the Mean Value Theorem to each term gives

\begin{aligned} Δ f = & Δ x_{1} \frac{\partial f (x_{1} + θ_{1} Δ x_{1}, x_{2} + Δ x_{2}, \dots, x_{d} + Δ x_{d})}{\partial x_{1}} \\ + Δ x_{2} \frac{\partial f (x_{1}, x_{2} + θ_{2} Δ x_{2}, x_{3} + Δ x_{3}, \dots, x_{d} + Δ x_{d})}{\partial x_{2}} \\ + \dots + Δ x_{d} \frac{\partial f (x_{1}, \dots, x_{d - 1}, x_{d} + θ_{d} Δ x_{d})}{\partial x_{d}} \end{aligned}

where $0 < θ_{k} < 1$ for $k = 1, \dots, d$ . Dividing by $Δ t$ , taking the limit $Δ t \to 0$ and using the fact that $f$ is continuously differentiable, we get

\frac{d f (t)}{d t} = \sum_{k = 1}^{d} \frac{\partial f (x (t))}{\partial x_{k}} \frac{d x_{k} (t)}{d t} .

$◻$

As a first application of the Chain Rule, we generalize the Mean Value Theorem to the case of several variables. We will use this result later to prove a multivariable Taylor expansion result that will play a central role in this chapter.

THEOREM (Mean Value) Let $f : D \to R$ where $D \subseteq R^{d}$ . Let $x_{0} \in D$ and $δ > 0$ be such that $B_{δ} (x_{0}) \subseteq D$ . If $f$ is continuously differentiable on $B_{δ} (x_{0})$ , then for any $x \in B_{δ} (x_{0})$

f (x) = f (x_{0}) + \nabla f (x_{0} + ξ p)^{T} p

for some $ξ \in (0, 1)$ , where $p = x - x_{0}$ . $♯$

One way to think of the Mean Value Theorem is as a $0$ -th order Taylor expansion. It says that, when $x$ is close to $x_{0}$ , the value $f (x)$ is close to $f (x_{0})$ in a way that can be controlled in terms of the gradient in the neighborhood of $x_{0}$ . From this point of view, the term $\nabla f (x_{0} + ξ p)^{T} p$ is called the Lagrange remainder.

Proof idea: We apply the single-variable result and the Chain Rule.

Proof: Let $ϕ (t) = f (α (t))$ where $α (t) = x_{0} + t p$ . Observe that $ϕ (0) = f (x_{0})$ and $ϕ (1) = f (x)$ . By the Chain Rule and the parametric line example,

ϕ^{'} (t) = \nabla f (α (t))^{T} α^{'} (t) = \nabla f (α (t))^{T} p = \nabla f (x_{0} + t p)^{T} p .

In particular, $ϕ$ has a continuous first derivative on $[0, 1]$ . By the Mean Value Theorem in the single-variable case

ϕ (t) = ϕ (0) + t ϕ^{'} (ξ)

for some $ξ \in (0, t)$ . Plugging in the expressions for $ϕ (0)$ and $ϕ^{'} (ξ)$ and taking $t = 1$ gives the claim. $◻$

3.2.2. Second-order derivatives#

One can also define higher-order derivatives. We start with the single-variable case, where $f : D \to R$ with $D \subseteq R$ and $x_{0} \in D$ is an interior point of $D$ . Note that, if $f^{'}$ exists in $D$ , then it is itself a function of $x$ . Then the second derivative at $x_{0}$ is

f^{″} (x_{0}) = \frac{d^{2} f (x_{0})}{d x^{2}} = lim_{h \to 0} \frac{f^{'} (x_{0} + h) - f^{'} (x_{0})}{h}

provided the limit exists.

In the several variable case, we have the following:

DEFINITION (Second Partial Derivatives and Hessian) Let $f : D \to R$ where $D \subseteq R^{d}$ and let $x_{0} \in D$ be an interior point of $D$ . Assume that $f$ is continuously differentiable in an open ball around $x_{0}$ . Then $\partial f (x) / \partial x_{i}$ is itself a function of $x$ and its partial derivative with respect to $x_{j}$ , if it exists, is denoted by

\frac{\partial^{2} f (x_{0})}{\partial x_{j} \partial x_{i}} = lim_{h \to 0} \frac{\frac{\partial f}{\partial x_{i}} (x_{0} + h e_{j}) - \frac{\partial f}{\partial x_{i}} (x_{0})}{h} .

To simplify the notation, we write this as $\partial^{2} f (x_{0}) / \partial x_{i}^{2}$ when $j = i$ . If $\partial^{2} f (x) / \partial x_{j} \partial x_{i}$ and $\partial^{2} f (x) / \partial x_{i}^{2}$ exist and are continuous in an open ball around $x_{0}$ for all $i, j$ , we say that $f$ is twice continuously differentiable at $x_{0}$ .

The matrix of second derivatives is called the Hessian and is denoted by

\begin{array}{r} H_{f} (x_{0}) = (\begin{array}{c} \frac{\partial^{2} f (x_{0})}{\partial x_{1}^{2}} & \dots & \frac{\partial^{2} f (x_{0})}{\partial x_{d} \partial x_{1}} \\ ⋮ & ⋱ & ⋮ \\ \frac{\partial^{2} f (x_{0})}{\partial x_{1} \partial x_{d}} & \dots & \frac{\partial^{2} f (x_{0})}{\partial x_{d}^{2}} \end{array}) . \end{array}

$♮$

Like $f$ and the gradient $\nabla f$ , the Hessian $H_{f}$ is a function of $x$ . It is a matrix-valued function however.

When $f$ is twice continuously differentiable at $x_{0}$ , its Hessian is a symmetric matrix.

THEOREM (Symmetry of the Hessian) Let $f : D \to R$ where $D \subseteq R^{d}$ and let $x_{0} \in D$ be an interior point of $D$ . Assume that $f$ is twice continuously differentiable at $x_{0}$ . Then for all $i \neq j$

\frac{\partial^{2} f (x_{0})}{\partial x_{j} \partial x_{i}} = \frac{\partial^{2} f (x_{0})}{\partial x_{i} \partial x_{j}} .

$♯$

Proof idea: Two applications of the Mean Value Theorem show that the limits can be interchanged.

Proof: By definition of the partial derivative,

\begin{aligned} \frac{\partial^{2} f (x_{0})}{\partial x_{j} \partial x_{i}} & = lim_{h_{j} \to 0} \frac{\frac{\partial f}{\partial x_{i}} (x_{0} + h_{j} e_{j}) - \frac{\partial f}{\partial x_{i}} (x_{0})}{h_{j}} \\ = lim_{h_{j} \to 0} lim_{h_{i} \to 0} \frac{1}{h_{j} h_{i}} {[f (x_{0} + h_{j} e_{j} + h_{i} e_{i}) - f (x_{0} + h_{j} e_{j})] - [f (x_{0} + h_{i} e_{i}) - f (x_{0})]} \\ = lim_{h_{j} \to 0} lim_{h_{i} \to 0} \frac{1}{h_{i}} {\frac{[f (x_{0} + h_{i} e_{i} + h_{j} e_{j}) - f (x_{0} + h_{i} e_{i})] - [f (x_{0} + h_{j} e_{j}) - f (x_{0})]}{h_{j}}} \\ = lim_{h_{j} \to 0} lim_{h_{i} \to 0} \frac{1}{h_{i}} {\frac{\partial}{\partial x_{j}} [f (x_{0} + h_{i} e_{i} + θ_{j} h_{j} e_{j}) - f (x_{0} + θ_{j} h_{j} e_{j})]} \\ = lim_{h_{j} \to 0} lim_{h_{i} \to 0} \frac{1}{h_{i}} {\frac{\partial f}{\partial x_{j}} (x_{0} + h_{i} e_{i} + θ_{j} h_{j} e_{j}) - \frac{\partial f}{\partial x_{j}} (x_{0} + θ_{j} h_{j} e_{j})} \end{aligned}

for some $θ_{j} \in (0, 1)$ . Note that, on the third line, we rearranged the terms and, on the fourth line, we applied the Mean Value Theorem to $f (x_{0} + h_{i} e_{i} + h_{j} e_{j}) - f (x_{0} + h_{j} e_{j})$ as a continuously differentiable function of $h_{j}$ .

Because $\partial f / \partial x_{j}$ is continuously differentiable in an open ball around $x_{0}$ , a second application of the Mean Value Theorem gives for some $θ_{i} \in (0, 1)$

\begin{aligned} lim_{h_{j} \to 0} lim_{h_{i} \to 0} \frac{1}{h_{i}} {\frac{\partial f}{\partial x_{j}} (x_{0} + h_{i} e_{i} + θ_{j} h_{j} e_{j}) - \frac{\partial f}{\partial x_{j}} (x_{0} + θ_{j} h_{j} e_{j})} \\ = lim_{h_{j} \to 0} lim_{h_{i} \to 0} \frac{\partial}{\partial x_{i}} [\frac{\partial f}{\partial x_{j}} (x_{0} + θ_{j} h_{j} e_{j} + θ_{i} h_{i} e_{i})] \\ = lim_{h_{j} \to 0} lim_{h_{i} \to 0} \frac{\partial^{2} f (x_{0} + θ_{j} h_{j} e_{j} + θ_{i} h_{i} e_{i})}{\partial x_{i} \partial x_{j}} . \end{aligned}

The claim then follows from the continuity of $\partial^{2} f / \partial x_{i} \partial x_{j}$ . $◻$

EXAMPLE: Consider the quadratic function

f (x) = \frac{1}{2} x^{T} P x + q^{T} x + r .

Recall that the gradient of $f$ is

\nabla f (x) = \frac{1}{2} [P + P^{T}] x + q .

To simplify the calculation, let $B = \frac{1}{2} [P + P^{T}]$ and denote the rows of $B$ by $b_{1}^{T}, \dots, b_{d}^{T}$ .

Each component of $\nabla f$ is an affine function of $x$ , specifically,

\frac{\partial f (x)}{\partial x_{i}} = b_{i}^{T} x + q_{i} .

Row $i$ of the Hessian is simply the gradient transposed of $\frac{\partial f (x)}{\partial x_{i}}$ which, by our previous results, is

{(\nabla \frac{\partial f (x)}{\partial x_{i}})}^{T} = b_{i}^{T} .

Putting this together we get

H_{f} (x) = \frac{1}{2} [P + P^{T}] .

Observe that this is indeed a symmetric matrix. $⊲$

Self-assessment quiz (with help from Claude, Gemini, and ChatGPT)

1 What does it mean for a function $f$ to be continuously differentiable at $x_{0}$ ?

a) $f$ is continuous at $x_{0}$ .

b) All partial derivatives of $f$ exist at $x_{0}$ .

c) All partial derivatives of $f$ exist and are continuous in an open ball around $x_{0}$ .

d) The gradient of $f$ is zero at $x_{0}$ .

2 What is the gradient of a function $f : D \to R$ , where $D \subseteq R^{d}$ , at a point $x_{0} \in D$ ?

a) The rate of change of $f$ with respect to $x$ at $x_{0}$

b) The vector of all second partial derivatives of $f$ at $x_{0}$

c) The vector of all first partial derivatives of $f$ at $x_{0}$

d) The matrix of all second partial derivatives of $f$ at $x_{0}$

3 Which of the following statements is true about the Hessian matrix of a twice continuously differentiable function?

a) It is always a diagonal matrix.

b) It is always a symmetric matrix.

c) It is always an invertible matrix.

d) It is always a positive definite matrix.

4 Let $f (x, y, z) = x^{2} + y^{2} - z^{2}$ . What is the Hessian matrix of $f$ ?

a) $(\begin{matrix} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & - 2 \end{matrix})$

b) $(\begin{matrix} 2 x & 2 y & - 2 z \end{matrix})$

c) $(\begin{matrix} 2 & 0 \\ 0 & 2 \end{matrix})$

d) $(\begin{matrix} 0 & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{matrix})$

5 What is the Hessian matrix of the quadratic function $f (x) = \frac{1}{2} x^{T} P x + q^{T} x + r$ , where $P \in R^{d \times d}$ and $q \in R^{d}$ ?

a) $H_{f} (x) = P$

b) $H_{f} (x) = P^{T}$

c) $H_{f} (x) = \frac{1}{2} [P + P^{T}]$

d) $H_{f} (x) = [P + P^{T}]$

Answer for 1: c. Justification: The text states, “If $f$ exists and is continuous in an open ball around $x_{0}$ for all $i$ , then we say that $f$ is continuously differentiable at $x_{0}$ .”

Answer for 2: c. Justification: From the text: “The (column) vector $\nabla f (x_{0}) = (\frac{\partial f (x_{0})}{\partial x_{1}}, \dots, \frac{\partial f (x_{0})}{\partial x_{d}})$ is called the gradient of $f$ at $x_{0}$ .”

Answer for 3: b). Justification: The text states: “When $f$ is twice continuously differentiable at $x_{0}$ , its Hessian is a symmetric matrix.”

Answer for 4: a). Justification: The Hessian is the matrix of second partial derivatives:

\begin{array}{r} (\begin{array}{c} \frac{\partial^{2} f}{\partial x^{2}} & \frac{\partial^{2} f}{\partial x \partial y} & \frac{\partial^{2} f}{\partial x \partial z} \\ \frac{\partial^{2} f}{\partial y \partial x} & \frac{\partial^{2} f}{\partial y^{2}} & \frac{\partial^{2} f}{\partial y \partial z} \\ \frac{\partial^{2} f}{\partial z \partial x} & \frac{\partial^{2} f}{\partial z \partial y} & \frac{\partial^{2} f}{\partial z^{2}} \end{array}) = (\begin{array}{c} 2 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & - 2 \end{array}) \end{array}

Answer for 5: c. Justification: The text shows that the Hessian of the quadratic function is $H_{f} (x) = \frac{1}{2} [P + P^{T}]$ .

Background: review of differentiable functions of several variables

Contents

3.2. Background: review of differentiable functions of several variables#

3.2.1. Gradient#

3.2.2. Second-order derivatives#