Third and higher order Taylor expansions in several variables

In this post we show that it is possible to derive third and higer-order Taylor expansions for functions of several variables. Given that the gradient of a function f:\mathbb{R}^n \to\mathbb{R} is vector-valued and its Hessian is matrix-valued, it is natural to guess that its third-order gradient will be tensor-valued. However, not only is the use of tensors not very convenient, but in this context it is also unnecessary.
Often Taylor expansions of functions f:\mathbb{R}^n\to\mathbb{R} at a point x\in\mathbb{R}^n are meant along a given direction d\in\mathbb{R}^n. This facilitates a lot out understanding even for first-order expansions.

Let f\in\mathcal{C}^3 and define a function \phi:\mathbb{R}\to\mathbb{R} by \phi(\tau) = f(x+\tau d) which describes function f along a direction. Then \phi is three times continuously differentiable and the third-order Taylor expansion of \phi about \tau=0 is

\begin{aligned} \phi(\tau) = \phi(0) + \tau \phi'(0) + \tfrac{t^2}{2!}\phi''(0) + \tfrac{t^3}{3!}\phi'''(0) + o(t^3). \end{aligned}

But \phi'(0) is related to the directional derivative of f at x along the direction d which is

\begin{aligned} \phi'(0) &= \lim_{h\to 0}\frac{\phi(h) - \phi(0)}{h}\\ &= \lim_{h\to 0}\frac{f(x+h d) - f(x)}{h}\\ &= \langle \nabla f(x), d \rangle \end{aligned}

Let us denote this by \nabla_{d}f(x).

Similary, \phi''(0) can be interpreted as the directional Hessian of f at x along the directions d and d, that is

\begin{aligned} \phi''(0) = \langle \nabla^2 f(x)d, d \rangle \end{aligned}

Let us denote this by \nabla^2_{d,d}f(x).

The term \phi'''(0) – a Terssian if we may call it so – is more difficult to represent. If fact, it will be a tensor. However, we are merely interested in the directional Terssian of f at x along directions d, d and d. This construct is actually used in the context of convex optimization theory and in particular the theory of self-concordant functions and is denoted by \nabla^3_{d,d,d}f(x) and we may write

\begin{aligned} \nabla^3_{d,d,d}f(x) = \langle \nabla^3 f(x)[d]d, d \rangle \end{aligned}

where \nabla^3 f(x) is the third-order gradient of f at x which, in my opinion, is best understood via its directional variant:

\begin{aligned} \nabla^3f(x)[d] = \lim_{h\to 0} \frac{\nabla^2 f(x+\alpha d) - \nabla^2 f(x)}{h} \end{aligned}

Here \nabla^3f(x)[d] is a matrix – it is a directional Hessian. Essentially, \nabla^3f(x)[d] describes how the Hessian of f changes at x along the direction d.

Similarly we may produce fourth and higher-order approximations.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s


Exploring and venting about quantitative issues

Look at the corners!

The math blog of Dmitry Ostrovsky

The Unapologetic Mathematician

Mathematics for the interested outsider

Almost Sure

A random mathematical blog


Mathematix is the mathematician of the village of Asterix

%d bloggers like this: