# Third and higher order Taylor expansions in several variables

In this post we show that it is possible to derive third and higer-order Taylor expansions for functions of several variables. Given that the gradient of a function $f:\mathbb{R}^n \to\mathbb{R}$ is vector-valued and its Hessian is matrix-valued, it is natural to guess that its third-order gradient will be tensor-valued. However, not only is the use of tensors not very convenient, but in this context it is also unnecessary.
Often Taylor expansions of functions $f:\mathbb{R}^n\to\mathbb{R}$ at a point $x\in\mathbb{R}^n$ are meant along a given direction $d\in\mathbb{R}^n$. This facilitates a lot out understanding even for first-order expansions.

Let $f\in\mathcal{C}^3$ and define a function $\phi:\mathbb{R}\to\mathbb{R}$ by $\phi(\tau) = f(x+\tau d)$ which describes function $f$ along a direction. Then $\phi$ is three times continuously differentiable and the third-order Taylor expansion of $\phi$ about $\tau=0$ is

\begin{aligned} \phi(\tau) = \phi(0) + \tau \phi'(0) + \tfrac{t^2}{2!}\phi''(0) + \tfrac{t^3}{3!}\phi'''(0) + o(t^3). \end{aligned}

But $\phi'(0)$ is related to the directional derivative of $f$ at $x$ along the direction $d$ which is

\begin{aligned} \phi'(0) &= \lim_{h\to 0}\frac{\phi(h) - \phi(0)}{h}\\ &= \lim_{h\to 0}\frac{f(x+h d) - f(x)}{h}\\ &= \langle \nabla f(x), d \rangle \end{aligned}

Let us denote this by $\nabla_{d}f(x)$.

Similary, $\phi''(0)$ can be interpreted as the directional Hessian of $f$ at $x$ along the directions $d$ and $d$, that is

\begin{aligned} \phi''(0) = \langle \nabla^2 f(x)d, d \rangle \end{aligned}

Let us denote this by $\nabla^2_{d,d}f(x)$.

The term $\phi'''(0)$ – a Terssian if we may call it so – is more difficult to represent. If fact, it will be a tensor. However, we are merely interested in the directional Terssian of $f$ at $x$ along directions $d$, $d$ and $d$. This construct is actually used in the context of convex optimization theory and in particular the theory of self-concordant functions and is denoted by $\nabla^3_{d,d,d}f(x)$ and we may write

\begin{aligned} \nabla^3_{d,d,d}f(x) = \langle \nabla^3 f(x)[d]d, d \rangle \end{aligned}

where $\nabla^3 f(x)$ is the third-order gradient of $f$ at $x$ which, in my opinion, is best understood via its directional variant:

\begin{aligned} \nabla^3f(x)[d] = \lim_{h\to 0} \frac{\nabla^2 f(x+\alpha d) - \nabla^2 f(x)}{h} \end{aligned}

Here $\nabla^3f(x)[d]$ is a matrix – it is a directional Hessian. Essentially, $\nabla^3f(x)[d]$ describes how the Hessian of $f$ changes at $x$ along the direction $d$.

Similarly we may produce fourth and higher-order approximations.

mathbabe

Exploring and venting about quantitative issues

Look at the corners!

The math blog of Dmitry Ostrovsky

The Unapologetic Mathematician

Mathematics for the interested outsider

Almost Sure

A random mathematical blog

Mathematix

Mathematix is the mathematician of the village of Asterix