Chapter 1: Introduction to Numerical Differentiation

When working with a function 𝑓 defined on an equidistant grid, calculating its derivative at the same grid points is essential. In previous discussions, I explored utilizing the Fast Fourier Transform for this purpose. Today, I will shift focus to employing matrices and interpolating polynomials for differentiation.

To derive a numerical differentiation scheme, we will evaluate the derivative at a specific grid point 𝑥_𝑘 by utilizing that point along with its two neighboring points on either side. Subsequently, we will construct a unique polynomial that passes through these points using Lagrange's interpolation method, which I elaborated on in a prior article. With the interpolating polynomial established, we can differentiate it analytically and then substitute our grid points.

For a practical implementation, we will utilize Python's SymPy library to define our symbols:

from sympy import symbols

x = symbols('x')

Next, we will create a function to perform polynomial interpolation according to Lagrange's method:

def lagrange_interpolation(points):

# Implementation of Lagrange interpolation goes here

The result will yield an interpolant 𝑝(𝑥), and the derivative of this interpolant can be expressed as follows:

p_prime = p.diff(x)

To evaluate this derivative at a grid point, we will substitute for 𝑥 and the grid points accordingly. This leads us to approximate the derivative at the grid.

If we extend our interpolation to utilize five points by adjusting the offsets from [-1, 0, 1] to [-2, -1, 0, 1, 2], the resulting calculations may be complex, but the derivative's value at the grid can be clearly defined. With five interpolation points, we obtain a polynomial of degree 4, suggesting our discretized derivative's accuracy is of the order O(Δ𝑥⁴).

To represent all grid points as column vectors, our aim is to express the corresponding derivative vector in terms of a differentiation matrix 𝐷, which acts on the grid points vector.

Ignoring the values of the first and last rows for the moment, this leads to the following expression:

D.dot(grid_vector)

A challenge arises with the first and last rows, as shifting non-zero entries upward would eventually drop values from the matrix. Two solutions exist: one is to compute coefficients that are asymmetrical using one-sided offsets, while the other is to apply periodic boundary conditions. By asserting that the function is periodic, coefficients can "wrap around," resulting in a circulant Toeplitz matrix. This matrix features constant terms along the diagonals, which are circulant, leading to a mostly sparse matrix. Such a structure offers significant speed and memory advantages when using sparse matrix libraries.

Despite the seeming limitation of only working with periodic functions, this assumption holds true in many physical scenarios.

Let's proceed to construct differentiation matrices using Python code, leveraging the sparse matrix functions available in SciPy. The following function, sparse_circulant, will generate the circulant matrix based on the first column input.

import numpy as np

from scipy.sparse import spdiags

def sparse_circulant(column):

"""Creates a circulant matrix in sparse form using the first column."""

nrows = len(column)

e = np.ones(nrows)

diags_data = []

diags_offsets = []

# Build the lower triangle of the matrix

for off, c in enumerate(column):

if c != 0:

diags_data.append(c * e)

diags_offsets.append(-off)

matrix = spdiags(diags_data, diags_offsets)

return matrix - matrix.T # Since the matrix is skew-symmetric

The differentiation matrix can be constructed using the diff_matrix function, which takes various parameters.

def diff_matrix(deriv, coefs, offsets, npoints, h):

"""Constructs the differentiation matrix."""

column = np.zeros(npoints)

for off, coef in zip(offsets, coefs):

if off > 0:

column[npoints-off] = coef

else:

column[-off] = coef

return sparse_circulant(column) / h**deriv

To verify this implementation, we can apply it to a small grid with a grid spacing of 1.

# Adjust numpy output options for clear matrix display

np.set_printoptions(edgeitems=30, linewidth=100000,

formatter=dict(float=lambda x: "%8.3f" % x))

N = 8 # Number of grid points

h = 1 # Grid spacing

print(diff_matrix(1, [-1./12, 8./12, -8./12, 1./12], [-2, -1, 1, 2], N, h).toarray())

The output should appear correct. Now, we will apply this method to a larger grid to compute the maximum error in the derivative of the function based on the grid size.

Ns = np.logspace(1, 4, 20, dtype=int)

def calc_errors(coefs, offsets):

errs = []

deriv = 1 # First derivative

for N in Ns:

# Set up the grid

h = 2 * np.pi / N

x = np.arange(1, N + 1) * h

f = np.exp(np.sin(x))

# Exact derivative

f1_ex = np.cos(x) * f

# Approximate derivative

D = diff_matrix(deriv, coefs, offsets, N, h)

f1 = D.dot(f)

# Maximum error on the domain

err = np.max(np.abs(f1 - f1_ex))

errs.append(err)

return errs

Finally, we visualize the results using a logarithmic plot:

import matplotlib.pyplot as plt

plt.loglog(Ns, calc_errors(coefs=[-0.5, 0.5], offsets=[-1, 1]), 'o', label=r'$O(Delta x^2)$')

plt.loglog(Ns, calc_errors(coefs=[1/12, -8/12, 8/12, -1/12], offsets=[-2, -1, 1, 2]), 'o', label=r'$O(Delta x^4)$')

plt.grid()

plt.xlabel('$N$')

plt.ylabel('Error')

plt.legend()

plt.show()

The results confirm that the second-order scheme has a slope of -2 in the double-log plot, indicating an error that behaves like 1/Δ𝑥². Conversely, the fourth-order scheme demonstrates a slope of -4, suggesting an error that behaves like 1/Δ𝑥⁴. This means that to achieve equivalent accuracy, fewer grid points are necessary with the higher-order scheme. As we include more coefficients, the sparse matrix gradually becomes denser, but the computational efficiency gained by needing fewer grid points outweighs the drawbacks of denser matrices.

Chapter 2: Video Tutorials on Matrix Differentiation

Explore the basics of calculating the derivative of a matrix in this informative video, designed for data science enthusiasts looking to enhance their understanding of differentiation matrices.

Delve into matrix differentiation techniques with this comprehensive video, focusing on derivatives with respect to matrices, ideal for those in the realm of programming and data science.