View in NBViewer Open in binder Download notebook View source


7.4. Stiff solvers, Jacobians and automatic differentiation

This page reports a set of experiments on how to use the stiff (implicit) solvers of DifferentialEquations.jl with the pseudo-spectral right-hand side of 7.1. Evolution via method of lines, and on the current state of automatic differentiation (AD) in Julia for this kind of code. The two approaches that came out of it are implemented in 7.2. Implicit solvers with the analytic Jacobian-vector product and in 7.3. Exponential time differencing with ETDRK4.

The experiments described here are not executed when this page is built: their results were obtained once, on the machine described in the Environment section, and are reported as static tables. The timings, in particular, are indicative only: they are single runs, after a warm-up call that compiles each solver, with no attempt at statistics, and they depend on the hardware, on the number of threads and on the load of the machine. Only the relative comparison between the methods is meaningful, and even that may change from one machine to another. The main benchmark can be run again, on any machine, with the script benchmarks/benchmark_stiff_solvers.jl in the repository, and the timings printed in the other two pages are obtained when those pages are built, so they may differ from the static tables below.

The problem

Stiff solvers need the Jacobian of the right-hand side, or at least its action on vectors, and by default it is computed with forward-mode AD via ForwardDiff.jl. The right-hand side nsepervort_hat_rhs! of 7.1. Evolution via method of lines has three features that get in the way:

It also has a fourth feature which is not related to AD, but which is unsafe with any adaptive or implicit solver: it mutates its input, by zeroing the dealiased modes of vort_hat. The new pages dealias the initial condition once instead; since the output of the right-hand side is dealiased, the state stays dealiased.

Environment

Note that in OrdinaryDiffEq v7 only a few solvers are exported by OrdinaryDiffEq itself; QNDF, FBDF, KenCarp47, TRBDF2, ETDRK4 and others must be loaded from their sub-packages (OrdinaryDiffEqBDF, OrdinaryDiffEqSDIRK, OrdinaryDiffEqExponentialRK, ...). Also, the autodiff keyword of the solvers takes an ADTypes.jl object, such as AutoForwardDiff(), AutoFiniteDiff() or AutoEnzyme(), instead of a boolean.

The current state of automatic differentiation

The AD ecosystem in Julia is now organized around two packages: ADTypes.jl, which defines the types used to select a backend (AutoForwardDiff(), AutoEnzyme(), AutoMooncake(), AutoFiniteDiff(), ...), and DifferentiationInterface.jl, a common interface to all the backends, which is what OrdinaryDiffEq uses internally to compute Jacobians and Jacobian-vector products. The main backends are:

For differentiating through the solver, e.g. with respect to parameters, the relevant package is SciMLSensitivity.jl. Sparse Jacobians are handled by AutoSparse together with SparseConnectivityTracer.jl and SparseMatrixColorings.jl, but they do not help here, since our Jacobian is dense.

Here is what happened with the original right-hand side, unchanged:

Backend / testResult
Rodas5P() with the default ForwardDifffails: ForwardDiff.DerivativeConfig has no method for a ComplexF64 state
same, with ForwardDiff.can_dual(::Type{ComplexF64}) = true (a workaround formerly used in 7.1. Evolution via method of lines)fails in the same way; the workaround no longer has any effect
rfft of an Array{<:ForwardDiff.Dual}fails: type ForwardDiff.Dual{...} not supported
FFTW plan applied to an Array{<:ForwardDiff.Dual}fails: no conversion from Dual to Float64
Rodas5P(autodiff = AutoEnzyme())fails: Enzyme has no derivative rule for the FFTW C library calls (No forward mode derivative found for ... fftw_alignment_of)
Mooncakenot tested; it would also need rules for the FFTW calls
Rodas5P(autodiff = AutoFiniteDiff()), \(N = 32\)works, building a dense Jacobian by finite differences

So, the issues from a few years ago are still there: ForwardDiff does not handle complex states in OrdinaryDiffEq, and neither ForwardDiff nor Enzyme can go through FFTW.

ForwardDiff can be made to work, and we checked that it does (with Rodas5P, QNDF and KenCarp4 at \(N = 32\), and with Jacobian-free Newton-Krylov at \(N = 128\)), but this requires four changes to the right-hand side:

  1. storing the state as a real array, reinterpret(reshape, Float64, vort_hat), and viewing it as complex inside the right-hand side;

  2. wrapping the work arrays with PreallocationTools.DiffCache, so they can hold dual numbers;

  3. writing an FFT wrapper for arrays of dual numbers, which, using the linearity of the transform, applies the plan to the value and to each partial derivative separately (about twenty lines);

  4. not mutating the input.

The resulting Jacobian agreed with the one from finite differences to within \(5 \times 10^{-5}\), relative, which is the accuracy of the finite differences. However, the right-hand side is bilinear, so the Jacobian is easy to write down by hand, and that is both simpler and faster. We did not pursue the AD version any further.

The analytic Jacobian

With \(F(\omega) = g + \nu\Delta\omega - B(\omega, \omega)\), with \(B\) bilinear, the Jacobian is \(J(\omega)h = \nu\Delta h - B(\omega, h) - B(h, \omega)\). Each product \(J(\omega)h\) costs six FFTs, against four for \(F\). Since \(F\) is quadratic, the centered difference \((F(\omega + \varepsilon h) - F(\omega - \varepsilon h))/2\varepsilon\) equals \(J(\omega)h\) exactly, which gives a sharp test: the two agree to about \(4 \times 10^{-14}\), relative.

Three caveats:

Timings

All runs are for \(N = 128\) (Nsub = 84), \(\nu = 1\), \(t \in [0, 1]\), starting from a random four-mode vorticity field and with a single-mode steady forcing, as in 7.1. Evolution via method of lines. The relative error is measured at \(t = 1\) against a reference solution computed with Vern9() and abstol = reltol = 1e-12. Adaptive methods use the default tolerances.

These tables were obtained on the machine described in the Environment section and are kept here as a record; the same benchmark can be run on another machine with the script benchmarks/benchmark_stiff_solvers.jl in the repository.

Matrix-free implicit methods with the analytic Jacobian-vector product

The number of evaluations is the solver's own count, sol.stats.nf, which includes the Jacobian-vector products; most of the evaluations are, in fact, Jacobian-vector products, one per GMRES iteration (in a separate run with counters, for instance, QNDF used 217 evaluations of \(F\) and 1475 of \(J(\omega)h\); the counts vary slightly from run to run, since multithreaded FFTs change the round-off errors and, with them, the number of GMRES iterations).

MethodTime (s)Relative errorEvaluations (\(F\) + jvp)Steps
Tsit5() (explicit)1.541.6e-95871978
Vern9() (explicit)3.052.5e-1112402775
QNDF + jvp + GMRES0.893.0e-71817186
FBDF + jvp + GMRES1.608.2e-62633224
KenCarp47 + jvp + GMRES1.542.9e-7330319
TRBDF2 + jvp + GMRES1.061.9e-5242928

On \(t \in [0, 10]\), converging to the steady state, QNDF + jvp + GMRES took 220 steps and reached a distance of about \(8 \times 10^{-7}\) from the steady state.

Exponential time differencing

Here the stiff linear part \(\nu\Delta\), diagonal in spectral space, is passed as a DiagonalOperator in a SplitODEProblem and integrated exactly, and no Jacobian is needed at all. ETDRK4 is used with a fixed time step.

MethodTime (s)Relative errorSteps
Tsit5() (explicit)1.424.2e-10979
Vern9() (explicit)2.881.5e-11775
ETDRK4(), dt = 1e-20.178.2e-5100
ETDRK4(), dt = 5e-30.265.5e-6200
ETDRK4(), dt = 2.5e-30.384.0e-7401
ETDRK4(), dt = 1.25e-30.822.8e-8801

The errors decrease at the expected fourth order. On \(t \in [0, 10]\), with dt = 1e-2, ETDRK4 reached a distance of about \(8 \times 10^{-7}\) from the steady state.

With \(\nu = 0.1\) (in an earlier run, with a different random initial condition), on the other hand, dt = 1e-2 is too large and the solution blows up (ReturnCode.Unstable), while dt = 1e-3 gives a relative error of \(3.7 \times 10^{-5}\) in about 1.6 s, against 0.35 s for Tsit5() with an error of \(6.7 \times 10^{-6}\): with less viscosity the problem is less stiff and the time step is limited by the nonlinear term, which favours the explicit, adaptive methods.

The IMEX multistep method CNAB2(), with dt = 1e-3, worked but was very slow in this setup (about 87 s for \(t \in [0, 1]\), with a relative error of \(7 \times 10^{-4}\)); we did not investigate why.

Small grids, dense Jacobian

At \(N = 32\), with the dense Jacobian assembled column by column from the analytic jvp, QNDF, KenCarp4 and FBDF work, as does Rodas5P, once tgrad is given.

A strongly nonlinear case

With \(\nu = 0.01\), the initial vorticity used here is large enough for the flow to be strongly nonlinear, with small differences growing quickly. All methods, including Vern9(), then give relative errors of order one at \(t = 1\), at the default tolerances, compared with the tight-tolerance reference. This reflects the sensitivity to the initial conditions of the flow, not a failure of the methods, and such cases are not useful for comparing accuracy.

Conclusions



Last modified: November 21, 2026. Built with Franklin.jl, using the Book Template.