This paper was converted on www.awesomepapers.org from LaTeX by an anonymous user.
Want to know more? Visit the Converter page.

Differentiable Programming for Differential
Equations: A Review

cundo Sapienza \email[email protected]
\affilDepartment of Statistics, University of California, Berkeley, USA \ANDrdi Bolibar
\affilUniv. Grenoble Alpes, CNRS, IRD, G-INP, Institut des Géosciences de l’Environnement, Grenoble, France
TU Delft, Department of Geosciences and Civil Engineering, Delft, Netherlands \ANDank Schäfer
\affilCSAIL, Massachusetts Institute of Technology, Cambridge, USA \ANDian Groenke
\affilTU Berlin, Department of Electrical and Computer Engineering, Berlin, Germany
Helmholtz Centre for Environmental Research, Leipzig, Germany \ANDik Pal
\affilCSAIL, Massachusetts Institute of Technology, Cambridge, USA \ANDctor Boussange
\affilSwiss Federal Research Institute WSL, Birmensdorf, Switzerland \ANDtrick Heimbach
\affilOden Institute for Computational Engineering and Sciences, University of Texas at Austin, USA
Jackson School of Geosciences, University of Texas at Austin, USA \ANDles Hooker
\affilDepartment of Statistics and Data Science, University of Pennsylvania, USA \ANDrnando Pérez
\affilDepartment of Statistics, University of California, Berkeley, USA \ANDr-Olof Persson
\affilDepartment of Mathematics, University of California, Berkeley, USA \ANDristopher Rackauckas
\affilMassachusetts Institute of Technology, Cambridge, USA
JuliaHub, Cambridge, USA
Abstract

The differentiable programming paradigm is a cornerstone of modern scientific computing. It refers to numerical methods for computing the gradient of a numerical model’s output. Many scientific models are based on differential equations, where differentiable programming plays a crucial role in calculating model sensitivities, inverting model parameters, and training hybrid models that combine differential equations with data-driven approaches. Furthermore, recognizing the strong synergies between inverse methods and machine learning offers the opportunity to establish a coherent framework applicable to both fields. Differentiating functions based on the numerical solution of differential equations is non-trivial. Numerous methods based on a wide variety of paradigms have been proposed in the literature, each with pros and cons specific to the type of problem investigated. Here, we provide a comprehensive review of existing techniques to compute derivatives of numerical solutions of differential equations. We first discuss the importance of gradients of solutions of differential equations in a variety of scientific domains. Second, we lay out the mathematical foundations of the various approaches and compare them with each other. Third, we cover the computational considerations and explore the solutions available in modern scientific software. Last but not least, we provide best-practices and recommendations for practitioners. We hope that this work accelerates the fusion of scientific models and data, and fosters a modern approach to scientific modelling.

Key words. differentiable programming, sensitivity analysis, differential equations, inverse modelling, scientific machine learning, automatic differentiation, adjoint methods.

This manuscript was conceived with the goal of shortening the gap between developers and practitioners of differentiable programming applied to modern scientific machine learning. With the advent of new tools and new software, it is important to create pedagogical content that allows the broader community to understand and integrate these methods into their workflows. We hope this encourages new people to be an active part of the ecosystem, by using and developing open-source tools. This work was done under the premise open-science from scratch, meaning all the contents of this work, both code and text, have been in the open from the beginning and that any interested person can contribute to the project.

1 Introduction

Models based on differential equations (DEs), including ordinary differential equations (ODEs) and partial differential equations (PDEs), play a central role in describing the behaviour of dynamical systems in applied, natural, and social sciences. For instance, DEs are used for the modelling of the dynamics of the atmospheric and ocean circulation in climate science, for the modelling of ice or mantle flow in solid Earth geophysics, for the modelling of the spatio-temporal dynamics of species abundance in ecology, and for the modelling of cancer cell evolution in biology. For centuries, scientists have relied on theoretical and analytical methods to solve DEs. Allowing to approximate the solutions of large, nonlinear DE-based models, numerical methods and computers have lead to fundamental advances in the understanding and prediction of physical, biological, and social systems, among others