Differentiable programming

Differentiable programming is a programming paradigm in which a numeric computer program can be

differentiated throughout via automatic differentiation.[1] [2][3][4][5] This allows for gradient-based
optimization of parameters in the program, often via gradient descent, as well as other learning approaches
that are based on higher order derivative information. Differentiable programming has found use in a wide
variety of areas, particularly scientific computing and artificial intelligence.[5] One of the early proposals to
adopt such a framework in a systematic fashion to improve upon learning algorithms was made by the
Advanced Concepts Team at the European Space Agency in early 2016.[6]

Approaches
Most differentiable programming frameworks work by constructing a graph containing the control flow and
data structures in the program.[7] Attempts generally fall into two groups:

Static, compiled graph-based approaches such as TensorFlow,[note 1] Theano, and MXNet.

They tend to allow for good compiler optimization and easier scaling to large systems, but
their static nature limits interactivity and the types of programs that can be created easily
(e.g. those involving loops or recursion), as well as making it harder for users to reason
effectively about their programs.[7] A proof of concept compiler toolchain called Myia uses a
subset of Python as a front end and supports higher-order functions, recursion, and higher-
order derivatives.[8][9][10]
Operator overloading, dynamic graph based approaches such as PyTorch and AutoGrad.
Their dynamic and interactive nature lets most programs be written and reasoned about
more easily. However, they lead to interpreter overhead (particularly when composing many
small operations), poorer scalability, and reduced benefit from compiler optimization.[9][10] A
package for the Julia programming language – Zygote (https://github.com/FluxML/Zygote.j
l) – works directly on Julia's intermediate representation, allowing it to still be optimized by
Julia's just-in-time compiler.[7][11][5]

A limitation of earlier approaches is that they are only able to differentiate code written in a suitable manner
for the framework, limiting their interoperability with other programs. Newer approaches resolve this issue
by constructing the graph from the language's syntax or IR, allowing arbitrary code to be
differentiated.[7][9]

Applications
Differentiable programming has been applied in areas such as combining deep learning with physics
engines in robotics,[12] solving electronic structure problems with differentiable density functional
theory,[13] differentiable ray tracing,[14] image processing,[15] and probabilistic programming.[5]

See also
Differentiable function
 Machine learning

Notes
1. TensorFlow 1 uses the static graph approach, whereas TensorFlow 2 uses the dynamic
graph approach by default.

References
1. Izzo, Dario; Biscani, Francesco; Mereta, Alessio (2017). "Differentiable genetic
programming" (https://link.springer.com/chapter/10.1007/978-3-319-55696-3_3). European
Conference on Genetic Programming (EuroGP). Lecture Notes in Computer Science. 18:
35–51. arXiv:1611.04766 (https://arxiv.org/abs/1611.04766). doi:10.1007/978-3-319-55696-
3_3 (https://doi.org/10.1007%2F978-3-319-55696-3_3). ISBN 978-3-319-55695-6.
S2CID 17786263 (https://api.semanticscholar.org/CorpusID:17786263).
2. Baydin, Atilim Gunes; Pearlmutter, Barak; Radul, Alexey Andreyevich; Siskind, Jeffrey
(2018). "Automatic differentiation in machine learning: a survey" (http://jmlr.org/papers/v18/1
7-468.html). Journal of Machine Learning Research. 18: 1–43.
3. Wang, Fei; Decker, James; Wu, Xilun; Essertel, Gregory; Rompf, Tiark (2018).
"Backpropagation with Callbacks: Foundations for Efficient and Expressive Differentiable
Programming" (http://papers.nips.cc/paper/8221-backpropagation-with-callbacks-foundation
s-for-efficient-and-expressive-differentiable-programming.pdf) (PDF). In Bengio, S.; Wallach,
H.; Larochelle, H.; Grauman, K (eds.). NIPS'18: Proceedings of the 32nd International
Conference on Neural Information Processing Systems (https://dl.acm.org/doi/proceedings/1
0.5555/3327546). Curran Associates. pp. 10201–12. Retrieved 2019-02-13.
4. Innes, Mike (2018). "On Machine Learning and Programming Languages" (http://www.sysml.
cc/doc/2018/37.pdf) (PDF). SysML Conference 2018.
5. Innes, Mike; Edelman, Alan; Fischer, Keno; Rackauckas, Chris; Saba, Elliot; Viral B Shah;
Tebbutt, Will (2019), ∂P: A Differentiable Programming System to Bridge Machine Learning
and Scientific Computing, arXiv:1907.07587 (https://arxiv.org/abs/1907.07587)
6. "Differential Intelligence" (https://www.esa.int/gsp/ACT/projects/differential_intelligence/).
October 2016. Retrieved 2022-10-19.
7. Innes, Michael; Saba, Elliot; Fischer, Keno; Gandhi, Dhairya; Rudilosso, Marco Concetto;
Joy, Neethu Mariya; Karmali, Tejan; Pal, Avik; Shah, Viral (2018-10-31). "Fashionable
Modelling with Flux". arXiv:1811.01457 (https://arxiv.org/abs/1811.01457) [cs.PL (https://arxi
v.org/archive/cs.PL)].
8. Merriënboer, Bart van; Breuleux, Olivier; Bergeron, Arnaud; Lamblin, Pascal (3 December
2018). "Automatic differentiation in ML: where we are and where we should be going" (http
s://papers.nips.cc/paper/2018/hash/770f8e448d07586afbf77bb59f698587-Abstract.html).
NIPS'18. Vol. 31. pp. 8771–81.
9. Breuleux, O.; van Merriënboer, B. (2017). "Automatic Differentiation in Myia" (https://www.sys
ml.cc/doc/2018/39.pdf) (PDF). Retrieved 2019-06-24.
10. "TensorFlow: Static Graphs" (https://pytorch.org/tutorials/beginner/examples_autograd/tf_two
_layer_net.html). Tutorials: Learning PyTorch. PyTorch.org. Retrieved 2019-03-04.
11. Innes, Michael (2018-10-18). "Don't Unroll Adjoint: Differentiating SSA-Form Programs".
arXiv:1810.07951 (https://arxiv.org/abs/1810.07951) [cs.PL (https://arxiv.org/archive/cs.PL)].
12. Degrave, Jonas; Hermans, Michiel; Dambre, Joni; wyffels, Francis (2016-11-05). "A
Differentiable Physics Engine for Deep Learning in Robotics". arXiv:1611.01652 (https://arxi
v.org/abs/1611.01652) [cs.NE (https://arxiv.org/archive/cs.NE)].
13. Li, Li; Hoyer, Stephan; Pederson, Ryan; Sun, Ruoxi; Cubuk, Ekin D.; Riley, Patrick; Burke,
Kieron (2021). "Kohn-Sham Equations as Regularizer: Building Prior Knowledge into
Machine-Learned Physics" (https://doi.org/10.1103%2FPhysRevLett.126.036401). Physical
Review Letters. 126 (3): 036401. arXiv:2009.08551 (https://arxiv.org/abs/2009.08551).
Bibcode:2021PhRvL.126c6401L (https://ui.adsabs.harvard.edu/abs/2021PhRvL.126c6401
L). doi:10.1103/PhysRevLett.126.036401 (https://doi.org/10.1103%2FPhysRevLett.126.0364
01). PMID 33543980 (https://pubmed.ncbi.nlm.nih.gov/33543980).
14. Li, Tzu-Mao; Aittala, Miika; Durand, Frédo; Lehtinen, Jaakko (2018). "Differentiable Monte
Carlo Ray Tracing through Edge Sampling" (https://people.csail.mit.edu/tzumao/diffrt/). ACM
Transactions on Graphics. 37 (6): 222:1–11. doi:10.1145/3272127.3275109 (https://doi.org/1
0.1145%2F3272127.3275109). S2CID 52839714 (https://api.semanticscholar.org/CorpusID:
52839714).
15. Li, Tzu-Mao; Gharbi, Michaël; Adams, Andrew; Durand, Frédo; Ragan-Kelley, Jonathan
(August 2018). "Differentiable Programming for Image Processing and Deep Learning in
Halide" (https://cseweb.ucsd.edu/~tzli/gradient_halide). ACM Transactions on Graphics. 37
(4): 139:1–13. doi:10.1145/3197517.3201383 (https://doi.org/10.1145%2F3197517.320138
3). S2CID 46927588 (https://api.semanticscholar.org/CorpusID:46927588).

Retrieved from "https://en.wikipedia.org/w/index.php?title=Differentiable_programming&oldid=1157986176"