Hybrid seminar at GERAD and on Zoom.

Derivatives are everywhere in scientific computing, yet not all ways of obtaining them are equal. Finite differences are easy to write, but their accuracy hinges on a step size trapped between truncation and round-off error, and a gradient costs one evaluation per variable. Symbolic differentiation is exact, but it needs a formula rather than a program, and the expressions it produces swell as dimension grows. Automatic differentiation (AD) is the third option, the one that powers most numerical software and all of deep learning, yet it is still widely treated as a black box.

This talk answers the title question from scratch, assuming nothing beyond the chain rule. We build forward mode, which pushes directional derivatives \(J*v\) through a computation, and reverse mode, which pulls gradients \(Jᵀ*u\) back through it, together with the surprise that a gradient costs a constant multiple of the function regardless of dimension. A separate question is how AD is implemented: operator overloading against source transformation.

AD does not return an m x n Jacobian. It returns an operator, one matrix-vector product at a time, so a full Jacobian costs n forward or m reverse passes. In high dimension that is hopeless, unless the Jacobian is sparse. How to leverage that sparsity is the goal of the sequel.