Neural Approximations and the Algebra of Gradients

We characterize neural networks as approximators of functions and dynamic systems. Neural approximations, leading to nonlinear minimization in highly dimensional spaces, require effective gradient calculation typically realized by gradient backpropagation. We discuss the use of gradient back-propagation for static and for dynamic systems. We also show the essential difference between the common chain rule and backpropagation, which is rarely acknowledged. (author)