Where can I find the proof of the universal approximation theorem?

Summary

This partial ingest is based on a clipped AI Stack Exchange Q&A. The thread collects proof pointers for universal approximation results, starting from classical feed-forward neural-network theorems and expanding to later architecture-specific and quantitative variants. It is not itself a proof, but it is useful as a map of which original papers and expository resources to consult.

Key Claims

• Universal approximation results show that certain neural-network classes can approximate broad function classes, usually continuous functions on compact domains.
• These results do not say how to train a network to find the approximating parameters.
• Cybenko’s 1989 result is presented as a classical proof for multilayer perceptrons with sigmoidal activation functions.
• Later results relax or change activation assumptions and extend universality claims to convolutional, recurrent, graph, deep, narrow, and quantitative settings.
• The thread distinguishes qualitative universality from quantitative guarantees that bound the number of parameters needed for a desired approximation rate.

Methods / Formalism

• Standard compact-domain form; see Universal Approximation Statement for the reusable theorem schema:

$$\forall f\in C(K),\forall \epsilon >0,\exists N(x)=\sum _{i=1}^m{a}_i\sigma (w_i^{\top }x+b_i)\text{such that}\underset{x\in K}{\sup }|f(x)-N(x)|<\epsilon .$$

• The activation assumptions vary by theorem: classical versions use sigmoidal functions, while later results cover ReLU and other non-polynomial or architecture-specific cases.
• The Q&A also separates approximation of continuous functions from exact memorization or memory-capacity results.

Evidence / Experiments

• The source is a community Q&A with bibliographic pointers, not an original theorem or empirical study.
• It cites Cybenko (1989), Hornik-style results, Barron-style quantitative approximation, Yarotsky-style deep ReLU rates, Kidger and Lyon, convolutional universality, recurrent universality, and graph-neural-network universality pointers.
• The references should be treated as a reading path; exact theorem statements require checking the original papers.

Connections

• Core source for Universal Approximation Theorem.
• Universal Approximation Statement stores the compact formal statement and caveats.
• Links directly to Sigmoid Functions because classical approximation theorems are often stated for sigmoidal activations.
• Useful in Deep Learning Fundamentals for separating expressivity results from optimization and generalization behavior.
• Helps contextualize Double Descent: a model class can be expressive enough to fit functions while still having nontrivial generalization dynamics.

Open Questions

• Which original universal-approximation papers should be fully ingested into source notes next?
• What theorem version is most useful for the wiki: classical sigmoidal MLPs, ReLU networks, convolutional networks, or architecture-agnostic density results?
• How should the wiki distinguish approximation, interpolation, memorization, and learnability?

Citation

Leroy Od, nbro, AB_IM, and Loren Rosen. (2019-2021). Where can I find the proof of the universal approximation theorem? AI Stack Exchange.