Universal Approximation Statement

Context

The universal approximation theorem is often invoked as a broad claim about neural-network expressivity. This annex stores a compact statement schema and the main caveats, while the concept page keeps the navigational distinction between approximation, optimization, and generalization.

Formal Statement

For a compact set K \subset \mathbb{R}^d, a continuous target function f \in C(K), and any tolerance \varepsilon>0, suitable one-hidden-layer networks can approximate f uniformly:

$$\exists m,a_i,w_i,b_{i}N(x)=\sum _{i=1}^m{a}_i\sigma (w_i^{\top }x+b_i)$$

such that

$$\underset{x\in K}{\sup }|f(x)-N(x)|<\epsilon .$$

The exact assumptions on \sigma vary across theorem versions. Classical statements use sigmoidal activations; later variants cover ReLU, non-polynomial activations, and architecture-specific settings.

Derivation / Construction

This batch did not ingest a full proof. The clipped Q&A points to proof families rather than supplying one:

• Cybenko-style density results for superpositions of sigmoidal functions.
• Hornik-style qualitative approximation results for neural networks.
• Barron- and Yarotsky-style quantitative rates for selected function classes and architectures.
• Later depth, narrowness, convolutional, recurrent, and graph-network universality variants.

Implications

• The theorem is an expressivity guarantee, not a training algorithm.
• Uniform approximation on compact domains does not guarantee extrapolation outside the domain.
• Approximation capacity does not settle generalization; finite-data behavior is tracked separately by concepts such as Double Descent.

Links

• Universal Approximation Theorem
• Sigmoid Functions
• LeroyOd2019 - Universal Approximation Theorem Proof Pointers