Simplifying Deep Temporal Difference Learning

Summary

This partial ingest is based on the extracted full PDF text. Gallici et al. revisit deep Temporal Difference Learning and argue that regularization plus vectorized online sampling can remove much of the engineering machinery inherited from DQN. Their theory gives a TD stability criterion and shows that LayerNorm plus small l2 regularization can stabilize nonlinear/off-policy TD. Their algorithm, PQN, is a parallelized normalized Q-learning method without target networks or large replay buffers, evaluated across Atari, Craftax, Smax, Overcooked, and Hanabi.

Key Claims

• Target networks and large replay buffers stabilize DQN-style training, but they also add memory overhead, implementation complexity, and delayed parameter updates.
• TD instability can be decomposed into off-policy instability and nonlinear-function-approximation instability.
• LayerNorm with small l2 regularization yields provably stable TD updates in the paper’s wide-network analysis.
• Parallelized online sampling over vectorized environments can make Q-learning practical without a large replay buffer.
• PQN is competitive with PPO or stronger value-based baselines in several single-agent and multi-agent settings, sometimes with much lower wall-clock time.

Methods / Formalism

• The paper studies TD parameter updates of the form:

$${\varphi }_{i+1}={\varphi }_i+{\alpha }_i(r+\gamma Q_{{\varphi }_i}(x^{\prime })-Q_{{\varphi }_i}(x)){\nabla }_{\varphi }{Q}_{{\varphi }_i}(x).$$

• The TD stability criterion requires the Jacobian of the expected update vector to point updates toward a fixed point; Theorem 1 states convergence under Robbins-Monro and regularity assumptions when this criterion holds.
• The analysis separates an off-policy term, driven by distribution shift between sampling and target policies, from a nonlinear term, driven by second derivatives of the function approximator.
• The LayerNorm Q-function bounds both sources of instability as width grows; Theorem 2 states that the regularized LayerNorm TD update satisfies the TD stability criterion above a finite width.
• PQN uses many parallel environment interactions, LayerNorm-regularized Q-networks, optional lambda-returns, and no target network or large replay buffer. Regularized TD Stability and PQN records the theorem schema and algorithmic pattern.

Evidence / Experiments

• Baird’s counterexample: LayerNorm plus l2 stabilizes a setting designed to make off-policy TD diverge.
• Atari: PQN outperforms PPO on Atari-10 in sample efficiency, score, and training time, and reaches competitive Atari-57 performance quickly relative to older DQN-family training budgets.
• Craftax: PQN-RNN is compared against PPO-RNN in a pure-GPU open-ended environment.
• Multi-agent tasks: PQN-VDN is evaluated on Smax, Overcooked, and Hanabi against MAPPO, QMIX, VDN, IPPO, and IQL-style baselines.
• The paper emphasizes simplicity, low memory, pure-GPU compatibility, and wall-clock speed, while acknowledging weaker exploration in hard Atari games with simple epsilon-greedy exploration.

Connections

• Connects Temporal Difference Learning to modern vectorized deep RL infrastructure.
• Uses Value Decomposition Networks as the straightforward multi-agent extension of PQN in cooperative tasks.
• Directly compares against Multi-Agent PPO baselines in several tasks, making it useful for interpreting later safe-MARL experiments that rely on MAPPO/IPPO.
• Belongs in Deep Learning Fundamentals because its core claim is about normalization, regularization, and stability of neural TD updates.

Open Questions

• How broadly does the LayerNorm stability result transfer to practical architectures and optimizer choices outside the paper’s assumptions?
• When does removing replay hurt exploration or long-horizon credit assignment despite improving memory and wall-clock cost?
• How should PQN-style baselines be included in safe-MARL comparisons where shields or constraints alter exploration?

Citation

Gallici, Matteo, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. 2025. “Simplifying Deep Temporal Difference Learning.” In International Conference on Learning Representations.