Regularized TD Stability and PQN

Context

Gallici2025 - Simplifying Deep Temporal Difference Learning argues that deep TD instability can be handled by normalization and regularization rather than by the full DQN stabilizer stack of target networks plus large replay buffers. The paper then builds PQN, a parallelized normalized Q-learning algorithm, around that analysis.

Formal Statement

For an action-value approximator Q_phi, a basic TD update has the form

$${\varphi }_{i+1}={\varphi }_i+{\alpha }_i(r+\gamma Q_{{\varphi }_i}(x^{\prime })-Q_{{\varphi }_i}(x)){\nabla }_{\varphi }{Q}_{{\varphi }_i}(x).$$

The paper studies the expected TD update vector and gives a TD stability criterion: around a fixed point, the update dynamics should locally point back toward the fixed point. Under Robbins-Monro step-size and regularity assumptions, their Theorem 1 states convergence when this criterion holds.

The analysis decomposes instability into:

• an off-policy term, from mismatch between the behavior distribution and the target-policy distribution;
• a nonlinear term, from curvature of the function approximator.

For LayerNorm Q-networks with small l2 regularization, their Theorem 2 states that at finite sufficient width the regularized TD update satisfies the TD stability criterion.

Derivation / Construction

• Start from the Jacobian of the expected TD update at a fixed point.
• Bound the off-policy component and the nonlinear component separately.
• Use LayerNorm to control representation scale and make the relevant Jacobian terms better behaved as width grows.
• Add small l2 regularization so the expected update has a restoring component.
• Build PQN by combining the regularized value network with many vectorized environment instances, online data collection, optional lambda-returns, and no target network or large replay buffer.
• Extend to cooperative MARL by using a VDN-style additive team value, yielding PQN-VDN.

Implications

The result is a useful theorem schema rather than a universal guarantee for every deep RL system. It suggests that some DQN engineering may be substitutable by normalization, regularization, and parallel online sampling, but the claim depends on architecture, optimizer, exploration, and distributional assumptions. For safe-MARL work, PQN and PQN-VDN are useful baselines because safety mechanisms can interact strongly with replay, exploration, and wall-clock training cost.

Links

• Temporal Difference Learning
• Value Decomposition Networks
• Gallici2025 - Simplifying Deep Temporal Difference Learning