Value-Decomposition Networks For Cooperative Multi-Agent Learning

Summary

This partial ingest is based on the extracted full PDF text. Sunehag et al. introduce Value Decomposition Networks (VDN) for cooperative MARL with a single team reward. VDN trains agent-local Q-functions through a centralized additive decomposition of the team Q-function, aiming to avoid spurious reward assignment and the “lazy agent” failure mode seen in naive centralized or independent learners under partial observability.

Key Claims

• Fully centralized learners can fail in simple cooperative MARL because the joint action space is combinatorial and one agent can become “lazy” once another agent finds a useful policy.
• Independent learners face non-stationarity and spurious team rewards that may be caused by teammates’ unobserved actions.
• A learned additive decomposition of the team action-value function can provide cleaner agent-local learning signals while preserving decentralized greedy execution.
• Weight sharing helps avoid symmetry-breaking lazy-agent failures in homogeneous teams, but role identifiers may be needed when agents should specialize.
• Information channels can improve tasks requiring coordination under partial observability, though they increase centralization and learning complexity.

Methods / Formalism

• The setting is a cooperative Dec-POMDP/Markov-game-style problem with local observations and one joint reward.
• VDN assumes the joint action-value can be approximated additively:

$$Q((h_1,\dots ,h_d),(a_1,\dots ,a_d))\approx \sum _{i=1}^d{\tilde{Q}}_i(h_i,a_i).$$

• The network backpropagates a team Q-learning loss through the sum to train the individual \tilde Q_i components without hand-designed individual rewards.
• If each agent acts greedily with respect to its local component, the joint action is equivalent to greedy maximization of the additive team value.
• The paper evaluates variants with value decomposition, shared weights, role identifiers, low/high communication channels, and centralization.
• Cooperative MARL Baseline Formalisms records the reusable team-objective and additive-factorization setup.

Evidence / Experiments

• The paper introduces seven two-player partially observable grid/maze domains designed to expose coordination, aliasing, and lazy-agent failures.
• Across normalized AUC and final-performance metrics, value-decomposition architectures outperform naive centralized agents and independent learners.
• Shared weights improve the hard one-corridor Fetch setting by preventing a lazy-agent collapse, while role information helps when non-identical behavior is beneficial.
• The learned decomposition in Fetch separates reward anticipation into agent-attributable component Q-functions despite observing only a team reward.

Connections

• Establishes an early CTDE-style value-factorization baseline for Cooperative Multi-Agent Reinforcement Learning.
• Yu2022 - The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games later compares PPO-style on-policy methods against VDN and other off-policy/value-decomposition baselines.
• Gallici2025 - Simplifying Deep Temporal Difference Learning uses a PQN-VDN variant to adapt simplified Q-learning to cooperative MARL tasks.
• Relates to Multi-Agent Non-Stationarity because VDN addresses spurious rewards and non-stationary teammates by centralizing training while preserving local execution.

Open Questions

• When is the additive Q-decomposition too restrictive for cooperative tasks with strong interaction effects?
• How should VDN-style decomposition be compared with monotonic mixers such as QMIX or modern attention/critic architectures?
• Does the learned decomposition remain interpretable as team size, partial observability, or communication complexity increases?

Citation

Sunehag, Peter, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. 2017. “Value-Decomposition Networks For Cooperative Multi-Agent Learning.” arXiv:1706.05296.