The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games

Summary

This partial ingest is based on the extracted full PDF text. Yu et al. give a broad empirical study showing that PPO-style on-policy methods can be strong cooperative-MARL baselines when configured carefully. They define MAPPO as PPO with centralized value-function inputs and IPPO as PPO with local actor/value inputs, evaluate on MPE, SMAC, Google Research Football, and Hanabi, and identify practical implementation factors that strongly affect performance.

Key Claims

• The common assumption that PPO is inherently sample-inefficient relative to off-policy MARL methods is too strong.
• MAPPO and IPPO can match or exceed off-policy baselines such as QMIX, MADDPG, RODE, SAD, and VDN across many cooperative benchmarks.
• PPO performance in MARL depends heavily on implementation details rather than domain-specific algorithmic architecture.
• The most important factors are value normalization, value-function input representation, training-data reuse, PPO clipping strength, and batch size.
• Properly configured PPO should be treated as a serious cooperative-MARL baseline.

Methods / Formalism

• The paper uses shared-reward DEC-POMDPs:

$$⟨S,A,O,R,P,n,\gamma ⟩,$$

where each agent chooses a_i from local observation o_i=O(s;i) and the team optimizes discounted shared reward.

• MAPPO uses decentralized actors but a centralized value function during training; IPPO uses local inputs for both actor and value function.
• Homogeneous agents share actor and critic parameters where possible.
• Practical suggestions include value normalization, value-function inputs that combine useful global and agent-specific local features, limited epoch/minibatch reuse in hard MARL tasks, clipping ratios below 0.2, and sufficiently large but tuned batch sizes.
• The source is empirical rather than a new theoretical PPO derivation.
• Cooperative MARL Baseline Formalisms records the shared-reward DEC-POMDP and MAPPO/IPPO baseline schema used by this comparison.

Evidence / Experiments

• MPE: MAPPO is comparable to or better than off-policy baselines, and IPPO is also strong despite no global value input.
• SMAC: MAPPO and IPPO perform at least as well as QMIX in most maps under comparable sample budgets; MAPPO is comparable or superior to RODE in many maps.
• Google Research Football: MAPPO outperforms QMIX in the reported academy scenarios and is competitive with stronger specialized methods.
• Hanabi: MAPPO performs strongly in 2-5 player self-play and improves over IPPO as agent count grows, suggesting centralized critic inputs become more important.
• Ablations support five concrete implementation suggestions, especially value normalization and limited sample reuse to reduce non-stationarity from policy changes.

Connections

• Complements Sunehag2017 - Value-Decomposition Networks for Cooperative Multi-Agent Learning by challenging the idea that off-policy value-decomposition methods should dominate cooperative MARL.
• Provides baseline context for Gu2022 - Multi-Agent Constrained Policy Optimisation, which builds safety-aware variants around MAPPO-style optimization.
• Links to Multi-Agent Non-Stationarity through its recommendation to limit epochs and minibatching in hard cooperative tasks.
• Connects to AI Evaluation and Benchmarking because the paper is largely a benchmark/implementation study across popular MARL testbeds.

Open Questions

• Which findings survive in competitive, mixed-motive, continuous-action, heterogeneous-agent, or safety-constrained settings?
• How much of MAPPO’s strength comes from parameter sharing and benchmark homogeneity rather than PPO itself?
• Can the empirical implementation suggestions be derived from a theory of non-stationarity and sample reuse in MARL?

Citation

Yu, Chao, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. 2022. “The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games.” In Advances in Neural Information Processing Systems 35, Datasets and Benchmarks Track.