Cooperative MARL Baseline Formalisms

Context

Sunehag2017 - Value-Decomposition Networks for Cooperative Multi-Agent Learning and Yu2022 - The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games provide reusable baseline formalisms for cooperative MARL: shared-reward partially observed team problems, value factorization, and centralized-training/decentralized-execution actor-critic baselines. Papadopoulos2025 - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks adds the evaluation warning that baseline strength depends on task family, observation modality, and training-time cost.

Formal Statement

A shared-reward cooperative MARL problem is often represented as a DEC-POMDP

$$⟨S,A,O,R,P,n,\gamma ⟩,$$

where n agents act through the joint action A=A_1 x ... x A_n, observe local information from O, transition by P, and optimize the shared discounted objective

$$J(\pi )=E_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }^{t}R(s_t,a_t)\right].$$

VDN assumes an additive team action-value decomposition:

$$Q((h_1,\dots ,h_n),(a_1,\dots ,a_n))\approx \sum _{i=1}^n{\tilde{Q}}_i(h_i,a_i).$$

MAPPO instead keeps decentralized actors but trains them with centralized value information when available. IPPO removes that centralized value input and trains local PPO actor/value functions.

A benchmark comparison can be summarized as a matrix over algorithms a and tasks b:

$$M[a,b]=(\mathrm{return},\mathrm{variance},\mathrm{steps},wall\_clock,\mathrm{compute}).$$

The matrix is more informative when b spans sparse rewards, joint exploration, collision dynamics, image observations, role assignment, and different agent counts.

Derivation / Construction

• VDN backpropagates a team TD loss through the additive sum, so each agent receives gradient information from the shared team return without hand-designed local rewards.
• Additivity makes decentralized greedy execution simple: maximizing each local \tilde Q_i maximizes the additive approximation to the team value.
• MAPPO uses the PPO clipped surrogate for each agent policy while estimating advantages with a centralized critic or richer value-function input.
• IPPO keeps the PPO recipe local; its strength on several shared-reward benchmarks shows that parameter sharing and benchmark homogeneity can matter as much as centralized information.
• Non-stationarity enters both families because replayed TD targets and on-policy advantages are generated by policy populations that may already have changed.
• Benchmarking adds a second layer: the same baseline family can look strong or weak depending on whether the task stresses coordination, perception, sparse reward, or teammate diversity.

Implications

Safe-MARL and coordination results should specify whether they compare against independent learners, VDN-style value factorization, MAPPO/IPPO-style policy gradients, or stronger modern baselines. A weak baseline or narrow benchmark can make a safety wrapper, constraint method, or task decomposition look better than it is. Conversely, strong and diverse baselines can expose whether the contribution is safety-specific, coordination-specific, or simply better optimization.

Links

• Cooperative Multi-Agent Reinforcement Learning
• Cooperative MARL Benchmarking
• Multi-Agent Coordination
• Value Decomposition Networks
• Multi-Agent PPO
• Multi-Agent Non-Stationarity
• Sunehag2017 - Value-Decomposition Networks for Cooperative Multi-Agent Learning
• Yu2022 - The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games
• Ahmed2022 - Deep Reinforcement Learning for Multi-Agent Interaction
• Papadopoulos2025 - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks