Cooperative MARL Benchmarking

Definition

Cooperative MARL benchmarking is the evaluation of multi-agent RL algorithms on shared-objective tasks in a way that tests the specific cooperative skills being claimed. A useful technical anchor is a score matrix M[a,b], where each algorithm a is evaluated across benchmark tasks b; broad cooperative competence requires performance, variance, and training cost to hold across task families rather than a single favored benchmark.

Why It Matters

Safe-MARL and cooperative-MARL methods often claim progress relative to baselines. Those claims are fragile when the benchmark set is narrow, implementation details are inconsistent, or only final return is reported. Fully cooperative tasks should test coordination, joint exploration, sparse rewards, partial observability, collision dynamics, role assignment, high-dimensional observations, and compute cost.

Formalism / Key Objects

• A benchmark report should specify the environment family, observation modality, reward density, number of agents, action structure, teammate policy assumptions, and training budget.
• A minimal result tuple is (return, variance, wall_clock_time, environment_steps, compute_footprint) for each algorithm-task pair.
• The Dec-POMDP setup remains the common mathematical base:

$$⟨S,A,P,r,F,O,N,\gamma ⟩,$$

with shared reward r, joint action A, partial observations from F, and policies conditioned on local histories.

• Baseline families include independent learners, value decomposition, centralized critics, actor-critic methods, exploration-enhanced methods, and image-encoder variants.
• Cooperative MARL Baseline Formalisms records core baseline equations; this concept tracks whether the evaluation matrix is broad enough to support the claim.

Connections

• AI Evaluation and Benchmarking supplies the general measurement frame.
• Benchmark Gaming is the stronger failure mode where systems exploit the evaluator; cooperative MARL also has a milder over-specialization risk when algorithms are tuned to SMAC/GRF-like environments.
• Cooperative Multi-Agent Reinforcement Learning provides the underlying learning problem.
• Multi-Agent PPO and Value Decomposition Networks are recurring baseline families, but their strength varies by task family.
• Multi-Agent Coordination identifies benchmark capabilities that simple team-game scores can miss.

Common Confusions

• “Works on SMAC” does not mean “solves cooperative MARL.”
• Higher return is hard to interpret without variance, training time, and implementation parity.
• Image-based observations can test representation learning as much as coordination.
• Fully cooperative does not mean easy: sparse rewards and joint exploration can make shared-reward tasks harder than some mixed-motive games.

Key Sources

• Papadopoulos2025 - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks
• Ahmed2022 - Deep Reinforcement Learning for Multi-Agent Interaction
• Yu2022 - The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games
• Sunehag2017 - Value-Decomposition Networks for Cooperative Multi-Agent Learning
• Rutherford2024 - JaxMARL