Deep Reinforcement Learning for Multi-Agent Interaction

Summary

Ahmed et al. provide a portfolio overview of the University of Edinburgh Autonomous Agents Research Group’s work on deep RL and MARL for autonomous agents that interact with other agents. The paper is not a single algorithmic contribution; it maps research strands around cooperative MARL benchmarking, shared-experience learning, emergent communication, ad hoc teamwork, agent/opponent modelling, autonomous driving goal recognition, multi-agent authentication, and open problems.

This is a partial ingest based on the full extracted text, emphasizing the map of problems and links to existing wiki pages.

Key Claims

• MARL progress is hard to measure without standardized benchmark environments, implementations, and task suites.
• Cooperative MARL spans independent learners, centralized policy-gradient learners, and value-decomposition learners, but empirical comparisons depend heavily on environment design and implementation choices.
• Ad hoc teamwork requires agents to cooperate with previously unknown teammates, including open teams where agents enter and leave.
• Agent modelling is central for interaction: agents need to reason about teammates’ goals, behaviors, capabilities, and composition from limited observations.
• Emergent communication can help coordination, but expressivity depends on the complexity and unpredictability of the task context.
• Open problems include generalization, causal RL, ad hoc communication, adaptive teammates, teammate generation, few-shot teamwork, and interpretable/verifiable goal recognition.

Methods / Formalism

• The paper surveys a research program rather than presenting one formal model.
• The cooperative MARL strand compares independent learners, centralized policy-gradient learners, and value-decomposition learners across multi-agent environments.
• The ad hoc teamwork strand assumes no prior coordination: a learner must infer enough about partner behavior and team composition online to cooperate.
• Graph-based policy learning (GPL) represents open teams with graph neural networks and action-prediction components, so teammate effects and uncertainty can enter the policy.
• The open-problems section frames causality as a route to more sample-efficient and generalizable RL: agents may learn causal models from interventions generated by their interactions.

Evidence / Experiments

• The paper summarizes prior group outputs rather than reporting one new experiment.
• It points to EPyMARL as an extended PyMARL repository, Level-Based Foraging (LBF), Multi-Robot Warehouse (RWARE), PressurePlate, and related environments as tools for reproducible MARL evaluation.
• It summarizes Shared Experience Actor-Critic (SEAC), Graph-based Policy Learning (GPL), LIAM-style agent modelling, emergent communication results, autonomous-driving goal recognition, and multi-agent interaction for authentication/key agreement.

Connections

• Reinforces Cooperative MARL Benchmarking as a persistent measurement problem that underlies later benchmark extensions such as Papadopoulos2025 - An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks.
• Connects Ad Hoc Teamwork with Multi-Agent Coordination: unknown teammates, open teams, and changing partner policies make coordination a modelling problem, not just a shared-reward optimization problem.
• Extends Emergent Communication by emphasizing that communication success depends on environmental pressure, partner variation, and generalization to new teammates.
• Links Multi-Agent Non-Stationarity to ad hoc teamwork because adaptive teammates are a central future challenge.
• Sits at the boundary of Safe Multi-Agent Reinforcement Learning and Strategic Reasoning: robust interaction with unknown agents is a safety concern even before explicit shields or constraints are introduced.

Open Questions

• Which benchmark suites actually measure general interaction capability rather than overfitting to fixed teammate policies?
• How should ad hoc agents communicate when they cannot assume a shared protocol or fixed partner identity?
• Can teammate generation produce the diversity needed for robust few-shot teamwork?
• How can goal-recognition models remain interpretable, fast, and verifiable under occlusion or partial observability?

Citation

Ahmed, Ibrahim H., Cillian Brewitt, Ignacio Carlucho, Filippos Christianos, Mhairi Dunion, Elliot Fosong, Samuel Garcin, Shangmin Guo, Balint Gyevnar, Trevor McInroe, Georgios Papoudakis, Arrasy Rahman, Lukas Schafer, Massimiliano Tamborski, Giuseppe Vecchio, Cheng Wang, and Stefano V. Albrecht. 2022. “Deep Reinforcement Learning for Multi-Agent Interaction.” AI Communications / arXiv:2208.01769.