Aspiration-Based Reinforcement Learning in Repeated Interaction Games: An Overview

Summary

This partial ingest covers Bendor, Mookherjee, and Ray’s overview of aspiration-based reinforcement learning in repeated interaction games. The paper surveys models where agents do not maximize an explicit expected-utility model; instead, they compare achieved payoffs to aspiration levels, reinforce satisfactory actions, and abandon or experiment after unsatisfactory outcomes. The source is especially useful for Strategic Reasoning because it shows how boundedly rational, myopic adaptation can converge to cooperation, Pareto-efficient coordination, or other non-Nash outcomes in repeated games.

Key Claims

• Aspiration-based reinforcement learning models agents as satisficers: an action is more likely to recur when its payoff meets or exceeds an aspiration level, and less likely to recur when the payoff disappoints.
• In repeated Prisoner’s Dilemma-style games, cooperative behavior can emerge and persist even when cooperation is strictly dominated in the one-shot stage game.
• Long-run outcomes can differ sharply from belief learning and evolutionary replicator dynamics because agents react to payoff satisfaction rather than optimizing against explicit beliefs or population fitness.
• Fixed but payoff-consistent aspirations can support equilibria concentrated on cooperative strategies; endogenous aspirations based on personal experience can make cooperation the unique long-run outcome under slow updating and rare trembles.
• Generational and peer-based aspiration models show how social inheritance or comparison classes can stabilize collusion, fairness, or cooperation without common-knowledge rationality.
• The authors emphasize that these are theoretical models of boundedly rational adaptation, not new experiments; empirical support comes from the broader reinforcement-learning and experimental-game literature they survey.

Methods / Formalism

• The base setting is a two-player finite stage game repeated at dates t = 1, 2, ..., with action sets A and B and payoff functions f(a,b) and g(a,b).
• A player’s behavioral state may be a pure action or a mixed strategy over actions. The state is updated from payoff experience relative to an aspiration level.
• In endogenous-aspiration models, the paper uses the anchor

$$A_{t+1}=\lambda A_t+(1-\lambda ){\pi }_t,$$

where A_t is the current aspiration, pi_t is the payoff just received, and lambda in (0,1) controls persistence.

• Positive reinforcement increases or preserves the probability of the chosen action when pi_t >= A_t; negative reinforcement triggers switching or experimentation when pi_t < A_t.
• In the canonical symmetric 2 x 2 cooperation game with payoffs (sigma, sigma) for mutual cooperation and (delta, delta) for mutual defection, with sigma > delta > 0, aspirations just below sigma make mutual cooperation the only jointly satisfactory state.
• The generational model characterizes pure stable outcomes: every pure stable outcome is individually rational and either efficient or a protected Nash equilibrium; conversely, strictly individually rational efficient outcomes and strict Nash equilibria are stable under the paper’s perturbation criterion.
• See Aspiration Dynamics in Repeated Games for the compact formal schema and stability conditions.

Evidence / Experiments

• The paper is a theory overview rather than an experimental report.
• It cites mathematical psychology, economics, and experimental-game work on reinforcement learning, including Bush and Mosteller, Roth and Erev, Selten, and Camerer and Ho.
• Applications discussed include repeated oligopoly, Prisoner’s Dilemma, coordination games, electoral competition, population bargaining, and peer-comparison models.

Connections

• Seeds Aspiration-Based Reinforcement Learning as a bounded-rationality mechanism for cooperation and coordination in repeated interaction.
• Aspiration Dynamics in Repeated Games records the reusable aspiration-update equation, cooperative-game setup, and pure-stable-outcome criteria.
• Connects to Opponent Shaping by offering an older, lower-cognition route to cooperation: agents need not model another learner’s gradient update if satisfaction-driven switching already changes the strategic state.
• Connects to Multi-Agent Non-Stationarity because each agent’s payoff experience and aspiration update change the effective environment faced by the other.
• Belongs in Decision Theory as a satisficing alternative to expected-utility maximization under limited information and limited modeling capacity.

Open Questions

• Which modern MARL environments would separate aspiration-driven cooperation from opponent-shaping cooperation?
• How should aspiration levels be estimated or represented for neural agents whose internal satisfaction criteria are not explicit?
• Can safety constraints be integrated into aspiration formation rather than treated only as external filters or penalties?
• Which peer-comparison aspiration models best capture institutional or population-level AI-agent deployment settings?

Citation

Bendor, J., Mookherjee, D., & Ray, D. (2001). Aspiration-Based Reinforcement Learning in Repeated Interaction Games: An Overview. International Game Theory Review, 3(2-3), 159-174.