Decision Theory

Overview

This hub tracks utility, preferences, risk, constraints, choice under uncertainty, and normative structure for agent design and evaluation.

Core Questions

• How should preferences, uncertainty, constraints, and trade-offs be represented and compared?
• Which assumptions about rationality, objectives, or risk are doing the real work in a paper?
• How do decision-theoretic lenses apply across RL, strategic interaction, ethics, planning, and embodied systems?
• How should interventions and strategic influence be evaluated when other agents learn in response?
• How should agent abilities be represented when choices are embedded in temporal, causal, or coalition logics?
• How should regret, dominance, and responsibility be compared when outcomes are value sets rather than scalar utilities?
• Which equilibrium notions remain meaningful when feasible choices are jointly constrained by safety requirements?
• How should policy choice be evaluated when other agents are adapting at the same time?
• When should a rational agent defer to oversight rather than act directly, and how does that change in groups?
• How should feasible action sets be restricted by safety filters while preserving useful reward optimization?
• When should bounded agents be modeled as satisficers with aspiration thresholds rather than maximizers with explicit beliefs?
• How should descriptive moral appraisal be separated from normative action selection when both are represented algebraically?
• When is safety best represented as a hard feasible set, an expected-cost constraint, a population-level distributional constraint, or a runtime action filter?
• When should probabilistic risk be spent as a live budget rather than collapsed into a hard feasible set?
• What should an explanation of constrained choice reveal: the objective, the constraint, the risk model, or the intervention rule?
• When is a shared team reward enough to justify decentralized execution, and when does credit assignment require a stronger factorization or centralized critic?
• When should another agent’s incentive signal be treated as helpful information, strategic manipulation, or reward tampering?
• How should agents choose under uncertainty over teammate type, subtask allocation, and partner goals?
• When can sampled experience replace an explicit transition model while preserving optimal-control guarantees?
• When should policy evaluation be treated as a counterfactual identification problem rather than a prediction problem?
• When does a sequential decision problem require equilibrium reasoning over multiple value functions rather than optimization of one objective?
• When can system-level choice be reduced to verified choices among reusable learned subsystems?
• When does maximizing observed reward stop tracking true utility because the agent can corrupt the feedback process?
• How should expected utility, equilibrium, and counterfactual evaluation interact when interventions change the game being played?
• When does a decision system need simulated counterfactual distributions rather than identified average effects?
• How should risk thresholds in AI safety specifications be chosen, audited, and verified?

Current Research Touchpoints

• Causal Games
• Multi-Agent Influence Diagrams
• Causal Game Intervention Semantics
• Goal Alignment
• Reward Tampering
• Corrupt Reward MDPs
• Opponent Shaping
• Stochastic Games
• Compositional Reinforcement Learning
• Constrained Markov Decision Processes
• Constrained Policy Optimization
• Mean-Field Reinforcement Learning
• Explainable Shielding
• Aspiration-Based Reinforcement Learning
• Probabilistic Controlled Invariant Sets
• Corrigibility
• Constrained Markov Potential Games
• Multi-Agent Non-Stationarity
• Cooperative Multi-Agent Reinforcement Learning
• Multi-Agent Coordination
• Ad Hoc Teamwork
• Social Learning in MARL
• Q-Learning
• Multi-Agent PPO
• Value Decomposition Networks
• Causal Decision Making
• Counterfactual Simulation
• Structural Equation Models
• Actual Causality
• Responsibility Anticipation
• Dyadic Morality
• Temporal Causal Models
• Alternating-Time Temporal Logic
• Strategic Reasoning
• Probabilistic Shielding
• Sound Value Iteration
• Causality
• Safe Multi-Agent Reinforcement Learning
• Guaranteed Safe AI
• Formal Methods

Key Concepts

• Strategic Reasoning
• Causal Games
• Multi-Agent Influence Diagrams
• Causal Game Intervention Semantics
• Goal Alignment
• Reward Tampering
• Corrupt Reward MDPs
• Constrained Markov Decision Processes
• Constrained Policy Optimization
• Mean-Field Reinforcement Learning
• Aspiration-Based Reinforcement Learning
• Probabilistic Controlled Invariant Sets
• Corrigibility
• Responsibility Anticipation
• Dyadic Morality
• Stochastic Games
• Compositional Reinforcement Learning
• Constrained Markov Potential Games
• Multi-Agent Non-Stationarity
• Cooperative Multi-Agent Reinforcement Learning
• Multi-Agent Coordination
• Ad Hoc Teamwork
• Social Learning in MARL
• Q-Learning
• Multi-Agent PPO
• Value Decomposition Networks
• Causal Decision Making
• Counterfactual Simulation
• Alternating-Time Temporal Logic
• Safe Reinforcement Learning
• Probabilistic Shielding
• Sound Value Iteration
• Distributional Value Iteration
• Probabilistic Model Checking
• Causality