A Review of Causal Decision Making

Summary

Ge et al. survey Causal Decision Making as a bridge between causal inference, policy learning, bandits, and reinforcement learning. The paper’s central organizing move is to decompose causal decision making into three tasks: causal structure learning (CSL), causal effect learning (CEL), and causal policy learning (CPL).

The review then classifies decision-making data into six paradigms determined by whether the data are collected under a fixed or adaptive policy and whether state dependence is independent, Markovian, or non-Markovian. This gives one map that covers single-stage treatment decisions, panel data, dynamic treatment regimes, contextual bandits, offline RL, online RL, and partially observable settings.

Key Claims

• Causal modeling is most important when unchosen actions have unobserved counterfactual outcomes and naive association cannot answer what would have happened under intervention.
• CSL, CEL, and CPL are connected rather than separate literatures: causal structure constrains effect identification, effect learning can inform policy choice, and policy learning can create better experimental designs or refine causal structure.
• Offline RL is fundamentally a causal problem because evaluating or optimizing a target policy from logged data requires counterfactual reasoning about actions not taken by the behavior policy.
• Online bandits and online RL also face causal issues because exploration collects action-dependent data and leaves alternative rewards missing.
• The usual causal assumptions are doing real work in policy learning: SUTVA/consistency, no unmeasured confounding, and positivity/overlap are needed for many value and effect estimands to be identifiable.
• Practical CDM must handle assumption violations such as unmeasured confounders, interference between units or agents, and poor overlap in offline data.

Methods / Formalism

• The survey aligns potential-outcome notation with RL notation by using state S, action A, and reward R.
• Potential outcomes and do-interventions are treated as equivalent views in the reviewed setup: R(a) == R{do(A=a)}.
• Causal identifiability is stated as invariance of a causal quantity across data-generating models that agree on the observed distribution under the assumed causal structure.
• CEL estimands include average treatment effects, heterogeneous treatment effects, and mediation effects.
• CPL estimands include policy value and optimal-policy regret, with off-policy evaluation estimating the value of a target policy from logged behavior-policy data.
• The paper reviews direct, inverse-probability-weighted, and doubly robust estimators across CEL and OPE, then connects them to fitted-Q evaluation, marginal importance ratios, outcome-weighted learning, Q-learning, A-learning, pessimism, and penalty-based offline RL.
• Causal Decision-Making Taxonomy records the paper’s reusable three-task/six-paradigm schema and core identification assumptions.

Evidence / Experiments

• This is a survey paper rather than a new algorithm paper. Its evidence consists of a structured literature review, task/paradigm tables, software-package mapping, and two demonstration case studies.
• In the MIMIC-III sepsis example, the paper uses NOTEARS-style CSL over selected variables, mediation-style CEL for IV fluid input, and Q-learning-style CPL for individualized treatment rules. The illustrative CEL result reports high IV input as associated with roughly a 20%-25% increase in mortality in the selected setup, motivating personalized policy learning rather than a blanket treatment rule.
• In the MovieLens example, the paper uses CEL estimates of genre preferences to initialize online Linear Thompson Sampling. The informative prior from logged data improves early online performance compared with an uninformative prior, while online exploration eventually improves beyond static greedy recommendations.
• The software survey maps packages such as causal-learn, GCastle, EconML, DoWhy, d3rlpy, CORL, RLlib, OBP, SCOPE-RL, DynTxRegime, SMPyBandits, ACME, TensorForce, and POMDP-PY to CSL/CEL/CPL tasks and paradigms.

Connections

• Seeds Causal Decision Making as the wiki page for causal inference as policy-evaluation and policy-optimization infrastructure.
• Extends Causality from explanation and structural semantics toward action selection, policy value, and decision support.
• Connects to Decision Theory by making policy choice depend on identified counterfactual value rather than observed reward alone.
• Connects to Probability and Statistics through identifiability, weighting, doubly robust estimation, and statistical inference under adaptive data collection.
• Connects to Structural Equation Models because SCMs provide the intervention language behind causal graphs, causal bandits, fairness constraints, and action explanations.
• Connects to Q-Learning because the survey treats Q-learning, fitted-Q evaluation, and fitted-Q iteration as policy-learning tools that can be read through a causal/off-policy lens.

Open Questions

• How should the wiki separate ordinary RL concepts from causal-policy-learning concepts when the methods overlap but the assumptions are different?
• Which CDM concepts deserve their own pages later: causal structure learning, causal effect learning, off-policy evaluation, causal bandits, dynamic treatment regimes, or interference?
• How much causal structure should safe-RL and MARL notes expose when discussing offline evaluation, confounding, or multi-agent interference?
• Can the paper’s six-paradigm taxonomy help route future raw intake in causality, RL, bandits, and econometrics without flattening their domain-specific assumptions?

Citation

Ge, L., Cai, H., Wan, R., Xu, Y., and Song, R. (2026). A Review of Causal Decision Making. Journal of Artificial Intelligence Research, 85, Article 41. https://doi.org/10.1613/jair.1.21001