Causal Decision Making

Definition

Causal decision making is decision making that evaluates actions, treatments, or policies by their intervention effects rather than by observed associations alone. Its technical center is the counterfactual gap between P(R | A=a) and P(R | do(A=a)): when these differ, a policy chosen from observed rewards may be biased.

Why It Matters

Many decisions are made from data where only the chosen action’s reward is observed. This turns policy evaluation into a counterfactual estimation problem: the system must estimate what would have happened under alternative actions or policies. Causal assumptions make clear when that estimation is valid, when it is underidentified, and which variables or experiments could repair it.

Formalism / Key Objects

• Causal structure learning (CSL): infer or assume a causal graph or structural model that says which variables are treatments, confounders, mediators, states, and outcomes.
• Causal effect learning (CEL): identify and estimate intervention effects such as:

$$ATE=E[R(1)-R(0)]=E[R|do(A=1)]-E[R|do(A=0)].$$

• Causal policy learning (CPL): evaluate or optimize policies using counterfactual value estimates. For a discounted policy pi, a standard value target is:

$$V^{p}i(s)=su{m}_{t>=0}gamm{a}^t{E}_{p}i[R_t|S_0=s].$$

• Identification assumptions: SUTVA/consistency, no unmeasured confounders, and positivity/overlap.
• Data paradigms: fixed-policy offline data versus adaptive-policy online data, each under independent-state, Markovian, or non-Markovian state dependence.
• Causal Decision-Making Taxonomy gives the compact task/paradigm grid from Ge2026 - A Review of Causal Decision Making.
• Causal Games generalize intervention-valued decision questions to strategic settings by quantifying over rational outcomes rather than a single fixed policy.
• Counterfactual Simulation samples from distributions such as p(W_do(X=x) | C=c) when the SCM is known but direct analytic counterfactual evaluation is hard.

Connections

• Structural Equation Models provide the do-intervention semantics and graph language for many CDM methods.
• Fox2024 - Causality and Strategic Reasoning distinguishes pre-policy interventions, where agents can adapt to the changed game, from post-policy interventions, where a policy profile is held fixed.
• Everitt2019 - Towards Safe Artificial General Intelligence uses causal graphs and do(pi) interventions to diagnose when an RL agent can influence its reward, observations, utility, or decision procedure.
• Multi-Agent Influence Diagrams provide the graphical game model that causal games lift into Pearl-style intervention and counterfactual semantics.
• Reward Tampering is a causal decision problem when the policy changes the feedback mechanism used to evaluate future actions.
• Decision Theory supplies the policy-choice frame: choose actions by expected value, regret, constraints, or other objectives.
• Probability and Statistics supplies identifiability, estimation, confidence intervals, weighting, matching, and doubly robust methods.
• Q-Learning becomes causal-policy-learning machinery when used to evaluate or optimize policies from logged observational data.
• Constrained Markov Decision Processes are adjacent when causal value estimates must respect cost, safety, fairness, or overlap constraints.
• Karvanen2024 - Simulating Counterfactuals connects CDM to individual fairness auditing by simulating how opaque model predictions change under sensitive-variable interventions.

Common Confusions

• Causal decision making is not just using a causal graph before an RL algorithm; it asks whether the policy-value target is identifiable from the data and assumptions.
• A good predictive reward model is not automatically a valid intervention model.
• A high observed reward under the current policy is not enough if the policy also caused the observation or reward channel to become unreliable.
• In a strategic setting, the effect of an intervention can differ depending on whether other agents learn about it before choosing their policies.
• Off-policy evaluation can look like ordinary supervised learning, but logged action choice creates selection bias unless the behavior policy and confounding assumptions are handled.
• Online exploration reduces some overlap problems, but it still creates adaptively collected, action-dependent data that need care for inference.

Key Sources

• Fox2024 - Causality and Strategic Reasoning
• Ge2026 - A Review of Causal Decision Making
• Everitt2019 - Towards Safe Artificial General Intelligence
• Karvanen2024 - Simulating Counterfactuals