Probabilistic Risk-Budget Shields

Context

Probabilistic Shielding turns safety into a live risk-budget invariant. The known-model version uses certified unsafe-reachability upper bounds; ProSh uses a learned backup cost critic in a CMDP. Both constructions augment the learner’s interface so that safety is enforced before an action is sampled.

Formal Statement

For unsafe-reachability shielding, the target constraint is:

$$M_{\pi }\models P_{\le p}(Reach(u)).$$

Let beta_M(s)=min_pi Pr_{M,pi,s}(Reach(u)), and let beta be an inductive upper bound with B_M(beta)(s) <= beta(s). The shield state is (s,q) with q >= beta(s). A shield action chooses a base action distribution and a next-budget prediction alpha, constrained so that:

$$\sum _a{v}_a\left(q-\sum _{s^{\prime }}P(s,a,s^{\prime })\alpha (s^{\prime })\right)\ge 0$$

and alpha(s') >= beta(s'). Any memoryless policy in this shield projects to a base-MDP policy satisfying the unsafe-reachability threshold.

For ProSh-style CMDP shielding, a state is (s,x) and the action is (a,y). Given a backup critic Q_b, the risk coordinate updates as:

$$x^{\prime }=y-Q_b(s,a)+Q_b(s^{\prime }).$$

If Delta_b=||Q_b-Q_b^*||_\infty, every Q_b-shielded policy satisfies:

$$C_{(s_0,x_0)}(\bar{\pi })\le x_0+\frac{2{\Delta }_b}{1-{\gamma }_c}.$$

Derivation / Construction

1. Choose the safety quantity: unsafe-reachability probability for a known MDP, or discounted expected cost for a CMDP.
2. Obtain a conservative risk object: an inductive upper bound beta, or a backup critic Q_b whose error can be bounded or at least monitored.
3. Add a scalar risk-budget coordinate to the state.
4. Restrict or transform actions so the expected next budget remains below the current budget.
5. Train an ordinary reward learner inside the shielded system.
6. Project the shielded policy back to the original MDP/CMDP while preserving the safety or cost guarantee stated by the risk object.

Implications

• Risk-budget shields are less conservative than hard action masks when safe optimality requires randomized risk allocation.
• Known-model shields can give strict formal guarantees, but they inherit the cost of computing conservative reachability bounds.
• Model-free shields are scalable and differentiable, but their guarantee moves onto backup-critic accuracy.
• The construction cleanly separates the reward learner from the safety certificate, making it reusable across PPO, TD3, and other RL algorithms.

Links

• HamelDeLeCourt2025 - Probabilistic Shielding for Safe Reinforcement Learning
• HamelDeLeCourt2025 - ProSh Probabilistic Shielding for Model-free Reinforcement Learning
• Probabilistic Shielding
• Shielding
• Sound Value Iteration