Shield Risk Decision Trees

Context

Rieder2025 - Explainably Safe Reinforcement Learning explains probabilistic shields by learning compact decision trees over risk categories, allowed actions, and unsafe-action evidence. The core formal object is the hierarchy of risk-based classifiers derived from an MDP and safety property.

Formal Statement

Given an MDP M=<S,s_0,A,P>, safety property phi, horizon h, and threshold epsilon, define the state risk as the minimum probability of violating phi within h steps under the safest available policy:

$$ris{k}_{M,\varphi }(s,h)=P_{M,\neg \varphi }^{min}(s,h).$$

After executing action a, risk propagates through successors:

$$ris{k}_{M,\varphi }(s,a,h)=\sum _{s^{\prime }\in S}P(s,a,s^{\prime })ris{k}_{M,\varphi }(s^{\prime },h-1).$$

An action is safe when:

$$ris{k}_{M,\varphi }(s,a,h)\le ϵ.$$

The state partition is:

• S_s: all actions are safe;
• S_c: at least one action is safe and at least one is unsafe;
• S_d: the state satisfies phi but all actions exceed the threshold;
• S_u: the state violates phi.

The shield allows safe actions and, in dangerous states, allows an action with minimum risk.

Derivation / Construction

• Use probabilistic model checking or dynamic programming to compute state/action risks.
• Learn a Level 1 decision tree that classifies states into safe, critical, dangerous, and unsafe categories.
• For a current critical or dangerous state, learn a Level 2 decision tree over the corresponding Level 1 leaf to explain which actions the shield permits.
• For a selected unsafe action, construct a Markov chain that takes that action once and then follows safest actions.
• Enumerate the most probable unsafe traces until their probability mass exceeds epsilon; prefix-merge them into a Level 3 execution tree.

Implications

• The explanation target is the shield’s formal abstraction, not the learned policy internals.
• User-defined predicates can make trees smaller and more intuitive, but they also become part of the explanation design.
• Dangerous-state explanations are especially important because the shield may permit the least risky action even though all actions exceed the safety threshold.
• The tree-size metric evaluates compactness, but deployment usefulness also depends on whether users understand and trust the predicates.

Links

• Source: Rieder2025 - Explainably Safe Reinforcement Learning
• Concepts: Explainable Shielding, Shielding, Probabilistic Model Checking