Discounted Safety Bellman Equation

Context

Fisac2019 - Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning adapts minimum-over-time Hamilton-Jacobi safety analysis to reinforcement learning. The central move is a discounted backup that is a contraction, unlike the ordinary undiscounted infinite-horizon safety backup.

Formal Statement

For a deterministic system with discrete approximation:

$$x^{+}=x+f(x,u)\Delta t,$$

and a safety payoff l(x) whose nonnegative values mark safe states, the undiscounted safety backup is:

$$V(x)=\min \{l(x),\underset{u\in U}{\max }V(x^{+})\}.$$

Fisac et al. define the discounted safety backup:

$$B[V](x)=(1-\gamma )l(x)+\gamma \min \{l(x),\underset{u\in U}{\max }V(x^{+})\},\gamma \in [0,1).$$

This operator is a contraction under the supremum norm with contraction constant gamma:

$$\Vert B[V]-B[\tilde{V}]{\Vert }_{\infty }\le \gamma \Vert V-\tilde{V}{\Vert }_{\infty }.$$

As gamma -> 1, the fixed-point solution converges to the undiscounted safety value.

Derivation / Construction

The discount has a terminal-event interpretation. In additive reward RL, 1-gamma can be read as the probability that the episode terminates before future rewards accrue. For minimum-over-time safety, if the episode terminates after the current step, the future minimum safety payoff is just the current l(x).

This gives a locally discounted minimum:

$$J(\xi )=(1-\gamma )l_0+\gamma \min \{l_0,(1-\gamma )l_1+\gamma \min (l_1,\dots )\}.$$

The state-action version supports Safety Q-learning:

$$Q(x,u)=(1-\gamma )l(x)+\gamma \min \{l(x),\underset{u^{\prime }\in U}{\max }Q(f(x,u),u^{\prime })\}.$$

The corresponding update is:

$$Q_{k+1}(x,u)\leftarrow Q_k(x,u)+{\alpha }_k\left((1-\gamma )l(x)+\gamma \min \{l(x),\underset{u^{\prime }}{\max }{Q}_k(f(x,u),u^{\prime })\}-Q_k(x,u)\right).$$

For finite state/action discretizations, sufficient exploration, and standard learning-rate conditions, the paper adapts Q-learning convergence arguments to show almost-sure convergence to the optimal state-action safety value.

Implications

• The construction makes safety analysis compatible with temporal-difference learning without pretending safety is an additive reward.
• Annealing gamma toward 1 extends the effective safety horizon and recovers the undiscounted safety value in the limit.
• The learned value can classify safe/unsafe states and supply a safety-preserving policy, but neural approximators do not inherit the tabular convergence guarantee.
• Model-free training may require failures, so the method is most appropriate for simulation, noncritical training environments, or post-training supervisory control.

Links

• Source: Fisac2019 - Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning
• Concept: Hamilton-Jacobi Reachability
• Safe-RL context: Safe Reinforcement Learning