Constrained Policy Optimization

Definition

Constrained Policy Optimization is a trust-region policy-search algorithm for CMDPs that updates a policy to improve surrogate reward while satisfying surrogate expected-cost constraints and an average KL trust-region bound.

Why It Matters

CPO is one of the cleanest safe-RL examples where safety is represented as explicit constraints rather than a hand-tuned scalar penalty. It also becomes a reference point for later safe MARL algorithms such as MACPO.

Formalism / Key Objects

• Starting from policy pi_k, CPO optimizes a local surrogate reward:

$$E_{s\sim d_{{\pi }_k},a\sim \pi }[A^{{\pi }_k}(s,a)].$$

• For each cost C_i, the surrogate feasibility condition is:

$$J_{C_i}({\pi }_k)+\frac{1}{1-\gamma }{E}_{s\sim d_{{\pi }_k},a\sim \pi }[A_{C_i}^{{\pi }_k}(s,a)]\le d_i.$$

• The update is constrained by an average KL trust region:

$${\bar{D}}_{KL}(\pi \Vert {\pi }_k)\le \delta .$$

• The practical neural-policy step linearizes reward and costs and quadratically approximates the KL term, yielding a constrained quadratic program.
• CPO Trust Region Update records the performance-difference bound, cost bound, and practical approximation.

Connections

• Achiam2017 - Constrained Policy Optimization introduces the algorithm and its empirical comparison against primal-dual and fixed-penalty baselines.
• Gu2022 - Multi-Agent Constrained Policy Optimisation adapts CPO-style reasoning to sequential multi-agent trust-region updates.
• Constrained Markov Decision Processes supply the problem class; CPO is a neural-policy-search method for that class.
• Shielding can be combined with or substituted for CPO depending on whether safety is better represented as cumulative cost constraints or action-level forbidden behavior.

Common Confusions

• CPO does not make every sampled trajectory safe; it constrains expected discounted costs under approximation.
• Per-iteration constraint satisfaction depends on the trust-region approximation and sampling quality.
• A Lagrangian baseline can converge to a constrained solution while still violating constraints during learning.

Key Sources

• Achiam2017 - Constrained Policy Optimization
• CPO Trust Region Update
• Gu2022 - Multi-Agent Constrained Policy Optimisation