wiki

Constrained Policy Optimization

3 min read

Constrained Policy Optimization

Summary

This partial ingest is based on the extracted full PDF text. Achiam et al. introduce Constrained Policy Optimization (CPO), a trust-region policy-search algorithm for Constrained Markov Decision Processes. The paper derives a tighter policy-performance bound in terms of an average policy divergence, uses the bound to justify surrogate reward and cost constraints, and implements a practical update that approximately improves reward while keeping auxiliary cost returns near specified limits during training.

Key Claims

  • CMDPs are a natural safe-RL formulation when a designer can specify reward and separate expected-cost constraints.
  • CPO is presented as the first general-purpose policy-search method for continuous CMDPs with per-iteration near-constraint-satisfaction guarantees.
  • A new performance-difference bound relates reward or cost-return changes between policies to expected advantages plus a divergence penalty.
  • Replacing direct off-policy constraint evaluation with on-policy surrogate cost constraints yields a practical trust-region method for neural policies.
  • Compared with primal-dual and fixed-penalty baselines, CPO tracks constraint limits more reliably while preserving return in the reported locomotion tasks.

Methods / Formalism

  • A CMDP augments an MDP with cost functions C_i:S x A x S -> R and thresholds d_i; feasible stationary policies satisfy J_Ci(pi) <= d_i.
  • The paper proves a bound for J(pi') - J(pi) using a surrogate advantage term under the old state distribution and an average total-variation/KL divergence term.
  • CPO approximately solves:

subject to surrogate cost constraints and an average KL trust-region constraint.

  • The finite-dimensional implementation linearizes reward and costs around policy parameters and uses a quadratic KL approximation, giving a constrained quadratic program.
  • Cost shaping tightens the constraints by optimizing upper-bound costs C_i^+ to reduce approximation-induced violations. CPO Trust Region Update records the reusable update and guarantee.

Evidence / Experiments

  • Tasks: Circle and Gather variants with Point, Ant, and Humanoid agents using neural policies with two hidden layers.
  • CPO is compared with primal-dual optimization (PDO), unconstrained TRPO, and fixed-penalty variants.
  • Reported learning curves show CPO driving cost returns close to the constraint limit across environments, while PDO can overcorrect, undercorrect, or violate constraints during training.
  • Cost shaping reduces approximation errors in the true cost return, especially in the Ant environments.
  • Fixed-penalty methods are highly sensitive to the penalty coefficient, whereas CPO adapts the trade-off through the constrained update.

Connections

  • Provides the single-agent constrained-policy baseline for Gu2022 - Multi-Agent Constrained Policy Optimisation, which extends trust-region constrained updates to Markov games.
  • Complements Shielding: CPO keeps the learner inside expected-cost constraints through optimization, while shields enforce action-level or trace-level safety at runtime.
  • Connects to Constrained Markov Potential Games by supplying the CMDP intuition that later multi-agent constrained games complicate.
  • Belongs under Decision Theory because the algorithm makes reward-versus-cost feasibility explicit rather than hiding safety in a scalar reward.

Open Questions

  • The guarantees are near-constraint satisfaction under approximations and sampling; how large can transient empirical violations become in high-stakes deployments?
  • How should expected discounted cost constraints be combined with hard state invariants or temporal-logic safety properties?
  • When does a runtime shield make CPO unnecessary, and when does CPO reduce the burden placed on a shield?

Citation

Achiam, Joshua, David Held, Aviv Tamar, and Pieter Abbeel. 2017. “Constrained Policy Optimization.” In Proceedings of the 34th International Conference on Machine Learning, PMLR 70.

Graph View

Backlinks