Boundary-Seeking Policy Gradient for Safe Reinforcement Learning

Reinforcement learning (RL) has achieved remarkable success in sequential decision-making tasks including game playing [1], autonomous driving [2], and robotics [3]. In safety-critical applications, however, maximizing reward alone is insufficient: a deployed policy must also satisfy operational constraints, a requirement commonly formalized through Constrained Markov Decision Processes (CMDPs) [4].

Existing constrained policy optimization follows several directions. Primal–dual methods [5, 6, 7, 8, 9, 10, 11] enforce the constraint through a learned multiplier, so the policy’s location relative to an active boundary is controlled only indirectly through the dual dynamics. Primal and projection-based methods [12, 13, 14, 15], and switching schemes such as CRPO [16], restrict or correct updates by feasibility — a one-sided condition that does not say how much of the remaining cost budget a feasible policy should use. Gradient-manipulation methods [17, 18, 19] coordinate the two objectives through gradient geometry, which captures local alignment but not where the policy sits relative to the cost limit. In short, standard updates do not separate reward improvement along a cost level set from signed regulation across cost levels toward an active boundary.

This separation matters because of a well-known structural property: when the CMDP model is known, safe RL admits an exact linear-programming solution over occupancy measures [4], and whenever the constraint is active at optimality, the optimal policy lies on the boundary of the feasible set. A strictly feasible policy therefore retains cost slack that can be locally converted into reward when the reward and cost gradients are positively aligned; Section IV makes both the global statement and this local trade-off precise.

Motivated by this structure, we propose Boundary-Seeking Policy Gradient (BSPG), which makes signed boundary regulation an explicit, separate component of the update. A tangential term removes from the reward gradient its component along the cost gradient, preserving cost to first order while retaining reward ascent; a normal term proportional to the signed residual decreases cost when the policy is infeasible and increases cost toward the limit when it is strictly feasible. The ideal update admits an algebraic Lagrangian representation with an induced coefficient and maintains no learned dual variable. Our contributions are:

1. 1.

   We characterize the boundary structure of constrained policy optimization: when no unconstrained reward maximizer is feasible, every constrained occupancy-measure maximizer uses the full cost budget, this target transfers to any policy class attaining the exact constrained optimum, and we identify when local slack supports first-order reward improvement.

2. 2.

   We propose BSPG and, for the ideal exact-gradient update, prove two-sided residual convergence with a finite-horizon $O(1/\sqrt{T})$ bound, reward ascent of the tangential component, and a characterization of any convergent run: the limit is stationary on the active set and satisfies the KKT conditions when it is also a local maximizer over the feasible set.

3. 3.

   On a standard Safety-Gymnasium navigation task, BSPG achieves higher reward with tighter boundary adherence than the compared baselines.

Primal–dual methods enforce safety through a Lagrange multiplier balancing reward against violation. PPO- and TRPO-Lagrangian [20] update the multiplier from the observed violation, with coupled dynamics that can be sensitive and oscillatory; PID-Lagrangian [5] stabilizes the dual update at the cost of extra tuning, and RCPO [21] and CAL [22] refine cost estimation and the coupled updates. In all cases the residual acts on the multiplier, which then reshapes the policy step: boundary regulation is not an explicit component of the update itself.

Primal methods restrict or correct updates by feasibility. CPO [12] solves a constrained local subproblem with guarantees but added computational cost [20]; PCPO [13], FOCOPS [14], and CUP [15] give first-order and projection-based variants; CRPO [16] alternates reward and constraint steps by feasibility status, and CPPO [23] adds an infeasibility-recovery mechanism. Feasibility, however, is one-sided: a policy with large slack and one at the boundary satisfy the same inequality, and when slack is locally convertible into reward (Section IV), feasibility alone does not determine how much of the budget to use.

Gradient-manipulation methods are closest to our geometric view. PCRPO [17] selects or combines directions by the reward–cost gradient angle, GradS [19] extends this to multiple constraints, and ESPO [18] uses gradient interaction to improve efficiency. But gradient geometry is not boundary location: two policies with similar gradient directions can hold very different slack, so geometry alone determines neither the direction nor the magnitude of motion across cost levels. These methods can reach an active boundary, yet their updates do not use the signed residual as a separate normal correction regulating the policy toward it.

We consider a CMDP defined by the tuple $(\mathcal{S},\mathcal{A},P,r,c,\gamma,\mu_{0},d)$, where $\mathcal{S}$ and $\mathcal{A}$ denote the state and action spaces, $P(s^{\prime}\mid s,a)$ is the transition kernel, $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ and $c:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ are the reward and cost functions, $\gamma\in(0,1)$ is the discount factor, $\mu_{0}$ is the initial-state distribution, and $d\in\mathbb{R}$ is the cost threshold. A policy $\pi:\mathcal{S}\to\Delta(\mathcal{A})$ maps each state to a distribution over actions. For a policy $\pi_{\bm{\theta}}$ parameterized by $\bm{\theta}\in\mathbb{R}^{p}$, define the expected discounted reward and cost as

Safe RL aims to solve the following constrained optimization problem:

To describe the constraint geometry in policy parameter space, define

Since our method is a policy-gradient algorithm, we next introduce the reward and cost gradients that determine the local update direction. These quantities also provide the basis for the geometric decomposition developed later. By the policy gradient theorem [24], define the normalized discounted state distribution $d_{\gamma}^{\pi_{\bm{\theta}}}(s)=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}\Pr_{\pi_{\bm{\theta}}}(s_{t}=s)$. Then

where $A_{r}^{\pi_{\bm{\theta}}}(s,a)=Q_{r}^{\pi_{\bm{\theta}}}(s,a)-V_{r}^{\pi_{\bm{\theta}}}(s)$ is the reward advantage, and $A_{c}^{\pi_{\bm{\theta}}}(s,a)=Q_{c}^{\pi_{\bm{\theta}}}(s,a)-V_{c}^{\pi_{\bm{\theta}}}(s)$ is the cost advantage.

The key observation during policy improvements is that the reward and cost gradients may not be aligned. For a local update $\bm{\theta}^{\prime}=\bm{\theta}+\alpha\bm{v}$, differentiability gives $J_{k}(\bm{\theta}^{\prime})-J_{k}(\bm{\theta})=\alpha\left\langle\bm{g}_{k},\,\bm{v}\right\rangle+o(\alpha)$, so $\bm{v}$ improves reward to first order if $\left\langle\bm{g}_{r},\,\bm{v}\right\rangle>0$ and reduces cost if $\left\langle\bm{g}_{c},\,\bm{v}\right\rangle<0$. When $\left\langle\bm{g}_{r},\,\bm{g}_{c}\right\rangle>0$, a naive ascent step along $\bm{g}_{r}$ increases both reward and cost to first order, the gradient conflict near an active constraint that our method is designed to handle.

We begin with a basic structural observation about the CMDP in (3) that motivates our algorithm design. When the constraint is active at the optimum, the optimal constrained solution should lie on the boundary of the feasible set rather than in its strict interior. Intuitively, if a feasible policy $\pi_{\bm{\theta}}$ satisfies the constraint with strict inequality, i.e., $J_{c}(\pi_{\bm{\theta}})<d$, then, by continuity of $J_{c}$ in $\bm{\theta}$, there exists a neighborhood around $\bm{\theta}$ in which the constraint remains inactive. In such a region, moving toward the feasible boundary can potentially improve reward without immediately violating the constraint. The following theorem formalizes this observation.

Suppose, for contradiction, that some constrained-optimal $\pi^{\star}$ satisfies $J_{c}(\pi^{\star})<d$, and write $\rho^{\star}=\rho_{\pi^{\star}}$. Fix any unconstrained maximizer $\pi_{r}^{\star}$ with occupancy measure $\rho_{r}$; by hypothesis $J_{c}(\rho_{r})>d$. If $J_{r}(\rho_{r})=J_{r}(\rho^{\star})$, then $\pi^{\star}$ would itself be a feasible unconstrained maximizer, contradicting the hypothesis; hence $J_{r}(\rho_{r})>J_{r}(\rho^{\star})$. For $\lambda\in(0,1]$ let $\rho_{\lambda}=(1-\lambda)\rho^{\star}+\lambda\rho_{r}\in\Lambda$ (convexity). By linearity, $J_{r}(\rho_{\lambda})=(1-\lambda)J_{r}(\rho^{\star})+\lambda J_{r}(\rho_{r})>J_{r}(\rho^{\star})$, and $J_{c}(\rho_{\lambda})=(1-\lambda)J_{c}(\rho^{\star})+\lambda J_{c}(\rho_{r})\leq d$ for every $\lambda\leq\bigl(d-J_{c}(\rho^{\star})\bigr)/\bigl(J_{c}(\rho_{r})-J_{c}(\rho^{\star})\bigr)$, a strictly positive threshold. Thus, for small $\lambda>0$, the policy induced by $\rho_{\lambda}$ is feasible with strictly larger reward than $\pi^{\star},$ a contradiction. Hence $J_{c}(\pi^{\star})=d$. $\blacksquare$

The proof, which relies only on the convexity of the occupancy-measure polytope and the linearity of $J_{r},J_{c}$ in the occupancy measure [4].

Indeed, any global maximizer $\bm{\theta}^{\star}$ attains the constrained value, so the induced policy $\pi_{\bm{\theta}^{\star}}$ is constrained-optimal in $\Pi$, and Theorem IV.1 gives $J_{c}(\pi_{\bm{\theta}^{\star}})=d$. Thus, nonconvex parameterization may create additional local stationary points, but it does not change the active cost level of a realizable global optimum.

Theorem IV.1 provides the basic theoretical motivation for BSPG: when the constraint is active at the optimum, the constrained optimum of the underlying CMDP uses the full cost budget. Thus, in safe RL, the objective is not merely to remain feasible, but to approach and optimize along the active constraint boundary, where the optimal constrained solution is attained.

Proposition IV.5 makes the intended claim precise: not every interior policy incurs a reward loss; rather, under positive gradient alignment, a fraction $\beta$ of the unused cost budget can be converted into a first-order reward gain of $\beta|\delta|\left\langle\bm{g}_{r},\,\bm{g}_{c}\right\rangle/\left\|\bm{g}_{c}\right\|^{2}$ while preserving strict feasibility. This result motivates the constraint residual as a diagnostic of boundary tracking. The residual alone is not a universal surrogate for reward, because the local reward change also depends on the alignment and magnitudes of the reward and cost gradients.

Theorem IV.1 and Corollary IV.2 place the global constrained optimum on the active constraint set $\mathcal{M}$ under their stated conditions; $\mathcal{M}$ may also contain non-optimal points. A natural algorithmic strategy is therefore to improve reward along $\mathcal{M}$ while regulating the iterates toward $\mathcal{M}$. Motivated by this, we propose Boundary-Seeking Policy Gradient (BSPG) Algorithm (algorithm 1) in this section, which implements this strategy through a gradient decomposition that separates the two goals into orthogonal components.

We now establish the convergence properties of the ideal BSPG update with exact gradients and $\hat{\delta}_{t}=\delta_{t}$; the statements below do not claim convergence of the stochastic normalized PPO implementation. Three results are proved: the constraint residual converges to zero from either side (with a finite-horizon rate), the tangential component is a reward-ascent direction on the boundary, and any convergent run is characterized at its limit. The proofs of Theorems VI.1–VI.4 are provided in Appendices -B–-D. We begin by collecting the regularity assumptions.

Assumptions 1 and 3 are standard in first-order constrained policy optimization [16]; if $0\leq c(s,a)\leq c_{\max}$ and $0\leq d\leq c_{\max}/(1-\gamma)$, the residual bound holds automatically with $D=\max\{d,\,c_{\max}/(1-\gamma)-d\}$. Assumption 2 is a nondegeneracy (constraint-qualification) condition excluding points where the cost gradient vanishes; it makes the local cost level sets regular and ensures the decomposition (13) is well defined at every iterate.

We first show that BSPG drives the constraint residual $J_{c}(\pi_{\bm{\theta}_{t}})-d$ to zero.

The corollary follows by summing (20) over $t<T$ and dividing by $\mu^{\prime}A_{T}$; see Appendix -B.

The Lyapunov function $V=\tfrac{1}{2}\delta^{2}$ is symmetric in the sign of the residual, so the ideal normal component regulates the target cost level from either side. This differs from a standard projected dual update: a negative residual can decrease the dual variable only until it reaches zero, after which the boundary signal is lost, whereas BSPG continues to apply a residual-proportional normal correction toward the prescribed cost level.

It is worth noting that the Lyapunov function $V=\frac{1}{2}\delta^{2}$ is symmetric in the sign of $\delta$, so the boundary-seeking mechanism operates identically whether the current iterate is feasible or infeasible. This contrasts with Lagrangian methods, where the multiplier update acts only on positive violations.

The boundary-attraction term drives $\delta\to 0$; however, it does not by itself guarantee reward improvement. The following theorem shows that the tangential component $\bm{g}_{r}^{\perp}$ is a valid ascent direction for $J_{r}$.

We next characterize convergent runs of the exact update.

As shown in Fig. 3, BSPG attains the highest episodic reward on SafetyPointGoal1-v0. At the same time, BSPG maintains episodic cost near the constraint limit, while both baselines remain more conservative and settle well inside the feasible region. This behavior is most clearly illustrated in the boundary-proximity plot, where BSPG remains consistently closest to zero, indicating tighter tracking of the active constraint boundary. Taken together, these results support the proposed boundary-regulation mechanism in the evaluated task; they do not by themselves imply a general performance ordering across environments.

In this paper, we proposed Boundary-Seeking Policy Gradient (BSPG), a safe reinforcement learning method that explicitly drives policy updates toward the active constraint boundary while improving reward along it. By combining a tangential reward term with a signed boundary-attraction term, BSPG encourages boundary-aligned optimization rather than conservative interior solutions. On the theory side, occupancy-measure analysis identifies the active boundary as the global constrained target when no unconstrained reward maximizer is feasible, and the realizability corollary transfers this target to a policy class attaining the exact constrained value. For the ideal exact-gradient update, BSPG drives the signed residual to zero from either side with a finite-horizon $O(1/\sqrt{T})$ bound, and any convergent run reaches a stationary point on the active constraint set, satisfying the KKT conditions when the limit is locally optimal over the feasible set. Empirically, BSPG attains a strong reward–safety trade-off in the evaluated task by making fuller use of the safety budget.

This work is partially supported by NSF ECCS Award #2534263.