Abstract:
ObjectivesThis study presents a learnability-aware safe reinforcement learning method for unmanned surface vehicle (USV) obstacle avoidance under complex sea conditions. The method is intended to alleviate policy mismatch, reduced learning efficiency, and performance degradation caused by excessive safety-layer intervention.
MethodsBuilt upon the Soft Actor-Critic (SAC) framework, a three-degree-of-freedom USV dynamic model and a Markov decision process for obstacle avoidance were formulated. A second-order HOCBF (Higher-Order Control Barrier Function) filter was introduced at the execution layer to enforce formal safety constraints, yielding an HOCBF-based safe reinforcement learning framework. On this basis, a learnability-aware optimization mechanism that jointly accounts for safety enforcement and policy learning was developed, resulting in LA-SafeSAC (Learnability-Aware Safe Soft Actor-Critic). Comparative experiments were conducted in calm-water, wave-disturbed, and wave-current-coupled scenarios.
ResultsLA-SafeSAC outperformed standard SAC+CBF in all three scenarios while maintaining stable-stage task completion performance comparable to that of the SAC baseline under complex disturbances. Relative to standard SAC+CBF, over the final 100 episodes, the success rate increased from 38.7%-62.0% to 79.7%-84.3%, the timeout rate decreased from 37.7%-54.3% to 15.7%-19.3%, and the CBF activation rate decreased from 35.9%-40.2% to 20.2%-24.1%. Training curves and safety-learning coordination metrics further indicate that the proposed method facilitates faster formation of effective policies under low collision risk, mitigates the long-term mismatch between nominal actions and safety-filtered actions, and progressively shifts the safety layer from persistent takeover to low-frequency corrective intervention.
ConclusionsThe proposed learnability-aware safety-layer optimization alleviates long-term conservative degradation, improves task completion capability and training stability, and enhances coordination between safety enforcement and policy learning while preserving formal safety guarantees.