Improving Control Performance of Tilt-Rotor VTOL UAV with Model-Based Reward and Multi-Agent Reinforcement Learning

Abstract

Tilt-rotor Vertical Takeoff and Landing Unmanned Aerial Vehicles (TR-VTOL UAVs) combine fixed-wing and rotary-wing configurations, offering optimized flight planning but presenting challenges due to their complex dynamics and uncertainties. This study investigates a multi-agent reinforcement learning (RL) control system utilizing Soft Actor-Critic (SAC) modules, which are designed to independently control each input with a tailored reward mechanism. By implementing a novel reward structure based on a dynamic reference response region, the multi-agent design improves learning efficiency by minimizing data redundancy. Compared to other control methods such as Actor-Critic Neural Networks (AC NN), Proximal Policy Optimization (PPO), Nonsingular Terminal Sliding Mode Control (NTSMC), and PID controllers, the proposed system shows at least a 30% improvement in transient performance metrics-including RMSE, rise time, settling time, and maximum overshoot-under both no wind and constant 20 m/s wind conditions, representing an extreme scenario to evaluate controller robustness. This approach has also reduced training time by 80% compared to single-agent systems, lowering energy consumption and environmental impact.