European Journal of Control, 2024
Towards Fully Autonomous Orbit Management for Low-Earth Orbit Satellites Based on Neuro-Evolutionary Algorithms and Deep Reinforcement Learning
- Robotics and Artificial Intelligence Group, Luleå University of Technology
Abstract
The recent advances in space technology are focusing on fully autonomous, real-time, long-term orbit management and mission planning for large-scale satellite constellations in Low-Earth Orbit (LEO). Thus, a pioneering approach for autonomous orbital station-keeping has been introduced using a model-free Deep Policy Gradient-based Reinforcement Learning (DPGRL) strategy explicitly tailored for LEO. Addressing the critical need for more efficient and self-regulating orbit management in LEO satellite constellations, this work explores the potential synergy between Deep Reinforcement Learning (DRL) and Neuro-Evolution of Augmenting Topology (NEAT) to optimize station-keeping strategies with the primary goal to empower satellite to autonomously maintain their orbit in the presence of external perturbations within an allowable tolerance margin, thereby significantly reducing operational costs while maintaining precise and consistent station-keeping throughout their life cycle. The study specifically tailors DPGRL algorithms for LEO satellites, considering low-thrust constraints for maneuvers and integrating dense reward schemes and domain-based reward shaping techniques. By showcasing the adaptability and scalability of the combined NEAT and DRL framework in diverse operational scenarios, this approach holds immense promise for revolutionizing autonomous orbit management, paving the way for more efficient and adaptable satellite operations while incorporating the physical constraints of satellite, such as thruster limitations.
Method
Station-keeping is posed as trajectory tracking: the unperturbed orbit is the reference, and the policy has to keep the satellite within a tolerance of it with low thrust. Two families of methods are trained in the same simulated environment.
- Deep reinforcement learning
- DDPG and TD3, each with actor and critic networks of two fully connected layers (400 and 300 units), temporally correlated exploration noise, a dense shaped reward, and a terminal constraint that ends an episode when the position error reaches 50 km.
- Neuro-evolution
- NEAT evolves the network topology itself, starting from two hidden layers, to see which structure the problem asks for and which state variables the controller needs.
Results
The target is a circular low-earth orbit at 70° inclination. Both reinforcement-learning algorithms solve the task, but DDPG trains unreliably, so TD3 is used for the comparison with NEAT. The topology that NEAT arrives at is close to the TD3 actor, and it does not use every state variable.
Figures are from the article, © 2024 The Authors, published under the CC BY 4.0 license.
This work was partially funded by the European Space Agency (ESA) open Invitations to Tender and innovation research grant in the OPTACOM project, in collaboration with OHB Sweden, under Grant Contract no. OPC-OSE-CC-0536.
Citation
@article{kyuroson2024orbit,
title = {Towards fully autonomous orbit management for low-earth orbit satellites based on neuro-evolutionary algorithms and deep reinforcement learning},
author = {Kyuroson, Alexander and Banerjee, Avijit and Tafanidis, Nektarios Aristeidis and Satpute, Sumeet and Nikolakopoulos, George},
journal = {European Journal of Control},
volume = {80},
pages = {101052},
year = {2024},
doi = {10.1016/j.ejcon.2024.101052}
}























