Alexander Kyuroson

European Journal of Control, 2024

Towards Fully Autonomous Orbit Management for Low-Earth Orbit Satellites Based on Neuro-Evolutionary Algorithms and Deep Reinforcement Learning

Earth with the inertial ECI axes and a satellite carrying its local LVLH frame with radial, tangential and normal axes. Earth with the inertial ECI axes and a satellite carrying its local LVLH frame with radial, tangential and normal axes.
The ECI and LVLH reference frames.
Loop between an agent, which holds a policy and an RL algorithm, and an environment that simulates the satellite dynamics. The environment sends the state vector of position and velocity and a reward; the agent returns a control vector of radial, tangential and normal thrust. Loop between an agent, which holds a policy and an RL algorithm, and an environment that simulates the satellite dynamics. The environment sends the state vector of position and velocity and a reward; the agent returns a control vector of radial, tangential and normal thrust.
Layout of the reinforcement-learning framework: the agent receives position and velocity and returns a thrust vector.

Abstract

The recent advances in space technology are focusing on fully autonomous, real-time, long-term orbit management and mission planning for large-scale satellite constellations in Low-Earth Orbit (LEO). Thus, a pioneering approach for autonomous orbital station-keeping has been introduced using a model-free Deep Policy Gradient-based Reinforcement Learning (DPGRL) strategy explicitly tailored for LEO. Addressing the critical need for more efficient and self-regulating orbit management in LEO satellite constellations, this work explores the potential synergy between Deep Reinforcement Learning (DRL) and Neuro-Evolution of Augmenting Topology (NEAT) to optimize station-keeping strategies with the primary goal to empower satellite to autonomously maintain their orbit in the presence of external perturbations within an allowable tolerance margin, thereby significantly reducing operational costs while maintaining precise and consistent station-keeping throughout their life cycle. The study specifically tailors DPGRL algorithms for LEO satellites, considering low-thrust constraints for maneuvers and integrating dense reward schemes and domain-based reward shaping techniques. By showcasing the adaptability and scalability of the combined NEAT and DRL framework in diverse operational scenarios, this approach holds immense promise for revolutionizing autonomous orbit management, paving the way for more efficient and adaptable satellite operations while incorporating the physical constraints of satellite, such as thruster limitations.

Method

Station-keeping is posed as trajectory tracking: the unperturbed orbit is the reference, and the policy has to keep the satellite within a tolerance of it with low thrust. Two families of methods are trained in the same simulated environment.

A satellite drifting off a dashed reference orbit; the position error is marked at successive time steps. A satellite drifting off a dashed reference orbit; the position error is marked at successive time steps.
Trajectory tracking: the error between the reference and the actual position at each time step.
Deep reinforcement learning
DDPG and TD3, each with actor and critic networks of two fully connected layers (400 and 300 units), temporally correlated exploration noise, a dense shaped reward, and a terminal constraint that ends an episode when the position error reaches 50 km.
Neuro-evolution
NEAT evolves the network topology itself, starting from two hidden layers, to see which structure the problem asks for and which state variables the controller needs.
Block diagram of DDPG: replay buffer, actor and critic, and their target networks with soft updates. Block diagram of DDPG: replay buffer, actor and critic, and their target networks with soft updates.

Scroll the figure sideways, or tap it to open it full size.

DDPG architecture.

Results

The target is a circular low-earth orbit at 70° inclination. Both reinforcement-learning algorithms solve the task, but DDPG trains unreliably, so TD3 is used for the comparison with NEAT. The topology that NEAT arrives at is close to the TD3 actor, and it does not use every state variable.

Two training curves of average episodic reward. DDPG rises, collapses around episode 5000 and recovers; TD3 rises within about 50 episodes and stays high with a few short dips. Two training curves of average episodic reward. DDPG rises, collapses around episode 5000 and recovers; TD3 rises within about 50 episodes and stays high with a few short dips.
Average episodic training reward for DDPG (top) and TD3 (bottom).
Network found by NEAT: six inputs for position and velocity, a few hidden nodes in two layers, three thrust outputs; edges in two colors mark negative and positive correlations. Network found by NEAT: six inputs for position and velocity, a few hidden nodes in two layers, three thrust outputs; edges in two colors mark negative and positive correlations.
Topology found by NEAT for the best fitness. Yellow nodes are clusters of neurons; red edges are negative correlations and blackwhite edges positive ones.
3D plot of one orbit: the reference, the trajectory under the reinforcement-learning policy and the trajectory under NEAT lie on top of each other. 3D plot of one orbit: the reference, the trajectory under the reinforcement-learning policy and the trajectory under NEAT lie on top of each other.
Trajectory correction with deep reinforcement learning and with NEAT, against the reference orbit.
Close-up of a segment of the orbit in the XY-plane, with the two corrected trajectories staying between the outer and inner zone of the ideal path. Close-up of a segment of the orbit in the XY-plane, with the two corrected trajectories staying between the outer and inner zone of the ideal path.
Close-up of a segment of the trajectory.
Three plots of position error in x, y and z over about 10,000 seconds for TD3, within roughly plus or minus 25 km. Three plots of position error in x, y and z over about 10,000 seconds for TD3, within roughly plus or minus 25 km.
Component-wise position error for TD3.
Three plots of position error in x, y and z over about 5,000 seconds for NEAT, within roughly plus or minus 25 km. Three plots of position error in x, y and z over about 5,000 seconds for NEAT, within roughly plus or minus 25 km.
Component-wise position error for NEAT.
Three plots of velocity error in x, y and z for TD3. The x and z components stay within about 0.03 km per second; the y component starts near 0.09 km per second and decays. Three plots of velocity error in x, y and z for TD3. The x and z components stay within about 0.03 km per second; the y component starts near 0.09 km per second and decays.
Component-wise velocity error for TD3.
Three plots of velocity error in x, y and z for NEAT, within plus or minus 0.03 km per second. Three plots of velocity error in x, y and z for NEAT, within plus or minus 0.03 km per second.
Component-wise velocity error for NEAT.

Figures are from the article, © 2024 The Authors, published under the CC BY 4.0 license.

This work was partially funded by the European Space Agency (ESA) open Invitations to Tender and innovation research grant in the OPTACOM project, in collaboration with OHB Sweden, under Grant Contract no. OPC-OSE-CC-0536.

Citation

BibTeX
@article{kyuroson2024orbit,
  title   = {Towards fully autonomous orbit management for low-earth orbit satellites based on neuro-evolutionary algorithms and deep reinforcement learning},
  author  = {Kyuroson, Alexander and Banerjee, Avijit and Tafanidis, Nektarios Aristeidis and Satpute, Sumeet and Nikolakopoulos, George},
  journal = {European Journal of Control},
  volume  = {80},
  pages   = {101052},
  year    = {2024},
  doi     = {10.1016/j.ejcon.2024.101052}
}

All projects     About Alexander Kyuroson