Collaborative Ground-Space Communications via
Evolutionary Multi-objective Deep
Reinforcement Learning

To ensure the availability of the DRL model when terminal numbers vary, the key point is to fix the action dimension associated with the transmit powers of the terminals. To this end, we first derive the relationship between the importance of the objectives 1 and 2 with the optimal transmit powers of the terminals. Specifically, we only consider one-time slot optimization and let $a$ and $b$ be the weights of these two objectives. Then, we can give a new optimization problem as follows:

Accordingly, $(\mathrm{P2})$ can be solved optimally or near-optimally by solvers. Consequently, the instantaneous transmit powers of terminals can be well-determined according to the objective weights $a$ and $b$. In this case, we can use the fixed-dimension weights $a$ and $b$ instead of the transmit powers of terminals as the actions of the MOMDP, which reduces the impact of terminal number varying.

Following this, the computing resources of the considered DCB-based terminal-to-satellite uplink communication are often constrained, which needs a swift training process for flexible parameter tuning and rapid model deployment. To accelerate the training process, we discrete the weights associated with optimization objectives 1 and 2 by using equidistant discretization [37]. As such, the DRL algorithms only need to consider a finite number of action options thereby facilitating the training speed.

In particular, let $\mathcal{K}=\{(a_{1},b_{1}),(a_{2},b_{2}),...,(a_{|\mathcal{K}|},b_{|\mathcal{K}|})\}$ denote the alternative weight set, and then $a_{k}$ and $b_{k}$ can be established as follows [37]:

Benefiting from the simplification above, we can re-formulate the optimization problem shown in Eq. (10) as an action space-reduced and more universal MOMDP. The key elements of the MOMDP are given as follows:

• •

  State Space: We assume the terminals possess a precise timer and maintain satellite orbit data, thus acquiring accurate real-time satellite positions. Simultaneously, the log system of the virtual antenna array can store the index of the last-connected satellite. Note that we employ accurate real-time satellite positions to derive the transmit powers of terminals without inputting them into the DRL model. Consequently, the observable state at time slot $t$ of the virtual antenna array is given by

  $$\bm{s}_{t}=\{t,s_{t-1}\}.$$

  (16)

• •

  Action Space: As aforementioned, the virtual antenna array can select the trade-off schemes in $\mathcal{K}$ instead of using the DRL model to determine the transmit powers of terminals at different time slots. Except for that, the virtual antenna array should select one satellite to connect at any time slot. Thus, the actions that can be adopted at time slot $t$ by the virtual antenna array contain

  $$\bm{a}_{t}=\{k_{t},s_{t}\},$$

  (17)

  where $k_{t}$ indicates the selected scheme from $\mathcal{K}$ at time slot $t$.

• •

  Reward Function: In DRL models, the environment furnishes immediate rewards after an action is performed, and then the agent adjusts its actions and learns the optimal policy according to the reward. Thus, it is essential to design a reasonable reward for enhancing the solving performance of such DRL models. To achieve long-term multi-objective optimization, the reward vector is

  	$\displaystyle\bm{r}(t)=$	$\displaystyle[r_{1}(t),\,r_{2}(t),\,r_{3}(t)]$		(18)
  	$\displaystyle=$	$\displaystyle[\rho_{1}\hat{R}(t),\,-\rho_{2}\sum_{i\in\mathcal{I}}P_{i}\Delta T,\,-\rho_{3}\kappa_{t}],$

  where $\rho_{1}$, $\rho_{2}$, and $\rho_{3}$ are three normalization parameters intended to bring them into the same order of magnitude. Additionally, if $R(t)>\overline{R}$, then $\hat{R}(t)=R(t)$; otherwise, $\hat{R}(t)=0$. Moreover, $\kappa_{t}$ is a parameter indicating whether the satellite changes (i.e., $\kappa_{t}=1$ denotes changes and vice versa). As can be seen, these three terms denote different objectives shown in Eq. (10).

Based on this, we obtain an MOMDP in which the reward is a vector containing multiple objective rewards. Next, we aim to propose a novel multi-objective DRL method to obtain several long-term policies representing different trade-offs.

In the proposed EMODRL-based solution, the $n$th learning task can be represented as a tuple $\Gamma_{n}=\langle\mathbf{w}_{n},\pi_{\theta_{n}}\rangle$, where $\mathbf{w}_{n}$ ($w_{m,n}>0$, $\sum_{1}^{3}w_{m,n}=1$) is a weight vector for optimization objectives and $\pi_{\theta_{n}}$ is the policy that seeks to achieve the best cumulative reward ($\sum\nolimits_{t\in\mathcal{T}}\mathbf{w}_{n}\bm{r}(t)$) under the current objective weights.

However, the action space of the MOMDP encompasses some deterministic low-reward actions, and the epsilon-greedy cannot avoid such actions and may make meanless attempts. This inadequacy may hinder the D3QN agent from swiftly acquiring strategies and converging effectively. To overcome this issue, we seek to enhance the action selection strategy of D3QN. Specifically, the optimal policy is characterized by the exclusion of the unavailable satellites imposed by constraints on angle and spectrum resources from the action set $\bm{a}_{t}$. This is due to the fact that switching to an unavailable satellite will not get positive rewards in both the current step and future moments. Based on this, we propose a legitimate action select method to mask such low-reward actions. Specifically, we define the legitimate action set at the $t$th time slot as $\bm{a}_{t}^{l}$, in which the actions of switching unavailable satellites have been excluded. Then, we propose an epsilon-greedy scheme as follows:

Next, we will present the learning tasks and introduce the interaction of these enhanced D3QN agents.

In this part, we present an EMODRL-enhanced D3QN (EMODRL-ED3QN) to obtain a set of Pareto near-optimal policies by learning from the feedback of the environment.

As shown in Fig. 3, EMODRL-ED3QN has the same structure as the multi-objective DRL frameworks in [39, 40], which has warm-up and evolutionary two stages. In the warm-up stage, EMODRL-ED3QN generates $N$ learning tasks and generates the initial task population by using the multi-task ED3QN scheme shown in Algorithm 1. The evolutionary stage will update the task population, and the Pareto policy archive based on the continuously generated offspring population. These two stages are detailed as follows.

$\bullet$ Warm-up Stage: This stage stochastically generates a set of $N$ learning tasks which are defined as $\bm{\Gamma}=\{\Gamma_{1},\ldots,\Gamma_{N}\}$. Note that these tasks share the same state space, action space, and reward vector, but have different objective weight vectors and neural network parameters. First, the weight vectors of these tasks are assigned as $\mathcal{W}=\{\mathbf{w}_{1},\mathbf{w}_{2},...,\mathbf{w}_{N}\}$, in which they are evenly distributed and sampled from a unit simplex [14]. Then, we randomly initialize $N$ Q-value networks $\{Q_{\pi_{\theta_{1}}},Q_{\pi_{\theta_{2}}},...,Q_{\pi_{\theta_{N}}}\}$. As such, $\pi_{\theta_{n}}$ can make decisions according to the Q-value networks and the weighted reward $\mathbf{w}_{n}\bm{r}(t)$.

Next, we utilize the multi-task ED3QN scheme to generate the initial task population. As illustrated in Algorithm 2, this multi-task ED3QN approach allows all learning tasks to gather data from the environment and adjust network parameters according to the main steps of the ED3QN agent. The learning tasks with the adjusted network parameters are the generated offspring task population.

As such, we can obtain a set of learning tasks with well-initialized policies, and the process of the evolutionary stage can unfold as follows.

$\bullet$ Evolutionary Stage: This stage explores better strategies by iteratively updating the task population. Each iteration contains three steps that are task population updating, Pareto policy updating, and offspring population generating.

As for task population updating, we need to update the task population $\mathcal{P}$ according to the newly generated offspring population $\mathcal{P}^{\prime}$ (As shown in Algorithm 3). In this case, it is essential to distinguish the nondominated policies and keep the population diversity. Thus, we introduce the buffer strategy [14] to reasonably update $\mathcal{P}$. Specifically, multiple buffers are set to store $\mathcal{P}$, in which $B_{num}$ and $B_{size}$ are defined as their total number and capacities, respectively. As such, the objective performance space is segmented into $B_{num}$ buffers, each capable of storing up to $B_{size}$ policies. We can set a reference point $\mathbf{Z}_{ref}$ [40] to prioritize these policies within the same buffer.

Accordingly, for any given buffer, tasks are sorted in descending order based on their distances to $\mathbf{Z}_{ref}$. If the number of tasks exceeds $B_{size}$, only the first $B_{size}$ tasks in that buffer are retained. Following this, the learning tasks from all buffers collectively constitute a new task population.

As for Pareto policy updating, a Pareto archive is utilized to retain nondominated policies discovered during the evolutionary stage. Specifically, this Pareto archive undergoes an update according to the offspring population $\mathcal{P}^{\prime}$. For the ED3QN policy $\pi_{\theta}$ of each task in $\mathcal{P}^{\prime}$, the policies dominated by $\pi_{\theta}$ are excluded, and $\pi_{\theta}$ is added to the Pareto archive only if no policies in the Pareto archive dominate $\pi_{\theta}$ (see step 11 of Algorithm 1).

As for offspring population generating, we choose the optimal task from $\mathcal{P}$ and still use the multi-task ED3QN approach to obtain the offspring task population. Specifically, we evaluate the objective function values $\mathbf{F}(\pi_{\theta_{q}})$ of each policy $\pi_{\theta_{q}}$ within $\mathcal{P}$. Then, for a given weight vector $\mathbf{w}_{n}\in\mathcal{W}$, we determine the best learning task in $\mathcal{P}$ based on $w_{n}$ and $\mathbf{F}(\pi_{\theta_{q}})$ ($q=1,\ldots,|\mathcal{P}|$) (as shown in Algorithm 4). Finally, the $N$ selected learning tasks are incorporated into $\Gamma^{\prime}$. We derive $P^{\prime}$ by executing multi-task ED3QN (see Algorithm 2) with $\Gamma^{\prime}$ and $T_{task}$ as its input, where $T_{task}$ represents the predefined number of task iterations.

This stage terminates if the predefined number of evolution generations are completed. In this case, all non-dominated policies stored in the Pareto archive will be output as the Pareto near-optimal policies for the formulated MOMDP as well as the optimization problem. These policies represent different trade-offs between the total uplink achievable rate, total energy consumption of terminals, and total satellite switching number. As such, the decision-makers can select one policy from them according to the current requirements and concerns.

We first consider the time complexity of the training EMODRL-ED3QN model. Specifically, in both the warm-up and evolutionary stages, the major complexity comes from the step of generating offspring population which involves the training of neural networks. Compared with this step, other steps (e.g., steps 8, 9, and 11 in Algorithm 1) are considered trivial and can be disregarded.

As shown in Algorithm 2, MMD3QN generates the offspring population, and its time complexity mainly depends on the training of neural networks. MMD3QN iteratively optimizes each learning task $\pi_{\theta_{n}}$ in the task set for $T_{iter}$ times (i.e., steps 2-8 in Algorithm 2), where $T_{iter}$ denotes the number of task iterations. Let $N_{data}$ denote the number of collected data, and $N_{epo}$ be the number of epochs for training the Q-value network. Note that the implemented Q-value network is the fully connected neural network, which consists of an input, an output, and $C$ fully connected layers. The numbers of neurons in the input and output layers are 2 and 2, respectively. Let $N_{c}$ denote the number of neurons in the $c$th fully connected layer, with $N_{0}=2$ and $N_{C+1}=2$. Consequently, the time complexity of MMD3QN is expressed as $O(n\cdot(T_{iter}\cdot N_{epo}\cdot N_{data}\cdot\sum_{c=1}^{C+1}N_{c-1}\cdot N_{c}))$ [39].

By considering the predefined number of maximum evolution generations ($T_{evo}$), the time complexity of training EMODRL-ED3QN is $O(T_{evo}\cdot n\cdot(T_{iter}\cdot N_{epo}\cdot N_{data}\cdot\sum_{l=1}^{L+1}N_{l-1}\cdot N_{l})$.

Moreover, we analyze the time complexity of using the trained EMODRL-ED3QN. Since EMODRL-ED3QN achieves multiple alternative policies to match the current preference, using EMODRL-ED3QN does not need transfer learning or other tuning. As such, the selected policy can quickly generate a solution to the problem through simple algebraic calculations. In this case, the time complexity of using the trained EMODRL-ED3QN is $O(T\cdot\sum_{c=1}^{C+1}N_{c-1}\cdot N_{c})$, where $T$ is the number of time slots [39].

In this work, we consider a terrestrial terminal to LEO satellite communication scenario, which includes the LEO satellite, terrestrial terminal, and communication-related parameters. First, we set up 110 periodically operated LEO satellites, of which 80 LEO satellites at an altitude of $5\times 10^{5}$ m and 30 LEO satellites at an altitude of $10^{6}$ m. Note that most of them are around the equatorial orbit and some of them have an inclination angle around $\pm\pi/8$, and the satellites in the same orbit are evenly distributed in this orbit [41, 25]. Second, we consider a $100\times 100$ terrestrial terminal area located near the equator, in which exists 10 terrestrial terminals and several sensors. Note that these devices can perform efficient information sharing and communication within the area. Finally, the carrier frequency, minimum to maximum transmit powers of each terminal, path loss exponent, and total noisy power spectral density are set as 2.4 GHz, 1-2 W, 2, and -157 dBm/Hz, respectively.

Additionally, we consider a timeline of 60 minutes. The radius of Earth $R_{e}$, gravitational constant $G$, and mass of Earth $M_{e}$ are set as $6.371\times 10^{6}$ m, $6.674\times 10^{-11}\,\text{m}^{3}\,\text{kg}^{-1}\,\text{s}^{-2}$, and $5.972\times 10^{24}$ kg, respectively.

In the proposed EMODRL-ED3QN, we set the number of the learning tasks $N$ as 10. In addition, the maximum evolution generations $T_{evo}$, the iteration number during the warm-up stage $T_{warm}$, and the iteration number for training each task $T_{task}$ are set as 300, 80, and 20, respectively. Finally, the number of performance buffers $B_{num}$ is designed to 50, and each buffer size $B_{size}$ is set to 2. For each learning task, the Q-value network has two fully connected layers with 2048 neurons and the tanh function serves as the activation function. Moreover, the learning rate is $10^{-4}$ and the discount factor is $\gamma=0.96$. The replay buffer size and batch size are set as $10^{5}$ and $256$, respectively.

To demonstrate the performance of the proposed EMODRL-ED3QN, we introduce and design various comparison algorithms and strategies as follows:

• •

  Non-DCB strategy: This strategy does not introduce DCB technology and only adopts one single terrestrial terminal to connect to the satellite directly.

• •

  Achievable rate greedy policy (ARGP): ARGP refers to the policy that any terminal $i\in\mathcal{I}$ employs the maximum transmit power $P_{max}$ and selects the satellite with the utmost uplink achievable rate at any time slot $t\in\mathcal{T}$. Note that ARGP achieves the upper bound of optimization objective 1.

• •

  State-of-the-art baseline algorithms: We design EMODRL-D3QN, EMODRL-Noisy-DQN, EMODRL-DDQN, EMODRL-PPO, EMODRL-TD3, and EMODRL-SAC as the baseline algorithms. Note that they are the variants of D3QN, Noisy-DQN [38], double DQN (DDQN) [42], proximal policy optimization (PPO) [43], twin delayed deep deterministic policy gradient algorithm (TD3) [44], and soft actor-critic (SAC) [45], respectively. We develop them by introducing the proposed evolutionary multi-objective reinforcement learning and multi-task frameworks for dealing with the formulated MOMDP.

As such, the comparison with non-DCB strategy shows the effectiveness of introducing DCB, the comparison with ARGP can assess the effect of the proposed evolutionary multi-objective DRL framework, and the comparison with other EMODRL algorithms can illustrate the optimization efficiency of EMODRL-ED3QN. In the following comparisons, we consider the average optimization objective values of these algorithms over timelines (i.e., $\bar{f}_{1}$, $\bar{f}_{2}$, and $\bar{f}_{3}$) as a performance metrics.

In this part, we compare the DCB-based policies and the non-DCB strategy to illustrate the effectiveness of the considered DCB-based uplink communication approach. Specifically, uplink achievable rates obtained by a policy of EMODRL-ED3QN, ARGP, and non-DCB strategy at each time slot are shown in Fig. 4. As can be seen, ARGP and EMODRL-ED3QN consistently surpass the threshold for uplink communication. In contrast, the non-DCB strategy struggles to attain an uplink achievable rate above the threshold. Moreover, the EMODRL-ED3QN policy achieves performance closely aligned with the upper bound (i.e., ARGP) at every time slot. These results show that the DCB-based uplink communication approach and EMODRL-ED3QN policy are both reasonable and suitable for the considered scenario.

We first evaluate the trade-offs obtained by the proposed EMODRL-ED3QN in solving the formulated problem. As shown in Fig. 5, we show the trade-offs among the considered three objectives obtained by multiple EMODRL baselines. As can be seen, all these algorithms obtain a set of Pareto policies with wide coverage among the considered three objectives. Thus, the considered EMODRL framework is effective and can obtain multiple policies that weigh each other. Moreover, EMODRL-ED3QN, EMODRL-Noisy-DQN, and EMODRL-DDQN outperform other comparison algorithms. This is because the three algorithms are offline reinforcement learning methods, which may save more periodic information in their replay buffer, thereby facilitating the learning of the periodicity of the considered system. Additionally, we can see that the proposed EMODRL-ED3QN outmatches other baselines. The reason is that the proposed legitimate action select method can well-balance the exploration and exploitation of the algorithms. Moreover, the structure of the selected D3QN is also the most suitable for the designed MOMDP and legitimate action select method, and thus enables the algorithm to approach optimal performance closely.

Second, we select one policy from the Pareto policy set of each algorithm for further comparisons and analyses. In most cases, the uplink achievable rate from the terrestrial terminals to LEO satellites is the most concerned optimization objective. As such, we choose the policy with the best optimization objective 1 from the Pareto policy set as the final policy. In this case, the numerical results in terms of the considered optimization objectives are shown in Table I. As can be seen, the proposed EMODRL-ED3QN achieves a similar objective 1 and much lower objectives 2 and 3 with the ARGP policy. This demonstrates that the proposed EMODRL-ED3QN uses lower energy consumption and satellite switching numbers to obtain a nearly optimal uplink rate. Moreover, compared with other comparison policies, EMODRL-ED3QN has a better balance among the three optimization objectives. Note that although EMODRL-PPO, EMODRL-TD3, and EMODRL-SAC achieve better optimization objectives 2 and 3, their optimization objective 1 is inadequate, making them unsuitable for terrestrial-to-satellite communication scenarios. Therefore, we can illustrate that EMODRL-ED3QN is most suitable for the considered scenario and can mitigate the ping-pong handover issue.

We first select different trade-off policies from the Pareto policy archive of EMODRL-ED3QN to illustrate the diversity performance of the obtained policy set. Specifically, we select four different trade-off policies which are policy favoring objective 1, policy favoring objective 2, policy favoring objective 3, and policy balancing objectives 1, 2, and 3, and the optimization objective values of these policies are shown in Fig. 7. It can be seen that the four policies all have obvious differences and show different objective tendencies when solving the formulated problem. In addition, these policies all achieve slightly weaker objective 1 but much better objectives 2 and 3 than ARGP. These results show that the policy set obtained by EMODRL-ED3QN has strong diversity.

Then, we evaluate the impacts of scenario changes on the policies obtained by the proposed EMODRL-ED3QN. Specifically, the satellite unavailable probability may have a significant effect on these policies. Thus, we depict the changes in the three optimization objectives with satellite unavailability probability $p$ in Fig. 6(a). We can observe that the policies still show different objective tendencies and no significant deterioration occurred compared with ARGP. Moreover, as aforementioned, we seek to propose a method in which one-time training can accommodate various terminal numbers. Fig. 6(b) shows the performance of these trained policies changed with the terminal numbers. As can be seen, the policies still show obvious objective tendencies and achieve good performance. The reason is that the proposed legitimate action select method can enable EMODRL-ED3QN to fully explore and utilize the high-value action space and obtain more valuable trade-off policies. Thus, one-time training of the EMODRL-ED3QN can obtain multiple trade-off policies with portability.

Ablation simulations are conducted to illustrate the effectiveness of the proposed enhanced methods. Specifically, we consider two strategies that are optimization without optimized $\bm{P}$ (OOP) and optimization without legitimate action select method (OLASM). In OOP, the transmit power of each terrestrial terminal is not optimized and randomly generated. In OLASM, the proposed legitimate action select method is not considered. Accordingly, the comparison results are shown in Fig. 8. As can be seen, the proposed EMODRL-ED3QN is significantly better than other ablated strategies. This shows that the proposed enhanced methods are effective and can boost the training performance of the traditional DRL algorithm in such scenarios.