Rationality based Innate-Values-driven
Reinforcement Learning

In our model, the randomness comes from three sources. The randomness in action is from the policy function: $A\sim\pi(\cdot|s)$; the needs weight function: $W\sim\omega(\cdot|s)$ makes the randomness of innate values; the state-transition function: $S^{\prime}\sim p(\cdot|s,a)$ causes the randomness in state.

Supposing at current state $s_{t}$ an agent has a needs weight matrix $N_{t}$ (Eq. (1)) in a mission, which presents its innate value weights for different levels of needs. Correspondingly, it has a utility matrix $U_{t}$ (Eq. (1)) for specific needs resulting from action $a_{t}$. Then, we can calculate its reward $R_{t}$ for $a_{t}$ through Eq. (2) at the state $s_{t}$.

In the process, the agent will first generate the needs weight and action based on the current state, then, according to the feedback utilities and the needs weights, calculate the current reward and iterate the process until the end of an episode. Fig. 3 illustrates the trajectory of state $S$, needs weight $W$, action $A$, and reward $R$ in the IVRL model.

According to the above discussion, we define the discounted return $G$ at $t$ time as cumulative discounted rewards in the IVRL model (Eq. (3)) and $\gamma$ is the discount factor.

At time $t$, the randomness of the return $G_{t}$ comes from the the rewards $R_{t},\cdots,R_{n}$. Since the reward $R_{t}$ depends on the state $S_{t}$, action $A_{t}$, and needs weight $W_{t}$, the return $G_{t}$ also relies on them. Furthermore, we can describe their randomness as follows:

Based on the discounted return Eq. (3) and its random factors – Eq. (4), (5), and (6), we can define the Action-Innate-Value function as the expectation of the discounted return $G$ at $t$ time (Eq. (7)).

$Q_{\pi,\omega}(s_{t},w_{t},a_{t})$ describes the quality of the action $a_{t}$ taken by the agent in the state $s_{t}$, using the needs weight $w_{t}$ generating from the needs weight function $\omega$ as the innate value judgment to execute the policy $\pi$.

Furthermore, we can define the State-Innate-Value function as Eq. (8), which calculates the expectation of $Q_{\pi,\omega}(s_{t},w_{t},a_{t})$ for action $A$ and reflects the situation in the state $s_{t}$ with the innate value judgment $w_{t}$.

The agent’s goal is to interact with the environment by selecting actions to maximize future rewards based on its innate value judgment. We make the standard assumption that a factor of $\gamma$ per time-step discounts future rewards and define the future discounted return at time $t$ as Eq. (3). Moreover, we can define the optimal action-value function $Q^{*}(s,a,w)$ as the maximum expected return achievable by following any strategy after seeing some sequence $s$, making corresponding innate value judgment $w$, and then taking action $a$, where $\omega$ is a needs weight function describing sequences about innate value weights and $\pi$ is a policy mapping sequences to actions.

Since the optimal action-innate-value function obeys the Bellman equation, we can estimate the function by using the Bellman equation as an iterative update. This is based on the following intuition: if the optimal innate-value $Q^{*}(s^{\prime},w^{\prime},a^{\prime})$ of sequence $s^{\prime}$ at the next time-step was known for all possible actions $a^{\prime}$ and needs weights $w^{\prime}$, then the optimal strategy is to select the reasonable action $a^{\prime}$ and rational innate value weight $w^{\prime}$, maximising the expected innate value of $r+\gamma Q^{*}(s^{\prime},w^{\prime},a^{\prime})$,

Furthermore, the same as the DQN mnih2015human , we use a function approximator (Eq. (11)) to estimate the action-innate-value function.

We refer to a neural network function approximator with weights $\theta$ as a Q-network. It can be trained by minimising a sequence of loss function $L_{i}(\theta_{i})$ that changes at each iteration $i$,

Where Eq. 13 is the target for iteration $i$ and $\sigma(s,w,a)$ is a probability distribution over sequences $s$, needs weights $w$, and action $a$ that we refer to as the needs-behavior distribution. We can approximate the gradient as follows:

Instead of computing the full expectations in the Eq. (14), stochastic gradient descent is usually computationally expedient to optimize the loss function. Here, the weights $\theta$ are updated after every time step, and single samples from the needs-behavior distribution $\sigma$ and the emulator $\epsilon$ replace the expectations, respectively.

Our approach is a model-free and off-policy algorithm, which learns about the greedy strategy $a=\max_{w,a}Q(s,w,a;\theta)$ following a needs-behavior distribution to ensure adequate state space exploration. Moreover, the needs-behavior distribution selects action based on an $\epsilon$-greedy strategy that follows the greedy strategy with probability $1-\epsilon$ and selects a random action with probability $\epsilon$. Fig. 5 illustrates the action-innate-value network generating Needs-Behavior distribution.

Moreover, we utilize the experience replay technique lin1992reinforcement , which stores the agent’s experiences at each time-step, $e_{t}=(s_{t},w_{t},a_{t},r_{t},s_{t+1})$ in a data-set $\mathcal{D}=e_{1},\dots,e_{N}$, pooled over many episodes into a replay memory. During the algorithm’s inner loop, we apply Q-learning updates, or minibatch updates, to samples of experience, $e\sim\mathcal{D}$, drawn at random from the pool of stored samples. After performing experience replay, the agent selects and executes an action according to an $\epsilon$-greedy policy, as we discussed. Since implementing arbitrary length histories as inputs to a neural network is difficult, we use a function $\phi$ to produce our action-innate-value Q-function. Alg. 1 presents the algorithm of the IVRL DQN.

Furthermore, we extend our IVRL method to the Advantage Actor-Critic (A2C) version. Specifically, our IVRL A2C maintains a policy network $\pi(a_{t}|s_{t};\theta)$, a needs network $\omega(w_{t}|s_{t};\delta)$, and a utility value network $u(s_{t},a_{t};\varphi)$. Since the reward in each step is equal to the current utilities $u(s_{t},a_{t})$ multiplying the corresponding weight of needs, the state innated-values function can be approximated by presenting it as Eq. (15). Then, we can get the policy gradient Eq. (16) and needs gradient Eq. (17) of the Eq. (15) deriving $V(s;\theta,\delta)$ according to the Multi-variable Chain Rule, respectively. We can update the policy network $\theta$ and needs network $\delta$ by implementing policy gradient and needs gradient, and using the temporal difference (TD) to update the value network $\varphi$.

Using an estimate of the utility $u$ function as the baseline function, we subtract the V value term as the advantage value. Intuitively, this means how much better it is to take a specific action and a needs weight compared to the average, general action and the needs weights at the given state Eq. (18). Fig. 6 illustrates the architecture of the IVRL actor-critic version and Alg. 2 presents the algorithm of the IVRL A2C.

In our experiments, we define four categories of utilities (health points, amount of ammo, environment rewards, and number of killed enemies), presenting three different levels of needs: low-level safety and basic needs, medium-level recognition needs, and high-level achievement needs. When the agent executes an action, it receives all the corresponding innate utilities, such as health points and ammo costs, and external utilities, such as environment rewards (living time) and the number of killed enemies. At each step, the agent can calculate the rewards for the action by multiplying the current utilities and the needed weight for them. In our experiments, the initial needs weight for each utility category is 0.25, which has been fixed in the benchmark DRL algorithms’ training, such as DQN, DDQN, and PPO. For more details about the experiment code, please check the supplementary materials.

a. Defend the Center – Fig. 7(a): For this scenario, the map is a large circle where the agent is in the middle, and monsters spawn along the wall. The agent’s basic actions are turn-left, turn-right, and attack, and the action space is 8. It needs to survive in the scenario as long as possible.

b. Defend the Line. – Fig. 7(b): The agent is located on a rectangular map, and the monsters are on the opposite side. Similar to the defend the center scenario, the agent needs to survive as long as possible. Its basic actions are move-left, move-right, turn-left, turn-right, and attack, and the action space is 32.

c. Deadly Corridor. – Fig. 7(c): In this scenario, the map is a corridor. The agent is spawned at one end of the corridor, and a green vest is placed at the other end. Three pairs of monsters are placed on both sides of the corridor. The agent needs to pass the corridor and get the vest. Its basic actions are move-left, move-right, move-forward, turn-left, turn-right, and attack, and the action space is 64.

d. Arena. – Fig. 7(d): This scenario is the most challenging map compared with the other three. The agent’s start point is in the middle of the map, and it needs to eliminate various enemies to survive as long as possible. Its basic actions are move-left, move-right, move-forward, move-backward, turn-left, turn-right, and attack, and the action space is 128.

The performance of the proposed IVRL DQN and A2C models is shown in the Fig 8. Fig. 7(a), 7(b), 7(c), and 7(d) demonstrate that IVRL models can achieve higher average scores than traditional RL benchmark methods (Fig. 8, 8, 8, and 8). Especially for the IVRL A2C algorithm, it presents more robust, stable, and efficient properties than the IVRL DQN model. Although the IVRL DQN shows better performance in the Arena scenario (Fig. 8), the small perturbation introduced by the innate-values utilities may have made the network weights in some topology difficult to reach convergence.

Furthermore, we also analyze their corresponding tendencies in different scenarios to compare the needs weight differences between the IVRL DQN and A2C models. In the defend-the-center and defend-the-line experiments, each category of the need weight in the IVRL DQN model does not split and converges to a specific range compared with its initial setting in our training (Fig. 8 and 8). In contrast, the weights of health depletion, ammo cost, and sub-goal (environment rewards) shrink to approximately zero, and the weight of the number of killed enemies converges to one in the IVRL A2C model. This means that the top priority of the IVRL A2C agent is to eliminate all the threats or adversaries in those scenarios so that it can survive, which is similar to the Arena task. According to the performance in those three scenarios (Fig. 8, 8, and 8), the IVRL A2C agent represents the characteristics of bravery and fearlessness, much like the human hero in a real battle. However, in the deadly corridor mission, the needs weight of the task goal (getting the vest) becomes the main priority, and the killing enemy weight switches to the second for the IVRL A2C agent (Fig. 8). They converge to around 0.6 and 0.4, respectively. In training, by adjusting its different needs weights to maximize rewards, the IVRL A2C agent develops various strategies and skills to kill the encounter adversaries and get the vast efficiently, much like a military spy.

In our experiments, we found that selecting the suitable utilities to consist of the agent innate-values system is critically important for building its reward mechanism, which decides the training speed and sample efficiency. Moreover, the difference in the selected utility might cause some irrelevant experiences to disrupt the learning process, and this perturbation leads to high oscillations of both innate-value rewards and needs weight.

Generally, the innate value system serves as a unique reward mechanism driving agents to develop diverse actions or strategies satisfying their various needs in the systems. It also builds different personalities and characteristics of agents in their interaction. From the environmental perspective, due to the various properties of the tasks, agents need to adjust their innate value system (needs weights) to adapt to different tasks’ requirements. These experiences also shape their intrinsic values in the long term, similar to humans building value systems in their lives. Moreover, organizing agents with similar interests and innate values in the mission can optimize the group utilities and reduce costs effectively, just like “Birds of a feather flock together." in human society.