On the Challenges of using Reinforcement Learning in Precision Drug Dosing:
Delay and Prolongedness of Action Effects

In this paper, we identify prolongedness and delay as fundamental roadblocks to using RL in precision drug dosing, and focus on addressing the former. To that end, we introduce the prolonged action effect partially observable Markov decision process (PAE-POMDP), a framework for modeling delayed action effects in decision making. We then present assumptions inspired by the pharmacology literature to convert the non-Markovian prolonged action effect problem into a Markov decision process (MDP), therefore enabling the use of recent advances in model-free RL. To the best of our knowledge, our work is the first to explore the prolonged effect of actions. We validate the proposed approach on a toy task, where the only violation of the Markov property comes from the prolonged effect of actions, and show that restoring the Markov assumption allows the RL agent we develop to significantly outperform previous baselines. We also address the challenging task of glucose control by optimal insulin dosing for type 1 diabetes patients by leveraging the open-sourced version of the FDA-approved UDA/Padova simulator (Dalla, MD, and C. 2009; Xie 2018). Although the glucose control task has garnered interest in the RL literature in the past (Tejedor, Woldaregay, and Godtliebsen 2020), there does not appear to be a widely adopted reward function. Therefore, we design a clinically motivated reward function that explicitly avoids overdosing while effectively controlling glucose levels. Our results show that our approach of converting the PAE-POMDP into a MDP is not only competitive in terms of performance, but is also remarkably more time and memory efficient than the baselines, making it more suitable for real-time dosing control systems. With these contributions, we aim to raise awareness of both the delayed and prolonged effects of drugs while tackling personalized drug dosing with RL, and hope that the proposed reward function will help break the entry barrier to control glucose levels for a closed-loop artificial pancreas, while fostering future research in this direction.

This paper does not aim to provide a general solution to tackle the prolonged effect of actions, but one crafted specifically for precision drug dosing that allows to quickly bring the problem back to the MDP framework in a compute- and memory-efficient way. We do not provide a new algorithm to tackle prolongedness, but instead propose a pharmacologically motivated effective way to enable the use of already existing RL algorithms for precision drug dosing. We are assuming drug action in isolation, instead of in combination with other drugs. Learning and predicting drug synergies is a growing body of research that can provide interesting future work.

Reinforcement Learning (RL) is a framework for solving sequential decision making problems where an agent interacts with the environment and receives feedback in the form of reward. The typical formal framework for RL is the Markov Decision Process (MDP). A MDP $\mathcal{M}$ is a 5-tuple $(\mathcal{S},\mathcal{A},r,\mathcal{P},\gamma)$, where $\mathcal{S}$ is a (finite) set of states, $\mathcal{A}$ is a (finite) set of actions, $\mathcal{P}$ is the state transition probability $\mathcal{P}(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)$, $r:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ is the reward function and $\gamma\in[0,1)$ is the discount factor. As the name suggests, a MDP obeys the Markov assumption that the future is in independent of the past given the present, which means that transitions and rewards depend only on the current state and action and not on the past history. The goal of an RL agent is to find a policy $\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ that maximizes the cumulative discounted return $\sum_{t=0}^{\infty}\gamma^{t}r_{t}$.

A Partially Observable MDP (POMDP) is a 7-tuple $(\mathcal{S},\mathcal{A},r,\mathcal{P},\Omega,\mathcal{O},\gamma)$, where $\Omega$ is the set of observations, $\mathcal{O}$ is the set of conditional observation probabilities, $O(\omega|s)$, also known as the emission function, and the rest of the elements are the same as in an MDP. In a POMDP, the agent does not have direct access to the identity of the states, instead needing to infer them through the observations. Note that while in a MDP, an agent which aims to act optimally with respect to the expected long-term return only needs to consider Markovian policies, $\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$, in a POMDP, policies need to either rely on the entire history of action and observations, or to infer the distribution of hidden states from this history (as in belief-based POMDP dolution methods). In recent work, recurrent networks have become the standard for implementing POMDP policies. We call recurrent policy a mapping $\pi:\mathcal{T}\times\mathcal{A}\rightarrow[0,1]$ where $\mathcal{T}$ is the space of trajectories $\tau=\{(o_{t},a_{t},r_{t})\}_{t=0}^{T}$ of up to $T$ time steps.

Q-Learning (Watkins and Dayan 1989) is a model-free RL algorithm that estimates $\mathcal{Q}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, a value function assessing the quality of action $a_{t}\in\mathcal{A}$ at state $s_{t}\in\mathcal{S}$. The value function can be estimated as: $Q(s_{t},a_{t})=Q(s_{t},a_{t})+\alpha\big{(}r_{t}+\gamma\max_{a}Q(s_{t+1},a)-Q(s_{t},a_{t})\big{)}$, where $\alpha\in(0,1)$ is the learning rate. In DQN (Mnih, Kavukcuoglu, and et al. 2015), the $Q$-values are estimated by a neural network minimizing the Mean Square Bellman Error: $L_{i}(\theta_{i})=\mathbb{E}_{s_{t},a_{t}\sim\pi_{b}}\big{[}(y_{i}-Q(s_{t},a_{t};\theta_{i}))^{2}\big{]}\\ y_{i}=(r_{t}+\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1};\theta_{i-1})$, where $\pi_{b}$ is the behavior policy, $y_{i}$ is the target and $\theta_{i}$ are the parameters of the $Q$-network at iteration $i$.

In this paper, we identified delay and prolongedness as a common roadblock for using RL in drug dosing. We defined PAE-POMDP, a sub-class of POMDPs in which action effects persist for multiple time steps, and introduced a simple and effective framework to convert PAE-POMDPs into MDPs which can be subsequently solved with traditional RL algorithms. We evaluated our proposed approach on a toy task and on a glucose control task, for which we proposed a clinically-inspired reward function for the glucose control simulator, which we hope will facilitate further research. Our quantitative results have shown that our proposed approach to convert PAE-POMDPs into MDPs is competitive and offers important advantages over the baselines. In particular, we have shown that by introducing domain knowledge on the prolongedness of action effects, we could build a solution capable of matching state-of-the-art recurrent policies such as ADRQN while being remarkably more time and memory efficient. Our qualitative results have further emphasized the benefits of the introduced Effective-DQN policy, which appeared to be more conservative than the recurrent policy, despite using the same reward function.

Our glucose control experiments are limited to a single virtual patient environment. Since humans exhibit similar profiles for prolonged action effects, similar performance on patients from other demographics is expected but should be evaluated. Although based on pharmacological literature we acknowledge that delay and prolongedness are present in any drug dosing scenario, we did not find other simulated environments to test our hypothesis. Hence, our experiments at the moment are limited to a single drug effect simulation. Although this work is only in the context of drug dosing, prolonged action effects seem to be present in other application areas such as robotics, where the decay assumption may not be justified. The recurrent solution is still the most general solution in such cases. This work does not explore those applications.

In this work, we have only considered discrete action spaces. Exploring continuous actions might lead to more interesting insights in the future. Delay and prolongedness appear across all drug dosing scenarios, so we hope future RL applications of autonomous drug dosing will consciously account for them. More broadly, future work includes coming up with more generalized solutions for action superposition that accommodate qualitative actions and are suitable for application areas beyond drug dosing.