An ocular biomechanics environment for reinforcement learning

Eye movement is one of the most complex, and the fastest movement that our body performs (Leigh and Zee, 2015); the different eye movement systems are tightly coupled with mental, cognitive and psychological states of the individual (Wong, 2008; Iskander et al., 2018a). One of the most studied eye movement systems is saccade, which shifts the gaze direction to a new point of interest rapidly (Gilchrist, 2011). Simulating the neural control of the muscles that could efficiently achieve eye movement through biomechanical simulation and analysis is an essential tool for studying different eye movement systems in normal and pathological cases Iskander et al. (2018c, d).

The horizontal, vertical and torsional eye movements are created through the activation of six extraocular muscles (EOM), Fig. LABEL:sub@fig:muscles Iskander et al. (2018c). The action/name of each of the six EOM are described in Table 1. To move the eye to the right, the right LR and the left MR are activated (agonists) while the right MR and left LR are inhibited (antagonists) (Sherrington, 1893; Hering et al., 1977). The opposite happens for leftward eye movement. To move the eye vertically, the SR, IR, SO and IO muscles are activated/inhibited (Purves et al., 2001; Scudder et al., 2002; Sparks, 2002). Next, we will introduce the concept of reinforcement learning as it will be used to control an ocular biomechanical system.

Reinforcement learning (RL) is a form of artificial intelligence that is different and fundamentally more difficult than supervised learning. According to Sutton and Barto (Sutton and Barto, 2018), RL is learning how to map a situation into an action such that a numerical reward is maximised. In the case of biomechanical studies, the situations is the biomechanical model state ($s$), and the actions is the muscle excitation signals ($a$), where as the numerical reward ($r$) reflects the desired movement. And, thus the RL agent learns to map each state into an efficient action that maximises the reward. In addition, we have a policy ($P(s)$) which defines the strategy the agent uses, Fig. LABEL:sub@fig:model. The policy $P(s)$ is the mapping from the $s$ to $a$. As the agent proceeds in the training phase, $P(s)$ evolves to produce the highest cumulative reward over time. The DRL training takes place in episodes. Each episode is a trial that allows the agent to explore the environment and to receive a reward based on how good or bad its behaviour was. The cumulative reward throughout an episode determines effectiveness of the training. RL has been used to model neural control of movements such as walking, running and standing (Hossny and Iskander, 2020; Kidziński et al., 2018a, b, 2020).

In the following, we present an ocular biomechanics simulation environment suitable for RL; and we train an agent to control the ocular biomechanics system by producing adequate muscle excitation signals to perform saccades.

The DRL environment used is made up of five components, as follows, the (1) neuro-musculoskeletal model; (2) the continuous state vector, that is produced from the environment; (3) the continuous action vector (muscle excitation signal), used to activate the neuro-musculoskeletal model; (4) the reward function to be maximise; and finally (5) the training mechanism used, which includes the actor and critic neural networks used.

During training, the agent achieved 26 milestones to reach the final trained state. Each milestone defined an increase in the cumulative reward and thus, a better policy ($P(s)$) was achieved, Fig. LABEL:sub@fig:milestone_reward.

Figure 2 shows the results of the first testing phase done on each milestone. Figure LABEL:sub@fig:milestone_reward shows the cumulative reward achieved at each milestone. As the training proceeded, the cumulative reward improved and approached zero which is the optimal reward value. Figures LABEL:sub@fig:M24 and LABEL:sub@fig:M25 show the rewards achieved at each step for the last two milestones. Figures LABEL:sub@fig:RA and LABEL:sub@fig:RL shows the muscle activation signal of the right and left eye, respectively during the episode number 2 of the last milestone. The object of interest was displaced by $dy$=0.1231 and $dz$=-0.1112. For the eyes to follow this object, both eyes has to be elevated; the right eye has to be adducted; and the left eye abducted. Figures LABEL:sub@fig:RA and LABEL:sub@fig:RL shows high activation of IO which causes elevation and abduction of the eye in contrast to SO which has a decreasing activation as it is an agonist muscle.

Figure 3 shows the muscle activation signals resulting from the second testing phase, each figure shows the mean activation of each muscle and the shaded part shows the standard deviation. Table 2 shows mean, maximum, minimum, and standard deviation of the distance between the right/left PoG and the object of interest for each object displacement case. The mean distance was approximately $6.1\pm 2.2$ cm, which is approximately equivalent to a deviation angle of $3.5\degree~{}\pm 1.25\degree$, respectively, since the object is 1 m away (x-direction). The statistics were calculated after removing the first 20 steps, equivalent to $20\times 0.01=0.2$ seconds. Figure 4 shows the change in the distance for the right and left eye over time.

In this paper, we presented an ocular environment that could be used in reinforcement learning. The environment was used to train a DRL agent to learn to move the eyes and fixate on a static object at different positions with a mean deviation angle, approximately $3.5\degree~{}\pm 1.25\degree$. It is noted that normal eyes are not stationary even at fixations but exhibits movement of small amplitude around the region of interest (Leigh and Zee, 2015). The DRL agent exhibited a similar behaviour; this is reflected in the shaded parts, presenting standard deviation, in Fig. 3 and Fig. 4.

The DRL agent adapted a model specific neural network that worked on capturing the EOM coordinated activation, Fig. LABEL:sub@fig:eomctrl. The main objective of the training was to drive the eyes to stabilise at or around the object of interest. Therefore, the agent tried to make use of all the muscles to get to the objective. The only constraint imposed was the inverse relationship between the right and left horizontal muscles (LR and MR) and the similarity between the other muscles of the left and right eyes.

The presented framework has its limitations too. As discussed, the agents main target was to maximise the reward; so it made optimal use of all muscles, without consideration of minimal effort or coordination between agonist and antagonist muscles (Wong, 2008), as it was not accounted for in the reward. The forces of the extraocular muscles are very small compared to the forces exerted by other skeletal muscles in the upper and lower limbs and consequently the metabolic cost is balanced by the benefits of having rapid and accurate fixations (Iskander et al., 2018d; Gilchrist, 2011; Leigh and Zee, 2015; Iskander et al., 2018c).

From Fig. 3, the agent relied on using the IO extensively which has a primary action of excyclotorsion and a secondary and tertiary action of elevation and abduction, respectively (Wong, 2008), Table 1. In contrast, LR and MR are used solely for abduction and adduction, respectively. Therefore, to compensate the high activation of IO, the IR was in most cases activated to compensate for the elevation caused by the IO.

From Figures 2 and 3, we can see that the agent was aspiring to achieve the best results; and it used most of the muscles all the time to achieve that objective. Although this may not be physiologically sound, it is a first step towards training a DRL agent for ocular motility. The next stage of this research will focus on fine tuning the actor and critic neural networks to achieve better results comparable to the ocular control theories established (Robinson et al., 1975; Jürgens et al., 1981; Scudder et al., 2002; Sparks, 2002). The environment can be also modified to simulate neck movement and thus, simulate vestibulo-ocular reflex and also, eye-hand coordination. Numerous scenarios can be simulated, from normal to pathological scenarios, and then analysis can be followed which will highlight how different control strategies can be used to perform improved eye movements. The DRL agent training can also be aligned with paediatrics research in ocular development.

None