Vitreoretinal Surgical Robotic System with Autonomous Orbital Manipulation using Vector-Field Inequalities

The automation of vitreoretinal tasks is an active research field. In recent years, He et al. [18] performed the bimanual control of instruments and proposed an automatic light pipe actuation system. Kim et al. [19] automated a tool-navigation task using deep imitation learning. Shin et al. [20] tackled the semi-autonomous extraction of lens fragments. Dehghani et al. [21] achieved autonomous docking of the instrument to the trocar.

Regarding object manipulation, some works addressed rigid multibody systems [22, 23] and collaborative deformable tissue manipulation [24]. However, these strategies cannot be directly applied to orbital manipulation, where the instruments penetrate the eyeball instead of grabbing it.

To the best of the authors’ knowledge, the works closest to our objectives are the ones from Wei et al. [13], where they proposed the mathematical model of orbital manipulation by separating intraocular manipulation from orbital manipulation, and Yu et al. [14] from the same group, where they demonstrated their method with a physical model.

In their mathematical model, effective for their purposes, intraocular and orbital manipulation were controlled individually. Our purpose is, instead, to automate orbital manipulation in a transparent way without any change in the description of the task. This autonomous orbital manipulation also requires the definition of hard limits for the motion of the eye, which cannot be achieved by prior work. In this context, we propose a new control strategy for orbital manipulation using inequality constraints based on the vector-field inequalities (VFIs) methodology [25].

The main contributions of this work are:

1. 1.

   new VFIs for orbital manipulation based on new distance functions and Jacobians unique to this task, and

2. 2.

   simulation and experimental results in a physical robotic system to evaluate the effects of orbital manipulation and its feasibility.

The quaternion set is

$\displaystyle\mathbb{H}\triangleq$

$\displaystyle\left\{h_{1}+\hat{\imath}h_{2}+\hat{\jmath}h_{3}+\hat{k}h_{4}\,:\,h_{1},h_{2},h_{3},h_{4}\in\mathbb{R}\right\},$

where $\hat{\imath}^{2}=\hat{\jmath}^{2}=\hat{k}^{2}=\hat{\imath}\hat{\jmath}\hat{k}=-1$. Elements of the set $\mathbb{H}_{p}\triangleq\left\{\boldsymbol{h}\in\mathbb{H}\,:\,\operatorname{\mathrm{Re}}\left(\boldsymbol{h}\right)=0\right\}$ represent translations in $\mathbb{R}^{3}$. The set of quaternions with unit norm, $\mathbb{S}^{3}\triangleq\left\{\boldsymbol{r}\in\mathbb{H}\,:\,\left\|\boldsymbol{r}\right\|=1\right\}$, represent rotations. We use the operator $\mathrm{v}_{4}$ to map a quaternion $\boldsymbol{h}$ $\in$ $\mathbb{H}$ into a column vector $\mathbb{R}^{4}$. Moreover, the Hamilton operators $\overset{+}{\boldsymbol{H}}_{4}$ and $\overset{-}{\boldsymbol{H}}_{4}$ [27, Def. 2.1.6] satisfy $\operatorname{v}_{4}\left(\boldsymbol{h}\boldsymbol{h}^{\prime}\right)=\overset{+}{\boldsymbol{H}}_{4}\left(\boldsymbol{h}\right)\operatorname{v}_{4}\left(\boldsymbol{h}^{\prime}\right)=\overset{-}{\boldsymbol{H}}_{4}\left(\boldsymbol{h}^{\prime}\right)\operatorname{v}_{4}\left(\boldsymbol{h}\right)$, $\boldsymbol{C}_{4}=\text{diag}(1,\,-1,\,-1,\,-1)$ satisfies $\operatorname{v}_{4}\left(\boldsymbol{h}^{*}\right)=\boldsymbol{C}_{4}\operatorname{v}_{4}\left(\boldsymbol{h}\right)$, and $\overline{\boldsymbol{S}}$ [25, Eq. (3)] has the properties $\operatorname{v}_{4}\left(\boldsymbol{h}\times\boldsymbol{h}^{\prime}\right)=\overline{\boldsymbol{S}}\left(\boldsymbol{h}\right)\operatorname{v}_{4}\left(\boldsymbol{h}^{\prime}\right)=\overline{\boldsymbol{S}}\left(\boldsymbol{h}^{\prime}\right)^{T}\operatorname{v}_{4}\left(\boldsymbol{h}\right)$ for $\boldsymbol{h},\,\boldsymbol{h}^{\prime}\in\mathbb{H}_{p}$.

We control the translations of the instruments’ tips using a centralized kinematic control strategy [15]. Let $\tilde{\boldsymbol{t}}_{i}\triangleq\tilde{\boldsymbol{t}}_{i}\left(\boldsymbol{q}_{i}\right)=\boldsymbol{t}_{i}-\boldsymbol{t}_{i\text{,d}}$ be the translation error between the current translation $\boldsymbol{t}_{\text{i}}\in\mathbb{H}_{p}$ and the desired translation $\boldsymbol{t}_{i\text{,d}}\in\mathbb{H}_{p}$ of the $i$-th robot’s end effector, with $i\in\left\{1,2\right\}$. Then, the desired control signal, $\boldsymbol{u}=\begin{bmatrix}\boldsymbol{u}_{1}^{T}&\boldsymbol{u}_{2}^{T}\end{bmatrix}^{T}$, is obtained as

	$\displaystyle\boldsymbol{u}\in\underset{\dot{\boldsymbol{q}}}{\text{arg min}}\$	$\displaystyle\beta\left(f_{\text{t,1}}+f_{\text{$\lambda$,1}}\right)+\left(1-\beta\right)\left(f_{\text{t,2}}+f_{\text{$\lambda$,2}}\right)$		(1)
	subject to	$\displaystyle\ \boldsymbol{W}\dot{\boldsymbol{q}}\preceq\boldsymbol{w},$

in which $\boldsymbol{q}=\left[\begin{array}[]{cc}\boldsymbol{q}_{1}^{T}&\boldsymbol{q}_{2}^{T}\end{array}\right]^{T}$, $f_{\text{t,i}}\triangleq\left\|\boldsymbol{J}_{\boldsymbol{t}_{i}}\dot{\boldsymbol{q}}_{i}+\eta\operatorname{v}_{4}\left(\tilde{\boldsymbol{t}}_{i}\right)\right\|_{2}^{2}$ are the cost functions related to the translation errors, $f_{\text{$\lambda$,}i}\triangleq\lambda\left\|\dot{\boldsymbol{q}}_{i}\right\|_{2}^{2}$ are the cost functions related to the joint velocity norm, and $\boldsymbol{J}_{\boldsymbol{t}_{i}}\in\mathbb{R}^{4\times n_{i}}$ are the translation Jacobians [28] that satisfy $\operatorname{v}_{4}\left(\dot{\boldsymbol{t}}_{i}\right)=\boldsymbol{J}_{\boldsymbol{t}_{i}}\dot{\boldsymbol{q}}_{i}$. In addition, $\eta\in\left(0,\infty\right)\subset\mathbb{R}$ is a tunable gain, $\lambda\in[0,\infty)\subset\mathbb{R}$ is the damping factor, and $\beta\in[0,\,1]\subset\mathbb{R}$ is a weight that defines the priority between the two robots. The $r$ inequality constraints $\boldsymbol{W}\dot{\boldsymbol{q}}\preceq\boldsymbol{w}$, in which $\boldsymbol{W}\triangleq\boldsymbol{W}\left(\boldsymbol{q}\right)\in\mathbb{R}^{r\times\left(n_{1}+n_{2}\right)}$ and $\boldsymbol{w}\triangleq\boldsymbol{w}\left(\boldsymbol{q}\right)\in\mathbb{R}^{r}$, are used to generate active constraints using VFIs [25].

The VFI method [25] uses signed distance functions $d\triangleq d\left(\boldsymbol{q},t\right)\in\mathbb{R}$ between two geometric primitives. The time-derivative of the distance is

$\displaystyle\dot{d}=$

$\displaystyle\underbrace{\frac{\partial\left(d\left(\boldsymbol{q},t\right)\right)}{\partial\boldsymbol{q}}}_{\boldsymbol{J}_{d}}\dot{\boldsymbol{q}}+\zeta\left(t\right)\text{,}$

where $\boldsymbol{J}_{d}\in\mathbb{R}^{1\times\left(n_{1}+n_{2}\right)}$ is the distance Jacobian and $\zeta\left(t\right)=\dot{d}-\boldsymbol{J}_{d}\dot{\boldsymbol{q}}$ is the residual that contains the distance dynamics unrelated to $\dot{\boldsymbol{q}}$. Then, by using a safe distance $d_{\text{safe}}\triangleq d_{\text{safe}}\left(t\right)\in[0,\infty)$, we define an error $\tilde{d}\triangleq\tilde{d}\left(\boldsymbol{q},t\right)=d_{\text{safe}}-d$ to generate safe zones or $\tilde{d}\triangleq d-d_{\text{safe}}$ to generate restricted zones. With these definitions, and given $\eta_{d}\in[0,\infty)$, the signed distance dynamics is constrained by $\dot{\tilde{d}}\geq-\eta_{d}\tilde{d}$ in both cases, which actively constrains the robot motion only in the direction approaching the boundary between the primitives so that the primitives do not collide. That is, the following constraint is used to generate safe zones,

$\displaystyle\boldsymbol{J}_{d}\dot{\boldsymbol{q}}\leq$

$\displaystyle\eta_{d}\tilde{d}-\zeta_{\text{safe}}\left(t\right)\text{,}$

(2)

for $\zeta_{\text{safe}}\left(t\right)\triangleq\zeta\left(t\right)-\dot{d}_{\text{safe}}$. Alternatively, restricted zones are generated by

$\displaystyle-\boldsymbol{J}_{d}\dot{\boldsymbol{q}}\leq$

$\displaystyle\eta_{d}\tilde{d}+\zeta_{\text{safe}}\left(t\right)\text{.}$

(3)

In geometrical terms, orbital manipulation is ensured by keeping the relative position between the RCMs of each instrument while they otherwise freely move. Let the distance between the RCMs be $d_{\text{OM}}\in\mathbb{R}^{+}$ as shown in Fig. 3, we constrain the squared distance²²2We use the squared distance since its time derivative, which we calculate in Section V-C, is defined everywhere. $D_{\text{OM}}\left(\boldsymbol{q}\left(t\right)\right)\triangleq D_{\text{OM}}=d_{\text{OM}}^{2}$ as follows

	$$\displaystyle-D_{\text{safe}}\leq D_{\text{OM}}-D_{\text{OM},\text{init}}\leq D_{\text{safe}}\iff$$
	$$\displaystyle\underbrace{D_{\text{OM}}-\left(D_{\text{OM},\text{init}}-D_{\text{safe}}\right)}_{\tilde{D}_{\text{OM}}^{+}}\geq 0,\ \underbrace{\left(D_{\text{OM},\text{init}}+D_{\text{safe}}\right)-D_{\text{OM}}}_{\tilde{D}_{\text{OM}}^{-}}\geq 0\text{,}$$		(6)

where $D_{\text{safe}}=0.5\,\mathrm{mm}$ and $D_{\text{OM},\text{init}}=D_{\text{OM}}\left(\boldsymbol{q}\left(t\right)\right)|_{t=0}$ are constants. Then, based on (2) and (3), these constraints in terms of joint velocities become

$\displaystyle\left[\begin{array}[]{c}-\boldsymbol{J}_{\text{OM}}\\ \boldsymbol{J}_{\text{OM}}\end{array}\right]\dot{\boldsymbol{q}}$

$\displaystyle\leq\eta_{\text{OM}}\left[\begin{array}[]{c}\tilde{D}_{\text{OM}}^{+}\\ \tilde{D}_{\text{OM}}^{-}\end{array}\right]\text{,}$

(11)

where $\eta_{\text{OM}}=0.1$, and $\boldsymbol{J}_{\text{OM}}\in\mathbb{R}^{1\times n}$ is the orbital manipulation Jacobian that relates the joint velocities $\dot{\boldsymbol{q}}$ to the time derivative of $\tilde{D}_{\text{OM}}^{+}$ and $\tilde{D}_{\text{OM}}^{-}$ in the form of $\dot{\tilde{D}}_{\text{OM}}^{+}=\boldsymbol{J}_{\text{OM}}\dot{\boldsymbol{q}}$ and $\dot{\tilde{D}}_{\text{OM}}^{-}=-\boldsymbol{J}_{\text{OM}}\dot{\boldsymbol{q}}$, respectively. To enforce (11), we have to find $D_{\text{OM}}$ and $\boldsymbol{J}_{\text{OM}}$. We address this in Section V-B and V-C.

The goal of this section is to find the distance function $D_{\text{OM}}$ as a function of $\boldsymbol{q}$ to enforce (11). We use the geometric relationships shown in Fig. 3. Let the world reference-frame $\mathcal{F}_{\text{$o$}}$ be at the center of the eyeball, and let the translations of the RCM points of both instruments be $\boldsymbol{t}_{\text{OM},i}\left(\boldsymbol{q}_{i}\right)\triangleq\boldsymbol{t}_{\text{OM},i}\in\mathbb{H}_{p}\ \left(i=1,\,2\right)$. Then, we have

$\displaystyle D_{\text{OM}}=$

$\displaystyle\left\|\boldsymbol{t}_{\text{OM},1}-\boldsymbol{t}_{\text{OM},2}\right\|^{2}\text{.}$

(12)

Next, we assume that, without loss of generality, $\boldsymbol{l}_{i}\left(\boldsymbol{q}_{i}\right)\triangleq\boldsymbol{l}_{i}\in\mathbb{S}^{3}\cap\mathbb{H}_{p}$ are the directions of the $z-$axes of the instruments pointing inside the eye and given by

$\displaystyle\boldsymbol{l}_{i}=$

$\displaystyle\boldsymbol{r}_{i}\hat{k}\left(\boldsymbol{r}_{i}\right)^{*}\text{,}$

(13)

where $\boldsymbol{r}_{i}\left(\boldsymbol{q}_{i}\right)\triangleq\boldsymbol{r}_{i}\in\mathbb{S}^{3}$ is the rotation of the instrument. Moreover, letting the distances between the tips and the RCM positions be $d_{i}\left(\boldsymbol{q}_{i}\right)\triangleq d_{i}\in\mathbb{R}$, we have

$\displaystyle\boldsymbol{t}_{\text{OM},i}$

$\displaystyle=\boldsymbol{t}_{i}-d_{i}\boldsymbol{l}_{i}.$

(14)

Using the law of cosines, the radius of the eyeball $r_{\text{eye}}\in\mathbb{R}^{+}-\left\{0\right\}$, and (14), we have

		$\displaystyle\begin{array}[]{cc}d_{i}^{2}&=\left\|\boldsymbol{t}_{i}\right\|^{2}+r_{\text{eye}}^{2}-2\left\langle\boldsymbol{t}_{i},\boldsymbol{t}_{\text{OM},i}\right\rangle\\ &=r_{\text{eye}}^{2}-\left\|\boldsymbol{t}_{i}\right\|^{2}+2d_{i}\left\langle\boldsymbol{t}_{i},\boldsymbol{l}_{i}\right\rangle\end{array}$
	$\displaystyle\iff$	$\displaystyle d_{i}^{2}-2\left\langle\boldsymbol{t}_{i},\boldsymbol{l}_{i}\right\rangle d_{i}+\left\|\boldsymbol{t}_{i}\right\|^{2}-r_{\text{eye}}^{2}=0$
	$\displaystyle\iff$	$\displaystyle d_{i}=\left\langle\boldsymbol{t}_{i},\boldsymbol{l}_{i}\right\rangle\pm\sqrt{\left\langle\boldsymbol{t}_{i},\boldsymbol{l}_{i}\right\rangle^{2}-\left\|\boldsymbol{t}_{i}\right\|^{2}+r_{\text{eye}}^{2}}.$

Because of our definition of the direction for $\boldsymbol{l}_{i}$, only the positive value is relevant.³³3The other value has an interesting mathematical meaning, because it will be the other point in the sphere that satisfies this same equation for $-\boldsymbol{l}_{i}$. Therefore, we have

$\displaystyle d_{i}$

$\displaystyle=\left\langle\boldsymbol{t}_{i},\boldsymbol{l}_{i}\right\rangle+\underbrace{\sqrt{\left\langle\boldsymbol{t}_{i},\boldsymbol{l}_{i}\right\rangle^{2}-\left\|\boldsymbol{t}_{i}\right\|^{2}+r_{\text{eye}}^{2}}}_{h_{1}}\text{.}$

(15)

Our next goal is to find the corresponding Jacobian $\boldsymbol{J}_{\text{OM}}$ to enforce (11). Since the time derivatives of $\tilde{D}_{\text{OM}}^{+}$ and $\tilde{D}_{\text{OM}}^{-}$ are $\dot{D}_{\text{OM}}$ and $-\dot{D}_{\text{OM}}$, we can get $\boldsymbol{J}_{\text{OM}}$ by finding the time derivative of $D_{\text{OM}}$ with respect to the joint velocities $\dot{\boldsymbol{q}}$. From (12), we have

$\displaystyle\dot{D}_{\text{OM}}=$

$\displaystyle 2\operatorname{v}_{4}\left(\boldsymbol{t}_{\text{OM},1}-\boldsymbol{t}_{\text{OM},2}\right)^{T}\operatorname{v}_{4}\left(\dot{\boldsymbol{t}}_{\text{OM},1}-\dot{\boldsymbol{t}}_{\text{OM},2}\right)\text{.}$

(16)

From (14), the time derivative of $\boldsymbol{t}_{\text{OM},i}\ \left(i=1,\,2\right)$ is

$\displaystyle\operatorname{v}_{4}\left(\dot{\boldsymbol{t}}_{\text{OM},i}\right)$

$\displaystyle=\boldsymbol{J}_{\boldsymbol{t}_{i}}\dot{\boldsymbol{q}}_{i}-\operatorname{v}_{4}\left(\dot{d}_{i}\boldsymbol{l}_{i}+d_{i}\dot{\boldsymbol{l}}_{i}\right).$

(17)

Then, we have to find $\dot{\boldsymbol{l}}_{i}$ and $\dot{d}_{i}$. From (13), we get

$$\displaystyle\operatorname{v}_{4}\left(\dot{\boldsymbol{l}}_{i}\right)=\underbrace{\left(\overset{-}{\boldsymbol{H}}_{4}\left(\hat{k}\left(\boldsymbol{r}_{i}\right)^{*}\right)+\overset{+}{\boldsymbol{H}}_{4}\left(\boldsymbol{r}_{i}\hat{k}\right)\boldsymbol{C}_{4}\right)\boldsymbol{J}_{\boldsymbol{r}_{i}}}_{\boldsymbol{J}_{\boldsymbol{l}_{i}}}\dot{\boldsymbol{q}}_{i}\text{,}$$

(18)

where $\boldsymbol{J}_{\boldsymbol{r}_{i}}\in\mathbb{R}^{1\times n_{i}}$ is the rotation Jacobian [28] that satisfies $\dot{\boldsymbol{r}}_{i}=\boldsymbol{J}_{\boldsymbol{r}_{i}}\dot{\boldsymbol{q}}_{i}$. As for $\dot{d}_{i}$, from (15), we have

$\displaystyle\dot{d}_{i}$

$\displaystyle=\overbrace{\left\langle\dot{\boldsymbol{t}}_{i},\boldsymbol{l}_{i}\right\rangle+\left\langle\boldsymbol{t}_{i},\dot{\boldsymbol{l}}_{i}\right\rangle}^{h_{2}}+\dot{h}_{1}.$

(19)

Then, considering $h_{1}$ is a positive value,⁴⁴4Since the tip of the instrument is kept inside the eyeball, $r_{\text{eye}}^{2}>\left\|\boldsymbol{t}_{i}\right\|^{2}\iff r_{\text{eye}}^{2}-\left\|\boldsymbol{t}_{i}\right\|^{2}>0$. we get

	$\displaystyle\dot{h_{1}}$	$\displaystyle=\frac{2\left(\left\langle\dot{\boldsymbol{t}}_{i},\boldsymbol{l}_{i}\right\rangle+\left\langle\boldsymbol{t}_{i},\dot{\boldsymbol{l}}_{i}\right\rangle\right)-2\left\langle\boldsymbol{t}_{i},\dot{\boldsymbol{t}}_{i}\right\rangle}{h_{1}}$
		$\displaystyle=\frac{1}{h_{1}}\left(2h_{2}-2\operatorname{v}_{4}\left(\boldsymbol{t}_{i}\right)^{T}\operatorname{v}_{4}\left(\dot{\boldsymbol{t}}_{i}\right)\right),$

and, from (18),

$\displaystyle h_{2}=$

$\displaystyle\underbrace{\left(\operatorname{v}_{4}\left(\boldsymbol{l}_{i}\right)^{T}\boldsymbol{J}_{\boldsymbol{t}_{i}}+\operatorname{v}_{4}\left(\boldsymbol{t}_{i}\right)^{T}\boldsymbol{J}_{\boldsymbol{l}_{i}}\right)}_{\boldsymbol{J}_{h_{2}}}\dot{\boldsymbol{q}}_{i}.$

(20)

Therefore, we have

$$\dot{h_{1}}=\underbrace{\left(2\boldsymbol{J}_{h_{2}}-2\operatorname{v}_{4}\left(\boldsymbol{t}_{i}\right)^{T}\boldsymbol{J}_{\boldsymbol{t}_{i}}\right)}_{\boldsymbol{J}_{h_{1}}}\dot{\boldsymbol{q}}_{i}.$$

(21)

We can now work backwards to find the corresponding Jacobian $\boldsymbol{J}_{\text{OM}}$. By substituting (20) and (21) into (19), we have

$$\dot{d}_{i}=\underbrace{\left(\boldsymbol{J}_{h_{2}}+\boldsymbol{J}_{h_{1}}\right)}_{\boldsymbol{J}_{d_{i}}}\dot{\boldsymbol{q}}_{i}\text{.}$$

(22)

Moreover, substituting (18) and (22) into (17) results in

$\displaystyle\operatorname{v}_{4}\left(\dot{\boldsymbol{t}}_{\text{OM},i}\right)$

$\displaystyle=\underbrace{\left(\boldsymbol{J}_{\boldsymbol{t}_{i}}-\operatorname{v}_{4}\left(\boldsymbol{l}_{i}\right)\boldsymbol{J}_{d_{i}}+d_{i}\boldsymbol{J}_{\boldsymbol{l}_{i}}\right)}_{\boldsymbol{J}_{\boldsymbol{t}_{\text{OM},i}}}\dot{\boldsymbol{q}}_{i}.$

(23)

Finally, from (16) and (23), we find

$\displaystyle\dot{D}_{\text{\text{R}}}=$

$\displaystyle\underbrace{2\operatorname{v}_{4}\left(\boldsymbol{t}_{\text{OM},1}-\boldsymbol{t}_{\text{OM},2}\right)^{T}\left[\begin{array}[]{cc}\boldsymbol{J}_{\boldsymbol{t}_{\text{OM},1}}&-\boldsymbol{J}_{\boldsymbol{t}_{\text{OM},2}}\end{array}\right]}_{\boldsymbol{J}_{\text{OM}}}\dot{\boldsymbol{q}}\text{.}$

(25)

When the eye is autonomously rotated, safe limits must be enforced to preserve the integrity of the eye muscles. To do so, we also propose additional constraints.

Fig. 4 shows the constraints $\text{C}_{\text{rot}1}$ and $\text{C}_{\text{rot}2}$. These constraints prevent the eyeball from rotating beyond a certain angle. To this purpose, we constrain the RCM positions $\boldsymbol{t}_{\text{OM},1}$ and $\boldsymbol{t}_{\text{OM},2}$ to be within a certain distance from the planes $\boldsymbol{\pi}_{\text{rot}1}$ and $\boldsymbol{\pi}_{\text{rot}2}$. The plane $\boldsymbol{\pi}_{\text{rot}1}$ is the plane perpendicular to the $x-$axis and contains the center of the eyeball. The plane $\boldsymbol{\pi}_{\text{rot}2}$ is the plane perpendicular to the $z-$axis and $d_{\boldsymbol{\pi}_{\text{rot}2}}$ away from the $xy-$plane. The distance $d_{\boldsymbol{\pi}_{\text{rot}2}}$ is half the radius of the eyeball.

These constraints can be enforced using the signed distances and Jacobians proposed in [25, Eq. (57), (59)]. Let the signed distances and Jacobians between $\boldsymbol{t}_{\text{OM},1}$ and $\boldsymbol{\pi}_{\text{rot}1},\,\boldsymbol{\pi}_{\text{rot}2}$ be $d_{\text{rot}1,1},\,d_{\text{rot}2,1}\in\mathbb{R}$ and $\boldsymbol{J}_{\text{rot}1,1},\,\boldsymbol{J}_{\text{rot}2,1}\in\mathbb{R}^{1\times n_{1}}$, and the signed distances and Jacobians between $\boldsymbol{t}_{\text{OM},2}$ and $\boldsymbol{\pi}_{\text{rot}1},\,\boldsymbol{\pi}_{\text{rot}2}$ be $d_{\text{rot}1,2},\,d_{\text{rot}2,2}\in\mathbb{R}$ and $\boldsymbol{J}_{\text{rot}1,2},\,\boldsymbol{J}_{\text{rot}2,2}\in\mathbb{R}^{1\times n_{2}}$. Then, to enforce $\text{C}_{\text{rot}j}\ \left(j=1,\,2\right)$, we can use

$\displaystyle\left[\begin{array}[]{cc}-\boldsymbol{J}_{\text{rot}j,1}&\boldsymbol{O}_{1\times n_{1}}\\ \boldsymbol{O}_{1\times n_{2}}&-\boldsymbol{J}_{\text{rot}j,2}\end{array}\right]\dot{\boldsymbol{q}}$

$\displaystyle\leq\eta_{\text{rot}}\left[\begin{array}[]{c}d_{\text{rot}j,1}\\ d_{\text{rot}j,2}\end{array}\right]\text{,}$

(30)

where $\eta_{\text{rot}}=1.0$.

The physical robotic setup shown in Fig. 1 was used for the experiment. To observe the orbital manipulation, the experiment was conducted without the face of the BionicEyE. For the simulation study, this setup was replicated in CoppeliaSim (Coppelia Robotics, Switzerland). Communication with the robot was enabled by the SmartArmStack⁵⁵5https://github.com/SmartArmStack. The quaternion algebra and robot kinematics were implemented using DQ Robotics [29] for Python3. The calibration of the system and the registration of the initial RCM positions were performed as described in [17, Section X].

Inequality constraints, in general, can increase or reduce the manipulability of the system in complex ways, and currently, there is no systematic way of analyzing manipulability in this case. In this study, to assess the impact of the proposed orbital manipulation on the manipulability of the system, the manipulability measure proposed in [30] was used as a strict lower-bound to compare our proposed orbital manipulation strategy and conventional fixed RCM-based control.

As shown in Fig. 5, the procedure was as follows. First, the surgical needle’s tip was positioned to a point on the fundus with and without enforcing the orbital manipulation. Then, the manipulability measures after positioning were calculated and compared. Positioning was conducted from the same initial pose to 149 positions covering a $14\,\mathrm{mm}$ diameter circle, which is the double size of the usual target area. To make conditions equal, the tip of the light guide was commanded to be still.

For a Jacobian $\boldsymbol{J}$, the manipulability measure is calculated as

$\displaystyle\boldsymbol{\omega}=$

$\displaystyle\sqrt{\text{det}\left|\boldsymbol{J}\boldsymbol{J}^{T}\right|},$

(31)

which is related to the manipulability ellipsoid proposed in [30]. Compatible with the original definition, we used the following augmented Jacobians

$$\displaystyle\boldsymbol{J}_{\text{w/o}}=\left[\begin{array}[]{cc}\boldsymbol{J}_{\boldsymbol{t}_{\text{1}}}&\boldsymbol{O}_{3\times n_{2}}\\ \boldsymbol{O}_{3\times n_{1}}&\boldsymbol{J}_{\boldsymbol{t}_{2}}\\ \boldsymbol{J}_{\boldsymbol{t}_{\text{OM},1}}&\boldsymbol{O}_{1\times n_{2}}\\ \boldsymbol{O}_{1\times n_{1}}&\boldsymbol{J}_{\boldsymbol{t}_{\text{OM},2}}\end{array}\right]\text{ and }\boldsymbol{J}_{\text{with}}=\left[\begin{array}[]{c}\begin{array}[]{cc}\boldsymbol{J}_{\boldsymbol{t}_{\text{1}}}&\boldsymbol{O}_{3\times n_{2}}\end{array}\\ \begin{array}[]{cc}\boldsymbol{O}_{3\times n_{1}}&\boldsymbol{J}_{\boldsymbol{t}_{2}}\end{array}\\ \boldsymbol{J}_{\text{OM}}^{{}^{\prime}}\end{array}\right],$$

for the fixed-RCM evaluation and the orbital manipulation evaluation, respectively. Note that $\boldsymbol{J}_{\boldsymbol{t}_{i}}$ are the translation Jacobians, and $\boldsymbol{J}_{\boldsymbol{t}_{\text{OM},i}}$ are as found in (23). Moreover, $\boldsymbol{J}_{\text{OM}}^{{}^{\prime}}$ is the proposed orbital manipulation Jacobian that satisfies $\boldsymbol{J}_{\text{OM}}^{{}^{\prime}}\dot{\boldsymbol{q}}=\frac{d}{dt}\left(\left\|\boldsymbol{t}_{\text{OM},1}-\boldsymbol{t}_{\text{OM},2}\right\|\right)$. That is because it is well known that, to calculate the manipulability, we need to unify the units of the augmented Jacobian. Hence, by letting $\boldsymbol{h}_{3}=\left\|\boldsymbol{t}_{\text{OM},1}-\boldsymbol{t}_{\text{OM},2}\right\|>0$, the Jacobian $\boldsymbol{J}_{\text{OM}}^{{}^{\prime}}$ can be calculated as follows.

$\displaystyle\dot{\boldsymbol{h}}_{3}=$

$\displaystyle\underbrace{\frac{1}{\boldsymbol{h}_{3}}\operatorname{v}_{4}\left(\boldsymbol{t}_{\text{OM},1}-\boldsymbol{t}_{\text{OM},2}\right)^{T}\left[\begin{array}[]{cc}\boldsymbol{J}_{\boldsymbol{t}_{\text{OM},1}}&-\boldsymbol{J}_{\boldsymbol{t}_{\text{OM},2}}\end{array}\right]}_{\boldsymbol{J}_{\text{OM}}^{{}^{\prime}}}\dot{\boldsymbol{q}}.$

This experiment was conducted to show the feasibility of the proposed method and compare it with a conventional approach. In this experiment, the tip of the surgical needle was tele-operatively controlled using an input device (Touch, 3D Systems, USA). The operator arbitrarily moved the tip parallel to the image plane of the microscope in two modes, with the conventional fixed-RCM and the proposed orbital manipulation strategy. The field-of-view of the microscope was larger than in actual surgery for a proper workspace analysis.