Control of Walking Assist Exoskeleton with Time-delay Based on the Prediction of Plantar Force

If the current time is $t$, the plantar force $\bm{\hat{\bm{y}}_{t+s}}$ at the future time $t+s$ is predicted from the past signals $[{\bm{x}_{t},\bm{x}_{t-1},\ldots,\bm{x}_{t-n+1}}]$ measured by the IMU sensors as follows:

$$\hat{\bm{y}}_{t+s}=f(\bm{x}_{t},\bm{x}_{t-1},\ldots,\bm{x}_{t-n+1}),$$

(1)

where $s$ is the prediction time. $\bm{x}_{t},\ldots,\bm{x}_{t-n+1}$ are the measured data from time $t$ to time $t-n+1$. If we wear $m$ IMU sensors on the human body, $\bm{x}$ is a vector with $6\times m$ elements (acceleration $\bm{a}$ and angular velocity $\bm{\omega}$), $\bm{x}=[\bm{a}_{1},\bm{\omega}_{1},\ldots,\bm{a}_{m},\bm{\omega}_{m}]^{T}\in\bm{R}^{\{6\times m\}\times 1}$. If there were $k$ cells in each plantar sensor, the predicted plantar force is a vector with $k\times 2$ elements, $\hat{\bm{y}}\in\bm{R}^{\{k\times 2\}\times 1}$.

As shown in Fig. 2, a deep learning model using LSTM and fully-connected layers is designed to define the prediction function (1). The input of the LSTM cell in every time $t$ is the motion data $\bm{x}_{t}$ (i.e., the signal measured from the IMU sensors). We set the number of hidden states of each LSTM cell to 32. When inputting $n$ flames of past measured signals, the last output of the hidden state is input to a 3-layer fully-connected network with ReLU activation. The size of the input layer of this network is the same as the number of the hidden states (i.e., 32 neurons). The size of the output layer is the same as the number of the cells in two plantar sensors (i.e., $k\times 2$ neurons). One hidden layer with 16 neurons is used to obtain the non-linear relationship between the phase and the predicted values.

For training the network, we take $n$ frames of the measured past signals of the IMU sensors as input. The measured plantar force after $s$ frames is used as the target output. To use the trained network, we input the measured past $n$-frame signals from the IMU sensors, and the network will output the predicted plantar force of the future $s$ frame. When using this method, only the measured past signals should be used. Therefore, the prediction can also be used to control the exoskeleton in real-time.

The walking phase is estimated based on the predicted plantar force. Result from many other studies shows that the plantar force is related to the walking phases (i.e., swing phase and stance phase). The detailed walking stage (i.e., heel-strike, support, toe-off, swing) can also be estimated.

Fig. 3 shows the change in the plantar force of the left foot during one walking cycle from a heel-strike to the next heel-strike. The plantar force starts on the heel $f_{h}$ and increases to the first peak during the heel-strike stage. Then, during the support stage, the plantar force wqualizes across all pressure cells on the foot. Before the foot swing, the plantar force on the tip $f_{t}$ increases to the second peak for kicking the ground. When foot start the swing, the plantar force is zero, $f=0$. By detecting the changing points for each plantar force ($f_{h}$, $f_{m}$, and $f_{t}$), the walking phases can also be estimated.

In the experiments, wearable plantar sensor and IMU sensor were used since they can applied easily even while wearing a lower-limb exoskeleton. Fig. 4(a) shows the wireless IMU sensor (TSND151, ATR-Promotions) used in this research. This sensor can measure acceleration and angular velocity simultaneously and transmit the data at up to 1000Hz via Bluetooth. Fig. 4(b) shows the wireless plantar sensor (Loadsol, Novel) we used. It is a soft insole-type sensor and can monitor the normal force between the plantar side of the foot and the shoe. There are three subareas (i.e., three cells) in the sensor to measure the change of the pressure on each foot. The normal contact forces from forefoot (toes), midfoot (middle), and hindfoot (heel) can be measured with an accuracy of 10N and a data transmission speed of up to 200Hz via Bluetooth.

As shown in Fig. 5, we placed the IMU sensors on the lower legs near the ankles and put the plantar sensors in the two shoes. The data measured from the four sensors was resampled to 100Hz.

We tested the proposed method by implementing it with a lower-limb exoskeleton device (AWN-02, Atoun inc.). As shown in Fig. 6, with this device, there are two motors on each side of the hips. The motors are connected to the thighs near the knees using the rigid links and velcro-tape attachments. By rotating the motors, a force can be applied to rotate the thigh forward. The exoskeleton can move the leg and foot up and reduce the force needed to lift and swing the legs. This exoskeleton only controls the movement of the thighs. The lower legs where we placed the IMU sensors are free from the exoskeleton, which is suitable for using our method to determine the wearer’s intention for motion.

We assembled a small single board computer (Raspberry Pi 3B) in the exoskeleton. This computer can measure the signals from the IMU sensors through the Bluetooth communications. The trained network can be copied to this computer to predict the walking phases in real-time from the measured IMU signals. This computer is also used to control the exoskeleton by sending control signals to the control board through serial communication.

By measuring the time difference between motion signal and motor power, we calculated the time-delay between signal measurement and motor response is about 0.1s (i.e., $t_{dm}+t_{dr}\approx 0.1\text{s}$). The time-delay of computation (i.e., the time for the prediction using the trained network) depends on the number of the used frames of measured past signals $n$. Table 1 shows the computation time for different $n$. For example, if we use 0.2s past signals, the prediction time $t_{dc}$ would be about 0.024s, so the total time-delay would be about 0.124s, $t_{d}=(t_{dm}+t_{dr})+t_{dc}\approx 0.124\text{s}$. The prediction time $s$ should be larger than it (i.e., $s>124$).

Fig. 8 shows the average prediction error for all the testing data from all the subjects using different length of input IMU signals and predicting different future plantar forces. As shown in Fig. 8(a), using more past IMU signals reduced the prediction error from about 6.7% to 4.2%. However, even using more signals, the change was small. Considering the computing performance of the small computer (Raspberry PI 3B) used in this research, 0.2s IMU signals ($n=20$) were used for the prediction. Fig. 8(b) shows the average prediction error for different futures when inputting the same measured IMU signals (0.2s, $n=20$). With an increase in the prediction time $s$, the prediction errors also become larger (from about 4.6% to 12.6%). To achieve high-accuracy predictions, the prediction time should be as shorter as possible. As described above, using these sensors and exoskeleton, the prediction time $s$ should be larger than 0.124s. Therefore, $s=20$ was used in the following experiments.

Fig. 9 shows the prediction result for a six-step walking by a subject in a trail. After 0.2s ($n=20$), the plantar force was predicted using the measured IMU signals for the past 0.2s ($s=20$). The graphs show the plantar force of each cell in order: left heel $f_{hl}$, left middle $f_{ml}$, left toe $f_{tl}$, right heel $f_{hr}$, right middle $f_{mr}$, and right toe $f_{tr}$. The red solid lines show the predicted force and the blue dashed lines show the measured values using the pressure sensors. In this six-step walking, the subject started walking with the left leg and stopped with the right leg. The predicted values do not deviate largely from the actual measured values. Even for the start and end of walking, the predicted plantar force shows the same change as the measured value.

Fig. 10 shows the average prediction errors of each subject for all frames and all ten trails. All prediction errors are less then 50N (23.5~48.8N), which is about 7.5% of the average body weight of subjects. The average prediction error for all trails and all subjects is 33.9N.

Based on the predicted plantar force, we detected the assistance timing needed to support walking during the swing stage. The assistance timing is planned for when the plantar force on toe is smaller than 50N ($f_{t}\leq 50N$). This level was set by trial and error, considering the measurement resolution of the pressure sensor and the prediction error. Note that when using other exoskeleton to support walking in other stages, the assistance timing could also be detected in other thresolds or methods. The vertical lines in Fig. 9 show the detected assistance timings based on the measured and predicted plantar force. The red vertical lines show the predicted assistance timing while the black dash lines show the assistance timing calculated from measured data. The assistance timings calculated from the predicted and measured data are very close to each other. Fig. 11 shows the differences in the assistance timings of all subjects. During the walks, the differences in the assistance timings of all subjects are less than 0.079s, and the average difference is 0.055s. When starting the walks, the assisting timing can also be detected. The difference of all subjects are smaller than 0.175s, and the average difference for all subjects is 0.093s. It is larger than the difference during the walking, but still small enough for detecting the walking stages.

In this experiment, we measured the walking data from the same nine subjects as in the previous experiment. Fig. 12 shows the predicted result of a six-step walking in a trail by a subject wearing the exoskeleton and using the switches for control. The red solid lines and the black dash lines show the predicted and measured plantar forces applied on each cell during the walking. The same change tendency can also be predicted for both the start of walking and during the walking even with the assistance.

Fig. 13 shows the average prediction errors of all subjects. As in the previous experiment, even when wearing the exoskeleton, the plantar force can still be predicted with relatively small prediction errors (27.3~62.0N). The average prediction error of all subjects is 40.3N. Even with the assist of the exoskeleton, almost the same prediction accuracy was obtained.

The vertical lines in Fig. 12 show the predicted and desired control timings. The red solid lines and the black dush lines are the timing determined from the predicted and measured plantar force. The blue solid lines are the switched timings measured by the subjects pushing the switches in their hands. Most of these timings are close to each other.

Fig. 14 shows the time differences of the support timing between predicted timing, measured timing, and the switch timing. The dark gray boxes show the difference between the timings calculated from the predicted and measured plantar forces. As in previous experiment, even assisted by the exoskeleton, the proposed method can still determine the assistance timing almost as accurately as using the measured values. As a comparison, the light gray boxes show the differences between predicted timing and the switch timing. To assist the walking just in time, subjects should push the switches before moving their thigh due to the time-delay of the exoskeleton. Therefore, the differences between predicted timing and switch timing are larger than compared to detecting the timing from the plantar force. The average difference between measured and predicted timing is about 0.05s during walking and about 0.08s when starting after having paused.

Finally, as show in Fig. 15, we copied the trained models of three subjects to the small computer in the exoskeleton to measure the IMU signals and control the exoskeleton in real-time. In this case the exoskeleton was controlled by the movement of subject’s leg only the measured signals of IMU sensors. After using this exoskeleton, in questionnaires the subjects all felt that the walking motion was supported with good timing by the proposed prediction method. This was also better than switched control. Finally, even the start of walking was supported with good timing.