Bidirectional GaitNet: A Bidirectional Prediction Model of Human Gait and Anatomical Conditions

Realistic simulation of human movement is one of the long-standing challenges in computer graphics. Musculoskeletal simulation has provided stepping stones to reproduce a range of human movements at the biomechanical and anatomical levels, adding significant realism to the movements created. It can also be used to predict how changes in anatomical conditions (e.g., bone deformity, muscle capacity/deficiency, mass distribution) and intrinsic/extrinsic factors (e.g., metabolic energy expenditure, fatigue, pain) affect human movement.

Accurate setting of anatomical conditions is the very first and fundamental process for generating realistic movements in musculoskeletal simulation as it creates the space for anatomically plausible human movements. A number of biomechanical studies have been conducted to accurately model these conditions through various experiments on human subjects and cadavers (Arnold et al., 2010; Rajagopal et al., 2016; Delp et al., 1990, 2007; Carbone et al., 2015). Several standard models for the typical/average human body have been adopted by the research community (Delp et al., 1990, 2007) and those average body models have significantly contributed to recent progress in biomechanics research (Dembia et al., 2020; Lee et al., 2019; Park et al., 2022). Building an accurate musculoskeletal model of a specific (healthy or impaired) person has long been a notoriously difficult challenge since accurate modeling of live organs and tissues often requires invasive measurements.

In this paper, we address the problem of estimating the physiological parameters of a musculoskeletal model from observed gait cycles. Our anatomy model describes the conditions of individual bones and muscles with 300+ parameters. Conceptually speaking, we want to uncover the relationship between human anatomy and gait. Although this relationship is well accepted empirically in biomechanics and clinical gait analysis, the relationship is probabilistic rather than a one-to-one deterministic mapping. For example, many people with different physical conditions can walk with a similar gait, and conversely, there is no guarantee that two people with similar physical conditions will walk with a similar gait. In this paper, we build a novel generative model, which we call Bidirectional GaitNet, based on a comprehensive full-body, simulation-ready, musculoskeletal model. The Bidirectional GaitNet consists of forward and backward models. The forward model is functionally equivalent to predictive gait simulation, which generates a bipedal gait for any anatomical model with a specific set of physical conditions. Conversely, the backward model is its inverse process that estimates the physical conditions of the model given a gait. More specifically, we first learn the forward model by distilling the simulation data generated by a state-of-the-art predictive gait simulator and then construct a conditional Variational Autoencoder (c-VAE) with the forward model as its decoder. Once it is learned, its encoder serves as the backward model. By the nature of VAE, our backward model generates a distribution of physical conditions that potentially produce the input gait in predictive gait simulation, from which many different physical conditions can be sampled.

We demonstrate the power of our model by showing results with a variety of healthy/impaired gaits. The simulated results are validated in comparison to not only unseen simulated gaits but also gaits from real patients. The effectiveness of the non-trivial system design choices that we made for developing our model are also validated by ablation studies. Code for this paper is available at https://github.com/namjohn10/BidirectionalGaitNet.

In musculoskeletal simulation, the human body is typically modeled by rigid bones and flexible musculotendon units. The bones are connected by rotational joints to which the musculotendon units are attached such that they can actuate the joints. The muscle dynamics is often formulated using Hill-type muscles (Zajac, 1989; Delp et al., 2007) composed of contractile and elastic elements and the dynamics of each element is determined by force-length and force-velocity curves. Accurate simulation requires adequate determination of all anatomical conditions of the bones and the muscles based on reliable measurements. A set of parameters for a typical/average human have been estimated by taking measurements from live tissues and cadavers (Delp et al., 1990; Rajagopal et al., 2016; Arnold et al., 2010; Carbone et al., 2015). Estimates thus obtained have been used as default parameters in many simulation-based studies (Delp et al., 2007; Lee et al., 2019; Dembia et al., 2020). Anthropometric scaling of a musculoskeletal model can generate a range of models with different heights, weights, and limb lengths (Ryu et al., 2021). Building accurate individualized models often requires expensive medical images (e.g., CT and MRI) and labor-intensive image labeling (Matias et al., 2009; Levin et al., 2011; Li et al., 2022).

Building robust dynamic controllers that can drive musculoskeletal models has been regarded as an open challenge for decades because musculoskeletal models are high-dimensional and highly nonlinear, and their control systems are often partly under-actuated and, at the same time, partly over-determined. In this paper, we will focus on legged locomotion, although there have been a series of studies for simulating other body parts, such as upper bodies (Lee et al., 2009, 2018), eyes (Nakada et al., 2018), faces (Ichim et al., 2017; Cong et al., 2016), hands (Sachdeva et al., 2015) and swimming (Si et al., 2014).

Biomechanics researchers have developed locomotion controllers to answer research questions, such as how weakening certain muscles affect gait patterns. Open-loop  (Sok et al., 2007; Liu et al., 2010; Al Borno et al., 2013; Falisse et al., 2019), model-based feedback control  (Geyer and Herr, 2010; Yin et al., 2007; Lee et al., 2010; Ye and Liu, 2010; Coros et al., 2010; Ha et al., 2012; Liu et al., 2016; Song and Geyer, 2015; Ong et al., 2019), and their combinations (Mordatch and Todorov, 2014) have been explored where the controller design is often motivated by structural and functional hypotheses on nervous and motor control systems. On the other hand, computer animation researchers have focused on allowing physically-simulated characters to move lifelikely as humans by adding actuation constraints imposed by muscle dynamics. Wang et al. (2012) developed walking controllers for 3D biped characters equipped with 8 hill-type muscles per leg with the muscle-reflex model proposed by Geyer and Herr  (2010). Stochastic optimization is used to determine feedback control parameters such that biped characters can maintain their balance while walking. Geijtenbeek et al. (2013) applied a similar approach to non-human musculoskeletal characters, where manually-specified initial muscle routings are further optimized to improve the motor skills of the characters. Lee et al. (2014) demonstrated interactive controllers for full-body musculoskeletal characters having more than 100 muscles, where it first runs offline optimization to refine the reference motion and then the online controller based on quadratic programming tracks the refined reference motion at runtime.

Recently, deep reinforcement learning (DRL) has successfully demonstrated its capabilities in solving high-dimensional, continuous control problems including human motion imitation (Yu et al., 2019; Peng et al., 2018; Bergamin et al., 2019; Park et al., 2019; Won et al., 2020; Peng et al., 2021; Merel et al., 2019; Peng et al., 2022; Won et al., 2022), motion control in complex environments  (Clegg et al., 2018; Liu and Hodgins, 2018; Won et al., 2021; Yang et al., 2022; Ye et al., 2022; Winkler et al., 2022) and non-human character control (Yu et al., 2018; Luo et al., 2020; Lee et al., 2022; Ishiwaka et al., 2022). The control of musculoskeletal characters is no exception for these technological innovations; in particular, controllers based on DRL have been significantly improved in terms of robustness against external perturbation, computational efficiency at runtime, and the scope of reproducible motor skills. Many simulation algorithms for controlling biped musculoskeletal models competed in the Learn-to-Move challenges (Kidziński et al., 2020) and DRL-based algorithms performed well. The winner of the final challenge proposed a model-based DRL algorithm equipped with an ensemble of probabilistic dynamics models and a risk minimization scheme by expanding the lower confidence bound of the value estimation (Kidziński et al., 2018). Lee et al. (2019) developed a two-level control architecture composed of a trajectory mimicking network and a muscle coordination network, by which comprehensive musculoskeletal characters with more than 300 muscles successfully reproduced highly dynamic human movements such as running, jumping, and cartwheel. Yifeng et al. (2019) proposed a new action space for DRL that mimics the behaviors of musculoskeletal simulation computationally more efficiently with torque-based simulation. Park et al. (2022) presented a predictive gait simulation framework Generative GaitNet, which can predict a broad spectrum of healthy and pathological gait of comprehensive musculoskeletal models over a high-dimensional parameter space spanned by anatomical (e.g., bone/muscle parameters, mass distribution, muscle capacity) and gait (e.g., stride and cadence) conditions.

The musculoskeletal model in this work is designed to represent healthy and pathological gaits often discussed in clinical gait analysis and medical engineering. Our model consists of 23 bones connected by 22 skeletal joints and 304 musculotendons. The activation of muscles generates contraction forces that drive skeletal joints. The individual bones and muscles are conditioned by anatomical parameters. Let $C_{\mathrm{anatomy}}=(C_{\mathrm{skeleton}},C_{\mathrm{muscle}})$ be the anatomical condition, where $C_{\mathrm{skeleton}}=(c_{\mathrm{head}},c_{\mathrm{trunk}},c_{1},\cdots,c_{8},\tau_{1},\tau_{2})$ represents the scaling factors of the head, the trunk, the lengths of four (upper and lower) limbs with respect to a reference (average) model of healthy adults. $\tau_{1}$ and $\tau_{2}$ are the torsional angles of femurs, which correspond to femoral anteversion and retroversion often observed in pathological gait. Each musculotendon is conditioned by two parameters: weakness and contracture. The weakness parameter of a muscle indicates the ratio of maximum isometric force the muscle can exert relative to the reference model. The higher the parameter, the larger the force the muscle can produce. The contracture parameter indicates the scaling factor of the muscle length relative to the reference model, which refers to the permanent shortening of a muscle that often limits the range of joint movements.

Conventionally, a complete two-step cycle of gait begins with a left heel strike, where both feet are in contact with the ground at the same time, and ends at the next left heel strike. To avoid the singularity at both ends of the gait cycle, two gait cycles are considered as basic units of gait in our system. Therefore, the gait pattern is represented by a time series of full-body poses $M=(Q_{0},Q_{\delta},Q_{2\delta},\cdots)$, where each pose $Q=(\mathbf{h},\mathbf{v},\mathbf{q}_{0},\mathbf{q}_{1},\mathbf{q}_{2},\cdots)$ denotes the root (pelvis) height $h$ from the ground, the root linear velocity $\mathbf{v}$ parallel to the ground plane, and joint rotations $\mathbf{q}_{i}$ with respect to their parent joints. $\mathbf{q}_{0}$ denotes the global orientation of the root body node. We use the first two columns of the rotation matrix to describe the rotation. The sample rate $\delta$ is 60 per two gait cycles in our implementation. The gait is conditioned by two parameters: stride and cadence.

The predictive gait simulation generates a gait pattern $M$ that likely occurs given anatomical and gait conditions (see Figure 1). Conversely, gait analysis is its inverse process of predicting the anatomical and gait conditions of a given gait pattern. The human musculoskeletal model is a dynamical system that is partly under-actuated because the root node is not actuated and, at the same time, partly over-actuated because the body has more muscles than minimally required to drive the skeletal joints. This redundancy makes the whole process probabilistic.

The predictive power of gait simulation stems from making full use of the laws of physics, providing accurate anatomical modeling, and designing reliable control policies for seemingly unstable biped locomotion. Recent approaches successfully derived robust control policies through deep reinforcement learning (Lee et al., 2019; Kidziński et al., 2018). In this case, the core of a predictive gait simulation is a policy network, which takes the current state of the musculoskeletal model as input and outputs the desired level of activation at all muscles. The state-of-the-art method learned control policies conditioned by high-dimensional conditioning vectors (more than 600 parameters) (Park et al., 2022). The musculoskeletal model with specific physical conditions driven by a policy network generates a trace of full-body poses in physics-based simulation.

In this paper, the key challenge is to find the inverse process of predictive simulation. Given a gait pattern, the goal of gait analysis is to find corresponding physical conditions. There are three issues to be addressed. First, the search space is high-dimensional. In our system, conditioning vectors are 280 dimensional. State space search or optimization in such a high-dimensional space is often computationally intractable. Secondly, the solution is not unique. As discussed before, the backward mapping is probabilistic, and thus we need to find a probabilistic distribution over the conditioning space rather than a single optimal solution. Lastly, the computational cost should be reasonable. Rolling out a single gait pattern from a predictive simulator requires the computational cost of physics-based simulation over a complete gait cycle, which is substantial with a stochastic sampling of gait patterns.

We address these issues by pretraining forward and backward networks. We transfer control policies from policy networks to a forward network representing a direct anatomy-to-gait mapping. The transfer process is similar to policy distillation. It takes samples from the policy network and learns the target network using supervised regression. The pretrained forward network has two advantages. The forward network generates physically-valid gait patterns without actually performing physics-based simulations because it imitates the behavior of the policy network. This direct anatomy-to-gait mapping is computationally more efficient at runtime than predictive gait simulation. This computational efficiency also makes it computationally feasible to pretrain the backward network employing a conditional Variational AutoEncoder (c-VAE) that models the probabilistic mapping between anatomy and gait using Gaussian distributions in latent space.

Let $C_{\mathrm{anatomy}}=(C_{\mathrm{skeleton}},C_{\mathrm{muscle}})$ and $C_{\mathrm{gait}}$ be anatomical and gait conditions, respectively. The gait pattern $M=(Q_{0},Q_{\delta},Q_{2\delta},\cdots,Q_{59\delta})$ includes two gait cycles, which are parameterized by phase $\phi\in[0,4\pi]$. Our Bidirectional GaitNet learns the relationship between anatomy and gait by conditional forward mapping $\mathbf{\hat{F}}(M|C_{\mathrm{anatomy}},C_{\mathrm{gait}})$ and backward mapping $\mathbf{\hat{B}}(C_{\mathrm{anatomy}},C_{\mathrm{gait}}|M)$. This formulation has several issues in designing network models. The size of the forward network depends on the frame rate $\delta$ and can be larger than actually needed. Given a gait pattern, its stride, cadence, body proportions, and limb lengths are readily available in the motion capture process or can be directly computed from the gait data. To address the issues, we simplify the forward and backward mappings, respectively, by $\mathbf{F}(Q_{\phi}|C_{\mathrm{anatomy}},C_{\mathrm{gait}},\phi)$ and $\mathbf{B}(C_{\mathrm{muscle}}|M,C_{\mathrm{skeleton}},C_{\mathrm{gait}})$ where we added the phase in the input layer of the forward network and reduced the output layer such that it generates a full-body pose $Q_{\phi}$ at phase $\phi$ instead of a full gait pattern. This design decision significantly reduces the complexity of the prediction and allows us to achieve improved accuracy with smaller regression networks. The backward network also becomes smaller by removing the skeleton and gait conditions from the output layer.

To train Bidirectional GaitNet, we collected approximately 500 hours long walking motions from our predictive gait simulator. This data generation takes approximately 10 hours in the cluster machine equipped with 20 Intel Xeon 6242 CPUs. We use Torch (Paszke et al., 2019) to implement our models. The encoder, the pre-decoder, and Forward GaitNet are modeled by feed-forward networks with [256, 256, 256], [256, 256, 256], and [512, 512, 512] layers, respectively. For the activation units, LeakyReLu/Linear, LeakyReLu/Sigmoid, and ReLu/Linear are used for the hidden/output layers. Both models are learned by following the standard procedure of supervised learning with 65536 (forward), 2048 (backward) batch sizes, Adam optimizer with $1e-5$ learning rate. The training takes 2, 1.5 hours for Forward GaitNet, and Backward GaitNet, respectively, by using the desktop equipped with Ryzen 3950, NVIDIA 2070, and 64GB ram. The dimension of $M$, $z$, $(C_{\mathrm{gait}},C_{\mathrm{skeleton}})$, and $C_{\mathrm{muscle}}$ are 6060, 32, 13, and 268, respectively.

We developed a novel generative model, called Bidirectional GaitNet, that learns the relationship between human anatomy and its gait. It consists of forward and backward models where the forward model predicts a gait pattern of a person with specific physical conditions, while the backward model estimates the physical conditions of a person when his/her gait pattern is provided. By constructing a c-VAE structure with the forward model learned by simulation data generated from the state-of-the-art predictive gait simulator, we were able to learn its inverse mapping (i.e., the backward model) effectively. We showed that the anatomical conditions predicted by our model were able to be realized for both unseen simulated gaits and real patients’ gaits.

There are still several limitations in our method. First, although joint angle prediction error is pretty accurate (less than 8 degrees on average), it needs to be further improved for some joints such as talus (ankle) and femur (upper leg) which show larger errors when compared to other joints. Note that those are actually the joints that have much wider variations in our dataset than others. The sophisticated network architectures (e.g., Transformer) or loss functions that can better capture wider variations in data would be helpful in general. Second, our method can be easily extended so that it includes the upper body muscle conditions. However, the prediction of those muscle conditions might be less reliable when compared to ones in the lower body due to weaker dependency between gaits and their muscle conditions. Simply speaking, the upper body motions have much larger freedom and could be almost arbitrary. In addition, some of anatomical features such as the nervous system, flexible tendon model, and skin are missing in the musculoskeletal model we used in this research, which might affect the prediction of gait and physical condition differently.

We envision several promising future directions that we want to study further. The input of the current system should be given as mocap data, which might be cumbersome and expensive to obtain. It would be interesting if we can use a monocular video as input to our system, which could be done by using existing video-to-mocap solutions or other training schemes. Another interesting direction would be exploring more complex full-body or facial musculoskeletal models that include volumetric muscles, which will have a practical impact on animation industries because those models have often been used in commercial films.