Beyond Convergence: Identifiability of Machine Learning and Deep Learning Models

Machine learning (ML) and deep learning models have demonstrated remarkable capacity to train and optimize model parameters based on data. The quality of adaptation in these models is commonly assessed using standard convergence measures such as mean squared error (MSE), mean absolute error (MAE), accuracy, etc. However, even with abundant training data and successful convergence, not all inverse problems are “identifiable,” in the sense that the model parameters may be uniquely determined from the available data.

Identifiability, a fundamental concept in system and control theory [1] and stochastic inference [2], rigorously examines the possibility of uniquely determining model parameters based on the (mathematical) description of the input-output relationship. Certain models may remain unidentifiable regardless of data volume, quality, and model accuracy due to the data-parameter relationship. Unidentifiable problems appear at the heart of many data modeling and stochastic inference problems, including retrieving time-series data from noisy measurements, estimating likelihoods or log-likelihoods in Bayesian inference problems, or training the weights of shallow and deep neural networks [3, 4]. Although unidentifiability is an intrinsic property of the data model that relates the measurements and parameters, it may go unnoticed, particularly in complex ML and deep learning models with thousands or millions of hidden parameters. This often occurs when models exhibit successful convergence during training or when the framework is fully data-driven without explicit data models. Despite the apparent training convergence, unidentifiable models can pose potential misinterpretation, especially when the underlying data models are not analytically available or easy to study.

In this work, we study the notion of identifiability through a case study focused on human gait and motion analysis. We employ the well-known bipedal-spring mass human walk dynamics model developed by Geyer et al [5]. Our objective is to identify subject-wise parameters, such as weight, height, and leg stiffness, by analyzing a person’s walking data, with potential applications in identifying pathological cases. We will use Geyer’s model as the base model for generating synthetic human motion data. The synthetic data will then be used to train deep learning models to estimate subject-wise parameters. Our case study will highlight the intrinsic and recurring challenge of identifiability in machine learning, an aspect that has received limited attention in the literature.

While we do not provide a definitive solution for unidentifiable ML problems, we shed light on this relatively unexplored aspect, which is crucial for ensuring the generalizability and explainability of state-of-the-art machine and deep learning models. By underscoring this recurrent issue, we emphasize the necessity of considering identifiability during model development and analysis, ultimately leading to more informed models across diverse domains. Furthermore, we demonstrate the potential of model-based machine learning to identify and potentially diagnose unidentifiable models.

The paper is organized as follows: In Section II, we present Geyer’s human gait model, which we will use for generating synthetic data and explore the identifiable and unidentifiable parameters of this model. In Section III we present the deep learning architecture, followed by the training, validation and test results in Section IV, to estimate subject-wise motion parameters. The paper concludes with a discussion on potential ideas for studying model identifiability in general machine learning and deep learning models, offering insights into the identifiability challenges in the context of model-based machine learning and deep learning.

The model proposed by Geyer et al. provides a simplified physics-based representation of human walking and running dynamics [5]. The model describes the fundamental dynamics of human walking and running through the concept of compliant leg behavior. It incorporates the interaction between the center of mass and the compliant legs during different phases of human gait, including single support, double support, and the transition between steps, as shown in Fig. 1. This compliant leg behavior, characterized by the mechanical properties of the legs to absorb and release energy during the stance phase of gait, plays a crucial role in accurately capturing the observed dynamics of human locomotion.

According to Fig. 1, the center of mass dynamics of the bipedal-spring mass human walking model can be described as follows for a single step from apex $i$ to $i+1$: (i) left leg single support: $m\ddot{x}=Px$, $m\ddot{y}=Py-mg$; (ii) intermittent double support: $m\ddot{x}=Px-Q(d-x)$, $m\ddot{y}=Py+Qy-mg$; (iii) right leg single support: $m\ddot{x}=-Q(d-x)$, $m\ddot{y}=Qy-mg$, where $m$ is the total mass of the human body; $k$ is the stiffness parameter representing the compliant behavior of the legs; $\ell_{0}$ is the equilibrium leg length, denoting the length of the legs at rest; g is the gravitational acceleration; $x$ and $y$ denote the horizontal and vertical positions of the center of mass, respectively. Additionally, the model employs two stiffness parameters to represent the compliant behavior of the legs during different phases of gait: $P=k(\ell_{0}/\sqrt{x^{2}+y^{2}}-1)$ for single support phases and $Q=k(\ell_{0}\sqrt{(d-x)^{2}+y^{2}}-1)$ for double support phases, where $d=\text{FP}_{i+1,x}-\text{FP}_{i,x}$ represents the horizontal distance covered in each step. The foot point (FP) refers to the point on the ground where the leg stance springs attach during gait. These equations collectively define the dynamics of the bipedal-spring mass human walk model [5].

In the sequel, we will use Geyer’s model as a forward model to generate synthetic human walk data with different subject-wise parameters $(m,k,\ell_{0})$. The research question is whether we can fit this model onto real data and train a neural network to estimate the model parameters $m$, $k$, and $\ell_{0}$ from real measurements. The application of such a model includes extracting physics-based features from motion sensors attached to patients, allowing for the study and classification of their walking patterns for normal and pathological cases such as tremulous patients.

We are interested in designing a deep learning model to estimate subject-wise gait parameters $m$, $k$ and $\ell_{0}$, from motion sensor data collected during human walking.

To assess the feasibility of this goal, we utilize the Geyer’s model detailed in Section II, to generate synthetic data and will use the data to train a neural network for subject-wise parameter estimation. This synthetic data generation process enables us to build a robust dataset representing a wide range of gait patterns and conditions. The trained model can eventually used to identify subject-wise parameters from real data.

To create diverse synthetic data, we used Geyer’s model to generate 1,000,000 records, each corresponding to 15 s of data with a time step of 0.1 s. The gravitational acceleration was fixed to $\textsl{g}=10$ m/s². The subject-wise parameters were selected from random distributions: $m\sim U[60,75]$ kg, stiffness $k\sim U[8000,10000]$ N/m, equilibrium leg length $\ell_{0}\sim U[0.6,0.8]$ m, initial horizontal position $x_{0}\sim U[0,0.1]$ m, and initial vertical position $y_{0}\sim U[0,0.1]$ m, where $U[a,b]$ denotes uniform distribution between $a$ and $b$.

The neural network architecture designed for parameter estimation consists of multiple layers, including an input layer, 7 fully connected layers with 50 nodes, and an output layer as shown in Fig. 2. The number of model training epochs was set to 500, with a training batch size of 1000. The 1-million record synthetic dataset was split into 80% for training, 10% for validation, and 10% for testing, ensuring a comprehensive evaluation of the deep learning model’s performance.

Two regression scenarios were studied to investigate the notion of identifiability:

1. 1.

   Using $(x,y)$ as inputs to estimate all three parameters: $(m,k,\ell_{0})$ in the output (Fig. 3(a)).

2. 2.

   Using $(x,y)$ as inputs to estimate $(\ell_{0},\rho)$ in the output (Fig. 3(b)).

From Section II, we know that the first scenario is unidentifiable; but the second scenario may potentially be identifiable (to be verified by model training on the synthetic data).

The training and validation results of the two scenarios described in Section III for estimating the parameters $(m,k,\ell_{0})$ and $(\rho,\ell_{0})$ are presented below.

In this study, we explored the notion of identifiability in the context of parameter estimation from motion sensor data using a deep learning model, based on the well-known bipedal-spring mass human walk dynamics model proposed by Geyer et al. [5]. The results demonstrated that while certain parameters were identifiable from the observation data, others were unidentifiable, indicating that they cannot be uniquely determined solely from the available data. Importantly, the unidentifiability of all or certain parameters does not indicate any flaw in the data model or learning algorithm; rather, it is an intrinsic limitation of the experimental setup, requiring a reevaluation of the data collection and experimental scenario.

The concept of unidentifiability is not unique to our study; various domains encounter similar challenges. For instance, subject localization from two-dimensional images, estimating stochastic parameter priors from aggregated parameter sets, and inferring deep hidden layer parameters from a limited number of observations are all examples of potentially unidentifiable problems. Addressing these issues necessitates revisiting data models and machine/deep learning model architectures, and the development of new data-driven approaches, including algorithms for data fusion and parameter estimation.

To address the challenges posed by unidentifiability in machine learning and deep learning models, several key areas warrant further attention and future studies:

In conclusion, our study provides insights into identifiability in the context of parameter estimation from motion sensor data using deep learning models. We showed that certain parameters can be uniquely determined from the observation data, while others remain unidentifiable — an intrinsic limitation of the experimental setup. Identifiability extends beyond this case study, impacting diverse domains in machine learning applications. Addressing unidentifiability requires novel data-driven approaches, data source integration, and interdisciplinary collaboration. Future research should prioritize multimodal data integration, model-based machine learning, and the development of identifiability metrics. Collaborative efforts across disciplines will also enable a comprehensive investigation of the notion of identifiability, leading to more informed models and applications.

I would like to thank Ms. Taniel Winner for bringing to my attention the compliant leg model by Geyer et al. (2006) in the BMI-532 course on “Model-based Machine Learning” and the fruitful conversations with the students, which resulted in a lecture on identifiability of physics-informed data models and the case-study investigated in this research.