Learning Linear Causal Representations from Interventions under General Nonlinear Mixing

Modern generative models such as GPT-4 [59] or DDPMs [26] achieve tremendous performance for a wide variety of tasks [7]. They do this by effectively learning high-level representations which map to raw data through complicated non-linear maps, such as transformers [89] or diffusion processes [79]. However, we are unable to reason about the specific representations they learn, in particular they are not necessarily related to the true underlying data generating process. Besides their susceptibility to bias [20], they often fail to generalize to out-of-distribution settings [5], rendering them problematic in safety-critical domains. In order to gain a deeper understanding of what representations deep generative models learn, one line of work has pursued the goal of causal representation learning (CRL) [75]. CRL aims to learn high-level representations of data while simultaneously recovering the rich causal structure embedded in them, allowing us to reason about them and use them for higher-level cognitive tasks such as systematic generalization and planning. This has been used effectively in many application domains including genomics [52] and robotics [43, 94].

A crucial primitive in representation learning is the fundamental notion of identifiability [37, 70], i.e., the question whether a unique (up to tolerable ambiguities) model can explain the observed data. It is well known that because of non-identifiability, CRL is impossible in general settings in the absence of inductive biases or additional information [29, 50]. In this work, we consider additional information in the form of interventional data [75]. It is common to have access to such data in many application domains such as genomics and robotics stated above (e.g. [17, 58, 57]). Moreover, there is a pressing need in such safety-critical domains to build reliable and trustworthy systems, making identifiable CRL particularly important. Therefore, it’s important to study whether we can and also how to perform identifiable CRL from raw observational and interventional data. Here, identifiability opens the possibility to provably recover the true representations with formal guarantees. Meaningfully learning such representations with causal structure enables better interpretability, allows us to reason about fairness, and helps with performing high-level tasks such as planning and reasoning.

In this work, we study precisely this problem of causal representation learning in the presence of interventions. While prior work has studied simpler settings of linear or polynomial mixing [84, 88, 2, 71, 11], we allow for general non-linear mixing, which means our identifiability results apply to complex real-world systems and datasets which are used in practice. With our results, we make progress on fundamental questions on interventional learning raised by [75].

Concretely, we consider a general model with latent variables $Z$ and observed data $X$ generated as $X=f(Z)$ where $f:\mathbb{R}^{d}\to\mathbb{R}^{d^{\prime}}$ is an arbitrary non-linear mixing function. We assume $Z$ satisfies a Gaussian structural equation model (SEM) which is unknown and unobserved. A Gaussian prior is commonly used in practice (which implies a linear SEM over $Z$) and further, having a simple model for $Z$ allows us to learn useful representations, generate data efficiently, and explore the latent causal relationships, while the non-linearity of $f$ ensures universal approximability of the model. We additionally assume access to interventional data $X^{(i)}=f(Z^{(i)})$ for $i\in I$ where $Z^{(i)}$ is the latent under an intervention on a single node $Z_{t_{i}}$. Notably, we allow for various kinds of interventions, we don’t require knowledge of the targets $t_{i}$, and we don’t require paired data, (i.e., we don’t need counterfactual samples from the joint distribution $(X,X^{(i)})$, but only their marginals). Having targets or counterfactual data is unrealistic in many practical settings but many prior identifiability results require them, so eliminating this dependence is a crucial step towards CRL in the real world.

Contributions    Our main contribution is a general solution to the identifiability problem that captures practical settings with non-linear mixing functions (e.g. deep neural networks) and unknown interventions (since the targets are latent). We study both perfect and imperfect interventions and additionally allow for shift interventions, where perfect interventions remove the dependence of the target variable from its parents, while imperfect interventions (also known as soft interventions) modify the dependencies. Below, we summarize our main contributions:

1. 1.

   We show identifiability of the full latent variable model, given non-paired interventional data with unknown intervention targets. In particular, we learn the mixing, the targets and the latent causal structure.

2. 2.

   Compared to prior works that have focused on linear/polynomial $f$, we allow non-linear $f$ which encompasses representations learned by e.g. deep neural networks and captures complex real-world datasets. Moreover, we study both imperfect (also called soft) and perfect interventions, and always allow shift interventions.

3. 3.

   We construct illustrative counterexamples to probe tightness of our assumptions which suggest directions for future work.

4. 4.

   We propose a novel algorithm based on contrastive learning to learn causal representations from interventional data, and we run experiments to validate our theory. Our experiments suggest that a contrastive approach, which so far has been unexplored in interventional CRL, is a promising technique going forward.

Causal representation learning [75, 74] has seen much recent progress and applications since it generalizes and connects the fields of Independent Component Analysis [13, 28, 30], causal inference [81, 61, 63, 64, 82] and latent variable modeling [40, 3, 78, 96, 32]. Fundamental to this approach is the notion of identifiability [36, 15, 93]. Due to non-identifiability in general settings without inductive biases [29, 50], prior works have approached this problem from various angles — using additional auxiliary labels [36, 27, 6, 76]; by imposing sparsity [75, 42, 54, 103]; or by restricting the function classes [41, 9, 104, 22]. See also the survey [32]. Moreover, a long line of works have proposed practical methods for CRL (which includes causal disentanglement as a special case), [19, 95, 16, 97, 44, 14] to name a few. It’s worth noting that most of these approaches are essentially variants of the Variational Autoencoder framework [39, 68].

Of particular relevance to this work is the setting when interventional data is available. We first remark that the much simpler case of fully observed variables (i.e., no latent variables), has been studied in e.g. [23, 62, 83, 34, 18] (see also the survey [82]). In this work, we consider the more difficult setting of structure learning over latent variables, which have been explored in [46, 42, 6, 104, 1, 84, 88, 2, 71, 11, 72]. Among these, [46] assumes that the intervention targets are known, [42] specifically consider instance-level pre- and post- interventions for time-series data and [6, 104, 1] assumes access to paired counterfactual data. In contrast, we assume unknown targets and work in general settings with non-paired interventional data, which is important in various real-world applications [85, 99]. The work [48] require several graphical restrictions on the causal graph and also require $2d$ interventions, while we make no graphical restrictions and only require $d$ interventions (which we also show cannot be improved under our assumptions). [71, 11, 84, 88] consider linear mixing functions $f$ whereas we study general non-linear $f$. Finally, [2] consider polynomial mixing in the more restricted class of deterministic do-interventions. Several concurrent works study related settings [100, 35, 65, 45, 92]. For a more detailed comparison to the related works, we refer to Appendix C.

Our proposed algorithm is based on contrastive learning. Contrastive learning has been used in other contexts in this domain [31, 27, 55, 104, 91] (either in the setting of time-series data or paired counterfactual data), however the application to non-paired interventional settings is new to the best of our knowledge.

Notation   In this work, we will almost always work with vectors and matrices in $d$ dimensions where $d$ is the latent dimension, and we will disambiguate when necessary. For a vector $v$, we denote by $v_{i}$ its $i$-th entry. Let $\mathrm{Id}$ denote the $d\times d$ identity matrix with columns as the standard basis vectors $e_{1},\ldots,e_{d}$. We denote by $N(\mu,\Sigma)$ the multivariate normal distribution with mean $\mu$ and covariance $\Sigma$. For a set $C$, let $\mathcal{U}(C)$ denote the uniform distribution on $C$. For two random variables $X,X^{\prime}$, we write $X\overset{\mathcal{D}}{=}X^{\prime}$ if their distributions are the same. We denote the set $\{1,\ldots,d\}$ by $[d]$. For permutation matrices we use the convention that $(P_{\omega})_{ij}=\bm{1}_{j=\omega(i)}$ for $\omega\in S_{d}$. We use standard directed graph terminology, e.g. edges, parents. When we use the term “non-linear mixing” in this work, we also include linear mixing as a special case.

We now formally introduce our main settings of interest and the required assumptions. We assume that there is a latent variable distribution $Z$ on $\mathbb{R}^{d}$ and an observational distribution $X$ on $\mathbb{R}^{d^{\prime}}$ given by $X=f(Z)$ where $d\leq d^{\prime}$ and $f$ is a non-linear mixing. This encapsulates the most general definition of a latent variable model. As the terminology suggests, we observe $X$ in real-life, e.g. images of cats and $Z$ encodes high-level latent information we wish to learn and model, e.g. $Z$ could indicate orientation and size.

Such an assumption is standard in the literature. Injectivity is needed in order to identify $Z$ from $X$ because otherwise we may have learning ambiguity if multiple $Z$s map to the same $X$. Differentiability is needed to transfer densities. Note that 1 allows for a large class of complicated nonlinearities and we discuss in Appendix E why proving results for such a large class of mixing functions is important.

Next, we assume that the latent variables $Z$ encode causal structure which can be expressed through a Structural Causal Model (SCM) on a Directed Acyclic Graph (DAG). We focus on linear SCMs with an underlying causal graph $G$ on vertex set $[d]$, encoded by a matrix $A$ through its non-zero entries, i.e., there is an edge $i\to j$ in $G$ iff $A_{ji}\neq 0$.

For an illustration of the model, see Fig. 1(a). It is convenient to encode the coefficients of linear SCMs through the matrix $B=D^{-1/2}(\mathrm{Id}-A)$. Then $Z=B^{-1}\epsilon$ and both $D$ and $A$ can be recovered from $B$, indeed the diagonal entries of $B$ agree with the entries of $D^{-1/2}$. Note that we can and will always assume that the entries of $D$ are positive since $\epsilon\overset{\mathcal{D}}{=}-\epsilon$.

Assuming that the latent variables follow a linear Gaussian SCM is certainly restrictive but nevertheless a reasonable approximation of the underlying data generating process in many settings. The Gaussian prior assumption is also standard in latent variable modeling and enables efficient inference for downstream tasks, among other advantages. Importantly, under the above assumptions, our model has infinite modeling capacity [53, 86, 33], so they’re able to model complex datasets such as images and there’s no loss in representational power.

The goal of representation learning is to learn the non-linear mixing $f$ and the high-level latent distribution $Z$ from observational data $X$. In causal representation learning, we wish to go a step further and model $Z$ as well by learning the parameters $B$ which then lets us easily recover $A,D$ and the causal graph $G$. For general non-linear mixing, this model is not identifiable and hence we cannot hope to recover $Z$, but in this work we use additional interventional data, similar to the setups in recent works [2, 84, 88].

For an illustration, see Fig. 1(b). An intervention on node $t_{i}$ has no effect on the non-descendants of $t_{i}$, but will have a downstream effect on the descendants of $t_{i}$. In particular, $A$ is modified so that the weight of the edge $k\to t_{i}$ could potentially be changed if it exists already but no new incoming edges to $t_{i}$ may be added. Also, the noise variable $\epsilon_{t_{i}}$ is also allowed to be modified via a scale intervention as well as a shift intervention. Note that [84, 2] do not allow shift interventions, whereas we do.

Again, this can be written concisely as $Z^{(i)}=(B^{(i)})^{-1}(\epsilon+\eta^{(i)}e_{t_{i}})$ where $B^{(i)}=(D^{(i)})^{-1/2}(\mathrm{Id}-A^{(i)})$. Let $r^{({i})}$ denote the row $t_{i}$ of $B^{(i)}$, then we can write $B^{(i)}=B-e_{t_{i}}(B^{\top}e_{t_{i}}-r^{({i})})^{\top}$. We assume that interventions are non-trivial, i.e., $Z^{(i)}\overset{\mathcal{D}}{\neq}Z$. In our model, we observe interventional distributions $X^{(i)}=f(Z^{(i)})$ for various interventions $i\in I$.

We call a single-node intervention imperfect if it is not perfect (some prior works call it a soft intervention). Note that perfect interventions are never of pure noise type because they necessarily change the relation to the parents. For an illustration of perfect and imperfect interventions, see Fig. 1(c), Fig. 1(b) respectively. Finally, we require an exhaustive set of interventions.

This assumption was also made in the prior works [84, 2, 88]. When the interventions don’t cover all the nodes, we have non-identifiability as described in Section 3 and Appendix D. We also extensively discuss that none of our other assumptions can simply be dropped in these sections.

To simplify the notation, it is convenient to use $B^{(0)}=B$, $\eta^{(0)}=0$ for the observational distribution. We also define $\bar{I}=I\cup\{0\}$. Then, all information about the latent variable distributions $Z^{(i)}$ and observed distributions $X^{(i)}$ are contained in $((B^{(i)},\eta^{(i)},t_{i})_{i\in\bar{I}},f)$.

We can now state our main results for the setting introduced above.

This result says that for the interventional model as described in the previous section, we can identify the non-linear map $f$, the intervention targets $t_{i}$, the parameter matrices $B,D,A$ up to permutations $P_{\omega}$ and diagonal scaling $\Lambda$. Moreover, we can recover the shifts $\eta^{(i)}$ exactly and also the underlying causal graph $G$ up to permutations.

When we drop the assumption that the interventions are perfect, we can still obtain a weaker identifiability result. Define $\prec_{G}$ to be the minimal partial order on $[d]$ such that $i\prec_{G}j$ if $(i,j)$ is an edge in $G$, i.e., $i\prec_{G}j$ if and only if $i$ is an ancestor of $j$ in $G$. Note that any topological ordering of $G$ is compatible with the partial order $\prec_{G}$. Then, our next result shows that under imperfect interventions, we can still recover the partial order $\prec_{G}$.

Theorem 1 and Theorem 2 generalize the main results of [84] which assume linear $f$ while we allow for non-linear $f$. The key new ingredient of our work is the following theorem, which shows identifiability of $f$ up to linear maps.

The proof of this theorem is deferred to Appendix A. In Appendix B we then review the results of [84] and show how their results can be extended to obtain our main results, Theorem 1 and Theorem 2. Now we provide some intuition for the proofs of the main theorems.

Proof intuition   The recent work [84] studies the special case when $f$ is linear, and the proof is linear algebraic. In particular, they consider row spans of the precision matrices of $X^{(i)}$, project them to certain linear subspaces and use those subspaces to construct a generalized RQ decomposition of the pseudoinverse of the linear mixing matrix $f$. However, once we are in the setting of non-linear $f$, such an approach is not feasible because we can no longer reason about row spans of the precision matrices of $X$, which have been transformed non-linearly thereby losing all linear algebraic structure. Instead, we take a statistical approach and look at the log densities of the $X^{(i)}$. By choosing a Gaussian prior, the log-odds $\ln p^{(i)}_{X}(x)-\ln p^{(0)}_{X}(x)$ of $X^{(i)}$ with respect to $X^{(0)}$ can be written as a quadratic form of difference of precision matrices, evaluated at non-linear functions of $x$. For simplicity of this exposition, ignore terms arising from shift interventions and determinants of covariance matrices. Then, the log-odds looks like $\theta(x)=f^{-1}(x)^{\top}(\Theta^{(i)}-\Theta^{(0)})f^{-1}(x)$, where $\Theta^{(i)}$ is the precision matrix of $Z^{(i)}$.

At this stage, we again shift our viewpoint to geometric and observe that $\Theta^{(i)}-\Theta^{(0)}$ has a certain structure. In particular, for single-node interventions, it has rank at most $2$ and for source node targets, it has rank $1$. This implies that the level set manifolds of the quadratic forms $\theta(x)$ also have a certain geometric structure in them, i.e., the DAG leaves a geometric signature on the data likelihood. We exploit this carefully and proceed by induction on the topological ordering until we end up showing that $f$ can be identified up to a linear transformation, which is our main Theorem 3. Here, the presence of shift interventions adds additional complexities, and we have to generalize all of our intermediate lemmas to handle these. Once we identify $f$ up to a linear transformation, we can apply the results of [84] to conclude Theorems 1 and 2.

On the optimality and limitations of the assumptions    We make a few brief remarks on our assumptions, deferring to Appendix D a full discussion and the technical construction of several illustrative counterexamples.

1. 1.

   Number of interventions: For our main results, Theorems 1 and 2, we assume that there are at least $d$ interventions (Assumption 4). This cannot be weakened even for linear mixing functions [84]. In addition, we also show in 1 that for the linear identifiability proved in Theorem 3, $d-2$ interventions are not sufficient. Thus, the required number of interventions is tight in Theorems 1, 2 and is tight up to at most one intervention in Theorem 3.

2. 2.

   Intervention type: In the setting of imperfect interventions, the weaker identifiability guarantees in Theorem 2 cannot be improved even when the mixing is linear [84, Appendix C]. We also show in 2 that if we drop the condition that interventions are not pure noise interventions, we have non-identifiability. Concretely, we show that when all interventions are of pure noise type, any causal graph is compatible with the observations. Finally, in contrast to the special case of linear mixing, we show in Lemma 8 that we need to exclude non-stochastic hard interventions (i.e., $\mathrm{do}(Z_{i}=z_{i})$) for identifiability up to linear maps.

3. 3.

   Distributional assumptions: We assume Gaussian latent variables, while allowing for very flexible mixing $f$. This model has universal approximation guarantees and is moreover used ubiquitously in practice. While the result can potentially be extended to more general latent distributions (e.g., exponential families), we additionally show in Lemma 9 that the result is not true when making no assumption on the distribution of $\epsilon$.

In this section, we explain our experimental methodology and the theoretical underpinning of our approach. Our main experiments for interventional causal representation learning focus on a method based on contrastive learning. We train a deep neural network to learn to distinguish observational samples $x\sim X^{(0)}$ from interventional samples $x\sim X^{(i)}$. Additionally, we design the last layer of the model to model the log-likelihood of a linear Gaussian SCM. Due to representational flexibility of deep neural networks, we will in principle learn the Bayes optimal classifier after optimal training, which we show is related to the underlying causal model parameters. Accordingly, with careful design of the last layer parametric form, we indirectly learn the parameters of the underlying causal model. Similar methods have been used for time-series data or multimodal data [27, 32] but to the best of our knowledge, the contrastive learning approach to interventional learning is novel.

Denote the probability density of $x\sim X^{(i)}$ (resp. $z\sim Z^{(i)}$) by $p_{X}^{(i)}(x)$ (resp. $p_{Z}^{(i)}(z)$). The next lemma describes the log-odds of a sample $x$ coming from the interventional distribution $X^{(i)}$ as opposed to the observational distribution $X^{(0)}$. We focus on the identifiable case (see Theorem 1) and therefore only consider perfect interventions. As per the notation in Section 3 and Section A.1, let $s^{(i)}$ denote the row $t_{i}$ of $B^{(0)}$ and let $\eta^{(i)},\lambda_{i}$ denote the magnitude of the shift and scale intervention respectively.

The proof is deferred to Appendix F. The form of the log-odds suggests considering the following functions

where $h(\cdot,\theta)$ denotes a neural net parametrized by $\theta$, parameters $w^{(i)}$ are the rows of a matrix $W$ and $\alpha_{i}$, $\beta_{i}$, $\gamma_{i}$ are learnable parameters. Note that the ground truth parameters minimize the following cross entropy loss

Note that (compare (6) and (7)) $W$ should learn $B=D^{-1/2}(\mathrm{Id}-A)$ thus its off-diagonal entries should form a DAG. To enforce this we add the NOTEARS regularizer [102] given by $\mathcal{R}_{NOTEARS}(W)=\text{tr}\exp(W_{0}\circ W_{0})-d$ (see Section F.3) where $W_{0}$ equals $W$ with the main diagonal zeroed out. We also promote sparsity by adding the $l_{1}$ regularization term $\mathcal{R}_{REG}(W)=\lVert W_{0}\rVert_{1}$. Thus, the total loss is given by

for hyperparameters $\tau_{1}$ and $\tau_{2}$. Our identifiability result Theorem 1 implies that when we assume that the neural network has infinite capacity, the loss in (9) is minimized, $\tau_{1}$ is large and $\tau_{2}$ small, and we learn Gaussian latent variables $h(X^{(i)},\theta)$, then we recover the ground truth latent variables up to the tolerable ambiguities of labeling and scale, i.e., $h$ recovers $f^{-1}$ and $W$ recovers $B$ up to permutation and scale. Thus, we estimate the latent variables using $\hat{Z}=h(X,\theta)$ and estimate the DAG using $W_{0}$. Full details of the practical implementation of our approach are given in Appendix H.

Our experimental setup is similar to [84, 2], we consider $d$ interventions with different targets (our theory holds in full generality) and therefore, we can arbitrarily assign $t_{i}=i$ based on the intervention index $i$ which removes the permutation ambiguity. We focus on non-zero shifts because the cross entropy loss together with the quadratic expression for the log-odds results in a non-convex output layer which makes it hard to find the global loss minimizer, as we will describe in more detail in Section G.1. We emphasize that this is no contradiction to the theoretical results stated above. Even when there is no shift intervention the latent variables are identifiable, but our algorithm often fails to find the global minimizer of the loss (9) due to the non-convex loss landscape. For the sake of exposition and to set the stage for future work, we also briefly describe an approach via Variational Autoencoders (VAE). VAEs have been widely used in causal representation learning and while feasible in interventional settings, they are accompanied by certain difficulties, which we detail and suggest how to overcome in Section F.4.

In this section, we implement our approach on synthetic data and image data. Complete details (architectures, hyperparameters) are deferred to Appendix H. Additional experiments investigating the effect of varsortability [66] and the noise distribution can be found in Appendix G.

Data generation   For all our experiments we use Erdös-Rényi graphs, i.e., we add each undirected edge with equal probability $p$ to the graph and then finally orient them according to some random order of the nodes. We write $\mathrm{ER}(d,k)$ for the Erdös-Rényi graph distribution on $d$ nodes with $kd$ expected edges. For a given graph $G$ we then sample edge weights from $\mathcal{U}(\pm[0.25,1.0])$ and a scale matrix $D$. For simplicity we assume that we have $n$ samples from each environment $i\in\bar{I}$. We only consider the setting where each node is intervened upon once and thus the latent dimension is also known. We consider three types of mixing functions. First, we consider linear mixing functions where we sample all matrix entries i.i.d. from a Gaussian distribution. Then, we consider non-linear mixing functions that are parametrized by MLPs with three hidden layers which are randomly initialized, and have Leaky ReLU activations. Finally, we consider image data as described in [2]. Pairs of latent variables $(z_{2i+1},z_{2i+2})$ describe the coordinates of balls in an image and the non-linearity $f$ is the rendering of the image. The image generation is based on pygame [77]. A sample image can be found in Figure 3.

Evaluation Metrics   We evaluate how well we can recover the ground truth latent variables and the underlying DAG. For the recovery of the ground truth latents we use the Mean Correlation Coefficient (MCC) [37, Appendix A.2]. For the evaluation of the learned graph structure we use the Structural Hamming Distance (SHD) where we follow the convention that we count the number of edge differences of the directed graphs (i.e., an edge with wrong orientation counts as two errors). Since the scale of the variables is not fixed the selection of the edge selection threshold is slightly delicate. Thus, we use the heuristic where we match the number of selected edges to the expected number of edges. As a metric that is independent of this thresholding procedure, we also report the Area Under the Receiver Operating Curve (AUROC) for the edge selection.

Methods    We implement our contrastive algorithm as explained in Section F where we use an MLP (with Leaky ReLU nonlinearities) for the function $h$ for the linear and nonlinear mixing functions and a very small convolutional network for the image dataset, which are termed “Contrastive”. For linear mixing function we also consider a version of the contrastive algorithm where $h$ is a linear function, termed “Contrastive Linear”. As baselines, we consider a variational autoencoder with latent dimension $d$ and also the algorithm for linear disentanglement introduced in [84]. Since the variational autoencoder does not output a causal graph structure we report the result for the empty graph here which serves as a baseline.

Results for linear $f$    First, for the sake of comparison, we replicate exactly the setting from [84], i.e., we consider initial noise variances sampled uniformly from $[2,4]$ and we consider perfect interventions where the new variance is sampled uniformly from $[6,8]$. We set $d=5$, $d^{\prime}=10$, $k=3/2$, and $n=50000$. Results can be found in Table 1. The linear contrastive method identifies the ground truth latent variables up to scaling. The failure of the nonlinear method to recover the ground truth latents will be explained and further analyzed in Appendix G.1. Note that the nonlinear contrastive and the linear contrastive method recover the underlying graph better than the baseline.

Results for nonlinear $f$    We sample all variances from $\mathcal{U}([1,2])$ (initial variance and resampling for the perfect interventions) and the shift parameters $\eta^{(i)}$ of the interventions from $\mathcal{U}([1,2])$. Results can be found in Table 2. We find that our contrastive method can recover the ground truth latents and the causal structure almost perfectly, while the baseline for linear disentanglement cannot recover the graph or the latent variables (which is not surprising as the mixing is highly nonlinear). Also, training a vanilla VAE does not recover the latent variables up to a linear map as indicated by the $R^{2}$ scores. In Figure 2 we illustrate the dependence on the dimension $d$ of our algorithm in this setting.

Results for image data   Finally, we report the results for image data. Here, we generate the graph as before and consider variances sampled from $\sigma^{2}\sim\mathcal{U}([0.01,0.02])$ and shifts $\eta^{(i)}$ from $\mathcal{U}([0.1,0.2])$ (i.e., the shifts are still of order $\sigma$). We exclude samples where one of the balls is not contained in the image which generates a slight model misspecification compared to our theory. Again we find that we recover the latent graph and the latent variables as detailed in Table 3.

In this work, we extend several prior works and show identifiability for a widely used class of linear latent variable models with non-linear mixing, from interventional data with unknown targets. Counterexamples show that our assumptions are tight in this setting and could only potentially be relaxed under other additional assumptions. We leave to future work to extend our results for other classes of priors, such as non-parametric distribution families, and also to study sample complexity and robustness of our results. We also proposed a contrastive approach to learn such models in practice and showed that it can recover the latent structure in various settings. Finally, we highlight that the results of our experiments are very promising and it would be interesting to scale up our algorithms to large-scale datasets such as [49].

We thank anonymous reviewers for useful comments and suggestions. We acknowledge the support of AFRL and DARPA via FA8750-23-2-1015, ONR via N00014-23-1-2368, and NSF via IIS-1909816, IIS-1955532. We also acknowledge the support of JPMorgan Chase & Co. AI Research. This work was also supported by the Tübingen AI Center.