Role of electron-phonon coupling in excitonic insulator candidate Ta₂NiSe₅

Correlated electron systems often contain multiple instabilities and intertwined orders, where numerous degrees of freedom jointly determine the material properties [1]. A notable example is the iron-based superconductors, where the electronic, orbital and spin channels all contribute to the rotational symmetry-breaking ground state [2]. In these systems, order parameters of the same symmetry often couple to each other, and the common reductionistic approach of ascribing the transition to a single dominant channel becomes ambivalent [3]. Another more recent example is the renewed search for bulk excitonic insulators, where electron-hole pairs spontaneously form in the ground state and cause charge insulation [4, 5]. In indirect band gap semiconductors such as 1T-TiSe₂ [6], both the plasmon and phonons soften to zero energy below the phase transition temperature, indicating contributions from both the lattice and electronic degrees of freedom. Indeed, it has been extensively discussed that the presence of electron-phonon coupling, regardless of whether there is pre-existing electron-hole Coulomb attraction, can also lead to a structural distortion with an ordering wave vector matching what connects the conduction band bottom and the valence band top [7, 8]. Such a “chicken-or-egg” dilemma originates from the pursuit of a reductive description, which is not always possible in correlated materials where relevant interaction energy scales are not well-separated. The lack of a clear energy hierarchy severely limits the construction and application of reductive frameworks. One common manifestation of this issue is the often ambivalent interpretations of observations made via different experimental techniques on the same system. A promising approach to avoid this dilemma is to include all relevant interactions on equal footing in the theoretical model, determine the interaction parameters by comparing the simulation with experimental observables through various tuning methods, and subsequently make further predictions for model validation and for material design guidance [9].

In this work, we will adopt such a framework to address the ongoing search for bulk excitonic insulators in low-dimensional materials. Recent studies revealed evidence of exciton formation and strong electron-phonon coupling (EPC) in the quasi-one-dimensional (quasi-1D) ternary chalcogenide Ta₂NiSe₅ [10, 11, 12, 13, 14, 15]. Upon warming, a $q=0$ monoclinic-to-orthorhombic structural transition happens at $T_{s}\sim$ 329 K [see Fig. 1(a)], above which a semi-metallic electronic structure is supposedly enforced by the mirror symmetry of the crystal and opposite parities of the low-energy orbitals near the Fermi momentum [16]. However, an insulating behavior is observed to persist up to 550K [10]. This is in striking contrast to archetypal metal-to-insulator transition (MIT) systems, such as the perovskite nickelates [17] and the chain compound TTF-TCNQ [18], where the higher-symmetry structure occurs concurrently with the metallic electronic state (usually called the normal state). The nature of the high-temperature electronic state in Ta₂NiSe₅ remains controversial [19, 20, 21, 22, 23, 24, 25, 26], including suggestions of either a regular gapless semimetal or a pseudogapped state with “preformed excitons” [25, 20, 26]. In addition, contentions remain regarding whether the system’s insulating ground state hosts an exciton condensate [11, 12, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28].

Combining high-energy x-ray diffraction (XRD) and high-resolution angle-resolved photoemission spectroscopy (ARPES) techniques, we first acquire high-resolution experimental results that will be used to test minimal many-body models. The experimental observations mainly focus on normal state properties, where the symmetry is not yet broken. These include an electronic pseudogap state over a wide temperature range above $T_{s}$ in Ta₂NiSe₅, which occurs concomitantly with a strong unidirectional fluctuation of the lattice; a similarly pseudogapped electronic structure during the insulator-to-semimetal transition upon electron injection below $T_{s}$; and concomitant negative electronic compressibility (NEC), which suggests the thermodynamic inevitability of EPC’s contribution. We then combine experimental results with both first-principles and many-body simulations to estimate the microscopic interaction parameters, including the kinetic energy of the conduction and valence electrons, the strength of the Coulomb interaction, and the EPC vertex. Considering both interactions on an equal footing, we discuss the roles of Coulomb interaction and EPC by comparing the experimental data to the numerically simulated spectra with both, either, and none of the interaction terms. Informed by such a comparison, we attribute the pseudogap state and NEC in Ta₂NiSe₅ primarily to the interband electron-hole excitation facilitated by a fluctuating lattice. On the contrary, direct Coulomb attraction between electrons and holes alone is insufficient to account for the insulating gap of the system.

The organization of this paper is as follows. We report the lattice fluctuations evidenced from XRD and the pseudogap state measured by ARPES in the high-temperature phase in Secs. II and III respectively. Next, we discuss the non-thermal tuning in the low-temperature phase, the emergence of NEC, and bound the Coulomb interaction strength in Sec. IV and Sec. V. We then construct a minimal lattice model, estimate the EPC vertex strength, and explain the experimental spectra using the first-principles and many-body simulations in Sec. VI. The combined impacts of Coulomb interactions and electron-phonon couplings are discussed in Sec. VII. Finally, we conclude by discussing the implication of this framework to other quantum materials with intertwined orders, and make a few predictions for the Ta₂NiSe₅ system in Sec. VIII.

High-quality Ta₂NiSe₅ single crystals were grown via the chemical vapor transfer method with iodine (I₂) as the transport agent [10, 12, 24]. Starting materials, composed of Ta powder (99.99%), Ni powder (99.99%), Se powder (99.99%) with a nominal molar ratio 2:1:5, were fully ground and mixed inside the glovebox. An additional 50 mg of iodine was then added to the mixed powder before it was vacuumed, backfilled with 1/3 Argon, and sealed inside a quartz tube (inner diameter 8 mm, outer diameter 12 mm, and a length of 120 mm). The sealed quartz tube was placed horizontally inside a muffle furnace during the growth. The hot end reaction temperature was set to 950 ^∘C and the cold end was left in air with the temperature stabilized at 850 ^∘C. Long and thin single crystals were harvested by quenching the furnace in air after one week of reaction. Residue iodine on the surface of the crystals was removed with ethanol.

Ta₂NiSe₅ crystallizes in a layered structure stacked via van der Waals interactions [Fig. 1(a)]. Within each layer, the Ta and Ni atoms form a chain structure along the a-axis of the crystal. Our XRD data reveals that the system undergoes a structural phase transition from a high-temperature orthorhombic Cmcm phase to a low-temperature monoclinic C2/c phase at $T_{s}$, in line with the previous report [10]. During the transition, the Ta atoms slightly shear along the chain direction, resulting in an increase of the $\beta$ angle from $90^{\circ}$ to $90.53^{\circ}$ [exaggerated in Fig. 1(a)]. The second-order nature of this transition is indicated by the continuous separation of the $\mathrm{\pm}$ 1 3 4 and $\mathrm{\pm}$ 1 3 7 nuclear Bragg peaks, which are identical above $T_{s}$ [Fig. 1(b)]. Here, the quantity $|\beta-90^{\circ}|$, or equivalently the average displacement $x$ of the Ta atoms, is used as the structural order parameter of this phase transition. We further compare the width of the Bragg peaks below and above $T_{s}$ [Fig. 1(c)]. The peak width is nearly identical across the structural transition, ruling out any vestigial static monoclinic phase in the high-temperature normal state.

In addition to the immediate nullification of the lattice order parameter above $T_{s}$, we also observe a pronounced decrease of the 2 0 0 nuclear Bragg peak intensity [Fig. 1(c)], indicating the presence of strong lattice fluctuations. Specifically, this fluctuation originates from the squared average of the atomic displacement $x$, and is further corroborated by the dramatically enhanced c*-direction diffuse scattering signal over a broad temperature range above $T_{s}$ [Figs. 2(a)-(b)]. Such a diffuse scattering signal can occur when the atomic position deviates from perfect periodicity along the c*-direction. Since static lattice disorder is ruled out, a strong dynamic lattice fluctuation in the form of inter-chain sheering is the most plausible scenario. This aligns with previous reports of the soft transverse acoustic phonon [29] and the electronically coupled optical phonons in Ta₂NiSe₅ [13, 14, 15].

The structural phase transition at $T_{s}$ could potentially drive an MIT transition in the electronic channel, taking the route of band folding in mean-field theory, which is discussed in many charge density wave (CDW) materials [30, 31]. This synchronization of structural symmetry breaking and electronic MIT is also expected according to the first principles calculation of the electronic structure of Ta₂NiSe₅  based on density functional theory (DFT). As shown in Fig. 3(a), the relaxed ground-state structure exhibits a lattice order parameter $\beta$ comparable to the experimental data (detailed in Sec. VI), and the calculated band structure exhibits an energy gap. When the structure is constrained to maintain orthorhombicity, the system is a gapless semimetal. However, previous transport studies have indicated only a small change of the resistivity slope across the structural transition at $T_{s}$, with an insulating behavior persisting up to 550 K, more than 60% above $T_{s}$ [10].

To reveal the origin of this asynchronized electronic MIT and structural symmetry breaking, we investigate the single-particle spectral function of Ta₂NiSe₅ with high-resolution ARPES. Figure 3(b) presents the APRES data along the $\Gamma$-X high symmetry direction [geometry illustrated in Fig. S1] for various temperatures below and above $T_{s}$. Here, the high statistics and energy resolution enable spectra restoration up to $\sim$150 meV above the Fermi level ($E_{F}$) after dividing the resolution-convolved Fermi-Dirac function [32]. As illustrated in Fig. 3(b), the low-temperature spectra ($T<T_{s}$) show a pronounced single-particle gap, where the dispersion and orbital compositions are consistent with the DFT calculations in Fig. 3(a). Therefore, the electronic state below $T_{s}$ aligns with the mean-field theory for a second-order phase transition.

In contrast, the high-temperature electronic structure ($T>T_{s}$) exhibits anomalous behavior beyond the DFT prediction. We observe a pronounced spectral weight depletion around $E_{F}$ ($\mathrm{\pm}$ 100 meV), despite the disappearance of the low-temperature flat valence band top [Fig. 3(b)]. Such a strong intensity depletion cannot be addressed in the generic band theory: drastic orbital character change alone cannot account for the missing spectral weight in both Ta and Ni states probed in both photon polarization channels. The dipole transition matrix-element effect is also unlikely, given the robust spectral weight depletion seen across a broad range of photon energies and Brillouin zones [Fig. S2].

The relative evolution of this spectral depletion to the structural phase transition can be better illustrated from the integrated energy distribution curves (EDCs) [Fig. 3(c)]. The structure-related band reconstruction at high binding energy (marked by the shifting of the valence band from -0.11 eV to -0.17 eV) happens within a relatively small temperature range below $T_{s}$  in line with the second-order phase transition. In contrast, the low-energy spectral weight ($E_{F}\pm 30$ meV), drops continuously from 380 K to 120 K with little sensitivity to the structural phase transition [Fig. 3(d)]. Such persistent single-particle spectral depletion at $E_{F}$ naturally accounts for the insulating behavior in resistivity and optical conductivity above $T_{s}$ [10, 28].

Despite this spectral weight depletion, however, the band dispersions above $T_{s}$ do not suffer from apparent gap opening or band hybridization. This can be seen from the band dispersions extracted from the ARPES spectra, normalized along the energy axis to remove the intensity depletion effect. As shown in Fig. 3(e), we identify uninterrupted conduction and valence band dispersions (recovered up to $\mathrm{\sim}5k_{B}T_{s}$ above $E_{F}$) consistent with the DFT calculation. Such a single-particle “gapped” spectrum on top of a metallic dispersion, without a global symmetry breaking of the system, is analogous to the pseudogap state found in high-$T_{c}$ cuprates [33, 34, 35], and is recently shown to also relate to electron-hole excitations [36].

Summing up the experimental results of Ta₂NiSe₅ upon thermal tuning, an electronic gapped state is found below $T_{s}$ in line with the second-order structure phase transition and the DFT calculation. Meanwhile, we observe concomitant evolutions of strong lattice fluctuation and electronic pseudogap state above $T_{s}$, without any global symmetry breaking, suggesting potential involvement of strong EPC behind the MIT transition in Ta₂NiSe₅.

While the concomitant evolution of the high-temperature electronic pseudogap and strong lattice fluctuation suggest a connected microscopic mechanism, one cannot quantitatively delineate the contributions of EPC and direct electron-hole Coulomb interaction (excitonic instability) to the broken-symmetry ground state. Non-thermal tuning methods offer more possibilities to separate the above two channels. While pump-probe studies could access excited states beyond the temperature range in equilibrium ARPES measurements, the non-equilibrium dynamics involve non-equilibrated electronic and lattice temperatures, as well as other non-thermal effects [37]. Therefore, contentions remain as to which interaction term determines the eventual restoration of metallicity in Ta₂NiSe₅ [21, 22, 23, 24].

Tuning the carrier density proves to be another efficient non-thermal tuning method in Ta₂NiSe₅, which can restore the gapless state at low temperatures [38, 39]. Here, through the combined use of a micron-spot synchrotron ARPES and in-situ potassium (K) dosing in the low-temperature broken-symmetry state, we achieve a quantitative measure of both the charge doping and chemical potential shift [see Fig. 4]. As will be discussed below, this measurement is necessary to examine the thermodynamic stability of the electronic subsystem. The relative potassium dosage is accessed through the core-level x-ray photoemission spectroscopy (XPS) of the K $3p$ orbital, and the electron doping per unit cell is determined by the Luttinger volume at different dosing levels (see Appendix C for details).

Enabled by these improvements, our experiments reveal a three-stage isothermal spectral evolution across the dosing-tuned MIT [Fig. 4(a) and Fig. S3]. This evolution takes a striking resemblance to the thermal phase transition (insulator-pseudogap-semimetal). Starting from a low-temperature gapped state, the valence band rapidly downshifts upon dosing (shaded in blue). A pseudogap concurrently appears at $E_{F}$, similar to the spectral features right above $T_{s}$. In the second stage (shaded in green), the energy position of the conduction band barely changes while the missing spectral weight in the pseudogap is gradually replenished, resembling the pseudogap filling at higher temperatures. Finally, a semi-metallic state is restored with the full conduction and valence bands intersecting each other (shaded in orange). This semi-metallic state achieved with heavy potassium dosing is qualitatively identical to the normal state of Ta₂NiSe₅ in the high-temperature orthorhombic phase, as suggested by the pump-probe experiments [23] and the DFT calculation [Fig. 3(a)]. Such resemblance indicates that upon electron doping, the system may have a similar tendency to naturally recover the orthorhombic lattice structure, which is indeed confirmed by our first-principles calculation as will be detailed in Sec. VI.

Meanwhile, thermodynamic inevitability of the lattice’s participation is evidenced by the anomalous chemical potential evolution during the electron injection process. Here, the chemical potential change is evaluated from the energy shift of the fully filled Ta $4f$ core-level and cross-referenced with the Se valence band shift [Fig. 4(b)], to rule out extrinsic contributions [40]. The dosage-independent linewidth indicates insignificant inelastic scattering from potassium disorder. Both the Ta $4f$ core-level and the Se valence band are found to evolve non-monotonically with charge doping, in sharp contrast to the non-interacting scenario where a rigid shift of bands towards higher binding energy is expected. Such a non-monotonic behavior of the chemical potential can be described in terms of electronic compressibility. NEC, where the energy of the electronic subsystem decreases despite the addition of electrons, appears when the pseudogap is being replenished in the second stage of charge doping [the green region in Fig. 4(c-d)]. Considering that purely electronic systems can only repel added electrons, such an NEC behavior requires the involvement of additional degrees of freedom, such as the lattice excitations, in order to maintain the thermodynamic stability of the full system.

An important consequence of accessing the semi-metallic normal state via different tuning methods is that it provides a reference to determine the Coulomb interaction strength, which are hard to estimate in the broken symmetry phase without bias. Here, we quantitatively compare the valence-conduction band overlaps $E_{g}$ in the semi-metallic normal state reached via heavy potassium dosing with the other two normal states achieved under dramatically different dielectric environments - equilibrium heating and neutral photodoping [see Fig. 5]. To study the influence of direct inter-band Coulomb interaction on this $E_{g}$, we consider a three-orbital model when lattice effects are not included. The inter-band Coulomb interaction term reads as

where $V$ is the Coulomb interaction strength. $n^{c}_{i}$, $n^{c}_{i+1}$, and $n^{f}_{i}$ are the density operators of two degenerate conduction bands (Ta $5d$) and the valence band (Ni $3d$). If we compare normal states only, the system is resistant to exciton condensation (detailed in Sec. VI) and each band has conserved electron numbers (for specific global doping). We denote their total electron numbers (considering the conduction-band degeneracy) as $N_{c}$ and $N_{f}$, respectively. Thus, the effective Hamiltonian can be mapped into

for the conduction band and

for the valence band, where $N$ is the number of unit cells. As a result, in addition to excitonic instability, which disappears in those normal states, the Hartree part of the Coulomb interaction can be directly read out from the relative shift of both conduction and valence band site energies. The doping-induced change of the band overlap $E_{g}$ corresponds to $\mathrm{\Delta}\mu={2V(\mathrm{\Delta}N_{c}-{\mathrm{\Delta}N}_{f})}/{N}$. Therefore, the $E_{g}$ is found to be linearly reduced with increasing $V$ [see Fig. S4]. Here $(\mathrm{\Delta}N_{c}-{\mathrm{\Delta}N}_{f})/N$ can be estimated from the potassium dosed ARPES spectrum. According to the Luttinger theorem [Appendix C], the doped electron density for the 15^th-dosing [Fig. 4(a)] reaches $\sim$0.4 per unit cell, indicating $(\mathrm{\Delta}N_{c}+{\mathrm{\Delta}N}_{f})/N\ =0.46$. As the Fermi velocity is comparable in the valence and two conduction bands, it leads to $(\mathrm{\Delta}N_{c}-{\mathrm{\Delta}N}_{f})/N\ \approx 0.152$. On the other side, $\mathrm{\Delta}\mu$ can be estimated by comparing the change of band overlap $E_{g}$ in the normal state [Fig. 5]. The heavily K-dosed spectra ($E_{g}\cong 280$ meV, heavily screened situation with enhanced $n_{e}$), the high-temperature spectra ($E_{g}\cong 270$ meV, intrinsic $n_{e}$ + $n_{p}$), and the pump-probe photodoped spectra [23] ($E_{g}\cong 290$ meV, photodoping enhanced $n_{e}$ + $n_{p}$) give an upper bound to the change of $E_{g}$ ($\mathrm{\Delta}\mu$) of $\sim 20$meV. Therefore, we can stringently place an upper bound of direct Coulomb interaction strength $V\sim 70$ meV in the system. This Coulomb interaction, in stark contrast to the $\sim$ eV scale strength required in previous Coulomb-only mean field calculations [16, 20], is not nearly enough to account for the experimentally observed low-temperature gap by itself [further discussed in Fig. 9].

Summing up our results on low-temperature charge doping, a semi-metallic electronic structure is recovered with heavy electron injection through a pseudogap stage, resembling the normal state reached at sufficiently high temperatures. Moreover, the thermodynamic inevitability of the lattice’s participation is revealed through the appearance of the pseudogap state and the NEC, while the direct Coulomb interaction strength is estimated to be insufficient to account for the insulating state by itself.

The stringent upper bound for the allowed Coulomb interaction and the indispensable contribution from the lattice, revealed by the above experimental data, point to the need to reexamine the proposal of excitonic condensate formation in Ta₂NiSe₅. Exciton condensation was first proposed as a viable route toward MIT soon after the success of the BCS theory of superconductivity [41]. In Ta₂NiSe₅, one leading contention is whether it is the electron-phonon coupling or direct electron-hole Coulomb attraction that “dominates” the transition. Based on the new experimental evidence discussed in Sec. II-IV, we quantify the EPC and Coulomb terms, and systematically examine the roles of each term using first-principles and many-body methods in this section.

We first focus on the mechanism of the semi-metallic normal state restoration by charge doping in Fig. 4(a). To describe this process, we simulate the system at different doping levels (see Appendix B for details), using first-principles DFT method with the numerically stable r²SCAN functional [42]. This functional includes the kinetic energy density contribution, whose accuracy in correlated materials is beyond the widely used standard functionals within local density approximation (LDA) or generalized gradient approximation (GGA) [43]. To reveal the ground-state structures, we perform lattice relaxation at each doping level to achieve the most energetically favorable lattice structure. As illustrated in Fig. 6(a), starting from the low-temperature monoclinic gapped state, the system gradually restores semi-metallicity with a transition at the doping level of $x_{\text{doping}}\sim 0.15$ (extra electrons per unit cell). Such a dramatic change in the electronic bands is mainly caused by the change of the lattice structure as shown in Fig. 6(b). We follow the definition in Sec. II and use the $\beta$ angle to depict the structural transition. Cross-checked with different DFT functionals, $\beta$ decreases as additional electrons dope into the system and eventually reaches 90^∘ at $x_{\text{doping}}\sim 0.2-0.3$. This simulated doping evolution reflects that the lattice structure of Ta₂NiSe₅ has a natural tendency to reshape into the high-symmetry orthorhombic phase (the high-temperature lattice structure in Sec. II) upon charge doping. It is worth noting that DFT is a ground-state theory and the functionals we adopted do not include excitonic effects. But the impact of phonons has been included on the level of Born-Oppenheimer approximation through the structural optimization at each doping level. Therefore, the experimentally observed restoration of the high-symmetry lattice structure upon charge doping (see Sec. IV), can be explained as a concomitant effect caused by changes in the lattice structure, without invoking the influence of additional excitonic instability.

To quantify the impact of phonons, we further evaluate the electron-phonon coupling strength near the Fermi level. Following Ref. 44, the coupling matrix element among Kohn-Sham orbitals is

where $|u_{n\boldsymbol{k}}\rangle$ is the Bloch periodic part of the Kohn-Sham wavefunction with momentum $\boldsymbol{k}$ of the $n$-th band; the $\Delta^{(\nu)}_{\boldsymbol{q}}V^{\rm KS}$ is the variation of Kohn-Sham potential induced by the $\nu$-th phonon mode at momentum $\boldsymbol{q}$. The definition of the dimensionless operator $\Delta^{(\nu)}_{\boldsymbol{q}}$ and details of the simulation are described in Appendix B.

The orthorhombic phase of Ta₂NiSe₅ is unstable at low temperature [see Figs. 3(a-b) and Fig. 6(a)]. Forcing the lattice structure to stay in a high-symmetry orthorhombic phase, our first-principles simulation identifies two unstable phonon modes responsible for the orthorhombic-monoclinic structural transition. However, one cannot directly evaluate the electron-phonon coupling matrix elements within the DFT framework for these unstable phonons. Instead, we employ the reference system Ta₂NiS₅ to estimate these couplings, whose orthorhombic phase is stable at low temperature. In Ta₂NiS₅, we identify two phonon modes, labeled by their irreducible representation $B_{2g}$ and $B_{1g}$, that have the maximum eigenvector overlap with the two unstable modes in Ta₂NiSe₅, respectively. In Ta₂NiS₅, these two phonon modes couple to three Kohn-Sham bands near Fermi surface: the top valence band ($v_{1}$), the lowest conduction band ($c_{1}$), and the second lowest conduction band ($c_{2}$). The coupling matrix elements (at the long-wavelength phonon limit $q=0$) in this three-orbital subspace are

The calculated phonon frequencies at $\boldsymbol{q}=0$ are $\omega_{B_{2g}}=1.91$ THz and $\omega_{B_{1g}}=1.54$ THz, consistent with those identified in Ta₂NiSe₅  by Raman spectra [13, 14, 15]. Therefore, the phonon parameters extracted here will be used for further analysis in Ta₂NiSe₅.

As we will show later, simply considering electron-phonon coupling at the mean-field phonon level (Born-Oppenheimer approximation) cannot address the asynchronization between the structural phase transition and electronic spectral evolution along the temperature axis, discussed in Sec. III. Previous theoretical investigations have concluded that an eV-scale Coulomb interaction is required to break the symmetry [16, 20], also without considering the non-mean-field phonon effects. With the parameters estimated from the experimental data and the first-principles simulations, we consider a many-body model explicitly including both electronic interactions and electron-phonon coupling and all orders of fluctuations. Here, we employ a model Hamiltonian in a similar form to Ref. 45:

but the band and interaction parameters are chosen to match the experimental data in this paper and the phonon parameters are chosen according to our first-principles simulations. We write the Hamiltonian in momentum space here: $c^{\dagger}_{k\sigma}$ ($c_{k\sigma}$) creates (annihilates) an electron at the conduction band (primarily Ta $5d$) for momentum $k$ and spin $\sigma$, with dispersion given by $\varepsilon^{c}_{k}$, while the $f^{\dagger}_{k\sigma}$ ($f_{k\sigma}$) creates (annihilates) an electron at the valence band (primarily Ni $3d$), with dispersion given by $\varepsilon^{v}_{k}$. The $n^{c}_{i\sigma}$ and $n^{f}_{i\sigma}$ are the density operators for the conduction and valence band, respectively. The fitted band structures via experimental results [Fig. 3(e)] read as

The direct hybridization between the conduction and valence bands around $\Gamma$ is forbidden in the orthorhombic phase by the inversion symmetry [16]. However, this hybridization is enabled by an $B_{2g}$ lattice distortion, parametrized as a momentum-dependent displacement $x_{q}^{(B2g)}$ and quantized as the phonon mode $x_{q}^{(B2g)}=a_{q}+a^{\dagger}_{-q}$. As the Fermi momentum $k_{F}$ is much smaller than $2\pi/a_{0}$, we further restrict the e-ph coupling to the zone center $g_{q}=g{\delta}_{q}$. The inter-orbital Coulomb interaction term takes the same nearest-neighbor approximation as Eq. (1). Ideally, the conduction band should contain two sub-bands due to the two Ta chains per unit cell. However, we consider only the $B_{2g}$ coupling corresponding to the lower conduction band [45], where the phonon mode reflects the strongest Raman anomaly across the transition[13, 14, 15].

This model allows beyond-Born-Oppenheimer lattice fluctuations that couple to the electrons, and captures both the thermal and non-thermal evolution of the spectral function. We simulate single-particle spectral functions of the model in Eq. 6 using the exact diagonalization (ED), which accurately describes all correlation effects in a finite system (see Appendix B for details). Here, $g$ is chosen as $70$ meV, slightly larger than the number from the Ta₂NiS₅ simulation in Eq. (VI) to compensate the ignored electronic and lattice degrees of freedom, and we first exclude the Coulomb interaction by setting $V=0$ (see Sec. VII for the discussion about the impact of Coulomb interaction). Enhanced by the low dimensionality, strong lattice fluctuation above $T_{s}$ readily enables the conduction-valence band hybridization, and results in an electronic pseudogap without causing a global symmetry breaking [Fig. 7(a), green]. Further reducing temperature results in a divergence of the phonon number, which signifies the phonon condensation and drives the transition to the monoclinic phase. Subsequently, a hard hybridization gap forms below $T_{s}$ [Fig. 7(a), blue]. On the other hand, increasing electron carrier density lifts the Fermi level from the charge-neutral point and reduces the interband phonon dressing. Thus, doping ultimately drives the monoclinic phase into the orthorhombic phase even at low temperatures, which in turn reflects the first-principles results in Fig. 6. The observed NEC also reflects strong electron-phonon coupling [46, 47] [see Appendix D], where additional electronic energy from the added carriers is absorbed into the lattice degree of freedom. Therefore, the phase diagram of Ta₂NiSe₅  along both the temperature and electron-doping axes can be depicted in Fig. 7(b), where the region shaded in green represents the lattice fluctuation induced pseudogap state above $T_{s}$. Clearly, both the thermal and quantum fluctuation effects are strong in this state.

When the phonon fluctuations are sufficient to drive the symmetry breaking through EPC, the many-body model in Eq. (6) can reproduce the sharp band gap of DFT simulations within the Born-Oppenheimer approximation. To show this, we simulate the spectral function by projecting the phonon fluctuations to a classical displacement (so-called the frozen phonon), which is equivalent to a coherent-state phonon wavefunction. Specifically, we assume that the ground state collapses to a broken-symmetry state characterized by a finite $X$. Treating this $X$ as a classical variable, we obtain the frozen-phonon equation for electrons

Using the $\langle(x_{q=0}^{\rm(B2g)})^{2}\rangle$ obtained by ED simulations for various temperatures, we estimate the frozen-phonon displacement by $X_{\rm ED}=\langle(x_{q=0}^{\rm(B2g)})^{2}\rangle^{1/2}$ and evaluate the spectral functions for the corresponding broken-symmetry states. As shown in the left panel of Fig. 7(a), this mean-field treatment for phonons reproduces the gapped ARPES spectra with well-defined single-particle band folding.

We emphasize that our discussion of the EPC effects does not exclude the presence of electronic Coulomb interactions, which naturally exist in all materials. Instead, the quantification of a minimal lattice model provides a way to delineate the contributions from both interactions, avoiding the “chicken-or-egg” dilemma. We discuss the combined impact of EPC and electronic Coulomb interactions in this section. Figure 8 shows the single-particle spectra with the EPC $g$ and interband Coulomb interaction $V$, whose upper bound is set by normal-state experimental results in Sec. IV. Comparing Fig. 7(a) with Fig. 8, we find that the Coulomb interaction within the allowed regime does not obviously influence the electronic structure, both in the broken-symmetry and pseudogap states. In contrast, when the EPC is ignored, the $\leq 100$ meV Coulomb interaction can only repel the conduction and valence bands in their entirety at the long-wavelength limit [Fig. 8 dashed red box]. Such a relative band shift is not helpful – and can even be destructive – to interband hybridizations.

More intuitions about the impact of EPC and Coulomb interactions can be obtained by analyzing the mean-field order parameters when both interactions are included. Thus, we further conduct a mean-field simulation for Hamiltonian in Eq. (6) with two order parameters: the electronic (excitonic) order parameter is defined as $\Delta_{\rm ext}=\sum_{i\sigma}\langle f^{\dagger}_{i\sigma}c_{i\sigma}\rangle/2N$, reflecting the hybridization between Ta to Ni orbitals; the lattice order parameter is $X_{\rm MFT}=\langle x_{q=0}^{\rm(B2g)}\rangle$, reflecting the symmetry breaking of the crystal and demonstrating the same information as $|\beta-90^{\circ}|$ in Sec. II [48]. As shown in Figs. 9(a) and (b), these two order parameters follow the same dependence on interactions at the mean-field level. For weak interactions, the EPC ($g$) and Coulomb interaction ($V$) are cooperative in driving the hybridization $\langle f^{\dagger}c\rangle$ and forming excitons. However, since the Hartree part of the Coulomb interaction [i.e. Eqs. (2) and (3)] separates the two bands, it competes with the hybridization or exciton formation. When $V$ is larger than some critical value ($\sim 80$ meV), its suppression on the excitonic or lattice order overwhelms the latter.

To connect the order parameters with the single-particle spectra, we simulate the single-particle band gap within the mean-field framework. Note that these gaps are different from the “pseudogap” from Figs. 7 and 8, the latter of which contains many-body fluctuations beyond the mean field. Figure 9(c) shows the mean-field gap size for different combinations of both interactions. For weak Coulomb interaction, this gap is proportional to the excitonic order parameter $\Delta_{\rm ext}$ (and the lattice order parameter $X_{\rm MFT}$). They start to deviate from each other for a relatively strong $V$, denoted by the dashed line. In this case, the Hartree part of the Coulomb interaction dominates and a band gap opens, suppressing the hybridization. Obviously, overlapping conduction and valence bands are always observed in ARPES experiments in the normal state of Ta₂NiSe₅, further supporting the conclusion about limited Coulomb interaction strength. To avoid the impact of this band gap, we further solve an artificial mean-field equation by eliminating the Hartree terms. This treatment leads to a pure hybridization gap [see Fig. 9(d)], depending quadratically on $g$ and slightly on $V$. Detailed discussions about the hybridization gap and the impact of the Hartree terms are presented in Appendix E.

The inefficiency of the interband Coulomb interaction in Ta₂NiSe₅ originates from the material’s mirror symmetry with respect to the Ni-chain at small momenta, where the exact symmetry is present at $k=0$ [16]. To illustrate this relation, we take the multi-band hopping parameters from Ref. 16 and extract the component corresponding to the Ta $5d_{x^{2}-y^{2}}$ - Ni $3d_{xy}$ hopping integral, denoted as $t_{\rm TaNi}$. Due to the different parities, $t_{\rm TaNi}$ is an odd function of the momentum $k$ and its strength at the $k_{F}$ (determined by experiments) is extremely small compared to the bandwidth [see Fig. 9(e)]. Adding this interband hopping $t_{\rm TaNi}$ in our two-band model, we conduct the same mean-field simulation for the order parameters and single-particle gap. Figure. 9(f) shows the critical Coulomb interaction $V_{c}$ necessary to generate a 200-300 meV hybridization gap ($\Delta E_{\rm hyb}$) [49], without contributions from phonons. Due to the competition from the Hartree term, a hybridization gap comparable to experiments can not be reached given the experimentally bounded $V$. Excluding the contribution from the Hartree part of the mean-field decomposition can relax the requirement on $V_{c}$ to a certain extent (dashed line) [20]. However, compared to other excitonic insulator candidates, the required $V_{c}$ in Ta₂NiSe₅ is still far beyond the experimentally identified upper bound (blue shade), due to its uniquely small Fermi momenta (black dashed line).

Our result reveals the important role of EPC in both the normal and broken-symmetry states of Ta₂NiSe₅, based on the electrons’ coupling to a strongly fluctuating lattice. Combining advanced spectroscopy and computation methods, we determine and quantify a minimal lattice model that captures the pseudogap state, the broken symmetry state, and the negative electronic compressibility. The direct electron-hole Coulomb interaction may not be zero but is not required to capture the above experimental observations. In the pseudogap phase without any global structural symmetry lowering, phonons cannot be treated under the Born-Oppenheimer approximation, under which the lattice’s impact on the electronic structure is only reflected by a nonzero lattice distortion $\langle x\rangle$. Instead, phonons act on the electronic structure through a fluctuating state where $\langle x\rangle=0$ but ${\langle x}^{2}\rangle\ \neq\ 0$. In this regard, the pseudogap state may be conceptually likened to the ‘preformed excitons’ proposed in recent studies [26], except that the binding is facilitated by interband electron-phonon coupling. In comparison, fluctuating CDW states in cuprates, K_0.3MoO₃, ZrTe₃, NbSe₃, and NbSe₂ are usually intertwined with electronic instabilities induced by strong Coulomb interaction [35, 50, 51, 52, 53]. In Ta₂NiSe₅, however, the major distinction lies in the small momentum low-energy band, which prohibits Coulomb interaction-induced charge transfer to the leading order. Such distinct lattice-symmetry-protected low-energy band crossings can be used to engineer accentuated lattice fluctuation effects in correlated materials.

The strong EPC in the Ta₂NiSe₅ system also leads to predictions of distinct material properties upon further thermal and nonthermal tuning. First, in the strong-coupling scenario, the phase transition should not be tied to the Fermi surface topology. Recent studies showed S-doping as an effective method to control the conduction and valence band overlap in Ta₂NiSe₅ [12]. The minimal many-body model shall produce an evolution of the phase transition temperature that is insensitive to the S-doping-tuned semimetal-to-semiconductor transition in Ta₂Ni(Se,S)₅, much different from the popular expectation of the dome-shaped excitonic insulator phase diagram upon band-gap tuning [41]. Second, given such strong EPC, the phonon self-energy will exhibit anomalies over a broad momentum range extending towards $\mathbf{q}\rightarrow 0$, rather than a narrow region sharply centered around $\mathbf{q}\sim 2\mathbf{k}_{F}$ [54]. This can be verified in the dynamic structural factor accessed with high-resolution inelastic x-ray or neutron scattering experiments.

While we qualitatively explain the observed spectral behaviors using a single phonon mode, further experiments have indicated that multiple phonons are involved in this transition: the diffuse scattering and inelastic x-ray scattering results point to extensive transverse acoustic phonon softening [29]; Raman scattering also suggests strong involvement of electronically coupled optical phonons [13, 14, 15]. Distinct from the acoustic-phonon-driven structural transition in 3D perovskite PrAlO₃ [55], the strong fluctuations in Ta₂NiSe₅ seem to suggest additional contributions from the lower system dimension and the optical phonons. The strong lattice fluctuations and electron-phonon coupling in this system offer a knob to realize optically, electrically, and mechanically controlled MIT [56], and a platform to investigate the role of electron-phonon coupling behind phase instabilities in low dimensional systems.

Last but not least, we demonstrated a general method to unambiguously resolve the typical chick-or-egg problems in correlated material systems. First-principles and many-body numerical simulations can be combined with spectroscopic experiments not only to determine the minimal microscopic lattice model, but also to quantify the microscopic interaction parameters – especially through numerous tuning methods. The minimal many-body model can be considered a faithful description of the material under investigation, and the roles of each interaction – if a reductionist view is desired – may be discussed by performing numerical experiments considering one term at a time. Most importantly, further material property predictions or engineering guidance can then be made by solving the perturbed minimal many-body model containing experimentally determined interaction parameters. This approach is ready to be generalized in low-dimensional correlated materials amid the rapid development of advanced spectroscopy, in situ material tuning, and modern computational methods.

We thank E. Baldini, S. D. Chen, E. W. Huang, A. Kemper, D. H. Lee, Y. B. Li, D. Y. Qiu, M. Trigo, Z. X. Shen, M. Yi, A. Zong, and W. T. Zhang for helpful discussions. Use of the Stanford Synchrotron Radiation Light Source, SLAC National Accelerator Laboratory, is supported by the US Department of Energy, Office of Science, Office of Basic Energy Sciences under Contract No. DE-AC02-76SF00515. This research used resources of the Advanced Light Source, a US DOE Office of Science User Facility under Contract No. DE-AC02-05CH11231. This work is based upon research conducted at the Center for High Energy X-ray Sciences (CHEXS) which is supported by the National Science Foundation under award DMR-1829070. Work at Lawrence Berkeley National Laboratory was funded by the U.S. Department of Energy, Office of Science, Office of Basic Energy Sciences, Materials Sciences and Engineering Division under Contract No. DE-AC02-05-CH11231 within the Quantum Materials Program (KC2202) and within the Theory of Materials Program (KC2301) which provided the DFT calculations. S.D. and Y.W. acknowledge support from the National Science Foundation (NSF) award DMR-2038011. The electron-phonon model simulations were performed on the Frontera computing system at the Texas Advanced Computing Center. The DFT calculations were performed using computation resources at the National Energy Research Scientific Computing Center (NERSC). The work at Yale University is partially supported by the National Science Foundation under DMR-2132343.