Competing Lattice Instability and Magnetism on the Surface of Kagome Metals

Kagome materials, characterized by a corner-sharing triangular sublattice, are a fertile playground for various correlated electronic orders Wen et al. (2010); Kiesel and Thomale (2012); Wang et al. (2013); Kiesel et al. (2013). The discovery of charge density wave (CDW) in non-magnetic kagome superconductors Ortiz et al. (2020); Jiang et al. (2021); Tan et al. (2021); Liang et al. (2021); Chen et al. (2021) sparked immediate interest because of its close relationship with the potential time-reversal symmetry breaking and superconductivity. In contrast, magnetic kagome materials showing CDW are rare, primarily due to the lack of lattice instability in most systems and the competition between magnetism and other low-energy scale orders.

The recent discovery of CDW in the magnetic kagome metal FeGe Teng et al. (2022) highlights the intricate interplay between magnetism and lattice instability Teng et al. (2023); Miao et al. (2023); Wang (2023); Chen et al. (2023), enabling the exploration of intertwining between these orders. This FeGe-type structure is derived by excluding the $R$ element from the extensively studied magnetic kagome compounds $R$Mn₆Sn₆ ($R$ = Y, Gd-Lu) Ghimire et al. (2020); Riberolles et al. (2022); Yin et al. (2020); Zhang et al. (2020); Li et al. (2021); Xu et al. (2022); Li et al. (2022); Asaba et al. (2020); Ma et al. (2021); Gao et al. (2021); Zhang et al. (2022a), where no charge instability has been reported. This structural resemblance raises the possibility of an underlying lattice instability in $R$Mn₆Sn₆, potentially suppressed by magnetism and observable when magnetism is diminished. The observation of CDW transition in the non-magnetic ScV₆Sn₆ Arachchige et al. (2022); Tan and Yan (2023); Cao et al. (2023); Korshunov et al. (2023), as opposed to its magnetic counterpart ScMn₆Sn₆ without CDW, suggests such a potential. However, the absence of CDW in other non-magnetic $R$V₆Sn₆ variants underscores the complexity of these materials. This complexity prompts questions about the existence of competition between lattice instability and magnetism and its manifestation in the wider range. Our recent Scanning Tunneling Microscopy (STM) studies reveal significant distortions on the surface of ScV₆Sn₆ Cheng et al. (2024), hinting at the possibility of observing this competition in a decreased dimension.

In this study, we elucidate the interplay between lattice instabilities and magnetism in surfaces of $R$V₆Sn₆ and $R$Mn₆Sn₆. We reveal that the kagome layer on $R$V₆Sn₆ surfaces inherently exhibits charge instabilities, manifesting as $\sqrt{3}\times\sqrt{3}$ and $2\times 2$ surface charge orders (SCOs), driven by phonon softening and characterized by distinct V trimer and clover-like distortions. Introducing magnetic Mn in place of non-magnetic V on these kagome surfaces leads to a significant weakening of Mn spin polarization due to surface distortions, increasing the magnetic energy via Hund’s rule. This energy increase suppresses SCOs, further evidenced by the absence of SCOs in magnetic $R$Mn₆Sn₆. Remarkably, substituting Mn with V on $R$Mn₆Sn₆ kagome surfaces allows SCOs to reappear, underlining a potent competition between surface lattice instability and magnetism and highlighting a pathway to potential high-temperature SCOs. Our findings unveil complex dynamics in surfaces of kagome magnets and beyond.

Bulk properties of $R$V₆Sn₆. The crystalline structure of $R$V₆Sn₆ comprises a Sn honeycomb layer, a $R$Sn triangular layer, and two V-Sn kagome layers in a buckled configuration, as shown in Fig. 1(a). Prior experimental observations have identified a complex ordering of the $R$ 4$f$ magnetic moments below $\sim$4 K Pokharel et al. (2022); Rosenberg et al. (2022); Zhang et al. (2022b); Lee and Mun (2022). ARPES analysis has revealed the minimal influence of the 4$f$ magnetism on the low energy physics Peng et al. (2021); Hu et al. (2022); Sante et al. (2023). Our calculations demonstrate robust SCOs not only in the ferromagnetic TbV₆Sn₆ but also in paramagnetic GdV₆Sn₆ and nonmagnetic YV₆Sn₆. Consequently, we will omit discussions regarding $f$-electron magnetism. Given the similarity of results across all $R$V₆Sn₆ compounds, we will primarily illustrate our findings with TbV₆Sn₆.

The bulk band structure of TbV₆Sn₆ is depicted in Fig. 1(b), characterized by typical kagome band features, such as van Hove singularities at $M$/$L$, Dirac points at $K$/$H$, and a flat band at approximately 0.3 eV. Notice that all $R$V₆Sn₆ materials exhibit similar band structures Pokharel et al. (2021); Peng et al. (2021); Hu et al. (2022); Tan and Yan (2023); Ding et al. (2023). In Fig. 1(c), the phonon spectrum of TbV₆Sn₆ confirms its bulk dynamical stability by showing the absence of imaginary phonon modes. This absence extends to all $R$V₆Sn₆ compounds (except for ScV₆Sn₆ Tan and Yan (2023)), indicating all $R$V₆Sn₆ compounds (except for Sc) are stable in their bulk forms. It’s worth noting that, among these stable compounds, LuV₆Sn₆ exhibits the softest phonon branch on the $k_{z}$ plane, which may suggest its susceptibility to structural phase transitions under external influences like strain (see Supplementary II.A SM for additional details).

SCOs of $R$V₆Sn₆. The layered structure of $R$V₆Sn₆ promises four different terminations, i.e., the $R$Sn triangular surface, Sn honeycomb surface, and two kagome surfaces, V₃Sn and SnV₃, as shown in Fig. 1(a)&(d), These different surfaces have been observed in various experiments Yin et al. (2020); Cheng et al. (2024); Hu et al. (2022); Peng et al. (2021); Rosenberg et al. (2022); Li et al. (2022). The primary differentiation between the two kagome surfaces arises from the respective chemical configurations underneath. In the V₃Sn termination, the V-kagome sublattice is positioned above the closely associated Sn sublattice from the same buckled V-Sn layer, with the subsequent layer being the Sn honeycomb layer not . In contrast, the SnV₃ termination situates the V-kagome sublattice below the closely associated Sn sublattice, with the next layer the $R$Sn triangular layer. Both terminations show similar SCOs and are collectively called kagome surfaces within this work.

Figures 2(a)-(d) display the surface phonon band structures of the four different terminations of TbV₆Sn₆, obtained from thick slab model calculations. Notably, the V₃Sn surface exhibits imaginary phonon modes across the entire surface Brillouin zone [Fig. 2(a)]. The interpretation of these modes unfolds as follows: i) The imaginary phonon at $\Gamma$ results in a $1\times 1$ surface modulation, primarily involving the surface V layer trimerization [Fig. 2(e)]. This modulation reduces the surface’s rotational symmetry from 6-fold to 3-fold while preserving the translational symmetry. However, such a trimerized surface is marked by a significant imaginary phonon at the $K$ point (see Supplementary II.C SM ), indicating its instability. ii) The imaginary phonon at $K$ induces a $\sqrt{3}\times\sqrt{3}$ SCO, as depicted in Fig. 2(f). In this phase with broken translational symmetry, the surface V-kagome layer transforms into a clover-like pattern, resulting in a total energy decrease of about 307 meV per surface primitive unit cell (u.c.), relative to the pristine surface (see Table 1). iii) The imaginary phonon at $M$ drives a $2\times 2$ SCO, combining V trimers and clovers, as illustrated in Fig. 2(g). This phase exhibits a lower total energy of $-$394 meV/u.c. The dynamical stability of these two SCOs is confirmed by the absence of imaginary phonon modes in their respective phonon spectra (see Supplementary II.C SM ). Typical scanning tunneling microscopy (STM) images at the charge neutral point are presented in Fig. 2(h)-(j) with additional results available in Supplementary II.H SM .

In Fig. 2(b), the phonon spectrum of the SnV₃ termination reveals exclusively imaginary phonons at $\Gamma$ point. The $\Gamma$ imaginary phonon instigates the 1$\times 1$ surface modulation with V trimers as illustrated in Fig. 2(e), which reduces the total energy by $-$17 meV/u.c.. Intriguingly, such a V trimerized SnV₃ surface shows a substantial imaginary phonon at $K$, akin to the scenario in V₃Sn, as detailed in Supplementary II.C SM . Furthermore, the $\sqrt{3}\times\sqrt{3}$ and $2\times 2$ SCOs displayed in Fig. 2(f)&(g) elicit more pronounced reductions in the total energy of the SnV₃ surface, amounting to $-$26 and $-$21 meV/u.c., respectively. An insightful deduction from the phonon band structure is that an imaginary phonon mode at a reciprocal space momentum generally implies the presence of a stable structure modulated by the same momentum. The SnV₃ surface, however, presents a paradoxical case by revealing the absence of a stable structure modulated by the $\Gamma$ imaginary phonon and the unexpected emergence of energy-favored structures driven by the non-imaginary phonons at $M/K$. This peculiarity might arise from the effective screening of the long-range $\Gamma$ imaginary phonon to the short-range $M/K$ imaginary phonons, which manifests only after the elimination of the $\Gamma$ imaginary phonon. Notably, this $\Gamma$ imaginary phonon screening effect may also apply to the V₃Sn surface, exhibiting strong imaginary phonons at $\Gamma$ and weak ones at $M/K$. STM images of SCOs for the SnV₃ surface at the charge neutral point are presented in Fig. 2(k)-(m), with additional details available in Supplementary II.H SM .

The phonon spectra of the Sn and TbSn terminations in Fig. 2(c)&(d) notably lack any soft modes. The above-established SCOs in kagome terminations are also confirmed to be unstable on both surfaces. It is crucial to emphasize that these two surfaces can be achieved by overlaying one Sn (TbSn) layer atop the SnV₃ (V₃Sn) surface. Consequently, introducing additional layers effectively mitigates the instabilities observed in the kagome surfaces. This comparison underscores the pivotal role of exposing the kagome layer at the surface in attaining SCOs in $R$V₆Sn₆. Furthermore, it is worth noting that there are numerous flat phonon bands evident across all surfaces, which could potentially influence the phonon dynamics, as discussed in Hu et al. (2023).

We apply the above SCOs to the kagome terminations of all other $R$V₆Sn₆. After adequate surface relaxation, these SCOs consistently reduce the total energy, as summarized in Table 1. Similar to the Tb compound, the $2\times 2$ SCO consistently exhibits a lower total energy in the V₃Sn termination, while the $\sqrt{3}\times\sqrt{3}$ SCO attains a lower total energy in the SnV₃ termination across all $R$ compounds. Remarkably, while the $2\times 2$ SCO can be achieved in the SnV₃ termination of GdV₆Sn₆ following surface relaxation, the $2\times 2$ distorted SnV₃ surfaces of Y and Dy-Lu materials spontaneously revert to the $1\times 1$ trimerized surface configuration after relaxation, effectively restoring the broken translational symmetry. Phonon calculations additionally reveal that the $1\times 1$ trimerized kagome surfaces of YV₆Sn₆ exhibit a pronounced imaginary phonon at $K$ (resulting in the $\sqrt{3}\times\sqrt{3}$ SCO), notwithstanding the pristine SnV₃ surface of the Y compound only manifesting an imaginary phonon at the $\Gamma$ point (elaborated in Supplementary II.D SM ). Given the resemblance of these materials, we conjecture that the $1\times 1$ trimerized kagome surfaces exhibit instability across all $R$V₆Sn₆ compounds (except for Sc). However, their total energies are included in Table 1 for reference and comparative analysis. These SCOs are pure surface effects, which are robust against the slab thickness as comprehensively elucidated in Supplementary II.E SM .

We have also extended our analysis of SCOs to the extensively studied ScV₆Sn₆. In the pristine bulk phase, while two SCOs reduce the energy of the V₃Sn termination, they are unstable in the SnV₃ termination, and only the $1\times 1$ surface trimerization reduces the total energy. This phenomenon may be attributed to the significantly higher in-plane chemical pressure in ScV₆Sn₆ (see Supplementary II.A SM ). Furthermore, we applied the $\sqrt{3}\times\sqrt{3}$ SCO to kagome terminations of ScV₆Sn₆ in bulk $\sqrt{3}\times\sqrt{3}\times 3$ CDW phase Arachchige et al. (2022), and found it to be only stabilized in the V₃Sn termination (Table 1).

Suppression of SCO by magnetism. Now we explore the impact of magnetism on SCOs by substituting the outermost V kagome layer on TbV₆Sn₆ kagome terminations with a magnetic Mn kagome layer. Calculations indicate that the local magnetic moments of Mn form a ferromagnetic ordering. This magnetic substitution leads to the disappearance of SCOs, following sufficient surface relaxation. Indeed, such Mn-doped surfaces with frozen SCOs show much higher total energies (Table 2). Moreover, phonon spectra analyses of these Mn-doped kagome surfaces reveal no lattice instabilities, as shown in Figs. 3(a). The interplay between SCOs and magnetism is further elucidated by studying magnetic counterpart TbMn₆Sn₆. Namely, the absence of imaginary phonon modes on kagome surfaces Mn₃Sn and SnMn₃ indicates no SCOs (see Supplementary II.G SM ). Conversely, reintroducing V into kagome surfaces of TbMn₆Sn₆ stabilizes the SCOs. Notably, the cross substitution between V and Mn does not exert a noticeable strain effect, attributed to the similar effective radii of V and Mn ions (V 0.640 Å versus Mn 0.645 Å Shannon and Prewitt (1969)). Thus, the stabilization or destabilization of SCOs in these kagome surfaces is more closely related to magnetic effects rather than mechanical strain. Notice that the $f$-electron magnetism of Tb does not affect SCOs.

These observations highlight the competition between lattice instabilities and magnetism. A detailed Bader charge analysis of Mn atoms of the Mn-doped kagome surfaces of TbV₆Sn₆ with frozen SCOs is presented in Table 2. It shows that while the total charge of each Mn atom is preserved, SCOs transfer charges from the spin-up channel to the spin-down channel, weakening the spin polarization heavily. For example, on the Mn-doped V₃Sn surface, Mn atoms have initial spin-up/down charges of 5.17$e$/1.65$e$, yielding a total charge of 6.82$e$ and magnetization of 3.52$\mu_{B}$. With the frozen $\sqrt{3}\times\sqrt{3}$ SCO, spin-up charges adjust to 4.84$e$ (Mn1) and 4.34$e$ (Mn2, labeled in Fig. 3(c)), and spin-down to 2.02$e$ (Mn1) and 2.53$e$ (Mn2), nearly preserving total charges (6.86$e$ for Mn1, 6.87$e$ for Mn2) but reducing magnetizations to 2.82$\mu_{B}$ (Mn1) and 1.81$\mu_{B}$ (Mn2). Magnetization densities are exemplified with Mn-doped V₃Sn in Fig. 3(b)-(d), where the Mn2 atoms have a much smaller magnetization density area under SCOs. Similar behaviors are confirmed for all other surfaces and SCOs (changes are smaller in the SnV₃ surface because of its smaller SCO distortion amplitudes). Results of TbMn₆Sn₆ kagome terminations are found in Supplementary II.G SM . According to Hund’s rule which favors high-spin states in magnetic ions for maximum spin polarization, the reduction in spin polarization leads to increased magnetic energy, which outweighs the kinetic energy loss from structural distortions. As a result, the total energy of the surface is increased, mitigating surface distortions. For TbV₆Sn₆ and V-doped TbMn₆Sn₆ kagome terminations, the elimination of surface magnetism ceases its antagonistic effect on SCOs, resulting in SCOs as previously discussed.

This mechanism contrasts the scenario for the CDW formation of the kagome magnet FeGe. In FeGe, structural distortions lead to kinetic energy gains and increased spin polarization, reducing more magnetic energy Miao et al. (2023); Wang (2023); Chen et al. (2023). Moreover, spin-splitting shifts a von Hove singularity into the Fermi level in the spin minority band Teng et al. (2023), enhancing Fermi surface nesting and promoting charge instability. The intricate interplay among spin, lattice, and charge degrees of freedom culminates in the CDW formation in FeGe and SCO suppression in $R$Mn₆Sn₆, illustrating the complex quantum ground states emergent from the intertwined dynamics of these variables.

Now we address the potential experimental detection of the predicted SCOs. SCOs significantly disrupt the surface kagome layers, obfuscating certain surface states (see Supplementary II.F SM ). ARPES appears as the initial experimental technique to consider. Notably, a limited number of ARPES experiments have been conducted on kagome terminations of $R$V₆Sn₆ Hu et al. (2022); Peng et al. (2021); Rosenberg et al. (2022); Sante et al. (2023); Cheng et al. (2024); Ding et al. (2023). While experiments observe some distinctive bands, ambiguous states are also present whose origin remains challenging to identify. Consequently, the determination of the presence of SCOs in these experiments, as well as the characterization of their range (long-range or short-range), remains elusive. We propose that microscope techniques such as STM, capable of detecting local atomic structures, could be an optimal approach for SCO detection. Nevertheless, relatively few STM experiments have been conducted on $R$V₆Sn₆. In the case of ScV₆Sn₆ within the bulk $\sqrt{3}\times\sqrt{3}\times 3$ CDW phase, recent STM experiments on kagome terminations observed a prominent $\sqrt{3}\times\sqrt{3}$ reconstruction Cheng et al. (2024). We do not conclude it the $\sqrt{3}\times\sqrt{3}$ SCO (Table 1) because the bulk CDW also exhibits a $\sqrt{3}\times\sqrt{3}$ in-plane modulation. However, it’s crucial to underscore that the bulk CDW of ScV₆Sn₆ exclusively involves Sc/Sn out-of-plane movements, with minimal distortion in the V kagome layer Arachchige et al. (2022); Tan and Yan (2023). This contrasts the STM experiment, underscoring the need for more comprehensive studies for $R$V₆Sn₆.

In the context of $R$Mn₆Sn₆, intriguing phenomena are anticipated. It is noteworthy to consider the possibility of observing SCOs in $R$Mn₆Sn₆ at high temperatures when the magnetic order ceases to exist. Confirmation of this phenomenon in experiments would establish $R$Mn₆Sn₆ as a rare class of symmetry-descending materials when temperature increases Wu et al. (2024). We recommend more STM experiments on $R$Mn₆Sn₆ at elevated temperatures to explore this intriguing prospect. Furthermore, the sizeable structural distortion accompanying SCOs might also suggest the potential for a significant surface magnetostriction Lee (1955) and piezomagnetic effect, which would benefit the application of antiferromagnetic.

In conclusion, we have revealed the pervasive $\sqrt{3}\times\sqrt{3}$ and $2\times 2$ SCOs across kagome terminations in $R$V₆Sn₆, which are suppressed in $R$Mn₆Sn₆ by magnetism due to the profound competition between lattice instability and magnetism through Hund’s rule. We call for additional experimental efforts to explore these fascinating surface phenomena. Our findings suggest the existence of undiscovered quantum states in general magnets, potentially accessible through the suppression of magnetism.

We thank Ilija Zeljkovic for fruitful discussions. H.T. also thanks Yizhou Liu for helpful discussions. B.Y. acknowledges the financial support by the European Research Council (ERC Consolidator Grant “NonlinearTopo”, No. 815869) and the ISF - Personal Research Grant (No. 2932/21) and the DFG (CRC 183, A02).