Three-dimensional nature of anomalous Hall conductivity in YMn₆Sn_6-xGa_x, $x\approx 0.55$

The crystal structure and atomic composition of YMn₆Sn_5.45Ga_0.55 (Y166-Ga) were determined using single crystal X-ray diffraction. Similar to the parent compound YMn₆Sn₆ (Y166), Y166-Ga adopts a hexagonal $P6/mmm$ structure $(a=b=5.4784$ Å, $c=8.925$ Å), consisting of kagome planes [Mn₃Sn] separated by two inequivalent layers Sn₂ and Sn₃YGa, as illustrated in Fig. 1.

The structure exhibits Sn-site doping with Ga, specifically at the Sn3 site (Wyckoff position 2c). This was confirmed through modeling efforts for partial Ga occupancy at other Sn sites, which either resulted in poorer fits to the data or nonsensical Ga occupancy values. Refinement analysis determined the Sn:Ga site occupancy to be 0.725:0.175, corresponding to the full chemical formula noted above, i.e., $\sim$ 0.55 Ga atoms per unit cell. Detailed crystallographic parameters from the single crystal X-ray diffraction experiment are summarized in Table 1.

Our DFT calculations, presented in Sec. III.3, also reveal a significant energy advantage for Ga substitution at the Sn3 site in the [Sn₃Y] layer, rather than in the [Mn₃Sn] or [Sn₂] layers. YMn₆Sn_5.45Ga_0.55 can thus be viewed as a moderately hole-doped ($\approx 0.09$ h/Mn) derivative of the parent compound Y166.

The temperature dependence of magnetic susceptibility ($\chi=M/B$) measured with a magnetic field $B$ of 0.1 T parallel to [100] ($\chi_{100}$) and along [001] ($\chi_{001}$) is shown in Fig 2 (a). These susceptibility data indicate that Y166-Ga undergoes a paramagnetic-to-ferromagnetic ordering below 350 K, consistent with previous reports [31, 41]. Additionally, the easy-plane behavior is evident from the significantly larger $\chi_{100}$ compared to $\chi_{001}$ below the transition temperature ($T_{c}$), with an anisotropy ratio of $\chi_{100}/\chi_{001}=5$ just below $T_{c}$. It is to be noted that parent compound Y166 orders with a commensurate antiferromagnetic helical structure below 345 K and exhibits an incommensurate double helical structure (DH) upon further cooling [17, 19, 42], while YMn₆Sn_1-xGa_x compounds show a ferromagnetic transition for doping concentrations $x>0.30$ [31].

Figures 2 (b) and (c) show the isothermal magnetization curves of Y166-Ga at some representative temperatures for $B$ $||$ [100] ($M_{100}$) and [001] ($M_{001}$), respectively. In the entire temperature range measured $M_{100}$ saturates below 0.5 T with a negligible hysteresis, while $M_{001}$ saturates at slightly larger $B$. At 1.8 K $M_{001}$ saturates at 2.2 T, which decreases with increasing temperature (1.7 T at 300 K), also with negligibly small hysteresis. At 1.8 K, saturation magnetization ($M_{sat}$) along [100] is 9.85 $\mu$_B/f.u. while it is 10.7 $\mu$_B/f.u along [001]. In either direction, $M_{sat}$ gradually decreases with the increase in temperature, which attains a value of 7.5 $\mu$_B/f.u. along [100] and 7.7 $\mu$_B/f.u along [001]. The ratio of the saturated magnetization at 1.8 to 300 K ($M_{sat,1.8K}$/ $M_{sat,300K}$) is 1.39 along the [001] direction and 1.31 along the [100] direction. For Tb166, the Mn moment shows the ratio of $\sim$ 1.05 and the Tb moment exhibits the ratio of around 1.66 [43, 12]. This suggests that the Mn moments in Y166-Ga experience more fluctuation between 1.8 and 300 K than in Tb166, where the Tb moments are the most fluctuating ones [12]. This observation is consistent with the expectation, as the Curie temperature of Tb166 is about 70 K higher than that of Y166-Ga.

To understand the doping-induced phase transition, we adopt the $J_{1}-J_{3}$ effective model proposed in Ref. [44]. The spin Hamiltonian is expressed as follows

where $J_{i}$ ($i=1-3)$ are the exchange interaction parameters as indicated in Fig. 1. The parameters were extracted using the least squared fitting of our DFT calculations into Eq. 3.

According to a thorough DFT-based analysis of the phase diagram for parent Y166 discussed in our earlier work [17], we observed that $U_{eff}=0.6$ best reproduce the magnetic states for parent Y166 observed in the experiments. Therefore, in our study, we considered the same $U$ parameter with one additional larger $U=2$ for comparison.

The results along with those of Y166, taken from Ref. [17], are summarized in TABLE 2. The fitting is achieved with excellent quality in all three cases, as all the relative energy differences between different magnetic states can be consistently and accurately reproduced by the model, especially in the case of $U=0.6$ where the average error is less than 1 meV. This suggests that the minimal model adopted here is appropriate and reliable.

In TABLE 2, one can see that while $J_{1}$ and $J_{2}$ consistently favor ferromagnetic alignment regardless of different U values, the Hubbard U correction tends to stabilize the FM states further as $J_{3}$ shifts from positive to negative as U increases. According to the analytically determined phase diagram Ref. [44] the spin model for all three U values yields the same correct FM ground state.

To assess the doping effect, we compare the results of the two compounds for the same $U=0.6$ (i.e. columns 3 and 5). In Y166, $J_{1}$ dominates and has the same (opposite) sign as $J_{3}$ ($J_{2}$). This arrangement leads to a competition between $J_{2}$ and $J_{3}$ which results in the formation of a helical magnetic state [17, 44]. However, with Ga-doping, $J_{2}$ becomes ferromagnetic and now comparable to $J_{1}$ in strength. Although $J_{3}$ becomes antiferromagnetic, it is too weak to induce frustration.

This qualitative shift aligns with expectations, considering the direct alteration of exchange pathways for $J_{2}$ and $J_{3}$ induced by the presence of doped-Ga. As a consequence of these changes, the frustration that was initially developed in the pure Y166 to promote the helical magnetic state is effectively mitigated. The system undergoes a transition, and the magnetic state collapses into a ferromagnetic (FM) order.

It is interesting to note that while all configurations predict the same correct ferromagnetic ground state, only $U=2$ gives the correct easy-plane anisotropy and $U=0$ and $0.6$ give very small easy-axis anisotropy.

The temperature dependence of electrical resistivity of Y166-Ga, measured with the electric current applied along the [100] direction ($\rho_{100}$, blue curve) and the [001] direction ($\rho_{001}$, red curve) over the temperature range 1.8-400 K, is shown in Fig. 3(a). The resistivity decreases as temperature decreases, indicating the metallic behavior of the sample. Residual resistivity ratio (RRR), calculated as $\rho(400K)/\rho(2K)$, is 9 for $I||[100]$ and is 18 for $I||[001]$. These values are smaller than those in Y166 [17, 19], likely due to disorder induced by doping. Both $\rho_{100}$ and $\rho_{001}$ exhibit a kink at 350 K, indicative of the onset of a ferromagnetic transition, as observed in the susceptibility measurements [Fig. 2(a)]. Across the entire temperature range, $\rho_{100}$ is greater than $\rho_{001}$. The conductivity anisotropy, $\sigma_{[001]}/\sigma_{[100]}$, is plotted in Fig. 3(b), showing a value $>$ 2, which suggests that the electronic transport in Y166-Ga is three dimensional. This behavior differs from that of the parent compound Y166, where in-plane conductivity is greater than the out-of-plane conductivity [19]). The enhanced $c$-axis conductivity observed in Y166-Ga is similar to that found in Ge doped YMn₆Sn₆ [10].

The anisotropic transport behavior was further investigated by measuring the angular-dependent magnetoresistance (AMR). In this measurement, an electric current $I$ was applied along the [100] direction, while the sample was rotated around the magnetic field within the crystallographic $bc$-plane, with $I\bot B$ held constant so that only transverse MR was measured. In this configuration, at $\theta=0^{\circ}(90^{\circ})$, $B||[120]([001])$. Since the largest magnetic saturation field is below 2.5 T (see Fig. 2), the 9T magnetic field aligns the magnetic moment $M$ perpendicular to $I$ at all times.

The AMR, defined as [{$\rho_{xx}(\theta)-\rho_{xx}(\theta=0)\}/\rho_{xx}(\theta=0)]\times 100$, measured at 5 K is shown in fig. 3(c), with maximum value of $-5.6\%$, indicating a substantial effect. To gain further insight into this large AMR, we compared the AMR with the ab initio calculated AMR using the GGA+U method. For this purpose, we used the all-electron WIEN2k package [45], varying $U_{eff}=U-J$ from 1.2 to 2 eV. Assuming an isotropic transport scattering rate, the longitudinal conductivity $\sigma_{xx}\propto\omega_{pl}^{2}\propto\sum_{\mathbf{k}}{\delta(E-E_{\mathbf{k}})v_{\mathbf{k}x}^{2}}$, where $v$ is the Fermi velocity. We calculated this quantity using the $19\times 19\times 11$ zone-centered k-point mesh and tetrahedron numerical integration, and an otherwise default setup. We found (Fig. 4) that AMR is very sensitive to correlation effects. The best agreement with the experiment occurs at $U_{eff}=1.3$ eV, yielding a calculated value of $1.6$%, which is over three times smaller than the experimental result but remains the same order of magnitude. This discrepancy may indicate the presence of an anisotropic scattering rate, or an underestimation of spin-orbit coupling (SOC) in the calculation, potentially related to the known underestimation of the AHC in TbMn₆Sn₆ [12, 46].

Hall resistivity measured with with $B||[001]$ and the current $I||[100]$ ($\rho_{yx}$) at representative temperatures is shown in Fig. 5(a). The zero-field value of $\rho_{yx}$ corresponds to the anomalous Hall resistivity $\rho_{yx}^{A}$. In Fig. 5(b), we show the longitudinal Hall conductivity, $\sigma_{xx}$ = $1/\rho_{xx}$, and the absolute value of the anomalous Hall conductivity, $\sigma_{yx}^{A}$ = $-\rho_{yx}/\rho_{xx}$ (valid when $\rho_{yx}$ $\ll$ $\rho_{xx}^{2}$, and $\rho_{xx}$ = $\rho_{yy}$), as a function of temperature between 1.8 and 300 K. This clearly indicates that $\sigma_{yx}^{A}$ varies with temperature across the entire temperature range. The scaling of the $\sigma_{yx}^{A}$, using the relation presented in Eq. 2, is shown in Fig. 5(c). This scaling law effectively fits the high-temperature data, yielding the coefficients $a$, $d$, and $c$, as presented in Table 3 and compared to those from Tb166. The coefficient $a$, representing impurity scattering, is about an order of magnitude larger in Y166-Ga compared to Tb166, as expected due to increased disorder scattering in the doped sample. The intrinsic anomalous Hall conductivity of Y166-Ga (121 S/cm) is comparable to that of Tb166 (140 S/cm). Interestingly, the magnitude of $d$ is smaller in the Y166-Ga than in Tb166, consistent with the hypothesis that the term $d/\sigma_{xx}$ in Eq. 2 is due to spin fluctuations [12]. Notably, $d\neq 0$ in Y166-Ga despite the absence of highly fluctuating Tb atoms, likely because, as discussed in Section III.2, Mn in Y166-Ga fluctuates significantly more than in Tb166. This implies that the $d/\sigma_{xx}$ is essential in the anomalous Hall scaling of $R$Mn₆Sn₆ compounds due to the presence of Mermin-Wagner fluctuations [17] involving either Mn or $R$ atoms.

The similar magnitude of intrinsic AHC in Y166-Ga and Tb166 suggests that the intrinsic AHC in $R$166 compounds is a general property of ferrimagnetism rather than a result of exotic Chern physics, consistent with the conclusion in Ref. [12]. This interpretation is further supported by photoemission spectroscopy data from the parent compound Y166 [47], which shows no such topological feature above the Fermi energy, with Ga doping shifting the Fermi energy even lower.

To further validate this interpretation, we measured the Hall conductivity of Y166-Ga by applying an in-plane magnetic field, ensuring that any 2D Chern gap contributions would be excluded if present. While measuring this in-plane configuration for Tb166 would require an exceptionally high magnetic field due to magnetic saturation constraints at low temperatures [48, 12], Y166-Ga can be measured under standard lab conditions. For consistency with the $\rho_{yx}$ measurement in Fig. 5, we applied the current along the [100] direction and applied the in-plane field along [120]. The Hall resistivity, $\rho_{zx}$, measured across a range of temperatures from 1.8 K to 300 K, is shown in Fig. 6(a).

In Fig. 6(b), we plot $\sigma_{xx}$ = 1/$\rho_{xx}$ alongside the absolute value of the anomalous Hall conductivity, $\sigma_{zx}$ = $-\rho_{zx}/(\rho_{xx}\rho_{zz})$, which is valid when $\rho_{zx}$ $\ll$ $\rho_{xx}$ and $\rho_{zz}$. The scaling of $\sigma_{zx}$, following the relation in Eq. 2, is presented in Fig. 6(c), with the fitting coefficients $a$, $d$, and $c$ presented in Table 3. Notably, the intrinsic AHC contribution, represented by coefficient $c$ ($\sigma^{A}_{zx,int}$), is significantly larger than $\sigma^{A}_{yx,int}$, indicating that the AHC in Y166-Ga has a 3D nature and can be large without invoking the Chern physics. The coefficient $|d|$ is also larger in this measurement geometry, potentially pointing to enhanced spin-fluctuations, though this cannot be directly compared to Tb166 due to differing Hall geometries and remains a topic of future work on $R$166 compounds. However, it is noteworthy that the $d/\sigma_{xx}$ term is also necessary in this case. Nevertheless, our findings reveal that the AHC in Y166-Ga displays 3D characteristics and supports theoretical calculations that suggest the AHC arises from predominantly 3D bands [12], as further discussed in Section III.6.

The intrinsic AHC can be calculated by integrating the Berry curvature over the Brillouin zone (BZ) [49]:

where $f(E_{n\vec{k}})$ is the Fermi-Dirac distribution, $\Omega_{n,\alpha\beta}(\vec{k})$ is the contribution to the Berry curvature from state $n$, and $\alpha,\beta=\{x,y,z\}$.

A notable aspect of the calculation is the pronounced and rapid oscillation observed in the Berry curvature across the BZ, requiring a dense $k$ mesh to ensure convergence. To accelerate the calculations, we implemented Eq. 4 in our recently-developed tight-binding (TB) code [50] and carried out the AHC calculations in $R$Mn₆Sn₆ where $R$ = Tb and Y. A realistic TB Hamiltonian was constructed using the maximally localized Wannier functions (MLWFs) method [51, 52, 53] implemented in Wannier90 [54] after the self-consistent density-functional-theory calculations performed using Wien2k. A set of 118 Wannier functions (WFs) consisting of Y-$4d$ (or Tb-$5d$), Mn-$3d$, and Sn-$sp$ orbitals offers an effective representation of the electronic structure near the Fermi level ($E_{\text{F}}$). The self-consistent DFT calculations were carried out with out-of-plane magnetization in TbMn₆Sn₆ and both in- and out-of-plane magnetization in YMn₆Sn₆. For the in-plane YMn₆Sn₆ configuration, the moment is along lattice vector $\mathbf{a}$, as denoted in Fig.7. A dense $256^{3}$ $k$-point mesh is used for the AHC calculations in TB.

Figure 7 presents the unit cell of YMn₆Sn₆ used in our calculation. The Cartesian coordinate system is chosen so that the lattice vector $\mathbf{b}$ is along the $y$-axis, $\mathbf{c}$ is along the $z$-axis, and $\mathbf{a}$ is along the $-30^{\circ}$ direction off the $x$-axis. Lattice vectors $\mathbf{a}$ and $\mathbf{b}$ point along the nearest neighboring (NN) Mn-Mn bond direction. For the $\sigma_{xy}$ calculations discussed below, the first subscript $x$ denotes the current direction, and the second subscript $y$ denotes the Hall-field direction.

Figure 8 shows the AHC values calculated at $T=0$ K as functions of Fermi energy using Eq. 4. In the out-of-plane orientation of both TbMn₆Sn₆ and YMn₆Sn₆, only $\sigma_{xy}$ exhibits substantial values, while $\sigma_{yz}$ and $\sigma_{zx}$ remain negligible, as illustrated in the top and middle panels of Figure 8. Conversely, with in-plane magnetization in YMn₆Sn₆, both $\sigma_{yz}$ and $\sigma_{zx}$ demonstrate appreciable values, while $\sigma_{xy}$ remains negligible, as depicted in the bottom panel of Figure 8. The band-filling calculation indicates that when doping YMn₆Sn₆ by $0.09$ hole/Mn, corresponding to a Fermi energy shift of -0.037 eV, $\sigma_{xy}=77.62$ S/cm, and $\sigma_{zx}=36.76$ S/cm. While the former closely aligns with the experimental value, the latter is approximately 10 times smaller than the experiment. These comparison has to be taken with caution, as random substitution of 0.55 Ga likely affects the electronic structure beyond a simple rigid Fermi energy shift. Nevertheless, our theoretical calculations predict that the Hall transport in $R$166 is 3D, consistent with the experimental findings.