Vacuum stability in the Standard Model with vector-like fermions

In the SM, interactions with the Higgs field are specified by the Higgs potential, which has the form

with $\kappa$ and $\lambda$ the quadratic and quartic couplings, $\Delta V$ the tree-level correction terms, and $\Phi$ the Higgs field given by

where $v=246$ GeV is the Higgs vacuum expectation value (VEV). Vacuum stability requires $V^{\prime}(\phi)>0$, or equivalently, $\lambda(\mu)>0$, with $\lambda$ the running Higgs coupling which depends on the scale, $\mu$, at which the SM breaks down. The issue is that the Higgs quadratic self-coupling is renormalized not only by itself ($\lambda$ increasing as the energy scale increases), but also by the Higgs (Yukawa) coupling to the top quark, which tends to drive it to smaller, even negative values at high scales $\mu$. At leading orders Altarelli:1994rb

yielding an estimate for the energy scale where $\lambda$ will become negative Altarelli:1994rb

with energy scales and masses measured in GeV. Taking into account all the uncertainties in the measurements of $m_{H},m_{t}$ and $\alpha_{s}$, the scale at which the SM fails is $\mu=10^{(9.4\pm 1.1)}$ GeV. Possible remedies include higher-dimensional operators in the effective theory, or additional symmetries (such as supersymmetry, which is associated to a new scale), or addition of new particles to the SM.

More explicitly, the variation of the coupling $\lambda$ with the energy scale involves evaluation of the one-loop beta function describing the running of $\lambda$ is Altarelli:1994rb

with $g_{1},g_{2},g_{3}$ the coupling constants for $U(1)_{Y},SU(2)_{L},SU(3)_{c}$ and $y_{t}$ the Yukawa coupling of the top quark. The renormalization group equations for these parameters are Tang:2013bz :

with initial conditions

Here $\alpha,\alpha_{s}$ are the weak and strong coupling constants, $\sin\theta_{W}=0.2312$ is the Weinberg angle, and we set $\mu_{0}=m_{Z}=91.188$ GeV. The radiative decay constant is Sirlin:1985ux

with $\xi=m_{H}^{2}/m_{Z}^{2}$, $G_{F}=1.16635\times 10^{-5}$ GeV^-2 and

with

where $c,s$ are abbreviations for $\cos\theta_{W},\,\sin\theta_{W}$.

Examining the running of the coupling parameters in this mode shows that the scalar couplings increase with increasing energy scales, while the Higgs coupling $\lambda$ decreases and becomes negative at around $10^{10}$ GeV.

In this section, we consider the simplest remedy to the stability problem by extending the particle content of the SM by an extra (singlet, as it is simplest) scalar boson which interacts solely with the SM Higgs, and we examine the constraints placed on its mass and mixing angle with the existing Higgs boson by the Higgs vacuum stability condition. The addition of a boson provides a positive boost to the coupling parameter, counteracting the effect of the top quark and contributing towards repairing the Higgs vacuum stability.

In this scenario, the Higgs doublet $\Phi$ from Eq. 2 interacts with the new scalar singlet $\chi$

where $u$ is the singlet VEV, through

After symmetry breaking, the singlet and doublet Higgs mix

yielding mass eigenvalues:

and the eigenvectors

As in the SM, we require $\lambda_{H}>0$ for a stable SM vacuum, and $\lambda_{S}>0$ for the new particle. In addition we impose that the potential is positive for asymptotically large values of the fields,

The Yukawa couplings can be expressed in terms of the physical masses as:

with $m_{H},\,m_{S}~{}(v,u)$ the masses (VEVs) of the physical fields, respectively and $\varphi$ their mixing angle. Requiring perturbativity up to Planck scales, we apply the Yukawa and Higgs sector RGEs:

Here $\lambda_{H}$ and $\lambda_{S}$ are the quartic self-couplings of $\Phi$ and $\chi$, and $\lambda_{SH}$ the coupling describing their mixing. Eqs. II.2 describe the coupling parameters at relatively small energy scales, and therefore serve as initial conditions to these RGEs.
Just as in the SM, we ignore the contributions of all Yukawa couplings except for that of the top quark, and also, we include electroweak radiative correction terms for increased accuracy. To this end, we replace the top Yukawa coupling and Higgs self-coupling boundary conditions with Hempfling:1994ar

where $\Delta_{H}(\mu)$ is the same correction as in the SM, and

with

and we include the RGEs for the gauge couplings as in the SM, Eq. 6.

Fig. 1 illustrates the running of the coupling parameters for a typical set of parameter values. Notice that in this model, the scalar couplings increase with increasing energy scales, compensating for the SM Higgs coupling, which becomes negative at around $10^{10}$ GeV. Therefore, the addition of an extra scalar boson to the SM rescues the theory from vacuum instability.

Of course, we may investigate the mass and mixing angle of this singlet scalar with the Higgs boson by eliminating all parameter values that do not satisfy Higgs vacuum stability. For this we perform a scan over a broad parameter space and disallow all parameter values which do not satisfy the vacuum stability conditions outlined in Eq. 20. The resulting allowed parameter space is illustrated in Fig. 2.

Note that, while the mass region allowed for the additional boson for $u=1$ TeV is quite restricted, for larger VEVs it is quite large, and increasing with the new boson VEV. However, in all cases, the mixing with the SM Higgs boson is required to be non-zero ($\varphi\neq 0$), consistent with the observation of Higgs potential instability in the absence of the additional boson.

Using the stable potential in the previous section, we investigate the effect of introducing vector-like fermions into this SM + additional scalar model. Unlike SM-like (chiral) fermions which act as doublets under $SU(2)_{L}$ if left-handed and as singlets if right-handed, and spoil the agreement of loop-induced production and decays of the SM Higgs, vector-like fermions have the same interactions regardless of chirality. They appear in many new physics models, such as models with extra dimensions, and many explanations put forward of deviations from SM physics include vector-like fermions.Thus it is important that, when considering the addition of a vector-like fermions to the SM, the presence of a new scalar boson is essential to ensure the stability of the Higgs potential, otherwise, as the fermions decrease the effective self coupling, the singular divergence of the Higgs quartic coupling would worsen compared to the one in the SM. As before we require the Higgs sector potential to be positive at asymptotically large values of the fields, up to Planck scale. The question we need to address is: what are the constraints on the masses of the vector-like fermions, and mixing angles with ordinary fermions, such as to maintain vacuum stability.

The new states interact with the Higgs bosons through Yukawa interactions. The allowed anomaly-free multiplet states for the vector-like quarks, together with their nomenclature, are listed in Table 1 Carmi:2012yp ; Ellis:2014dza ; Aguilar-Saavedra:2013qpa . The first two representations are $U$-like and $D$-like singlets, the next three are doublets (one SM-like, two non-SM like), and the last two are triplets. The various representations are distinguished by their $SU(2)_{L}$ and hypercharge numbers.

In these representations, the Yukawa and the relevant interaction terms between the vector-like quarks and SM quarks are

After spontaneous symmetry breaking, the Yukawa interactions generate mixing between the SM quarks and the vector-like quarks at tree level. The singlet vector-like quark and the triplet vector-like quark exhibit similar mixing patterns, while the doublet vector-like quarks have a different mixing pattern. To avoid conflicts with low energy experimental data (flavor changing neutral interactions), we limit the vector-like quarks mixing with the third generation of SM quarks only. This is reasonable also because of the large mass gap between vector-like fermions and the first two generations of quarks. The mixing patterns will be described below.

The gauge eigenstate fields resulting from the mixing can be written in general as,

The mass eigenstate fields are denoted as $(t,_{1},t_{2})$ and $(b_{1},b_{2})$ and they are found through bi-unitary transformations,

where

In the following we abbreviate $\cos\theta_{L}^{t}\equiv c_{L}^{t}$,…. Through these rotations we obtain the diagonal mass matrices

the eigenvectors now become, for instance for the top sector

with eigenvectors

where the mixing angles are

and similarly for the $b$-quark sector, with the replacement $t\to b$ and $\theta\to\beta$. Note that, because of their charge assignments, the $X$ and $Y$ fields do not mix with the other fermions and are therefore also mass eigenstates.

Relationships between mixing angles and mass eigenstates depend on the representation Aguilar-Saavedra:2013qpa ; Chen:2017hak .

and where

For doublet models, while the Higgs mixing is the same as in the previous section, the mixing between the top quark $t$ and the new vector-like singlet $T$, characterized by the mixing $\theta_{L}$, results in the shift in the Yukawa couplings as follows

with $x_{b}=m_{b}/m_{B}$, and as before $x_{t}=m_{t}/m_{T}$. We use these expressions as initial conditions to the RGE equations, Eq. VII.1.3. We review the mass bounds on vector-like quarks, then proceed with our numerical analysis in Sec. III.3.

Searches for vector-like quarks have been performed at the LHC and various limits exist ATLAS-CONF-2016-101 , all obtained assuming specific decay mechanisms. The Run 2 results from the LHC have improved previous limits from Run 1 by about 500 GeV moriond2018 . All measurements assume top and down-type vector-like quarks to decay into one of the final states involving $Z$, $W$ or Higgs bosons with 100% branching ratios. So far, lower limits around 800 GeV have been obtained. The most recent search at ATLAS obtains, with 95% C.L., lower limits on the $T$ mass of 870 GeV (890 GeV) for the singlet model, 1.05 TeV (1.06 TeV) for the doublet model, and 1.16 TeV (1.17 TeV) for the pure $Zt$ decay mode quark Aaboud:2017qpr. The current experimental lower bound on the mass of the down-type vector-like quark which mixes only with the third generation quark is around 730 GeV from Run 2 of the LHC and around 900 GeV from Run 1. The current lower bound for a vector-like quark which mixes with the light quarks is around 760 GeV from Run 1. In our analyses, we set a lower limit on all masses of 800 GeV, to allow for the consideration of the largest parameter space.

We proceed with analyzing the representations in turn, showing the effects of the additional fermions on the RGEs, and the mass and mixing angles constraints on the fermions and additional boson for each. All the relevant RGE for the Yukawa couplings, couplings between the bosons, and gauge coupling constants are given in the Appendix VII.

The evolution of the RGE’s under different vector-like fermion representations are illustrated in Fig. 3 for different values of the VEVs of the new scalar field ($u=1,2,4$ TeV). In the case of $u=1$ TeV, we have taken the mass of the scalar boson to be 0.8 TeV, because for $m_{S}=1$ TeV, the Higgs sector couplings diverge, leading to singularities, whereas in the case of $u=2$ TeV and $u=4$ TeV, we chose $m_{S}=1$ TeV because the smaller mass of 0.8 TeV is not large enough to ensure a positive Higgs quartic coupling.

As required, all of the Higgs sector couplings remain positive up to Planck scale. As expected, the fermion Yukawa couplings tend to decrease with increasing energy, while the scalar bosonic couplings tend to increase. As we discussed previously, the addition of extra scalar bosons to the Standard Model helps maintain a positive Higgs self-coupling at larger energy scales, while the addition of extra fermions only aids in lowering it further. A common trend with respect to the models is that the Yukawa couplings are generally negatively affected by added loops at higher energy scales, while the Higgs sector couplings are generally affected positively (they tend to increase with increasing energy).

The obvious exception here is the SM Higgs coupling, which strays dangerously close to zero at high energy scales, and even becomes negative for the additional singlet vector-like fermion case. The models that augment the scalar boson by vector-like representations vary significantly among each other in predictions for the various couplings with the scalar VEV $u$. Note in particular that for the the first case, for the singlet vector-like case, where $u=1$, the Higgs couplings increase, and $\lambda_{H}$ becomes negative at $\mu\sim 10^{11}$, rendering the theory unstable, while if the additional scalar VEV is increased to $u=2,$ or $4$ TeV, the problem is ameliorated. The same problem recurs for the doublet and triplet models (not shown), but the theory is safe for $u=2$ and $4$ TeV. Differences in the running RGE’s are more pronounced for $\lambda_{S}$, the new boson self-coupling, and negligible for the others. Note in particular, the difference between the values in Fig. 3 and 1. For the doublet and triplet vector-like fermions, the RGE evolutions are similar, and the Higgs self-coupling remains positive at all energies. The additional scalar quartic coupling does not lie close to the origin as its interactions with fermions are small. There are some variations among models in the new scalar coupling, and the one describing the mixing with the SM Higgs. We have put less emphasis on the vacuum stability bound for the additional scalar, since its mass and VEV are unknown, and thus limiting concrete information from a detailed study of its vacuum stability bound.

Imposing the same conditions on the positivity of the potential as in Eq. 20, we study the allowed masses and mixing angles corresponding to each vector-like fermion representation. In Figs. 4 and 5 we perform a scan over random values of the relevant vector-like quarks between 300 and 2200 GeV, and for the mixing angles $\sin\beta_{L}$ (in the bottom sector) and $\sin\theta_{L}$ (in the top sector) between -1 and 1. The allowed values of the scalar mass $m_{S}$ are plotted against the mixing angle in the scalar sector, $\sin\varphi$ for different values of the expectation values $u$ (1, 2 and 4 TeV), providing an illustration of the possible quantitative properties of the scalar boson in this model. The results are given for all models. In Fig. 4 we plot the scans for singlet vector-like $T$ (top row), singlet vector-like $B$ (second row), $(T,B)$ doublet (third row), $(X,T)$ doublet (fourth row), $(B,Y)$ doublet (bottom row). And in Fig. 5 we consider the $(X,T,B)$ triplet (top row), and $(T,B,Y)$ triplet (bottom row), providing an illustration of the possible quantitative properties of the scalar boson in these models. We remark from Figs. 4 and 5 that just as in the SM extension containing only an extra scalar boson, considered in the previous section, mass mixing between the two scalar bosons is always required, and this mixing is significant, $\sin\varphi\geq 0.2$. Also, consistent with previous discussions, increasing the VEV $u$ enlarges the parameter space, which is now quite restricted for $u=1$ TeV. As expected, the addition of vector-like fermions worsens the stability of the potential, but larger VEVs (mass scales) survive. The mixing in the singlet ${\cal U}_{1}$ model is the most effective counter-term addition, in fact pretty much ruling out the scenario where $u=1$ TeV (unless the additional scalar is light, 600-1000 GeV), while the other representations provide much milder bounds for the same VEV.

We now investigate the restrictions on the vector-like fermion masses and mixing from requiring the stability of the Higgs potential. We concentrate first on the vector-like $T$, which has the same change as the top quark, and which, through mixing can affect changes in the Higgs potential, both in the fermion and in the scalar sector. In order to investigate this, we perform the same scan over random values of $m_{S}$ and $M_{T}$ between 300 and 2200 GeV, and for the mixing angles $\sin\varphi$ and $\sin\theta_{L}$ between -1 and 1, and show the results in Fig. 6. The first row shows the results for the singlet $T$ vector-like quark, the second row shows the results for the doublet vector-like $(T,B)$, the third for the $(X,T)$ doublet, the fourth for the $(X,T,B)$ triplet and the last for the $(T,B,Y)$ triplet. Unlike the case of scalar mixing, here the mixing between the top quark and the vector-like one is required to be small, in general, for most cases, in the region $\sin\theta_{L}\in(-0.2,0.2)$ (with some exceptions, where it can be larger, discussed below), and it can be zero. The allowed masses of the $T$ quark are restricted for the scalar VEV $u=1$ TeV, and increase with increasing VEVs, so that in the singlet $T$ and doublet $(X,T)$ models cases, practically no $T$ masses are allowed for $u=1$ TeV, while masses up to 1400 GeV are allowed for $u=4$ TeV. For the SM like doublet $(T,B)$, for $u=1$ TeV, $m_{T}\leq 800$ GeV, for $u=2$ TeV, $m_{T}\leq 1600$ GeV, while for $u=4$ TeV, $m_{T}$ spans the whole axis. Note that here, like in the scalar sector, there are marked differences between the scenarios. For the doublet $(X,T)$, any mixing between the $T$ and $t$ quark is allowed. We expect this case to be somewhat similar to the singlet, however, the Yukawa coupling of the $X$ quark lowers the Higgs coupling further, the parameter space continues to be severely constrained, and the mass is allowed in a narrow region near $m_{T}=1$ TeV for all values of the additional singlet. Here, as an exception to small mixing, the constraints on the mixing with the top are lifted. The case with triplets $(X,T,B)$, affected by both the $X$ and $B$ vector-like quarks, exhibits a behaviour independent of the singlet VEV. Masses again are favoured to be near $m_{T}=1$ TeV (we rule out light masses, $\sim 500$ from direct searches) and the mixing is allowed to be small or large. The strong enhancements are for the cases where the $t$ and $T$ mix. The mixing is expected to be stronger than between $B$ and $b$ quarks, due to the differences between mass of the top and of the bottom (making the denominator in Eq. III.1 smaller). It is interesting to note here the effect of the $X$ vector-like quark, which, while not mixing with SM quarks, is nevertheless important for the mass of the $T$ vector-like quark (seen clearly if we compare the singlet $T$ model with the doublet $(X,T)$, and the doublet $(T,B)$ model with the triplet $(X,T,B)$).

The scans in Fig. 7 illustrate the allowed masses and mixing angles of the $B$ vector-like quark with the bottom quark for the SM augmented by the additional scalar. We show, in the top panel, the vector-like singlet $B$ model, in the second panel, the vector-like $(T,B)$ model, in the third panel, the vector-like $(B,Y)$ model, in the fourth panel, in the vector-like $(X,T,B)$ triplet, and in the bottom panel, the $(T,B,Y)$ triplet. We again perform the same scan over the $m_{B}$ and $m_{S}$ masses and mixing angles $\sin\beta_{L}$ constrained by vacuum stability requirement, and plot the resulting $m_{B}$ against the mixing the $b$-sector $\sin\beta_{L}$. The effect of the $B$ quark is markedly different from that of the $T$ quark due to weaker constraints on its angle (the denominator in $\tan\beta_{L}$ is larger than $\tan\theta_{L}$ ). For the $B$ singlet model, the mixing and mass range are restricted, especially for $u=1$ TeV, while for the $(T,B)$ model the mass restrictions are lifted, but the mixing limits still remain. For the $(B,Y)$ doublet and for the $(T,B,Y)$ triplet model, the situation is very similar to that of the $T$ mass and mixing in these models: the mixing is restricted everywhere except around 1000 GeV, and this result is independent on the scalar VEV.

Finally, we investigate constraints on the vector-like fermions with non-SM like hypercharge, $X$, with charge 5/3, and $Y$, with charge -4/3. As the additional vector-like quarks $X$ and $Y$ do not mix with SM particle, a plot of mass against the mixing angle does not make sense, Instead, in Fig. 8, the allowed values of the scanned fermion mass $m_{X}$ is plotted against $m_{T}$, and $m_{Y}$ is correlated with $m_{B}$. Note that in the $(X,T)$ quark doublet, the $X$ and $T$ masses are strongly correlated (as seen from the third row of Fig. 6) and the expected $m_{X}$ is required to be close to $1000$ GeV regardless of $m_{T}$ values. We see that, similarly, in the $(B,Y)$ doublet model, $m_{Y}$ must have an allowed mass of approximately 1000 GeV, regardless of $m_{B}$, or the VEV $u$, unless both $m_{X}$ and $m_{Y}$ would be much lighter (100-200 GeV) in agreement with our earlier results. This seems to severely constrain models with vector-like quarks with exotic hypercharges.

For completeness, all the relevant RGE for the Yukawa couplings, couplings between the bosons and coupling constants are included in the Appendix VII.

Rewriting Eq. IV explicitly in terms of the scalar loop contributions to the gauge boson two point functions