Predicting Job-Hopping Motive of Candidates Using Answers to Open-ended Interview Questions

Voluntary turnover, which represents the vast majority of all employee turnover, decreases organizational productivity and dampen employee morale [1] while inflicting direct financial costs related to rehiring such as sourcing, recruiting and onboarding. However making frequent voluntary job changes, known as job-hopping, has become a trend in the recent past [2]. The motivations for job-hopping, have been identified to be two-fold; advancement and escape [3]. The advancement motive represents the growth and career perspective, while the escape motive represents a withdrawal or dislike of the work environment, especially among those who are described as impulsive and unpredictable. The latter is identified as a psychological property and commonly known as the “hobo syndrome” [4]. Further studies have shown the relationship between personality and voluntary turnover [5, 6, 7, 8, 9].

The ability to assess a candidate’s motivation for job-hopping prior to selection can help both candidates and employers make better decisions and avoid future surprises and costs due to voluntary exits. The most frequently used approach for discovering patterns of job-hopping is to explore the employment history listed in an applicant’s resume. Sifting through resumes can be both time-consuming and unreliable, especially in situations of high volume recruitment. Resumes are also known to produce biased outcomes [10, 11]. Moreover, it is an ineffective method with novice job seekers, such as new graduates with insignificant job histories.

In this study, we examine whether answers given by candidates to interview questions related to past behaviour and situational judgement demostrate a correlation to their job-hopping motives as measured by the Job-Hopping Motives (JHM) Scale [3], a validated self-report measure of job-hopping motives. The basis for selecting interview answers as a possible predictor is two-fold. Firstly, one’s language use has been shown to be highly predictive of their personality. Personality traits have been successfully derived from informal (microblogs [12, 13, 14], social media posts [15, 16]), semiformal (blogs [17], interview questions [18]) and formal (essays [19]) contexts. Authors own prior work has shown that interview answers are a strong predictor of personality traits [18]. Secondly, structured interviews where the same questions are asked from every candidate in a controlled conversation flow and evaluated using a well-defined rubric have shown to reduce bias [20] and also increase the ability to predict future job performance [21]. Computational inference of job-hopping motive from interview responses further increases the utility of the structured interview and its applicability in high volume recruitment.

In this work, we make the following contributions to the crossroads of computational linguistics and organizational psychology.

1. 1.

   We demonstrate that responses to typical interview questions related to past behaviour and situational judgement can be used to reliably infer one’s job-hopping motive as measured by the JHM Scale.

2. 2.

   We evaluate multiple methods of text representations and establish that the Glove based word-embedding method achieves the highest correlation of r=0.35 between text and JHM Scale when used with a Random Forest regressor.

3. 3.

   We validate the positive correlation between job-hopping motive and Openness to experience (one of the personality traits in the HEXACO personality model), both derived from text (r=0.25). This is in line with previous findings using standard personality tests.

The rest of the paper is organised as follows. Section 2 presents a detailed background into the research on employee turnover, the role of personality on turnover and the link between language and personality. In section 3, we describe the methodology, including the data used and the five different text representation methods we evaluated, namely TF-IDF, LDA, Glove word embeddings, Doc2Vec document embeddings and LIWC. Results, in terms of the accuracies achieved by each text representation method, are presented in section 4 along with discussion and further analysis of salient correlations, demographics, and terms used. Section 5 concludes the paper with a summary and future research directions.

A study conducted by the Australian HR Institute in 2018 across all major industry sectors in Australia [22] found that on average companies face an annual turnover rate of 18% and within the age group of 18 to 35 it jumps to 37%. That is more than 1 in 3 people in the youngest age group leaving an organization within a year. A majority (63%) of respondents in the study, mostly HR staff, claimed that their organisation does not measure the financial cost of employee turnover. Employee turnover rate is much higher than the average for some industries such as hospitality [23, 1]. Cho et al. [23] report a staggering 115% turnover rate among non-managerial employees of hospitality firms while for managerial employees it is 35%.

Majority of employee turnover consists of voluntary turnover, that is, employee-initiated separations compared to involuntary turnover initiated by the employer such as layoffs and terminations due to poor performance. Significant costs have to be borne by an organization when an employee voluntarily leaves. These include replacement costs such as costs associated with advertising, screening and selecting a new candidate, employee training costs, and operational efficiency losses until a new employee reaches a sufficient level of productivity. These costs are exacerbated by the dampening of remaining employee morale leading to lower quality work and lower productivity [1]. A study conducted by the Work Institute in the US [24] found that voluntary exit of an employee costs a company 33% of the employee’s base salary, which the authors claim is a conservative estimate. The report also states that with a median base salary of $45,000, it is costing the US economy close to $600bn a year due to voluntary turnover. Similarly, in a large-scale meta-analysis (N $>$ 300,000), Park and Shaw [25] observed a significant and negative ($\rho=-0.15$) correlation between voluntary turnover rates on organizational performance.

Voluntary turnover has been associated with a number of negative job attributes such as low level of job satisfaction, lack of promotion opportunities, lack of work-life balance, lack of fairness of the firm’s procedures etc. [26], which are reasons originating from a misalignment between employee and employer expectations. Measures can be put in place by the employer to discover and address these issues where possible and methods such as employee engagement surveys and periodic review discussions serve this purpose.

The focus of our work is a type of voluntary turnover identified as “job hopping”, the frequent move from job to job [3] by some individuals than others. Studies have shown that there is a relationship between one’s personality and job-hopping motives [5, 6, 27, 7, 8, 9, 28, 29]. The hypothesis behind these studies is that one’s personality plays a key role in their intention to hop jobs and acts as a latent variable that mediates their desire to voluntarily leave an organization. Barton and Cattell [5] conducted one of the earliest longitudinal studies on the effect of personality on job promotion and job change and found that individuals who were more practical and down to earth recorded the lowest incidents on job turnover. Ghiselli [4] named this tendency the “hobo syndrome”, an internal impulse-driven action to move from one job to another shown by some employees irrespective of other more rational motives. Lake et al [3] refer to this as an “escape motive” or a sudden withdrawal from the work environment as opposed to an “advancement motive” that makes an employee leave for a perceived better opportunity. More recent studies have also shown a correlation between hobo syndrome and the Big 5 personality trait of Openness to experience [30] further validating the personality influence on job-hopping motives. Other studies have shown similar relationships between the Big-5 personality traits and turnover intention. For instance, Sarwar et at. [29] found personality traits Extraversion, Neuroticism, Conscientiousness and Agreeableness to be negatively correlated with turnover intention while Openness to experience was positively related with turnover intention. These results are in line with the Big-5 personality traits’ correlations with the intention to quit observed in a meta-analysis conducted by Zimmerman [9]. In a study conducted with call centre employees, Timmerman [8] reported slightly different results with Neuroticism, Conscientiousness and Agreeableness as negatively correlated with turnover while Extraversion and Openness to experience positively correlated with turnover. With the above studies, it is pertinent to conclude that overall there is a latent relationship between one’s personality and his/her turnover intentions.

Individual job-hopping motive can be measured using self-rating questionnaires similar to standard personality tests. One such validated self-rating item list is the Job-Hopping Motives Scale developed by Lake, Highhouse and Shrift [3]. It includes eight self-rating items with four items each validated with factor analysis to assess escape and advancement motives. Their study also confirms the positive correlation of escape motive with impulsivity (r=0.19) and a negative correlation with persistence (r=-0.16). They found the number of jobs the participants had voluntarily quit during their lifetime to be significantly related to both escape and advancement motives (r=0.08, p<0.05 and r=0.10, p<0.01). They also found the ratio of voluntarily quit jobs relative to the total number of jobs held over one’s life to be significantly related to both motives (r=0.11, p<0.01 and r=0.09, p<0.05). Further, assessing whether job-hopping motives could predict work history variables above and beyond established predictors, they found escape motive was significant with $\beta$=0.14 (p=0.02) in two separate regression models while the advancement motive was not.

However a challenge with administering self-rating based assessments is the need to have multiple statements to gain a measure of a single personality construct (for example four items are required to measure escape motive). It is not unusual to have over 100 items in such tests when you combine other measures such as personality traits. Applicant reactions to such personality tests have shown to be less favourable than interviews [31, 32]. Based on a meta-analysis of multiple studies on applicant reaction to selection methods, Anderson et al. [31] found that compared to job interviews and work sample tests, personality tests fall short of making a positive impression with candidates in areas of face validity, opportunity to perform, interpersonal warmth and respectful of privacy. These indicate candidates’ preference to express themselves and not be restricted to self-rating themselves on a pre-defined set of multiple-choice questions (typically over 100 items) as found in standard personality tests.

On the other hand, researchers have demonstrated that one’s language use is indicative of his/her personality attributes [33, 34, 35, 36, 37]. This allows structured interviews, which are much more engaging for applicants and permit open expression of applicant thoughts to be used as a source for inferring personality attributes. Authors have demonstrated in [18] how responses to open-ended interview questions can be used to reliably infer one’s personality attributes based on the six-factor HEXACO model [38].

Combining the relationship between job-hopping motives and personality with that of personality and language use, we hypothesised that one’s language use is closely associated with their job-hopping motives. In other words, the the job-hopping motive can be inferred from various characteristics of their language use. Based on this hypothesis, we envisioned building a language-based model that is able to predict a candidate’s job-hopping motives as measured by the JHM Scale from answers to regular open-ended interview questions. Such a model would have wide applicability in digitised forms of interviews, be it chat-based, voice or video where the textual content of the candidate answers can be used to infer JHM Scale scores in addition to the personality and communication skills that can be derived from text. This enables the use of digitised interviews, which are much more engaging than standard personality tests and preferred by candidates to be used as a multi-measure assessment scalable to high applicant volumes supported by algorithmic inferences.

In order to test the correlation between language use and job-hopping motive, we built a regression model to infer the JHM Scale rating (discussed above) using textual answers to open-ended interview questions. Given the importance of numerical representation of language in building a machine learning model, we compared the performance of five different text representation methods namely, terms (TF-IDF), topics (LDA), Glove word embeddings, Doc2Vec and LIWC. In this section, we describe the training dataset, the five different text representation methods and the regression model building approach.

We evaluated the trained ‘text to JHM Scale’ regression models on the remaining 20% of the data. The models are evaluated on the correlation coefficient between the actual JHM Scale score and the score predicted by the trained model. Figure 2 shows the accuracies in terms of correlation across different minimum response lengths and language modelling approaches.

Language use representation using Glove word embeddings with minimum response length of 150 words achieved the highest correlation of r=0.35. It is important to note that six of the correlations fell above r=0.3, typically considered as a correlational upper-limit in personality research when predicting behaviour [48]. It is also important to note that apart from the correlations for LIWC with minimum text length of 50, 100 and 150 all other correlations had a $p<0.001$. These results indicate that all open-vocabulary approaches across all minimum lengths recorded significant correlations, demonstrating that the language one uses in responding to typical interview questions are predictive of their job-hopping motive.

Overall, word embeddings based models recorded the highest corrections across all minimum lengths analysed. This is specifically important given that word embeddings based models are more generalizable to unseen words compared to models based on TF-IDF and LDA, which are limited to the vocabulary seen in the training corpus. The generalizability comes from the content used in training the word embedding model; In our case, the word embeddings used were trained on very generic content from web pages crawled from the Internet. Compared to the superior results achieved by word embedding based models, document embedding based models fell short of in terms of the accuracy. This, we believe, is due to the nature of the content used to train the document embedding model. We used content from Wikipedia to train our document embedding model and differences in actual content and writing style between Wikipedia and candidate responses may have contributed to the degraded performance of the document embedding based models. Further research is required to validate this and Doc2Vec models trained on other content such as tweets are options to consider.

Overall, minimum response length of 50 words achieved the weakest results confirming our hypothesis that responses that are too short might not have enough textual content to predict the candidates’ job-hopping motive. However, none of the models showed an increase in accuracy but a decrease or maintaining the same accuracy (with the exception of LIWC based model), when moving to a minimum length of 200 words. Given LIWC depends on counts of words related to pre-defined categories, we assume that more words in a response raise the possibility of finding more LIWC classified words in the response. However, the overall poor performance of LIWC highlights the limitations of closed-vocabulary approaches where a tediously developed lexicon is less effective in generalizing to unseen words.

Following sections describe some of the further analysis we performed on the best ‘text to JHM Scale’ model (min. response length=150, using word embeddings features) to get a deeper understanding of the model’s behaviour. These analyses were carried out on the remaining 20% of the data that were left out of training the regression model. The gender and role family composition of the this test data set can be found Tables 4 and 5.

Frequent movement from job to job or “job-hopping” as its commonly known is found to be associated with one’s personality. In this paper, we presented a novel approach to predicting job-hopping motive as measured by the Job-Hopping Motives (JHM) Scale using answers to typical interview questions related to past behaviour and situational judgement. Using data from over 45,000 individuals who answered open-ended interview questions and self-rated themselves using the JHM Scale, we built a regression model to infer the JHM Scale score. We compared the performance of four open-vocabulary text representation methods (namely terms, topics, word embeddings and document embeddings) and one closed-vocabulary method (LIWC). The Glove word embedding based model achieved the highest correlation of r=0.35 ($p<0.001$) between interview response text and the JHM Scale score. All other open-vocabulary representations achieved correlations above 0.25 ($p<0.001$), highlighting a statistically significant positive correlation between interview responses and job-hopping motive. We further demonstrated that one’s job-hopping motive is positively correlated (r=0.25) with the trait Openness to experience and negatively correlated with Agreeableness (r=-0.09) as found in the six-factor HEXACO personality model. In other words, the more open someone is for new experiences and less lenient with views of others, the more likely he/she will show job-hopping motivation.

We find the above outcome to be significant in at least two ways. It provides an alternative to resume based job histories as a source for inferring a job-applicants tendency to job hop. Resumes are known to induce bias in the hiring process and especially ineffective with newcomers to the job market with no significant prior job history. Secondly, the ability to infer job-hopping motive computationally from interview responses uplifts the utility of the interview as a multi-measure assessment that can be conducted digitally (e.g.text chat, video) at scale and cost-effectively giving every candidate an opportunity to express themselves. Interview as an assessment is preferred by applicants over traditional assessments such as personality tests.

Further work is required in assessing the predictive validity of the outcome, i.e. establishing the correlation between the inferred job-hopping motive scale score and actual job-hopping behaviour. This requires a longitudinal study or following the career journey of an applicant sample. In our current study, we used only the semantic level features (terms, topics etc). Exploring whether other types of features can further increase the accuracy, is another useful future extension. These can include the use of parts of speech (POS), use of emojis and multi-modal information such as audio and video signals captured while candidates answer the questions. Exploring the performance of other available regression algorithms, including neural network approaches, and using more advanced language representations such as BERT [54], may help increase the accuracy of the regression model further.

Both authors are employed at PredictiveHire, the creator of the FirstInterview product that was used to collect the data for the research.