Validating an Instrument for Teachers’ Acceptance of Artificial Intelligence in Education

Although measuring TAAI is an important topic and many studies have reported preliminary findings, the field is suffering from a lack of instruments with robust evidence of psychometric properties. Insufficient psychometric evidence will undermine the methodological rigor of the instrument, in turn significantly compromising the validity of the findings. To obtain a high-quality instrument, researchers must provide psychometric evidence in various aspects to ensure validity.

A valid instrument needs to be supported by validity evidence, including content, construct, convergent, and discriminant validity. To address the content validity, the developed instrument needs to be put through expert review, pilot test, and cognitive interview; otherwise, the instrument is in lacks evidence of face and content validity [4, 16]. We have rarely seen research documenting sufficient content validity evidence. With regard to construct validity, studies generally use exploratory factor analysis or confirmatory factor analysis, reporting the fit indices and factor loading to show the factor structure [15]. To maintain a high-level construct validity, the number of items in one dimension should be no less than three; otherwise, it could undermine the rigor of the measurement [17]. Furthermore, the presence of items within the same dimension with similar meanings has to be cautious. Overly similar items may be redundant and should be reduced. For example, instruments measuring technology acceptance have integrated “anxiety” as one of the dimensions within TAM and relevant models [14, 16, 15]. However, the dimension used for measuring anxiety had a small number of items focused on similarly negative feelings that people have with AI, resulting in some redundancy of the instrument. Such item design may yield better reliability results, but it reduces the information that could be obtained from the instrument. Additionally, it can confuse subjects when answering items with similar meanings.

Moreover, convergent and discriminant validity, as well as item discrimination, matter the instrument’s psychometrical quality. Convergent validity refers to whether items within the same dimension are constructed to represent most of their constructs and eventually converge in the same dimension, while discriminant validity means that each item should be strongly related to its own dimension and weakly related to other dimensions [18]. The two aspects of validity demand a clear conceptual framework for the instrument. Item discrimination indicates the discriminative power of items to distinguish between respondents with higher and lower scores. To maintain item descriptive, the developed items should be accurate and understandable to the respondents without confusing wording that can impact the appropriate response of subjects [19].

However, few studies have taken all the aspects of validity into consideration, leaving the existing instruments of TAAI without robust validity evidence. Other studies that developed instruments to assess teachers’ AI affection with sufficient validity evidence, but fall out of TAAI. For example, [20] developed an instrument to measure pre-service teachers’ AI perception based on the planned behavior theory. They provided evidence about construct reliability, convergent validity, and discriminant validity. However, the instrument is about measuring teachers’ perception of learning AI rather than TAAI, which this study focuses on. To sum up, there are limited well-validated instruments to measure TAAI in education to date. The field needs instruments with psychometric quality evidence specifically to measure teachers’ AI acceptance in education.

Participants’ responses would be impacted by their knowledge and understanding of the objects being measured. That is, teachers’ responses to AI acceptance can be impacted by their knowledge and understanding of AI. Without realizing this idea, TAAI instruments may yield invalid conclusions.

As an emerging technology, AI has been widely used in a variety of applications in our daily lives. However, many people are unaware of the conceptions of AI and unconscious of AI applications, let alone its potential application in education. A survey, conducted in 2022 encompassing 11,004 U.S. adults aimed at gauging people’s awareness of AI applications in daily life found that although 90% of the participants have heard about the term AI, merely one-third reported having extensive knowledge about AI and can correctly identify all the uses of AI provided in the survey (Pew Research Center, 2023). Similarly, the British Office for National Statistics released an article about public awareness, opinions, and expectations about AI and reported that only 17% of adults could identify AI usage in daily life [21]. The findings resonate with a recent study reporting a range of misunderstandings of AI [22]. Even though AI has been increasingly introduced into educational contexts, [23] found that many pre-service teachers were unaware of AI in their learning. Similarly, many teachers face the same problem of being unfamiliar with AI concepts and its various applications in daily life and education. Without knowing AI concepts and its various applications, teachers might provide incorrect or biased responses when reporting their acceptance and attitude toward AI in education.

The problem can be partly alleviated by offering stimuli, such as reading materials or video clips at the very beginning of the survey. The approach is considered to be useful in helping participants recall their memories of using technology and get a better understanding of the technology, thus eliciting more accurate responses during the test [4, 24, 25]. Studies on the acceptance of technology that involve people with limited knowledge and familiarity with the specific technology have used this methodology [4]. For example, to evaluate children’s perceptions of conversational agents (CA), [25] allowed children to interact with the CA during the study, providing them with proximal and in-person experiences to better elicit their perceptions. Kim et al. [24] provided reading articles to students when investigating their perceptions of AI teaching assistants, given these assistants are relatively new and have not yet been widely implemented. [4] provided reading material on their educational artificial intelligence tools (EAIT) to support teachers’ understanding of the concept of EAIT before evaluating their acceptance of EAIT. Additionally, [17] provided images of conversations with chatbots before surveying teachers’ attitudes towards chatbots in education. To be noted, as an emerging technology, AI has not been widely spread and integrated in classrooms, thus limiting teachers’ knowledge of AI in education. Therefore, providing stimuli is critical for obtaining reliable results when measuring TAAI. However, apart from the studies presented above, providing stimuli before the scale has not been generally seen in recent studies concerning TAAI [15].

AI shares common features with prior learning technologies while maintaining its distinct characteristics, such as stimulating human-like cognitive functions [33], presenting teachers with excitement and unique challenges as they embrace AI in teaching and learning settings. These challenges, in particular, are stressed by the automaticity, accessibility, and functionality of AI [34], which substantially impact teachers’ willingness to adopt AI in teaching and learning.

AI seldom acts in isolation in the classroom; instead, it is integrated with other technologies to facilitate teaching and learning. For example, Gerard and Linn (2022) [35] included AI-based automatic scoring in web-based inquiry to facilitate science learning. Latif and Zhai, et al. [Latif2024] included AI-Scorer in mobile learning to facilitate formative assessment practices in classrooms. These integrations increase the performance challenges for teachers due to the complexity of the applications, thus drawing teachers’ concerns about ease of use. What made it more complex was the unique applicability of machine algorithms for specific purposes and populations based on the training dataset. Research has suggested that machine algorithms trained by given datasets are expected to be used with new datasets with similar features to maintain objectivity and accuracy; violating this rule may result in errors or bias [36]. This requirement is substantially demanding as teachers are supposed to understand the conditions of AI and its applicability. Will teachers be able to use AI in a feasible manner? To what degree does this additional complexity impact teachers’ willingness to use AI? These are questions that TAAI has to address.

Although AI has recently received significant attention, many people are unaware of AI applications. According to Gartner Research Circle and Gartner’s 2019 CIO Agenda survey, 42% of respondents did not fully understand the benefits of implementing AI in the workplace and daily life [37]. In the educational field, although much research has been conducted on integrating AI into teaching practice, teachers’ actual use of AI in classrooms is still very limited [5]. Both in-service and pre-service teachers barely have experience learning AI knowledge and using AI technology [5]. Due to the circumstances, teachers’ perceived ease of use and perceived usefulness of AI can be important variables that impact their intention to accept AI. Thus, we adopt perceived ease of use and perceived usefulness as internal factors to explain teachers’ behavioral intentions. These three constructs are also empirically verified to be powerful for predicting and explaining user behaviors for new technologies in educational contexts [38, 14, 15]. As a result, we have perceived ease of use for AI, perceived usefulness of AI, and behavioral intention of AI integration.

In addition, we expanded the TAM by adding two more constructs (i.e., anxiety and self-efficacy) to assess teachers’ acceptance of AI in education. Teachers’ anxiety about AI is critical to be assessed and garners significant scholarly attention [39, 8, 40]. The characteristics of AI, notably its ability to learn, reason, solve problems, and make decisions by imitating human cognitive abilities, compared to other information technologies [33], raise critical concerns among teachers. One of the concerns is that the decision process of AI is not transparent and can potentially be biased, which makes it possibly to lead to unfair treatment of students [41]. Some educators also worry that AI can cause job replacement, resulting in job loss and a decline in the quality of education [42]. At the same time, since the functioning of AI applications is based on large and processing data, personal privacy also becomes an issue. With these concerns and the lack of trust in using AI in teaching, it will be hard for teachers to engage in the open and effective integration of AI in education [39].

Besides, anxiety has been considered one of the important variables in predicting acceptance of AI. Many studies in educational contexts have explored the relationship of anxiety with other constructs, including attitudes, perceived ease of use, etc., and have found some significantly negative correlation [43, 39, 14, 16]. At the same time, although teachers’ AI anxiety is increasingly investigated, existing research uncovered limited evidence of the relationship between teachers’ anxiety and their acceptance of AI. In this study, we proposed that teachers’ anxiety about using AI in teaching is a critical construct in the TAAI instrument.

Teachers’ self-efficacy refers to teachers’ belief in their ability to integrate new technologies into their professional practice to enhance students’ learning [14]. Previous studies have found both direct and indirect effects of self-efficacy on pre- and in-service teachers’ acceptance of using new technologies, including mobile devices [14], computers [44], assistant technologies for special education [45], etc. Self-efficacy, thus, is regarded as one of the key drivers in the adoption of AI [18]. Due to the low actual use of AI in current teaching practice, teachers are somehow unfamiliar with learning and teaching with AI applications. They may also have a low level of awareness about the use of AI. In fact, most teachers consider themselves unknown to the general characteristics of AI and how to apply it in teaching [5, 46]. The uncertainty can make teachers feel overwhelmed about using AI [18] and undermine their perceived usefulness of adopting AI in teaching [18]. Low self-efficacy in using AI in teaching, thus, leads to unwillingness to integrate AI into classroom practices. In this study, we also examined the teachers’ self-efficacy in using AI. Given the advantages of TAM and its wide application in related studies, this study proposed the constructs of interest—TAAI within a revised framework of TAM.

Table I presents the definitions of the construct through five sub-dimensions. Table I. Constructs and descriptions of the instrument dimensions of teachers’ acceptance of AIED

The initial version of the TAAI instrument includes three sections. The first section consists of reading material and 10 five-point scale questions. The reading material introduces the concept of AI, some daily applications, and how AI is integrated into educational settings. The questions ask participants to illustrate their previous experiences of using AI applications in their personal and professional lives. The reading material and the questions function as the stimulus in the instrument to help participants familiarize themselves with AI applications in their daily lives and education settings. The second section collects participants’ demographic information, including gender, grade, and majors, to better contextualize students’ responses to the following questions. The third section includes 32 survey items to assess the five dimensions of teachers’ acceptance of AI. The survey items were developed using a five-point Likert scale, ranging from strongly disagree (1) to strongly agree (5).

The construction of teachers’ AI acceptance guided the development of the item pool in the third section of the instrument. We developed the items based on previous technology acceptance instruments, mostly for teachers [17, 8, 47, 14, 40, 16]. Since some items focused on general AI acceptance rather than TAAI, we revised them to make the context more related to teachers’ using AI in their teaching. For example, the original item, “AI treats different people differently, which makes me anxious” [8], was revised to form the item, “I am concerned that AI algorithms may be biased, leading to varying accessibility to students with diverse backgrounds” in AN dimension. Meanwhile, we adopted some items from other non-AI instruments to be suitable in the TAAI context. For instance, the item “Using WBLS saves me time” [48] was modified into “Using AI technologies saves me time.” Additionally, we created new items focusing on teachers’ potential teaching practice with AI, such as the items “I am willing to use AI tools to help prepare lessons” and “I am willing to use AI tools to assess students,” which were developed as original items in the BI dimension to focus more on using AI for different teaching purposes.

To establish the face and content validity of the initial instrument, a panel of four experts —comprising two professors with specialization in AI in education, one with a focus on STEM education, and another with expertise in none-STEM education — conducted a comprehensive evaluation. They evaluated every part of the instrument concerning its intended purpose, phrasing, and the defined scope of each item’s corresponding dimensions. Furthermore, they provided insightful comments and suggestions for instrument revision at item and dimension levels. Based on expert feedback, we revised the instrument. We reduced the length of the reading material to make it more concise, clear, and readable. As for teachers’ experience of AI, we provided more examples of commonly used AI programs to make the items more familiar to respondents. For example, in the original item “I employ personal assistants in the mobile phone”, we added Siri as an AI assistant example that people usually use. For items that measure TAAI, we revised the wording of some items and also added concrete examples to make the items easier to read and understand. For example, we added “lesson preparation, grading, and other administrative tasks” as ways that teachers can use AI in teaching to save time. In total, 15 items in the instrument were modified.

To further establish face and content validity, we conducted think-aloud of the initial instrument with five pre-service teachers, including four females and one male, randomly selected from a class. Think-aloud protocol requires participants to articulate their feelings, thoughts, actions, and other cognitive processes while engaging in a set of activities to make thought processes as explicit as possible throughout task performance [49]. This allows researchers to gain insights into the participants’ cognitive processes rather than just their responding results for further instrument refinement [50]. Researchers provided guidance for relevant concepts, functions, and procedures of think-aloud before the test, followed by a demonstration. The participants read the guidance and then completed a short task with think-aloud, ensuring their understanding of the procedure. Afterward, they were given the questionnaire and started to think aloud. Researchers videotaped the entire think-aloud process. Based on the data obtained, we modified or deleted the items that did not function as expected. For example, for some original items wording as “I can use AI technologies to …,” participating teachers commented that “Technology seems difficult. I can use AI-based tools, but I don’t think I can use AI technology since I know nothing about that.” We thus rephrased “AI technologies” as “AI applications” or “AI tools.” Through the think-aloud procedure, six items were revised, and an additional item was added. The added item is “I have seen others using AI tools for teaching” in the AI using experience section, since a participating teacher mentioned that although she had no experience of using AI in teaching, she has seen other teachers used AI applications.

Using a random sampling approach, we distributed the revised version of the TAAI instrument to 301 pre-service teachers for field test through an online platform called “Wenjuanxing.” All participants were from a four-year university specializing in in-service and pre-service teacher education. To increase the diversity of participants, we recruited participants with various majors, including STEM majors such as mathematics, physics, biology, chemistry, geography, and technology, and non-STEM majors including Chinese, English, history, and politics. Among the 301 participants, we collected 274 valid responses for data analysis. The demographic distribution of the sample is in Table II.

To examine individual items’ discrimination power, we first conducted a t-test using SPSS to compare teachers’ scores between the upper and lower 27%. Meanwhile, using SPSS, we also calculated Cronbach’s alpha for the whole instrument, each dimension, and each item, to indicate the internal consistency of the scale and identify items that need to be deleted.

Then, we employed Mplus to conduct confirmatory factor analysis (CFA) to confirm the dimensionality of the instrument, providing evidence for construct, convergent, and discriminate validity. We used the weighted least squares means and variance adjusted (WLSMV) estimator option, considering the categorical and non-normal data. Factor structure was verified based on a solid theoretical foundation and by conducting CFA using empirical data. CFA can help examine whether the data fit a theoretical structure. Based on the results of CFA, many items were loaded onto the factors; however, the fit of the overall model was not satisfying enough. Thus, according to CFA results, including factor loadings and modification indices (M.I.), we removed ill-fitting items with factor loadings less than 0.32 (Tabachnick & Fidell, 2001). Meanwhile, factor loadings of the items belonging to the same dimension should be close ideally, and items with significantly lower factor loadings were also considered to have relatively poor performance. Furthermore, items with M.I. larger than 10 were “bad” [51] since M.I. shows the degree to which model fit improves if an item is allowed to be correlated with another factor or removed.

To select and exclude ill-items, we considered item complexity and content. For example, the item “Learning to use AI in teaching is difficult for me” showed relatively low factor loading (i.e., 0.44) and a high modification index (i.e., above 10). Although it stands for the “learning anxiety” construct in anxiety construct, the concept can be easily confused with “self-efficacy,” which can also be seen from the think-aloud result. This will result in a correlation between multiple factors for one item. Therefore, we decided to remove the item. Another example is the item “I will actively learn to adopt AI tools to assist teaching.” The factor loading was satisfying (i.e., 0.896) while the M.I. was above 10, the meaning expressed in the item is somewhat different from other items in the dimension, which concerned teachers’ willingness to use AI in teaching practice. Therefore, we also removed the item. On the contrary, for the item “I would like to use AI tools for student assessment” in the behavioral intention dimension, the factor loading (i.e., 0.741) was acceptable, while the M.I. was above 10. We decided to keep the item since assessment is an important part of teaching practice. The cull of items stopped until the overall fit of the model and the statistics of each item reached a satisfactory result.

To examine the item discrimination, we conducted a two-sample t-test to compare scores between the upper 27% and the lower 27% of participants for each item [52]. We found that all items show significant discrimination between the upper and lower groups of participating teachers (see Table III). The findings suggest the robustness of individual items in discriminating participants with varying levels of acceptance of AI.

To examine the reliability of the instrument, we calculated Cronbach’s alpha for the instrument, each dimension, and each item if deleted (see Table III). Results indicate that the reliability coefficient of the instrument is 0.92, and that of the sub-dimensions PU, PEU, BI, SE, and AN are 0.88, 0.91, 0.91, 0.91, and 0.77, respectively. All the values over 0.7 indicate a high consistency in each dimension and among the instrument [N](Nunnally, 1978). Meanwhile, we found that if an item was deleted, the Cronbach’s alpha was either dropped or similar to the original value, which means no item should be deleted.