Validation of an Instrument to Measure the use of Artificial Intelligence in the Teaching Process of Basic Education Teachers

Abstract

Research realities, characterized by a constant flow of information, are immersed in an ethical dilemma regarding reliable and rigorous processes in knowledge management. In this context, hyperstimulation mediated by Information and Communication Technologies, and especially AI, represents a phenomenon that cannot be compared to other technologies, as its impact has not yet been fully measured. Consequently, this reality has created a need in academia for reliable instruments that provide theoretical and methodological support for research as a generator of knowledge for humanity. 

Objective: To validate, through expert judgment, an instrument for measuring the use of artificial intelligence in the teaching process of basic education teachers. 

Materials and methods: The study employed a quantitative approach, using statistical tests of concordance for content validity. The psychometric study was based on content validity through expert judgment, involving five experts with doctorates in education and experience in basic and university education. Specific criteria were considered for their selection: being practicing teachers, having educational experience in basic education, using and integrating emerging technologies in teaching processes, and possessing extensive knowledge in the field of information and communication technologies. The degree of agreement among the judges was determined based on Aiken's V calculation as a decision criterion for instrument 1 (Questionnaire on Advances and Limitations of AI in Teaching Processes), which presented values ranging from 0.00 (no agreement) to 1.00 (total agreement), taking as a decision threshold a value ≤0.7 in the three indicators: clarity, coherence, and relevance. Regarding the scoring, the ratings were presented on a scale of 1 to 5, where 1 corresponded to “poor”, 2 to “fair”, 3 to “good”, 4 to “very good”, and 5 to “excellent”. In parallel, instrument 2 (AI Usability Scale in Teaching Processes - EUIAP-E) used Fleiss's Kappa as a reliability indicator for five levels of AI use in teaching, all on a 25-item Likert scale—namely: Level 1: Resistant; Level 2: Exploratory; Level 3: Operational; Level 4: Integrative; Level 5: Critical. The Likert scale consisted of 25 items that were rated according to: Always (A), Almost always (AA), Sometimes (S), Almost never (AN), and Never (N). The strength of agreement for Fleiss's Kappa coefficient ranged from 0.00 to 1.0, being: poor (0.00), slight (0.1-0.20), acceptable (0.21-0.40), moderate (0.41-0.60), considerable (0.61-0.80), and almost perfect (0.81-1.0). 

Results: The indicators of clarity (V = 0.94), consistency (V = 0.89), and relevance (0.94), represented by Aiken's overall V coefficient of 0.92, show that the AI Progress and Limitations in Teaching Processes Questionnaire is a solidly valid instrument. In detail, this instrument investigates the use of AI in the dimensions of teaching, learning, assessment, classroom management, educational risks, supports, obstacles, and teacher perception, all of which are inherent aspects of the teaching process. Meanwhile, the results for instrument 2 show that Fleiss's Kappa coefficient (k = 0.417) indicates a moderate strength of agreement among evaluators, suggesting an agreement greater than would be expected by mere chance. This test represents greater reliability than other statistical verifications such as mode, which is useful in this type of study. 

The results for the AI Usability Scale in Teaching Processes (EUIAP-E) are considered statistically significant, as demonstrated by the Z-test value of 13.106 and a p-value of 0.000, which is below the standard significance threshold (α = 0.05). The 95% asymptotic confidence interval for Kappa (0.354-0.480) indicates that the actual agreement in the population of judges is highly likely to fall within this range. 

Conclusions: The findings show that both instrument 1 (Questionnaire on Advances and Limitations of AI in Teaching Processes) and instrument 2 (EUIAP-E) are valid tools for measuring the uses of artificial intelligence in teaching processes developed by basic education teachers. As such, these instruments represent a relevant alternative for strengthening and improving educational contexts. 

In this regard, the integration of artificial intelligence in education requires constant support to ensure safe and reliable processes with a cross-cutting approach. In an era characterized by major transformations, the role of teachers must adapt to social dynamics, which implies new forms of teaching, training, and professional development. Tool validation is therefore an act that allows us to view these instruments as valuable tools that guarantee processes of continuous improvement—in this case related to the use of AI in teaching, which is a fundamental part of driving innovation and modernization mechanisms in the education sector. 

Likewise, information-gathering instruments enable informed discussion. Currently, the interaction between AI and education is giving rise to a series of ethical, moral, academic, social, and research debates. While the overwhelming rise of emerging technologies in many areas represents certain threats, it also offers valuable opportunities that are important to support through reliable instruments. 

Based on the foregoing, it is clear that this is a constantly evolving issue. However, it is important to consider the postulates regarding its use in the field of education, especially given the repercussions of AI on the academic and institutional life of society. While it is true that teaching will be transformed in the coming years, far from disappearing or remaining stagnant, teachers must ask themselves how, when, and why to teach, which, although it has been a commendable task, must be updated with the arrival of various technologies. Ultimately, tools such as those presented in this article will allow for the analysis of advances and possibilities for improvement that will help integrate AI into educational dynamics from an ethical and sustainable standpoint.