The Item Bank Calibration and Replenishment for Computerized Adaptive Testing in Small Scale Assessments: Method, Theory and Application

This research project will advance statistical estimation methods for computerized adaptive testing (CAT) item bank calibration and replenishment in small-scale assessments.

• Sponsor
  National Science Foundation Methodology, Measurement, & Statistics Program
  $300,000

• Principal investigator
  Shiyu Wang
  Associate professor, Department of Educational Psychology

• Co-principal investigator
  Yuan Ke
  Assistant professor, Franklin College of Arts and Sciences

• Active since
  August 2023

Visit project website

Abstract

This research project will advance statistical estimation methods for computerized adaptive testing (CAT) item bank calibration and replenishment in small-scale assessments. CAT has emerged as a powerful assessment tool and has been applied to the field of educational testing, quality of life measurement, health related measurement, and testing in industrial settings. Different from the traditional paper-pencil test, CAT allows for personalized assessment. However, the application of CAT remains limited in small-scale assessment scenarios, such as in classrooms or business daily routines.

This project will develop a series of statistical estimation methods, theories, algorithms, and software aimed at accelerating the development of CAT and promoting its applications in personalized educational assessment and learning. The project will address a fundamental challenge in educational research: the lack of efficient statistical methods to cope with limited sample sizes and missing data, especially in low-stakes assessment settings. The outcomes are expected to advance the fields of education and psychology and to positively impact society. The investigators will apply the new methods to educational data sets and develop publicly available software that will be directly applicable for educators and applied researchers. Graduate and undergraduate students will participate in the conduct of this research.

This research project will develop new statistical methods and theories to address the challenges of Item Response Theory (IRT) model calibration with small samples, sparse response data, and the lack of computationally efficient online estimation procedures with theoretical guarantee. The project will develop a dimension reduction method for a family of IRT models based on homogeneity structure learning, which belongs to a class of statistical and machine learning methods that aim to summarize useful information by discovering low-dimensional structures. An online inference approach that borrows the wisdom of some recent developments in stochastic gradient descent based online estimation and inference also will be developed. The investigators will validate these developed approaches through a set of simulation studies that mimic different aspects of small-scale assessments and analyze two real data sets from a computer-based classroom assessment and a large-scale CAT.