Supporting Instructional Decision Making

This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports. The project will also generate knowledge about teachers’ uses of 3D assessments and examine the potential of automatically scored 3D assessments.

• Sponsor
  National Science Foundation Discovery Research K-12
  $903,420
• Principal investigators
  Xiaoming Zhai
  Joseph S. Krajcik
  CREATE for STEM Institute Director and Professor, Michigan State University
  Gary Weiser
  Research Associate, WestEd
  Yue Yin
  Professor, University of Illinois at Chicago
• Active since
  September 2021

Visit the Project Website

Abstract

This project will study the utility of a machine learning-based assessment system for supporting middle school science teachers in making instructional decisions based on automatically generated student reports (AutoRs). The assessments target three-dimensional (3D) science learning by requiring students to integrate scientific practices, crosscutting concepts, and disciplinary core ideas to make sense of phenomena or solve complex problems.

Led by collaborators from the University of Georgia, Michigan State University , University of Illinois at Chicago , and WestEd , the project team will develop computer scoring algorithms, a suite of AutoRs, and an array of pedagogical content knowledge supports (PCKSs). These products will assist middle school science teachers in the use of 3D assessments, making informative instructional changes, and improve students’ 3D learning. The project will generate knowledge about teachers’ uses of 3D assessments and examine the potential of automatically scored 3D assessments.

The project will achieve the research goals using a mixed-methods design in three phases:

Phase I: Develop AutoRs

Machine scoring models for the 3D assessment tasks will be developed using existing data. To support teachers’ interpretation and use of automatic scores, the project team will develop AutoRs and examine how teachers make use of these initial reports. Based on observations and feedback from teachers, AutoRs will be refined using an iterative procedure so teachers can use them with more efficiency and productivity.

Phase II: Develop and test PCKSs

Findings from Phase I, the literature, and interviews with experienced teachers will be employed to develop PCKSs. The project will provide professional learning for teachers on how to use the AutoRs and PCKSs and will research how they use AutoRs and PCKSs to make instructional decisions. The findings will be used to refine the PCKSs.

Phase III: Classroom implementation

In this phase, a study will be conducted with a new group of teachers to explore the effectiveness and usability of AutoRs and PCKSs in terms of supporting teachers’ instructional decisions and students’ 3D learning. This project will create knowledge about and formulate a theory of how teachers interpret and attend to students’ performance on 3D assessments, providing critical information on how to support teachers’ responsive instructional decision making. The collaborative team will widely disseminate various products, such as 3D assessment scoring algorithms, AutoRs, PCKSs, the corresponding professional development programs, and publications to facilitate 3D instruction and learning.

The Discovery Research preK-12 program (DRK-12) seeks to significantly enhance the learning and teaching of science, technology, engineering, and mathematics (STEM) by preK-12 students and teachers through research and development of innovative resources, models, and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects.

This award reflects NSF’s statutory mission and has been deemed worthy of support through evaluation using the Foundation’s intellectual merit and broader impacts review criteria.