Skip to main content
Implement for Impact
Search Find
Explore Methodology

Methodology

Implement for Impact (I4I) is a free online tool that provides detailed information about different research-based interventions—including the budgetary, organizational, and policy contexts in which each intervention was carried out and the intervention’s results. We hope that by sharing these contexts alongside results, you will be better equipped to select, adopt, and successfully implement the most cost-effective interventions for your district or community.

To best assess and summarize these study components, we rely on a combination of existing, peer-reviewed studies and reports of quality and a large language model (LLM). To facilitate openness and trust of data, process, and methodology, we explain each aspect of this tool below, and we will continue to update this page through the project’s duration as needed.

Phase 1 - An alpha version of this tool will be rolled out beginning in November 2024. The initial release will contain a small subset of research studies and allow the I4I project team to develop and refine the site based on district needs and use cases.

Phase 2 - In early to mid 2025, the I4I team will conduct interactive pilot studies with school districts to further inform the tool’s design and offerings. During that time, I4I will expand the number of studies available, and the information reported about those studies, in service of assisting decision makers and implementers during their spring 2025 school-improvement planning cycles.

Phase 3 - In mid 2025 through 2026, the I4I team plans to incorporate more detailed intervention resource cost and budgetary information for interventions in the tool, including line-item resources required to implement a given intervention. Pilot participants will compare relevant cost details between interventions, indicate a desired intervention, and test a feature that will allow them to automatically receive a line-item budget proposal to implement the selected intervention in their own school/district as part of their regular, school and district budget planning processes. Once a proposal is approved by the district, the I4I team will support school leaders in tracking and assessing their intervention spending in real time using PowerSchool Allovue’s Manage solution.

Study/Report Data Sourcing

We source our list of interventions from a body of academic research studies, which evaluate the efficacy of those interventions among a sample group of students or participants. Here are a few ways we work to ensure quality, consistency, and reliability of the information presented on I4I:

  • Source Location: Currently, all studies for I4I are published and available on the What Works Clearinghouse (WWC) federal database. The WWC contains thousands of studies and will be I4I’s primary source. All studies on the WWC have been independently reviewed as meeting WWC’s quality standards for publication, which means the reviewed studies’ methodology and data design are rigorous enough that their findings are credible. The team will also explore other high-quality sources to more holistically inform the I4I tool.
  • Study Quality/Quality Control: The WWC only includes studies that meet peer-review and related criteria and that demonstrate causal effect evidence.
  • New Data Additions/Future Inclusion of Studies: I4I will only contain studies with causal inference and, whenever expanding beyond WWC, will assess and report here the specific source(s) and mechanisms for quality control.

I4I’s current content represents a specific subset of studies that are related to mathematics interventions and are also whole group/whole class (or MTSS Tier I) interventions.

Machine Learning Text-Based Paper/Report Analysis

I4I uses a machine learning-based approach to gather information about interventions. In particular, I4I uses LLAMA 3.1, a state-of-the-art large language model (LLM) from Meta, to summarize relevant information about interventions from each of the documents in our corpus of quality-approved academic research studies. As such, it generates data points about interventions that would be of interest to education practitioners as they browse programming and intervention options for their students. Any information not reported in a study and/or not found by the model will be reported as ‘N/A.’

To proactively address machine learning hallucination (the fabrication of information), the I4I team conducts extensive model testing and prompt refinement to ensure accuracy of information presented in the tool. The I4I team will scale this approach to continue to add more intervention information to the website that will help education practitioners make decisions about their programming.

Resource and Cost Calculations

In the alpha release of I4I, all resource and cost calculations will be lifted directly from a given source study or evaluation report. In late 2025, line-item planned expenses lifted directly from each study — and costs assigned to those line items via a price model — will populate cost information. We will publish the final technical methodology for that work when we complete Phase 2.