DreamBox Math Study with William Penn School District: ESSA Evidence Level III
Research & Impact
At-a-Glance
Type:
Case Study
Source:
LearnPlatform by Instructure
Grade Level:
K-5
Product:
DreamBox Math
Share This Report
In 2022, LearnPlatform, a third-party edtech research company, was contracted to explore the extent to which DreamBox Math helps accelerate learning for all students at William Penn School District in Pennsylvania.
This study examines the relationship between usage of DreamBox Math and student math outcomes. LearnPlatform designed the study to satisfy Level III requirements (Promising Evidence) according to the Every Student Succeeds Act (ESSA).
Source
LearnPlatform by Instructure, Senior Researchers Ashley Hunt, Ph.D. and Meetal Shal, Ph.D., Director of Research Mary Styers, Ph.D. “DreamBox Math (2021-22) ESSA Evidence Level III study conducted at The William Penn School District, PA”. Discovery Education, February 2,2023.
Who Was Included in the Study?
This study included 18,51 K-6 students across eight elementary schools in William Penn School District, PA. This student sample represents a diverse and historically vulnerable student population.
Researchers Asked:
- How were different DreamBox Math usage patterns related to grade K-6 students’ spring 2022 math achievement?
- Which usage pattern(s) of DreamBox Math had the greatest impact on grade K-6 students’ spring 2022 math achievement?
- What was the overall impact of DreamBox Math on grade K-6 students’ spring 2022 math achievement?
- How did the impact of DreamBox Math vary by student grade and racial identity?
- What was the impact for students who had free or reduced-price lunch (FRL) status?
Study Sample Student Demographics
What Data Were Evaluated?
Researchers investigated differential rates of learning acceleration for K-6 students who had access to DreamBox Math during the 2021-22 academic year and completed the fall 2021 Savvas MSDA math pretest and the spring 2022 Savvas MSDA math posttest.
Analysts used 2021-22 student-level DreamBox Math usage (i.e., average weekly lessons and weekly minutes). This usage data informed the extent to which students used the program during the school year and whether students’ use of DreamBox Math related to math learning outcomes on Savvas™ MSDA (the district’s end-of-year assessment).
Study Inclusion Requirements
Students in grades K-6 who completed both:
✓ Fall 2021 Savvas MSDA math pretest
✓ Spring 2022 Savvas MSDA math posttest
DreamBox Math Usage Groups
✓ Low usage = 0 to 3.4 weekly lessons
✓ Moderate usage = 3.5 to 6.7 weekly lessons
✓ High usage = More than 6.8 weekly lessons
How Much Time Did Students Use DreamBox Math Each Week?
DreamBox Learning recommends students complete five lessons per week (about one hour). Overall, students at William Penn School District demonstrated strong usage. During the 2021-22 school year, students at WPSD completed an average of 4.2 DreamBox Math lessons per week and spent an average of 70 minutes in DreamBox Math per week. 59% of students completed 3.5+ weekly lessons and 33% of students completed the recommended 5 weekly lessons.
Overall Distribution of DreamBox Math User Groups
What Were Some Key Takeaways from the Study?
Overall, students who completed more lessons and spent more time in DreamBox Math had higher math achievement at the end of the study.
✓ Students who completed at least 3.5 lessons (55 minutes) in DreamBox Math had higher end-of-year math achievement scores.
✓ This study controlled for students’ beginning-of-year math achievement and demographics.
DreamBox Math was equally impactful for students regardless of grade, race, and FRL status.
✓ Across all grades, students who completed more DreamBox Math weekly lessons had higher spring Savvas MSDA achievement scores.
✓ Students who identified as Black (African American) and completed more weekly lessons/spent more time (weekly minutes) in the program had higher end-of-year math achievement than those students who used the program less.
✓ Among students who had FRL status, there was a positive, statistically significant relationship that suggests students who completed more weekly lessons/spent more time using the program had higher end-of-year math achievement than those who didn’t use the program.
Education Intervention Impact
LearnPlatform researchers calculated standardized effect sizes (Hedge’s g) to determine the magnitude of changes in student outcomes. The effect size for students who used DreamBox Math more than 6.8 lessons per week was 0.29. This result suggests that DreamBox Math is 4Xs more effective than the average elementary school math intervention.*
* Lipsey, M. W., Puzio, K., Yun, C., Hebert, M. A., Steinka-Fry, K., Cole, M. W., Roberts, M., Anthony, K. S., & Busick, M. D. (2012). Translating the statistical representation of the effects of education interventions into more readily interpretable forms (NCSER 2013-3000). Washington, DC: National Center for Special Education Research, Institute of Educational Sciences, US Department of Education.