I Evaluating the Effectiveness of Vista’s Connect: Language, Literacy, Content for Improving Elementary Multilingual Learner’s Reading Skills and Reducing Academic Risk 2024 2023 Click here to learn more about Connect. Click here to contact us.
1 Evaluating the Effectiveness of Vista’s Connect: Language, Literacy, Content for Improving Elementary Multilingual Learners’ Reading Skills and Reducing Academic Risk Overview Many students fail to acquire the necessary reading skills during their early elementary education and are at risk academically. Multilingual learners (MLs) whose first language is not English are at particular risk, often arriving to school with significant reading deficits and underperforming their peers whose first language is English. During the 2022–2023 school year, SEG Measurement (SEG) conducted a study of the effectiveness of Vista’s Connect: Language, Literacy, Content for improving the reading skills of elementary-level MLs. The study was conducted with students in grades 1, 2, and 3 in a midwestern suburban U.S. school district. SEG used several research strategies, including a matched-groups, quasi-experimental study consistent with the guidelines for a Level 2 ESSA study (U.S. Department of Education, Office of Elementary and Secondary Education, 2016). This document describes the study procedures and findings. The first part of the document describes the intervention, the sample of students studied, the design of the study, and the research questions posed. The second part of the document presents the analyses and results. The findings and conclusions are summarized in the final section of the document. First, we examined the reading skills growth achieved by MLs who were provided instruction incorporating Connect. Second, we compared the level of reading skills growth for MLs receiving instruction incorporating Connect to the expectations established by the i-Ready norms and typical growth levels (Curriculum Associates, 2023). Third, we compared the performance of MLs receiving instruction incorporating Connect to a matched comparison group of non-MLs that received instruction that did not incorporate Connect. Fourth, we examined the impact of instruction incorporating Connect in reducing academic risk for MLs. Multilingual students receiving instruction that included Connect achieved substantial levels of reading skill growth exceeding typical levels of growth, and decreased their academic risk level.
2 Overall, SEG Measurement found that MLs receiving instruction that included Connect achieved substantial reading skill growth exceeding typical levels of growth seen, based on national norms. MLs also achieved reading skills growth comparable to their non-ML peers and decreased their academic risk level after receiving Connect instruction. Background The Importance of Effective Reading Skills The acquisition of reading skills during early elementary education is fundamental to academic success, especially for multilingual students whose first language is not English. Early reading in English is not just about learning a new language; it's about forming a foundation for academic learning across all subjects. The ability to read proficiently in English from an early age opens doors to higher educational opportunities and future career prospects. The National Literacy Trust (2012) emphasizes that literacy is strongly linked to employability and economic success. Insufficient reading skills can significantly limit MLs’ career options and earning potential in adulthood (Cunningham & Stanovich, 1997). Reading skills are essential for active participation in civic activities such as voting, and for engaging in community initiatives; MLs with reading deficits may feel excluded from these activities. Early Intervention Is Critical The early elementary grades are a critical period for literacy development. Children who have difficulty reading at the end of first grade rarely catch up by the end of elementary school (Torgesen, 2004; Francis et al., 1996; Juel, 1988; Shaywitz et al., 1999; Torgesen & Burgess, 1998). When students fail to master foundational reading skills early on (by third grade), there is a long-term negative impact on their reading ability and overall academic success. Students failing to read effectively by the end of third grade have a nearly five times greater rate of dropping out of high school before graduation. Early intervention is research-proven as an effective measure for reducing—and sometimes altogether preventing—reading disabilities for students at risk of reading failure (Connor et al., 2014; Gersten et al., 2008). ML Learners Face Unique Reading Challenges ML learners encounter specific challenges in developing reading skills when compared to their non-ML peers. The lack of English language skills among MLs impacts their ability to understand instructions, participate in class discussions, and comprehend academic materials (August and Shanahan (2017); Thomas & Collier, 2002; Cummins, 2000). MLs often exhibit a vocabulary gap compared to their non-EL peers (Laufer and Ravenhorst-Kalovski, 2010). MLs with reading skills deficits often struggle to perform well in various academic subjects, especially those heavily dependent on reading comprehension (Lesaux & Kieffer, 2010). MLs often come from diverse educational backgrounds with varying
3 levels of reading proficiency in their first language, which can impact their English reading skills development (Thomas and Collier, 2002). Reading skills deficits can lead to lower self-esteem and confidence. Not only is ML status associated with lower academic expectations from teachers and the students themselves (Kanno & Kangas, 2014), but also a lower likelihood of taking general and advanced courses in core subjects (Estrada, 2014; Umansky, 2016a, 2018). This has contributed to lower academic achievement (Umansky, 2016b). Soland and Sandilos (2020) found that students classified as ML at any point in their academic career had lower achievement than their non- ML peers. While acquiring reading skills can be challenging for students whose first language is not English, early and effective literacy interventions can bridge gaps and provide these students with the tools necessary for academic achievement. Federal Support for ML Learners One out of every ten students is classified as a multilingual learner at some point during their K–12 schooling (U.S. Department of Education (DOE), 2018). The scope of this challenge has not gone unnoticed by the federal government; assisting students whose first language is not English has been an educational priority for the federal government. Since 1968, the federal government has provided financial assistance to schools for elementary and secondary school programs to meet the special needs of “limited English-speaking” students. Since that time, eligibility has been expanded to all students with limited English proficiency (Stewner-Manzanares, 1988), culminating in the 2000s with No Child Left Behind and the Common Core State Standards (García & Kleifgen, 2018). Closing the Gap: Reducing Disparities in Multilingual Learners’ and Native Speakers’ Reading Skills The disparities in reading skills between multilingual learners and their native English– speaking counterparts are often large. MLs often enter school with lower English proficiency levels compared to native speakers, which initially reflects in their reading skills. Effective instruction can mitigate initial disparities in reading skills. By implementing targeted interventions and employing effective strategies, educators can facilitate reading skills growth, ultimately leading to reading skills parity with their non-ML peers. Studies by Lesaux and Geva (2006) show that with appropriate instruction, MLs demonstrate significant growth in reading skills. Goldenberg (2008) emphasizes that focused reading instruction in the early years can accelerate literacy development in MLs and enable them to close the gap with their native English–speaking peers. Slavin and Cheung (2005) report that the gap in reading skills between MLs and native English speakers tends to narrow over time, particularly when MLs receive consistent, highquality literacy instruction. The research is clear. While MLs may suffer from reading deficits in the early elementary grades, they are capable of remarkable growth, often catching up to or surpassing their native English–speaking peers, particularly when supported by effective literacy instruction.
4 The increasing need for solutions to help MLs acquire English language skills has led to the development of several products and services to help MLs develop language skills. Among those products is Connect, which is the focus of this study. Demonstrating Effectiveness Demonstrating that Connect is effective is a process of marshalling evidence in support of the claim that instruction incorporating Connect increases students’ reading skills. No single study can conclusively establish this claim; multiple sources of evidence enhance the claim that Connect is effective in developing students’ reading skills. The results of this study should be viewed along with other evidence of Connect effectiveness (e.g., Evaluating the Effectiveness of Vista’s Connect: Language, Literacy, Content for Improving Elementary Students’ Reading Skills, SEG Measurement, 2023). The Intervention Students identified as MLs in grade 1, 2, and 3 classrooms participating in this study were provided with sustained instruction incorporating Connect on a weekly basis. Teachers were provided with training in the use of Connect at the outset of the program and had access to district ML instructional support personnel for continued support in delivering intervention. Connect provides evidence-based instruction incorporating current research-driven learning science and language development practices. Connect is a comprehensive literacy program created for multilingual learners and striving readers. The program builds language and reading proficiency through research-driven lessons and activities and engaging texts and content-driven lessons. The program provides a variety of print and digital classroom resources that provide support and motivation. Connect provides tools for teachers to assess ongoing learning progress, discover skills gaps, and personalize instruction. Study Site This study was conducted in a suburban midwestern U.S. school district. The district serves nearly 6,000 students in eleven elementary schools and two middle schools. Sixty-three percent of the students were low income at the time of the study, and 46% were multilingual learners. The ethnic distribution of students was 42% Hispanic, 38% White, 11% Asian, 5% Black, 2% American Indian, and 3% mixed race. Connect provides evidence-based instruction incorporating current researchdriven learning science and language development practices.
5 Research Questions We posed four research questions to gather evidence supporting the claim that Connect is effective: First, we examined the level of reading skills growth achieved by multilingual learners using Connect from the beginning to end of instruction. • Question 1: Do grades 1, 2, and 3 multilingual learners in the district receiving instruction incorporating Connect (treatment group) achieve significant growth in reading skills during a school year? What is the magnitude of the growth achieved? Second, we compared the reading skills growth achieved by students receiving instruction incorporating Connect to the i-Ready norms and typical growth levels. • Question 2: Do grades 1, 2, and 3 multilingual learners in the district receiving instruction incorporating Connect (treatment group) achieve reading skills growth greater than what would be expected based on the i-Ready norms and typical growth levels? Third, we compared the reading skills growth achieved by ML learners receiving instruction incorporating Connect to the growth achieved by a matched sample of non-ML learners (receiving instruction not incorporating Connect). • Question 3: Do grades 1, 2, and 3 multilingual learners in the district receiving instruction incorporating Connect (treatment group) achieve reading skills growth similar to a comparison group of non-multilingual learners in the district not receiving instruction incorporating Connect (comparison group)? Fourth, we examined Connect’s effectiveness in reducing academic risk among ML learners receiving instruction incorporating Connect. • Question 4: Do academically at–risk multilingual learners in the district receiving instruction incorporating Connect (treatment group) reduce their level of academic risk after receiving Connect instruction? Data Collection and Measurement The district administered the i-Ready reading skills assessment (Curriculum Associates, 2023) at the beginning (pretest) and end of the school year (posttest) to all students. The district prepared a data file for use by SEG. The file contained all the grades 1, 2, and 3 students enrolled in the district, along with their beginning-of-year i-Ready assessment results (pretest), EL classification, the student’s grade level, sex, ethnicity, and an indication of whether the student was eligible for free or reduced-cost lunch, along with a unique student identifying number. At the end of instruction, the district merged the students’ end-of-year i-Ready results and provided the file to SEG.
6 About the i-Ready Assessment The i-Ready reading assessment used in this study is widely used throughout the U.S. The assessment is reported on a vertical scale ranging from 100–800 covering grades K–12; this allows comparability of scores both within and across grades. The reliability of i-Ready is above 0.90. Numerous studies supporting the validity of the i-Ready assessment have been conducted. The i-Ready reading assessment provides an overall score and several subtest scores. The overall score was used for the pretest and posttest for the study. The overall test scores were used since the subtests vary across grade levels limiting the ability to compare results across grades. Study Population and Sampling Creating Matched Samples Random sampling was not possible for this study. SEG Measurement used a matched-groups, quasi-experimental design. For each student in the treatment group, an individual from the control group who was similar in ability and background was selected. We created matched-treatment and control groups using propensity score matching based on multiple factors using logistic regression. We matched students based on the propensity score, a single value reflecting the composite status of each student with respect to the matching variables. For each treatment group student, a comparable student in the comparison group was selected. The treatment and comparison samples were matched based on initial reading skills ability, sex, race, ethnicity, and economic status. The sampling was done without replacement. Matched pairs were required to have propensity scores within 5% of each other. The population of 1773 first, second, and third grade students in the district were included in the matching analysis. We identified 436 treatment-comparison group pairs meeting the criteria, for a total matched sample size of 872 students. Comparability of Treatment and Comparison Groups The initial reading skills levels of the matched treatment and control group were comparable. The difference between the treatment group (mean=441.52) and the comparison group (mean=461.84) was approximately 20 points, or about a third of a standard deviation (SD=63.74). This is within the commonly applied one-half standard deviation guideline for statistical comparison (Schneider, Carnoy, Kilpatrick, Schmidt, and Shavelson, 2007), though modestly greater than the more conservative quarter of a standard deviation guidelines from the U.S. Department of Education (2016). See Table 1. Table 1 Matched Groups Sample Mean Initial Reading Skill Level Study Group Mean n SD Treatment 441.52 436 60.03 Comparison 461.84 436 65.75 Total 451.68 872 63.74
7 Matched Groups: Sex, Race, Ethnicity and Economic Status Composition The matched groups were similar in background, including the number of boys and girls in the treatment and comparison groups and the number of students from Hispanic and Non-Hispanic backgrounds. The students in both study groups were also similar in racial composition and economic status. This is summarized in Table 2 below. Table 2 Matched Groups Sample Number of Students by Sex, Race, Ethnicity, and Economic Status *Economic Level (Eligibility for Free and Reduced Lunch) Background Comparison n Treatment n Total Sex Boys 218 214 432 Girls 218 222 440 Total 436 436 872 Race Asian 81 60 141 Black 0 13 13 Mixed 14 18 32 Native American 51 58 109 White 290 287 577 Total 436 436 872 Ethnicity Hispanic Background 147 160 397 Not of Hispanic Background 289 276 565 Total 436 436 872 Economic Level* Not Eligible 158 148 306 Eligible for Reduced 55 55 110 Eligible for Free 223 233 458 Total 436 436 872
8 Attrition The final analysis sample was adjusted for attrition. Attrition included those students who took the pretest but left the district before the posttesting or otherwise did not have a posttest score available. Attrition among students in the matched sample was 39 students (4%). Thirteen students (3%) in the treatment group and 26 students (6%) in the comparison group were lost to attrition. Final Analysis Sample After accounting for attrition, 833 students remained in the final analysis sample, 423 students in the treatment group, and 410 students in the comparison group. Due to the relatively low attrition rate, there was little change in the comparability of the two study groups in the final analysis sample from the initial matched sample. The initial reading skills levels of the treatment and control groups within the final analysis sample were comparable. The difference between the treatment group (mean=441.80) and the comparison group (mean=462.90) in the final analysis sample was approximately 21 points, or about a third of a standard deviation (SD=63.71). The matched groups were similar in their demographic composition. The number of boys and girls, members of each racial and ethnic group, and economic level in the treatment and comparison groups for the final analysis sample largely mirrored that seen for the matched sample. The final analysis sample profile appears in Table 3 and 4. Table 3 Final Analysis Sample Mean Initial Reading Skill Level (Pretest) Study Group Mean n SD Treatment 441.80 410 60.25 Comparison 462.90 423 65.29 Total 452.52 833 63.71
9 Table 4 Matched Groups Sample Number of Students by Sex, Race, Ethnicity, and Economic Status *Economic Level (Eligibility for Free and Reduced Lunch) Background Comparison n Treatment n Total Sex Boys 208 201 409 Girls 215 209 424 Total 423 410 833 Race Asian 78 51 129 Black 0 12 12 Mixed 14 17 31 Native American 50 53 103 White 281 277 558 Total 423 410 833 Ethnicity Hispanic Background 142 150 292 Not of Hispanic Background 281 280 541 Total 423 410 833 Economic Level* Not Eligible 155 134 289 Eligible for Reduced 53 54 107 Eligible for Free 215 222 437 Total 423 410 833
10 Question 1: Determining the Impact of Connect Instruction on Treatment Group Yearly Growth Do multilingual learners receiving instruction incorporating Connect (treatment group) show significant growth in reading skills over the course of a school year? To answer Question 1, we examined the growth in reading skills achieved between the beginning and the end of the school year by the multilingual learners who received instruction incorporating Connect (treatment group). The analysis compared the beginning and end-of-year i-Ready scores. See Figure 1 below. Figure 1 Question 1 Study Design Reading Skills Growth Following Connect Instruction We examined the growth in grade 1, 2, and 3 reading skills for the ML learners included in the treatment group. At the beginning of the school year (fall of 2022), the treatment group receiving instruction incorporating Connect was administered the i-Ready reading skills pretest; the treatment group was again assessed at the end of the end of the year (posttest). Reading skills growth was operationalized as the mean difference in beginning-of-year i-Ready scores (pretest) and end-of-year i-Ready scores (posttest). We examined grades 1, 2, and 3 results. The overall reading skills growth was about 61 points in grade 1(t=33.40; df=287; p<0.001), about 54 points in grade 2 (t=30.12; df=287; p<0.001), and about 37 points in grade 3 (t=23.90; df=307; p<0.001). The pretest and posttest data are illustrated in Figure 2 and summarized in Table 5 below. i-Ready Reading Skills Pretest i-Ready Reading Skills Posttest Beginning of Year End of Year ML Instruction Using Connect
11 Table 5 Total Treatment Group Pretest and Postest i-Ready Reading Skills Scores Descriptive Statistics by Grade The ML learners receiving instruction incorporating Connect increased their reading skills by about one standard deviation (effect size=0.93). This effect size is generally considered to be a large effect size, indicating that students in the treatment group receiving instruction incorporating Connect achieved substantial reading skills growth. Question 2: Comparison of ML Treatment Group Reading Skills Growth to Typical Growth Levels Do grades 1, 2, and 3 multilingual learners receiving instruction incorporating Connect (treatment group) achieve growth targets based on i-Ready norms and established growth levels? The disparity in reading skill levels between multilingual learners and their nonmultilingual peers can present a major challenge to academic success. This is of particular concern if MLs are unable to close the gap in reading skills. Research points to the success in mitigating the disparity and narrowing the gap over time through appropriate instruction, particularly in the early grades (Goldenberg, 2008; Lesaux and Geva, 2006; Slavin and Cheung, 2005). We compared the level of reading skills growth achieved by the ML treatment group using Connect to expected levels of growth based on the i-Ready norms and typical growth levels. Specifically, we compared each ML student in the treatment group (Connect users) to the typical growth levels provided by Curriculum Associates (2023). The typical growth level is defined as the median score (by grade level) from the national sample of i Ready test takers used to create i ready norms. See Figure 3. Grade n Pretest Mean Pretest SD Posttest Mean Posttest SD Pre–Post Mean Diff Effect Size Grade 1 208 384.44 40.96 446,57 46.77 61.15 1.30 Grade 2 142 444.49 52.26 496.20 51.43 53.78 1.06 Grade 3 160 478.14 47.09 515.94 41.51 36.79 .88 All Grades 410 441.89 60.25 490.83 54.11 50.08 .93 Students in the treatment group receiving instruction incorporating Connect achieved substantial reading skills growth.
12 Figure 3 Question 2 Study Design Overall, the ML learners receiving instruction incorporating Connect began the year with a lower level of reading skills than the median in the i-Ready norms. Grade 1 Connect users achieved an average beginning-of-year i-Ready reading score of about 384; the grade 2 Connect users achieved an average i-Ready score of about 444; and the grade 3 Connect users achieved an average score of 478. This places the grades 1 and 3 students at about the 33nd percentile and the grade 2 students at the 40th percentile. This indicates that many ML learners in the district began the school year with reading deficits. After receiving instruction including Connect, ML learners achieved reading skills gains beyond the level of growth expected. The i-Ready publisher has identified typical growth levels based on the median achievement levels observed in large national samples of i-Ready users. The (partial) i-Ready norms for grades 1 2, and 3 and typical growth targets are provided in Tables 6 and 7 below. Table 6 Excerpted National i-Ready Norms (Reading) Percentile Grade 1 Grade 2 Grade 3 BOY EOY BOY EOY BOY EOY 25th Percentile 373 417 420 469 463 496 50th Percentile 403 454 460 505 502 534 75th Percentile 422 490 496 535 534 564 District ML Treatment Group i-Ready Reading Skills Pretest Meets Growth Target ML Instructio n Using Connect Does Not Meet Growth Target i-Ready Reading Skills Posttest
13 Table 7 i-Ready Typical Growth Targets (Median Student Growth) About two-thirds (63%) of the ML learners receiving instruction including Connect met or exceeded growth expectations. Two-thirds (67%) of the first graders and nearly two-thirds (63%) of the second graders met or exceeded growth expectations. Nearly two-thirds (60%) of grade 3 students met or exceeded growth expectations. The number of students that did and did not meet growth targets are presented in Table 8 and illustrated in Figure 4 below. Table 8 ML Learners (Treatment Group) Meeting Growth Targets by Grade While two-thirds of the MLs achieved their growth targets, one-third of the MLs did not. The large proportion of MLs able to meet the growth targets suggests that providing instruction including Connect was effective in making headway in reducing reading deficits and reducing disparities between MLs and their non-ML peers. Beginning-of-Year Classification Grade Level 1 2 3 On Grade Level, Mid, Late, or Above 37 22 17 On Grade Level, Early 47 29 22 One Grade Level Below 49 39 26 Two Grade Levels Below 54 44 33 Three or More Grade Levels Below 36 Grade Level Did Not Meet Growth Target Met Growth Target Total 1.0 36 72 108 2.0 53 89 142 3.0 64 96 160 All Grades 153 257 410
14 Meeting the typical growth target means that a student has achieved the median annual i-Ready scale score growth at the student’s grade and starting placement category. Achieving this target indicates that the student is maintaining status and not falling behind. Question 3: Comparison of Treatment and Comparison Group Reading Skills Growth Do multilingual learners receiving instruction incorporating Connect (treatment group) achieve levels of reading skills growth comparable to the comparison group of non-ML learners not receiving instruction incorporating Connect? Are there differences in reading skills growth attributable to sex, race, ethnicity, and economic status? To answer Question 3, the reading skills growth of the treatment group was compared to the growth of the comparison group using a quasi-experimental design. Using analysis of covariance (ANCOVA), we compared the reading skills growth of district ML learners receiving Connect instruction to the reading skills growth of a matched group of district students whose first language was English but who shared a similar starting reading skill level and background. We also explored the effects of sex, race, ethnicity, and economic status to determine if Connect was any more or less effective for students of particular backgrounds. See Figure 5. The large proportion of MLs able to meet the growth targets suggests that providing instruction including Connect was effective in making headway in reducing reading deficits and reducing disparities between MLs and their non-ML peers.
15 Figure 5 Question 3 Study Design The ANCOVA model examined the reading skills posttest scores (dependent variable) achieved by students in the treatment group and comparison group (independent variable) and Sex, race, ethnicity, and economic status (independent variables), To answer Question 3, the reading skills growth of the treatment group was compared to the growth of the comparison group using a matched groups, pre-post quasiexperimental design. Using ANCOVA, we compared the reading skills growth of district ML learners receiving Connect instruction to the reading skills growth of a matched group of district students whose first language was English but who shared a similar starting reading skill level and background. We also explored the effects of Sex, race, ethnicity, and economic status to determine if Connect was any more or less effective for students of particular backgrounds. See Figure 5. The ANCOVA model examined the reading skills posttest scores (dependent variable) achieved by students in the treatment group and comparison group (independent variable) and sex, race, ethnicity, and economic status (independent variables), adjusting for the pretest scores (covariate). Tests were conducted at the .05 level of significance. The relatively large sample sizes overall (833) and individual study group sample sizes (410 treatment group; 423 comparison group) provided reasonable levels of statistical power to detect differences between the two study groups if present. The statistical power to detect small differences (n effect size of .15) is greater than .80. Power for subgroup comparisons is lower due to smaller sample sizes. We examined the level of reading skills growth achieved by ML learners in the treatment group in comparison to the reading skills growth of the comparison group. We also examined potential differences in reading skills growth with respect to sex, race, ethnicity, and economic status to determine if Connect was more effective for any subgroups. ML Connect Treatment Group i-Ready Reading Skills Pretest Non-ML Comparison Group i-Ready Reading Skills Pretest i-Ready Reading Skills Posttest i-Ready Reading Skills Posttest Instruction Without Connect Instruction Using Connect
16 Comparing Treatment and Comparison Group Reading Skills Growth The treatment group (ML learners) receiving instruction incorporating Connect (mean=501.08) were not significantly different in their reading skills growth from the comparison group that did not receive instruction using Connect (mean=501.92) (F=0.25, df=1/832; p<0.617). The less than one point difference in means shows that the ML learners taught using Connect achieved about the same level of growth in reading skills as non-ML learners receiving traditional instruction. There were no significant differences between boys and girls, among racial groups, or between ethnic groups. A main effect for economic status was found (F=6.36; df=2/832; p<0.002). Students of lower economic status showed lower levels of reading skills growth than did those of higher economic status. However, the interaction between study group and economic status was not significant, indicating that the intervention was no more effective as a function of the students’ economic status. These results are summarized in Tables 9 to 14 below. Table 9 Analysis of Study Group and Student Background on Reading Skills Growth (ANCOVA) Source Type III Sum of Squares Df Mean Square F Significance Corrected Model 2314635.73 17 30861.81 44.86 .001 Intercept 275642.74 1 275642.74 400.70 .001 i-Ready Pretest 1726816.30 1 1726816.30 2510.27 .001 Study Group 171.984 1 171.98 .25 .617 Sex 7.65 1 7.65 .01 .916 Race 4436.34 4 1109.08 1.61 .169 Ethnicity 864.38 1 864.38 1.26 .263 Economic Status 8752.45 2 4376.22 6.36 .002 Study Group * Sex 1415.90 1 1415.90 2.06 .152 Study Group * Race 990.00 3 329.99 .48 .696 Study Group * Ethnicity 1167.15 1 1167.15 1.70 .193 Study Group * Economic Status 2488.32 2 1244.16 1.81 .165 Error 621131.58 815 703..90 Total 212819973.00 833 Corrected Total 2835375.93 832 Covariates appearing in the model are evaluated at the following values: i-Ready Reading Skills Pretest=452.15.
17 Table 10 Comparison of Treatment and Comparison Group Reading Skills Growth Mean i-Ready Reading Skills Posttest Scores (Adjusted for Initial Reading Skills Level) Covariates appearing in the model are evaluated at the following values: i-Ready Reading Skills Pretest=452.515. Table 11 Comparison of Reading Skills Growth by Sex Mean i-Ready Reading Skills Posttest Scores (Adjusted for Initial Reading Skills Level) Covariates appearing in the model are evaluated at the following values: i-Ready Reading Skills Pretest=452.515. Table 12 Comparison of Reading Skills Growth by Race Mean i-Ready Reading Skills Posttest Scores (Adjusted for Initial Reading Skills Level) Covariates appearing in the model are evaluated at the following values: i-Ready Reading Skills Pretest=452.515. Study Group Mean N SD Comparison Group (Non-MLs) 501.92 423 60.32 Treatment Group (MLs using Connect) 501.08 410 54.11 Total 501.46 833 58.38 Sex Mean N SD Boys 498.98 409 60.32 Girls 504.01 424 54.11 Total 501.45 833 58.38 Race Mean N SD Asian 505.41 129 56.73 Black 503.41 12 54.12 Mixed 506.45 31 55.70 American Indian 496.65 103 51.45 White 499.53 558 58.77 Total 501.46 833 58.38
18 Table 13 Comparison of Reading Skills Growth by Ethnicity Mean i-Ready Reading Skills Posttest Scores (Adjusted for Initial Reading Skills Level) Covariates appearing in the model are evaluated at the following values: i-Ready Reading Skills Pretest=452.515. Table 14 Comparison of Reading Skills Growth by Economic Status (Free and Reduced-Cost Lunch) Mean i-Ready Reading Skills Posttest Scores (Adjusted for Initial Reading Skills Level) Covariates appearing in the model are evaluated at the following values: i-Ready Reading Skills Pretest=452.515. Results by grade level: The treatment and comparison groups showed comparable levels of reading skills growth for all three grades. For grade 1, the comparison group outperformed the treatment group by about one point. In grade 2, the comparison group outperformed the treatment group by about five points. The comparison group mean in grade 3 was about six points greater than the treatment group mean. The results by grade level appear in Table 15. Hispanic Background Mean N SD From Hispanic Background 498.06 292 55.93 Not From Hispanic Background 504.60 541 57.69 Total 501.46 833 58.38 Economic Status Mean N SD No Eligibility 506.44 289 61.14 Partial Lunch Eligibility 502.87 107 54.48 Free Lunch Eligibility 497.22 437 55.61 Total 501.46 833 58.38
19 Table 15 Comparison of Reading Skills Growth by Study Group and Grade Level Mean i-Ready Reading Skills Posttest Scores (Adjusted for Initial Reading Skills Level) Covariates appearing in the model are evaluated at the following values: i-Ready Fall Pretest=394.95. Question 4: Reducing Academic Risk Do multilingual learners receiving instruction incorporating Connect (treatment group) who are at risk academically reduce their level of academic risk and at levels consistent with at-risk students in the comparison group of non-multilingual learners not receiving instruction incorporating Connect? Multilingual learners face significant academic and life challenges if they do not acquire adequate reading skills during the early grades. It is therefore critical to demonstrate that any intervention designed to improve reading skills, such as Connect, effectively reduces ML learners’ level of academic risk. Therefore, we examined the impact of Connect instruction on students’ academic risk reduction. See Figure 6. Grade Level Comparison Group Mean n Comparison Comparison SD Treatment Group Mean n Treatment Treatment SD F df Significance Grade 1 456,83 129 47.92 456.15 108 46.77 .038 1/237 .846 Grade 2 510.13 146 49.51 504.57 142 51.43 2.64 1/288 .106 Grade 3 535.20 148 51.18 529.24 160 41.51 3.80 1/308 .052 The treatment group (Multilingual learners receiving instruction incorporating Connect) and comparison group (non-ML learners not receiving instruction incorporating Connect) showed comparable levels of reading skills growth for all three grades.
20 Figure 6 Question 4 Study Design First, using ANCOVA, we evaluated the effect of study group membership (treatment or comparison group membership; independent variable) on the change in academic risk level (dependent variable) adjusting for the beginning of study reading skills level (pretest; covariate). We then examined the frequency of changes in academic risk among students from beginning to end of instruction for ML learners (treatment group). Measuring Academic Risk Each treatment and comparison group student was classified as being at high, moderate, or low academic risk. For this study, we made the classifications based on the students’ i-Ready scores in relation to the i-Ready norms. • High Risk (Level 3) Students with pretest scores ranging from the 1st to 25th percentile were classified as high risk. • Moderate Risk (Level 2) Those with scores in a range from the 26th percentile to the 50th percentile were considered to be of moderate risk. • Low Risk (Level 1) Those with scores in the range of the 51st to 99th percentile were considered to be of low risk. ML Connect Treatment Group Beginning-ofYear i-Ready Pretest End-of-Year i-Ready Posttest Moderate Risk High Risk Low Risk Moderate Risk High Risk Instruction Using Connect Low Risk Non-ML Comparison Group End-of-Year i-Ready Posttest Beginning-of- Year i-Ready Pretest Low Risk Moderate Risk High Risk Low Risk Moderate Risk High Risk Instruction Without Connect
21 The risk level at the beginning of instruction was determined based on the pretest scores and fall i-Ready norms. The end of instruction risk level was based on the posttest scores and spring i-Ready norms. To understand the change in academic risk from the beginning to the end of instruction, we examined the average (mean) difference between the starting and ending academic risk level change for the two study groups. Academic Risk Reduction in the Treatment and Comparison Group We examined the differences between the change in academic risk levels from the beginning to the end of instruction for both the treatment group (MLs using Connect) and comparison groups (non-MLs) using ANCOVA. The amount of change in risk reduction achieved by the treatment group was compared to the amount of change in risk reduction achieved by the comparison group adjusting for the initial reading skills level. Amount of change in risk was measured on a scale from -2 to +2 ranging from increasing risk levels by 2 to decreasing risk levels by 2; a value of 0 reflected no change in academic risk level. The treatment and comparison groups achieved a similar level of risk reduction over the course of the study; there was no significant difference in the amount of change in risk between the ML learners in the treatment group (mean reduction=0.344) and the nonML learners in the comparison group (mean reduction=0.360) (F=1.68; df=1/832; p<0.196). On average, students in the study decreased their risk level by about a third of a level. See Tables 16 and 17. Table 16 Treatment and Comparison Group Mean Risk Reduction Comparison (ANCOVA} (Adjusted Pretest Scores) Covariates appearing in the model are evaluated at the following values i-Ready Reading Skills Pretest=452.515. Source Type III Sum of Squares Df Mean Square F Significance Corrected Model 16.94 2 8.47 17.53 .000 Intercept 29.87 1 29.87 61.83 .000 I-Ready reading Skills Pretest 16.89 1 16.89 34.95 .000 Study Group .81 1 .81 1.68 .196 Error 401.00 830 .48 Total 521.00 833 Corrected Total 417.94 832
22 Table 17 Mean Change in Risk by Study Group Number of Students Showing Risk Reduction We explored the movement from higher levels of risk at the beginning of the year to lower levels of risk at the end of the year in more detail. We examined the number of students classified in the high and moderate academic risk categories at the beginning of instruction and their changes in risk levels at the end of instruction. Initial Risk Levels There were more treatment group students (n=187) than comparison group students (n=126) classified as at high risk at the beginning of instruction. However, there were a comparable number of treatment (n=120) students and comparison group (n=116) students classified as moderate risk at the beginning of the study. Overall Reduction of Academic Risk The treatment and comparison groups achieved similar levels of success in reducing their level of academic risk. Of the 410 ML learners in the treatment group, 238 (58%) showed no change in their level of risk from the beginning to end of instruction. Similarly, of the 423 students in the comparison group, 257(61%) showed no change in their level of risk from the beginning to end of instruction. One hundred and twelve (27%) ML learners in the treatment group reduced their initial academic risk level by one level and another 30 (7%) reduced their risk level by two levels. Of the comparison group students, 114 (26%) reduced their initial academic risk level by one level and another 30 (7%) reduced their risk level by two levels. Caution should be used in interpretation, as these numbers are unadjusted for the initial reading skill ability of the students. Recall, when adjusting for initial reading skill level using ANCOVA, there was no significant difference in risk reduction between the two study groups. Reduction of Academic Risk among Students Initially Classified as High or Moderate Risk Of the 187 ML learners in the treatment group classified as high risk at the beginning of instruction, 82 (44%) successfully reduced their risk level to moderate or low risk. Of the 126 students in the comparison group classified as high risk at the beginning of instruction, 71(56%) successfully reduced their risk level to moderate or low risk. Study Group Mean Risk Level Change SD N Comparison Group (Non-MLs) .360 .691 423 Treatment Group (MLs using Connect) .344 .728 410 (MTotal .351 .709 833
23 Of the 120 treatment group students initially classified as being at moderate academic risk, 60 students (50%) were able to reduce their risk level to low risk from the beginning to the end of instruction. Of the 116 comparison group students initially classified as being at moderate academic risk, 73 (62%) were able to reduce their risk to low risk from the beginning to the end of instruction. When examining the reduction of academic risk for ML learners in the treatment group at high and moderate academic risk, we found that of the 187 treatment group students at high academic risk initially, 82 (44%) students successfully reduced their risk level, moving to a classification of moderate or low risk by the end of instruction. Of the 120 students in the treatment group at moderate academic risk at the beginning of instruction, 60 students (50%) successfully moved to low risk by the end of instruction. In summary, many ML learners showed significant progress in reducing their academic risk. This level of risk reduction combined with those who maintained their risk level and avoided falling farther behind is of particular interest, given the instructional challenge of ML learners falling further behind. This is summarized in Tables 18, 19, 20, and 21 and is illustrated in Figures 7 and 8. Table 18 End-of-Year Risk Classification for Students Initially Classified as High Risk (n=313) End-of-Year Risk Classification Level No. of Students Initially Classified as High Risk Comparison Group (Non-MLs) Low Risk 30 Moderate Risk 41 High Risk 55 Total 126 Treatment Group (MLs using Connect) Low Risk 30 Moderate Risk 52 High Risk 105 Total 187 All Study Groups Low Risk 60 Moderate Risk 93 High Risk 160 Total 313
24 Table 19 End-of-Year Risk Classification for Students Initially Classified as Moderate Risk (n=342) Table 20 Change in Academic Risk From Beginning to End of Study by Study Group (Initially Classified as High Risk) End-of-Year Risk Classification Level No. of Students Initially Classified as Moderate Risk Comparison Group (Non-MLs) Low Risk 73 Moderate Risk 33 High Risk 10 Total 116 Treatment Group (MLs using Connect) Low Risk 60 Moderate Risk 48 High Risk 12 Total 120 All Study Groups Low Risk 133 Moderate Risk 81 High Risk 128 Total 342 Study Group Total No Change Reduce 1 Risk Category Reduce 2 Risk Categories Comparison (Non-MLs) 55 41 30 126 Treatment (MLs using Connect) 105 52 30 187 Total 160 93 60 313
25 Table 21 Change in Academic Risk From Beginning to End of Study by Study Group (Initially Classified as Moderate Risk) Summary and Conclusions SEG Measurement conducted a study of the effectiveness of ML instruction incorporating Connect with grades 1, 2, and 3 students in a midsized midwestern U.S. school district during the 2022–2023 school year. Students were assessed using the iReady reading skills assessment in the fall of 2022 (pretest) and in spring 2023 (posttest). First, we examined the reading achievement of multilingual learners receiving instruction incorporating Connect to assess the reading skills growth achieved from the beginning to the end of the school year. Next, we compared the level of reading skills growth for the ML learners using Connect to the i-Ready norms and growth targets suggested by the publisher. We then compared the reading skills growth of ML learners to a comparison group of non-multilingual learners in the district to evaluate whether MLs achieved reading skills growth comparable to their non-ML peers. Finally, we examined the reduction in academic risk achieved by ML learners in the treatment group and compared the reduction in academic risk to the risk reduction for the comparison group of non-ML learners. Study Group Total Increase 1 Risk Category No Change Reduce 1 Risk Category Comparison (Non MLs) 10 33 73 116 Treatment (MLs using Connect) 12 48 60 120 Total 22 81 133 236 “Overall, the MLs achieved reading skills gains beyond the level of growth expected…” Many ML learners showed significant progress in reducing their academic risk.
26 Findings Overall, we found that instruction incorporating Connect was effective in improving ML learners’ reading skills and reducing ML learners’ level of academic risk. Finding 1: Multilingual learners receiving instruction including Connect achieved about one standard deviation of reading skills growth from pretesting to post testing. The group of 423 ML learners in grades 1, 2, and 3 who received instruction incorporating Connect achieved statistically significant reading skills improvement. The comparison of their beginning-of-year reading skills assessment scores to the end-ofyear assessment scores revealed about one standard deviation of reading skills growth from pretest to posttest. This effect size is substantial, particularly for an educational intervention (Kraft, 2019; Lipsey et al, 2012). Finding 2: Multilingual learners taught using Connect achieved reading skills growth exceeding the expected growth levels identified for i-Ready. We compared the growth level of each ML learner in the treatment group (Connect users) to the typical growth level for i-Ready test takers, that is, the median scores from the national norms sample for i-Ready Curriculum Associates (2023). Overall, the ML learners receiving instruction incorporating Connect began the year with a lower level of reading skills than the median expectation reflected in the i-Ready norms (mean initial reading skills at the 33nd to 40th percentile). Overall, the MLs achieved reading skills gains beyond the level of growth expected, based on the median achievement levels. While two-thirds of the MLs achieved their growth targets, one-third of the MLs did not. The large proportion of MLs able to meet the growth targets suggests that providing instruction including Connect was effective at making headway in reducing reading deficits and reducing disparities between MLs and their non-ML peers. “… instruction incorporating Connect was effective in improving ML learners’ reading skills and reducing ML learners’ level of academic risk.”
27 Finding 3: Multilingual learners receiving instruction incorporating Connect grew at about the same rate as a matched group of their non-ML peers. We compared the performance of the ML learners in the treatment group to the performance of a matched sample of non-MLs of similar background, but whose first language was English, using a matched-groups, pre–post quasi experimental design. After adjusting for any differences in the beginning-of-year reading skills level, the MLs in the treatment group receiving instruction using Connect achieved levels of growth comparable to non-MLs in the comparison group. There was no significant difference in growth in reading skills between students receiving instruction with Connect and intervention without Connect. Finding 4: About half of the treatment group initially classified as high or moderate risk who received Connect instruction reduced their level of academic risk. We examined the academic risk levels of both the treatment and comparison groups and the change in academic risk over the course of instruction. Both study groups showed evidence of reducing their academic risk and were equally effective in reducing their risk level from the beginning to the end of instruction. Overall, nearly half (44%) of the MLs receiving instruction incorporating Connect reduced their level of academic risk and more than half (56%) of the comparison group students reduced their level of academic risk from start to finish. About two-thirds (66%) of the treatment group reduced their risk level from the beginning to the end of instruction. Of those MLs initially classified as high risk, nearly half (44%), reduced their risk level to moderate or low risk from the beginning to end of instruction. Half (50%) of the MLs initially classified as being at moderate academic risk reduced their risk level to low risk from the beginning to the end of instruction. The rate of risk reduction was comparable to the level of risk reduction seen within the non-ML comparison group. “About half of the treatment group initially classified as high or moderate risk who received Connect instruction reduced their level of academic risk.” “Multilingual learners receiving instruction incorporating Connect grew at about the same rate as a matched group of their non-ML peers.”
28 Conclusions This study provides multiple sources of evidence that support the claim that Connect is effective in improving grade 1, 2, and 3 multilingual learners’ reading skills. We found evidence supporting Connect’s effectiveness for all four research questions examined. Effective reading skills are critical to success in school, work, and life. It is critical for ML learners to create a foundation for academic learning across all subjects, as well as for lifelong learning and career success. The results of this study support the value of early intervention in general and the use of Connect in specific. These results demonstrate the effectiveness of Connect in reducing deficits for students at academic risk and the power of Connect to provide these students with the tools necessary for academic success. Effective instruction using Connect can mitigate initial deficits in reading skills and enable MLs to close the gap with their native English–speaking peers. Clearly, these results support Connect’s ability to assist MLs in achieving substantial levels of reading skill growth, reach established standards, and help MLs avoid falling further behind. The evidence supports Connect’s effectiveness in fostering MLs reading skills growth, meeting or exceeding growth targets, achieving growth comparable to their non-ML peers, and reducing their level of academic risk. While not all the participating MLs achieved high levels of reading success, the success rate for the ML learners in this study is impressive. The MLs receiving instruction incorporating Connect were, as a group, as successful as their non-ML peers. About two-thirds of the MLs receiving instruction incorporating Connect achieved their growth targets, and nearly half were able to reduce their level of academic risk. Nearly all (93%) of participating students maintained or reduced their academic risk level, enabling MLs, at minimum, to avoid falling further behind. This is a particularly significant outcome given the large and increasing population of MLs. Future research would do well to explore the conditions under which Connect is most effective to further build on the levels of success seen in this study. In conclusion, we found that ML learners receiving instruction incorporating Connect achieved significant reading skills growth, met growth targets, achieved growth comparable to their non-ML peers, and reduced their academic risk level. The evidence supports the conclusion that Connect is effective in improving ML learners’ reading skills. “The evidence supports Connect’s effectiveness in fostering MLs reading skills growth, meeting or exceeding growth targets, achieving growth comparable to their non-ML peers, and reducing their level of academic risk.” “…the success rate for the ML learners in this study is impressive. The ML learners receiving instruction incorporating Connect were, as a group, as successful as their non-ML peers.”
RkJQdWJsaXNoZXIy MjUyNzA0NQ==