Evaluating the Effectiveness of Vista’s Connect: Language, Literacy, Content for Improving Elementary Students’ Reading Skills 2023 Click here to learn more about Connect. Click here to contact us.
1 Evaluating the Effectiveness of Vista’s Connect: Language, Literacy, Content for Improving Elementary Students’ Reading Skills1 During the 2022–2023 school year, SEG Measurement conducted a study of the effectiveness of Vista’s Connect: Language, Literacy, Content for improving the reading skills of elementary level multilingual learners (MLs). The study was conducted with students in grades one, two, and three. We found that MLs receiving instruction incorporating Connect achieved substantial growth in reading skills and at levels exceeding what is typically seen for MLs nationally. Overview The first part of this document describes the intervention, the design of the study, and the research questions addressed. The second part of the document presents the analyses and results. First, we examined the extent of reading skills growth over the course of the school year achieved by MLs who were provided instruction incorporating Connect. Second, we compared the level of reading skills growth observed for the MLs receiving instruction incorporating Connect to the level of growth achieved by a national sample of MLs. Third, we examined the performance of the district MLs receiving instruction with Connect in relation to a comparison group of non-MLs in the district that received instruction that did not incorporate Connect. Overall, SEG Measurement found that MLs receiving instruction that included Connect achieved substantial reading skills growth exceeding typical levels of growth seen for MLs and reading skills growth similar to their non-ML peers in grades one, two, and three. A complete summary of the findings and conclusions are presented in the final section of the document. 1In addition to Connect, newcomer students (beginner/emergent/WIDA 1) used Vista’s Get Ready! Sail. Multilingual students in classes receiving instruction that included Connect achieved substantial levels of reading skills growth exceeding typical levels of growth for multilingual students, and at levels similar to their non-multilingual peers in grades one, two, and three.
2 Early Intervention Is Critical Independent, fluent reading is key to success in school, work, and life. Research has proven that children who have difficulty reading at the end of first grade rarely catch up by the end of elementary school (Torgesen, 2004; Francis et al., 1996; Juel, 1988; Shaywitz et al., 1999; Torgesen & Burgess, 1998). When reading difficulties are scrutinized further, findings reveal that when students do not master foundational skills early on (by third grade), there is a long-term negative impact on their reading ability and overall academic success. Students failing to read effectively by the end of third grade have a nearly five times greater rate of dropping out of high school before graduation. Early intervention is research proven as an effective measure for reducing—and sometimes altogether preventing—reading disabilities for students at risk of reading failure (Connor et al., 2014; Gersten et al., 2008). Acquiring reading skills can be challenging for students for whom English is not their first language. One out of every ten students is classified as EL (English Learner)/ML at some point during their K–12 schooling (U.S. Department of Education [DOE], 2018). Since 1968, the federal government has provided financial assistance to schools for elementary and secondary school programs to meet the special needs of “limitedEnglish-speaking” students. Since that time, eligibility has been expanded to all students with limited English proficiency (Stewner-Manzanares, 1988). The goal of support has shifted over time to developing English proficiency and assimilation into mainstream classrooms, with English-only services replacing many bilingual programs (Ovando, 2003). In the 2000s, this goal shift culminated with No Child Left Behind and the Common Core State Standards, placing English language arts skills front and center (Garía & Kleifgen, 2018). Research has shown that not only is EL/ML status associated with lower academic expectations from teachers and students themselves (Kanno & Kangas, 2014), but also with a lower likelihood of taking general and advanced courses in core subjects (Estrada, 2014; Umansky, 2016a, 2018). This has contributed to lower academic achievement (Umansky, 2016b). In terms of academic growth, Soland and Sandilos (2020) found that students classified as EL/ML at any point in their academic career had lower achievement than their non-EL/ML peers. The increasing need for solutions to help ELs/MLs acquire English language skills has led to the development of several products and services to help ELs/MLs develop language skills. Among those products are Connect and Get Ready!, which are the focus of this study. The Intervention Students identified as ML by the district in the grades one, two, and three classrooms examined in this study were provided with sustained instruction incorporating Connect on a weekly basis. In addition, newcomer students were provided instruction using Get Ready! Sail. Teachers were provided with training in the use of Connect and Get Ready! Sail at the outset of the program and had access to district ML instructional support personnel for continued support in delivering intervention.
3 Connect provides evidence-based instruction incorporating current research-driven learning science and language development practices. Connect is a comprehensive literacy program created for MLs/ELs and striving readers. The program builds language and reading proficiency through research-driven lessons and activities and engaging texts and content-driven lessons. The program provides a variety of print and digital classroom resources that provide support and motivation. Connect provides tools for teachers to assess ongoing learning progress, discover skills gaps, and personalize instruction. Research Questions/Demonstrating Effectiveness Demonstrating that Connect is effective is a process of marshalling evidence in support of the claim that instruction incorporating Connect increases student reading skills. No single study can conclusively establish this claim; multiple sources of evidence enhance the claim that Connect is effective in developing students’ reading skills. This document presents three pieces of evidence supporting the claim that Connect is effective: • First, we examine the level of reading skills growth achieved by MLs using Connect over the course of a school year, from pretesting to posttesting. • Second, we compare the level of growth achieved by students receiving instruction incorporating Connect to a national sample of MLs receiving ML instruction based on a range of instructional approaches and materials. • Third, we compare the level of reading skills growth achieved by MLs in the district receiving instruction incorporating Connect to the level of growth achieved by non- MLs in the district (receiving instruction not incorporating Connect). We posed the following three research questions. The answers to these questions provide evidence supporting the effectiveness of the Connect solution. Question 1: Do grade one, two, and three MLs in the district receiving instruction incorporating Connect (treatment group) achieve significant growth in reading skills over the course of a school year? What is the magnitude of the growth achieved? Question 2: Do grade one, two, and three MLs in the district receiving instruction incorporating Connect (treatment group) achieve reading skills growth greater than a national sample of MLs receiving reading skills instruction using other instructional approaches and materials(comparison group)? Question 3: Do grade one, two, and three MLs in the district receiving instruction incorporating Connect (treatment group) in the district achieve reading skills growth similar to a comparison group of non-MLs in the district not receiving instruction incorporating Connect (comparison group)?
4 Study Population and Sampling The sample of students participating in this study was drawn from two sources: the population of grade one, two, and three students in the school district studied in this research and a national sample of approximately 3,200 grade one, two, and three MLs. We drew the district sample from a file containing all students at the three grade levels, including approximately 278 MLs receiving instruction incorporating the treatment and the remaining 1,173 non-MLs who did not receive instruction incorporating Connect. The treatment group was composed of the 278 MLs receiving the instruction incorporating Connect; the comparison group was composed of a sample of the group of remaining non-MLs at the same grade levels not receiving instruction incorporating Connect. The national sample was drawn from the sample of MLs completing the MAP assessment in districts across the US. Study Sample for Question 1 The study sample used to answer Question 1 was the total treatment group of 278 MLs receiving instruction incorporating Connect. Four students were lost to attrition. These four students took the pretest but left the district or otherwise did not have a posttest score available. Study Sample for Question 2 The 278 ML student sample comprising the treatment group for Question 1 was augmented to include a national sample of MLs. NWEA provides a large research database containing nearly 60,000 students nationwide. The approximately 3,200 students classified as ML throughout their grade one, two, and three experience drawn from the NWEA research database were used as a basis for answering Question 2. Study Sample for Question 3 The sample of 310 students used to answer Question 3 included 155 students drawn from the treatment group and 155 students drawn from the comparison group, a sample of the remaining same grade level students not receiving instruction incorporating Connect. Creating Matched Samples Given that random sampling was not possible for this study, SEG Measurement used a matched groups quasi-experimental design. For each student in the treatment group, an individual from the control group similar in ability and background was selected. We created matched treatment and control groups using propensity score matching, an advanced statistical technique that permits matching based on multiple factors. Using a logistic regression-based propensity score model, students were matched on a composite propensity score created from initial reading ability (MAP English language assessment pretest scores), their sex, ethnicity, and free or reduced-cost lunch eligibility. Sampling was done without replacement and the composite propensity score for the students in the matched pair was required to be within 10% of each other.
5 One thousand four hundred and fifty-six (1,456) first, second, and third grade students were included in the matched group analysis, which included the 278 MLs in the treatment group (from Study 1) and the remaining 1,178 district non-MLs enrolled in first, second, or third grade. One hundred and fifty-five (155) treatment-comparison pairs meeting the criteria were identified for the final analysis sample of 310 students—half treatment, and half control. Six students (2%) were lost to attrition; these students left the district before the posttest or otherwise did not have a posttest score available. The descriptive statistics for the total population from which students were sampled and the sampling protocol are described under each study question. Data Collection and Measurement The participating school district administered the MAP reading skills assessment published by NWEA at the beginning and end of the school year to all students. The beginning- and end-of-year student scores for the assessment served as the pretest and posttest scores for the study. The district provided SEG Measurement with a data file including all the grade one, two, and three students in the district. The file contained the beginning- and end-of-year MAP reading assessment results, along with a designation of whether the student received instruction incorporating the Connect solution, the student’s grade level, sex, ethnicity, an indication of whether the student was eligible for free or reduced-cost lunch as an indicator of economic status, and a unique student identifying number. MAP Assessment The MAP reading assessment is reported on a scale ranging from 100–350, covering grades K through 12. The MAP scores are vertically scaled; this means that a single scale is used across all grades to allow comparability of scores both within and across grades. NWEA reports that the reliability for the grade one, two, and three reading assessments ranges from .96 to .97. NWEA has conducted numerous studies supporting the validity of the MAP, indicating that “validity evidence for MAP Growth assessments involves multiple sources including test content, internal structure, and relations to other variables” (NWEA, 2023). The MAP reading assessment provides an overall score and several subtest scores. However, the subtests vary between grades one, two, and three. For that reason, this study used the overall language score as the outcome measure.
6 MAP Reading Skills Pretest MAP Reading Skills Posttest ML Instruction Using Connect Question 1 Evaluation Determining the Impact of the Connect Solution on Treatment Group Student Growth Do MLs receiving instruction incorporating Connect (treatment group) show significant growth in reading skills over the course of a school year? To answer question 1, we examined the growth in reading skills achieved from the beginning to the end of the year by the treatment group, composed of MLs that received instruction incorporating Connect, using the beginning- and end-of-year MAP assessment. This study design is illustrated in Figure 1. Figure 1 Question 1 Study Design Sample There were 278 MLs included in the district data file who were included in the analyses. The students were drawn from all 6 elementary schools in the district. Four of the 278 students who began the study left prior to the posttest or were otherwise unable to take the posttest. This reflects a total attrition rate of less than 2%. The comparative analyses requiring both pretest and posttest results were completed using the remaining 274 students. The number and percentage of students in each subgroup category were calculated to characterize the sample. The results are summarized in Tables 1 and 2 below. Reading Skills Level The average (mean) MAP reading skills pretest score for the 278 students in the treatment group was 156.37 (SD=15.47). This is an indicator of the initial reading skills level of the treatment students at the beginning of the study. Grade Level About one-quarter of the students (26%) were enrolled in grade one. The remaining three-quarters of the participants were enrolled in grades two and three (74%). Beginning of Year End of Year
7 Table 1 Treatment Group Sample Number of Students by Grade Level Background Profile: Sex, Ethnicity, and Economic Level About three-fifths of the students were boys (61%) and the remaining two-fifths were girls (39%). Nearly two-thirds (63%) of the sample were Hispanic. Most of the remaining students were classified as Black (17%) or Asian (17%). Most of the students were of low economic status; nearly all (86%) of the sample qualified for free or reduced-cost lunch. Table 2 Treatment Group Sample Number of Students by Sex, Ethnicity, and Economic Level Grade Level N Percent Grade 1 72 26 Grade 2 112 40 Grade 3 94 34 Total 278 100.0 Background N Percent Sex Boys 169 61 Girls 109 39 Ethnicity Asian 47 17 Black 48 17 Hispanic 174 63 White 9 3 Economic Level No Eligibility 33 12 Reduced Eligibility 26 9 Free Eligibility 219 79 Total 278 100.0
8 Overall Growth in Reading Skills At the beginning of the school year (fall of 2022), the treatment group of MLs receiving instruction incorporating Connect were administered the MAP reading skills pretest; the treatment group was again assessed at the end of the school year (posttest). Reading skills growth from fall 2022 to spring 2023 (the mean difference in pretest and posttest scores) was about 19 points in grade one and about 18 points in grades two and three. The reading skills growth in grade one (t=13.89; df=273; p<.016), grade two (t= 18.11; df=273; p<.001), and grade three (t=14.23; df=273; p<.001) was statistically significant. The pretest and posttest data are illustrated in Figure 2 and summarized in Table 3 below. Table 3 Pretest to Posttest Reading Skills Growth by Grade Level Mean MAP Reading Skills Scores The students’ reading skills grew about one-and-a-half standard deviations. Specifically, the effect size computed across all grades was 1.6 (Cohen’s D). The effect size for grade one and grade two growth was 1.7, and the grade three effect size was 1.5. This is generally considered a large effect size, indicating that students in the treatment group receiving instruction incorporating Connect achieved substantial reading skills growth. Grade N Pretest Mean Pretest SD Posttest Mean Posttest SD Pre- Post Mean Diff Pre- Post Diff SD Grade 1 69 140.94 9.60 160.43 9.95 19.49 11.65 Grade 2 112 156.63 10.61 174,64 12.13 18.01 10.52 Grade 3 93 167.98 13.76 185.49 17.90 17.52 11.87 All Grades 274 156.53 15.44 174.75 16.84 18.22 11.23 This is generally considered a large effect size, indicating that students in the treatment group receiving instruction incorporating Connect achieved substantial reading skills growth.
9 Treatment Group Differences in Growth by Sex, Ethnicity, and Economic Status We then analyzed the results to determine if the treatment (instruction incorporating Connect) was more effective for any population subgroup. We looked at the differences in growth achieved by boys and girls, different ethnic groups, and different economic levels. The results are illustrated in Figure 3 and summarized in Tables 4 and 5, below. The ANCOVA model failed to detect any significant difference between any of the subgroups examined. There was a one-point difference in MAP reading skills growth between boys and girls, which was not statistically significant (F=.45; df=1/274; p<.503). The less than two-point difference in reading skills growth observed between different ethnic groups was also not significantly significant (F=.06; df=3/274; p<.980). There was a less than two-point difference in reading skills growth among different student economic levels. The differences were not statistically significant (F=.14; df=2/274; p<.871). These results indicate that the instruction incorporating Connect did not advantage one subgroup over another; it was equally effective for boys and girls, different ethnic groups, and different student economic levels. Table 4 ANCOVA Treatment and Comparison Group Mean MAP Reading Skills Posttest Scores Covariates appearing in the model are evaluated at the following values: MAP Reading Skills Pretest = 156.53. Source Type III Sum of Squares Df Mean Square F Significance Corrected Model 46851.83 19 2465.89 20.51 .001 Intercept 5369.83 1 5369.83 44.67 .001 MAP Reading Skills Pretest 4016513 1 40165.23 334.1 .001 Gender 54.11 1 54.11 .45 .503 Ethnicity 22.43 3 7.48 .06 .980 Economic Status 33.20 2 16.60 .14 .871 Sex * Ethnicity 139.66 2 69.83 .58 .560 Sex * Economic Status 36.60 2 18.30 .15 .859 Ethnicity * Economic Status 391.51 5 78.30 .65 .661 Sex * Ethnicity * Economic Status 451.13 3 150.38 1.25 .292 Error 30535.80 254 120.22 Total 8444505.00 274 Corrected Total 77387.62 273
10 Table 5 Comparison of the Treatment by Sex, Ethnicity, and Economic Status Mean MAP Reading Skills Posttest Scores (Adjusted for initial reading ability) Background N Mean SD Sex Boys 167 175.27 17.45 Girls 107 174.27 15.86 Total 274 174.85 16.85 Ethnicity Asian 46 175.96 16.34 Black 48 174.60 13.82 Hispanic 172 174.19 17.31 White 8 174.80 19.85 Total 274 174.85 16.85 Economic Level No Eligibility 33 175.73 17.26 Reduced Eligibility 26 174.90 18.59 Free Eligibility 215 173.93 16.56 Total 274 174.85 16.85 … the instruction incorporating Connect did not advantage one subgroup over another; it was equally effective for boys and girls, different ethnic groups, and different student economic levels.
11 District ML Treatment Group MAP Reading Skills Pretest MAP Reading Skills Posttest ML Instruction Using Connect National ML Comparison Sample ML Instruction without Connect Question 2: Evaluation: Comparison of ML Treatment Group Reading Skills Growth to National Sample of Multilingual Learners (Comparison Group) Do grade one, two, and three MLs receiving instruction incorporating Connect (treatment group) show levels of reading skills growth comparable to a national sample of MLs? Sample The sample was used to compare the level of reading skills growth achieved by the ML treatment group using Connect and a national sample of ML students using a range of other instructional approaches and materials. NWEA, the publisher of the MAP assessments, provides normative data and a research database for use by educators, students, and parents in understanding MAP scores (NWEA, 2023). While the interpretation of MAP scores should include reference to the content domain and standards established by educators, normative data remains an important source of interpretive information. This study design is illustrated in Figure 4 below. Figure 4 Question 2 Study Design The NWEA research database includes approximately 60,000 K–12 students; the data is identified with respect to sex, ethnicity, economic level, and identification of students as ML. The norms for the grades one, two, and three fall and spring MAP assessments examined in this study are provided in Table 6 below. There is less MAP assessment information available examining the performance of MLs. However, Johnson (2022) conducted a national study of approximately 55,000 students, examining the achievement and growth of MLs. A cohort of 3,971 students consistently classified as ML across their grade one through four elementary education were studied to obtain the research data and results reported in this document; the students were
12 assessed using the MAP math and reading assessments multiple times per year. This is a large national sample, but Johnson (2022) cautions that this may not be a representative sample. The reading skills of the treatment group students receiving instruction using Connect grew by 19 points in grade one, 18 points in grade two, and 18 points in grade three. Comparative data provided by Johnson (2022) for MLs in the national sample indicated that grade one MLs grew by about 15 points, while grade two MLs grew about 13 points and grade three non-MLs grew by about 11 points. The gains achieved by MLs in the district were significantly greater than those achieved by MLs in the national sample (Johnson, 2022). In grade one, the district MLs statistically outperformed the national sample of MLs by about four points (t=2.85; df= 69; p<.05). In grade two, the MLs receiving instruction incorporating Connect in the District outperformed the national sample by nearly five points. (t=4.78; df=114; p<.001). The third graders in the district significantly outperformed the national sample by about seven points (t=5.67; df= 94; p<.001). This is illustrated in Figure 5 and summarized in Tables 7 and 8 below. Table 7 MAP Reading Skills Assessment Scores for National Sample of EL/ML Students (Johnson, 2022) Mean MAP Reading Skills Scores Table 8 Comparison of District EL/ML Student Growth to National EL/ML Student Sample Growth Mean MAP Reading Skills Scores Grade Level EL/ML Student Fall Mean EL/ML Student Spring Mean Non-EL/ML Student Fall Mean Non-EL/ML Student Spring Mean Grade 1 151.58 166.82 161.20 177.81 Grade 2 168.80 183.00 178.48 193.36 Grade 3 183.69 194.54 192.97 203.63 Grade Level District EL/ML Mean Growth Diff District EL/ML Growth SD National EL/ML Sample Mean Growth Diff National EL/ML Sample Growth SD Grade 1 19.49 11.66 15.47 7.74 Grade 2 18.01 10.52 13.22 7.77 Grade 3 17.52 11.87 10.50 7.77
13 Question 3 Evaluation: Comparison of Treatment and Comparison Group Reading Skills Growth Do ML students receiving instruction incorporating Connect (treatment group) achieve levels of reading skills growth similar to a control group of students not receiving traditional instruction incorporating Connect? Question 3 examined the reading skills growth achieved by ML students in the district in relation to non-ML students in the district with similar starting reading ability and similar sex, ethnic, and economic background to provide evidence of the effectiveness of Connect. We compared the reading skills growth of district ML students receiving Connect instruction to the reading skills growth of a matched group of district students whose first language was English but who shared a similar starting reading skill level and a similar sex, ethnic and economic background. While the reading skills level of both the treatment and control group were comparable, as were the sex, ethnicity, and economic levels, we recognized that the MLs may face a number of academic and social challenges not faced by their peers whose first language was English. Our investigation was looking to determine the extent to which the district ML students achieve reading skills growth approaching the levels of growth achieved by a matched group of their non-ML peers The extent to which the reading skills growth of the ML group approaches that of the non-ML group provides evidence of the effectiveness of the Connect intervention. Study Design A quasi-experimental design was used to answer Question 3. This design compares the reading skills growth of students in the treatment group to the comparison group receiving instruction without the benefit of the treatment. We compared the treatment group to a comparison group drawn from the remaining grade one, two, three students in the district who did not receive instruction incorporating Connect. The level of reading skills growth for the treatment group was compared to the level of growth achieved by the comparison group. This study design is illustrated in Figure 6.
14 Figure 6 Question 3 Study Design Sample The sample used to compare the level of reading skills growth achieved by the ML treatment group using VHL Connect and a comparable group of non-ML students followinngue instruction. The treatment group used for the first part of the study served as the treatment group for this second part of the study as well. The comparison group was drawn from the remaining grades one, two, and three students in the district—those who were not categorized as ML and who were receiving reading instruction following traditional practices (i.e., not using Connect). The reading skills posttest scores (dependent variable) achieved by students in the treatment group (independent variable) and the comparison group (independent variable), adjusting for the pretest scores (covariate) were examined. Matched Groups The treatment and comparison samples were matched based on initial reading skills ability, sex, ethnicity, and economic status using propensity score matching. This procedure matches students based on multiple characteristics, in this case the initial reading skills level of the student, along with their sex, ethnicity and economic levels. A single value representing the composite status of each student with respect to the matching variables was created. For each student included in the treatment group, the procedure sought a comparable student in the comparison group. The matches were required to be within 10% of each other to be considered acceptable. One hundred fifty-five (155) matched pairs were identified, for a total analysis sample size of 310 students. The final analysis included 89 students from grade one, 123 second-grade students and 98 students from grade three. A profile including the initial reading skills ability and background (sex, ethnicity, and economic level) of the treatment and comparison group samples is provided below. The average (mean) pretest reading scores for the treatment and control groups, along MAP reading Skills Posttest gg MAP reading Skills Pretest MAP reading Skills Posttest Comparison Group Group h MAP Reading Skills Pretest Instruction Using Connect District ML Treatment Group MAP Reading Skills Posttest No Connect Use MAP Reading Skills Pretest MAP Reading Skills Pretest District Non-ML Control Group
15 with the number and percent of students in each category of each background variable are also provided below. Comparability of Treatment and Comparison Group Background Initial Reading Skill Level The initial reading skill level of the treatment and control groups were comparable. The difference between the treatment group (mean=163.40) and the comparison group (mean=159.59) was approximately four points. This represents about a quarter of a standard deviation ceiling (SD=16.37) difference; this is well within the1 standard deviation maximum and one-half standard deviation preferred for group differences recommended for statistical comparisons (e.g., Shavelson). This is summarized in Table 9 below. The observed difference was not statistically significant (F=3.85; df=1/308; p<.07). Any remaining differences in the starting ability level of the study groups were adjusted for in the ANCOVA comparisons described below. Table 9 Treatment and Comparison Group Sample Mean Initial Learning Skill Level Sex, Ethnic and Economic Composition The number of boys and girls in the treatment and comparison group were similar, as was the economic level of the two groups. Not surprisingly, ethnicity was similar except for the larger number of students classified as white in the comparison group and larger number of students classified as Hispanic in the treatment group. This is summarized in Table 10 below. Statistics N Mean Std. Deviation Treatment 155 163.40 15,12 Comparison 155 159.59 17.38 Total 310 161.50 16.37
16 Table 10 Treatment and Group Sample Number of Students by Sex, Ethnicity, and Economic Status Comparison of Treatment and Control Group Reading Skills Growth We examined the differences between the level of reading skills growth in the treatment and control groups using analysis of covariance (ANCOVA). This approach examines the Mean posttest reading skills scores for the treatment group to the Mean posttest reading skills scores for the control group adjusting for the pretest reading skills scores. This adjustment puts the treatment and control group on the same baseline; ensuring that the “starting place” for both the treatment and control groups provides the fairest comparison. The difference in the average performance for both the treatment and control group after this adjustment allows us to see the specific contribution of the treatment (Connect). Background Comparison N Treatment N Total Sex Boys 81 92 173 Girls 74 63 137 Total 155 155 310 Ethnicity Asian 28 35 63 Black 45 47 92 Hispanic 48 65 113 Mixed 6 0 6 White 28 8 36 Total 155 155 310 Economic Level No Eligibility 36 28 64 Reduced Eligibility 14 15 29 Free Eligibility 105 112 217 Total 155 155 310
17 Specifically, we used ANCOVA to examine the difference between the treatment and control group (independent variable) end-of-year reading skills growth (posttest dependent variable), adjusting for any differences in the beginning-of-year reading skills level (pretest covariate). ML students receiving intervention that included instruction using Connect (Mean=179.85) showed no significant difference in growth in reading skills than did students receiving intervention without Connect (mean=180.34) (F = .03, df=1/310; p<.683). The half-point (.49) difference in means is less than a tenth of a standard deviation. In short, the ML students taught using Connect achieved about the same level of growth in reading skills as non-ML students receiving traditional instruction. This is described in Tables 11 and 12 below. Table 11 Treatment and Comparison of Mean MAP Reading Skills Posttest Scores (ANCOVA) (Adjusted Pretest Scores) Table 12 Comparison of the Treatment and Comparison Group Reading Skills Growth Mean MAP Reading Skills Posttest Scores (Adjusted for Initial Reading Skills Level) Covariates appearing in the model are evaluated at the following values: MAP Reading Skills Pretest = 161.50. Source Type III Sum of Squares Df Mean Square F Significance Corrected Model 46851.83 19 2465.90 20.51 .001 Corrected Model 58300.54 2 29150.27 272.94 .001 Intercept 6102.24 1 6102.24 57.14 .001 MAP Reading Skills Pretest 57728.80 1 57728.80 540.53 .001 Study Group 17.85 1 17.85 .17 .683 Error 32787.75 307 106.80 Total 10145531.00 310 Corrected Total 91088.29 309 Study Group N Mean SD Comparison Group 155 180.34 18.15 Treatment Group 155 179.85 16.08 Total 310 180.09 17.17
18 Summary and Conclusions SEG Measurement conducted a year-long study of the effectiveness of ML instruction incorporating Connect with grade one, two, and three students in a single district during the 2022–2023 school year. Participating students were assessed using the MAP reading skills assessment published by NWEA in fall 2022 and again in spring 2023. The reading achievement of MLs receiving instruction incorporating Connect was examined to assess the reading skills growth achieved from the beginning to the end of the school year. We then compared the level of reading skills growth for this group of MLs using Connect to the level of growth achieved by a national sample of MLs using a range of instructional materials and approaches. We then compared the reading skills growth of MLs and non-MLs in the district to evaluate whether MLs achieved reading skills growth comparable to their non-ML peers. Findings We found that instruction incorporating Connect effectively increased the reading skills of the treatment group of 278 grade one, two, and three MLs. The group of 278 MLs receiving instruction incorporating the Connect intervention achieved statistically significant reading skills improvement. This was determined by comparing the beginning-of-year reading skills assessment scores to the end-of-year assessment scores (t= 28.3; df=277; p<.001); this comparison revealed a difference of about one-and-ahalf standard deviations of growth (ES=1.6). This effect size is substantial, particularly for an educational intervention. Lipsey et al. (2012) indicate that effect sizes above .25 can be considered large for an educational intervention. Kraft (2019) notes that traditional guidelines, such as those offered by Cohen and even those suggested by Lipsey et al, may be too rigid for use in evaluating educational interventions. …instruction incorporating Connect effectively increased student reading skills… …district students receiving instruction using Connect improved their reading skills substantially more than observed for the national sample of users implementing other ML instructional materials and programs.
19 We also examined how the reading skills growth seen for ML students receiving instruction using Connect stacked up against the level of growth seen in a broader national sample of ML students. We found that the district students receiving instruction using Connect improved their reading skills substantially more than was observed for the national sample of users implementing other ML instructional materials and programs. The district treatment group receiving instruction using Connect achieved growth consistent with an effect size of .5 to 1 when compared to the national sample of ML students. Finally, we compared the performance of the Connect treatment group to the performance of a matched sample of non-ML students of similar background, but whose first language was English. We looked to see if the ML students using Connect were able to achieve levels of growth similar to non-ML students in the district. We found that the ML students receiving instruction using Connect were able to achieve similar levels of reading skills growth as achieved by their non-ML counterparts of similar background. After adjusting for any differences in the beginning-of-year reading skills level, the ML students receiving intervention that included instruction using Connect (mean=179.85) showed no significant difference in growth in reading skills from students receiving intervention without Connect (mean=180.34). The ML students taught using Connect achieved about the same level of growth in reading skills as non-ML students receiving traditional reading instruction in the district. Conclusions The multiple sources of evidence obtained in this study strongly support the conclusion that Connect is effective in improving grade one, two, and three ML student reading skills. Positive evidence supporting the effectiveness of Connect was found for all three research questions examined. First, we found that instruction including Connect resulted in about one-and-a-half standard deviations of reading skills growth among ML students over the course of the school year. Second, we found that MLs in the district taught using Connect achieved reading skills growth exceeding the growth seen within a national sample of MLs receiving instruction using a variety of instructional practices by about four to seven points. This is about one-half to one standard deviation greater reading skills growth than found for ML students in the national sample. Third, we found that MLs in the district receiving instruction incorporating Connect grew at about the same rate as a matched group of their non-ML peers in the district; the comparability in growth was seen despite the challenges one might expect for MLs whose primary language is not English. In summary, the evidence supports the conclusion that Connect is effective in improving grade one, two, and three ML student reading skills.
20 References Bloom, H.S., Hill, C.J., Black, A.B., and Lipsey, M.W. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. Journal of Research on Educational Effectiveness, 1(4):289–328. Connor, C.M., Alberto, P.A., Compton, D.L., & O’Connor, R.E. (2014). Improving reading outcomes for students with or at risk for reading disabilities: A synthesis of the contributions from the Institute of Education Sciences Research Centers (NCSER 20143000). Washington, DC: National Center for Special Education Research, Institute of Education Sciences, US Department of Education. Francis, D.J., Shaywitz, E.E., Stuebing, K.K., Shaywitz, B.A., and Fletcher, J.M. (1996). Developmental lag versus deficit models of reading disability: A longitudinal, individual growth curves analysis. Journal of Educational Psychology, 88(1): 3–17. Gersten, R., Compton, D., Connor, C.M., Dimino, J., Santoro, L., Linan-Thompson, S., & Tilly, W.D. (2008). Assisting students struggling with reading: Response to Intervention (RtI) and multi-tier intervention in the primary grades. A practice guide. (NCEE 2009–4045). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education. International Literacy Association. (2020). Intensifying literacy instruction in the context of tiered interventions: A view from special educators [Literacy leadership brief]. https://www.literacyworldwide.org/docs/default-source/where-we-stand/ilaintensifying-literacy-instruction.pdf Johnson, A. (2022). Achievement and growth for multilingual learners. (NWEA Center for School and Student Progress Working Paper). Juel, C. (1988). Learning to read and write: A longitudinal study of 54 children from first through fourth grades. Journal of Educational Psychology, 80(4): 437–447. https://psycnet.apa.org/doiLanding?doi=10.1037%2F0022-0663.80.4.437 Lipsey, Mark W., Puzio, KMLey, Yun, Cathy, Hebert, Michael A., Steinka-Fry, Kasia, Cole, Mikel W., Roberts, Megan, Anthony, Karen S., Busick, Matthew D. (2012) Translating the statistical representation of the effects of education interventions into more readily interpretable forms. (NCSER 2013–3000) Washington, DC: National Center for Special Education Research, Institute of Education Sciences, US Department of Education, National Equity Project (n.d.). [Educational equity definition]. Retrieved from https://www.nationalequityproject.org/ on November 11, 2020. NWEA (2023) NWEA News.2020 MAP Growth Norms Overview and FAQ. Retrieved from https://Connection.nwea.org/ August 26, 2023. Schneider, B. M. Carnoy, J. Kilpatrick, W.H. Schmidt, R.J. Shavelson. (2007). Estimating causal effects: Using experimental and observational designs. Washington, D.C. American Educational Research Association.
21 Shaywitz, S.E., Fletcher, J.M., Holahan, J.M., Schneider, A.E., Marchione, K.E., Stuebing, K.K., Francis, D.J., Pugh, K.R., & Shaywitz, B.A. (1999). Persistence of dyslexia: The Connecticut longitudinal study at adolescence. Pediatrics, 104(6): 1351–1359. Swanson, H.L. (1999). Reading research for students with LD: A meta-analysis of intervention outcomes. Journal of Learning Disabilities, 32: 504–532. Torgesen, J.K. (2004). Avoiding the devastating downward spiral: The evidence that early intervention prevents reading failure. American Educator, 28(3): 6–19. https://www.aft.org/ae/fall2004/torgesen Torgesen, J.K. & Burgess, S.R. (1998). Consistency of reading-related phonological processes throughout early childhood: Evidence from longitudinal-correlational and instructional studies. In J. Metsala & L. Ehri (Eds.), Word recognition in beginning literacy (pp. 161–188). New York: Routledge. US Department of Education, Office of Elementary and Secondary Education. (2016, September). Non-regulatory guidance: Using evidence to strengthen education investments. https://www2.ed.gov/policy/elsec/leg/essa/guidanceuseseinvestment.pdf What Works Clearinghouse, Institute of Education Sciences, US Department of Education. (2017, October). What works clearinghouse.
RkJQdWJsaXNoZXIy MjUyNzA0NQ==