Sistema COC
Do high school Sistema COC students demonstrate higher achievement on a simulated national standardized exam?
In Brazil, students with the highest exam scores earn places at the best universities in the country. High schools must prepare their students for these exams, the most important of which is the Exame Nacional do Ensino Médio (ENEM) or National Exam of Upper Secondary Education.
Sistema COC is an integrated learning system for Brazilian private schools. Its multimedia resources and cross-curricular approach aim to prepare high school students to progress to higher education, particularly by achieving high marks on the ENEM.
Pearson explored the relationship between the use of Sistema COC with learner outcomes like students’ achievement in the simulated national assessments. We conducted a study comparing SimENEM results, a practice exam, for students in Sistema COC schools with similar students in public and private schools not using Sistema COC1.
We found that Sistema COC students statistically outperformed other similar non-Sistema COC public school attendees in SimENEM subjects language and natural science by a large margin (by 21 and 17 percentile points). There were no significant differences in math and human science. The research team equated comparison groups by matching examinees older than 18, on the previous year’s (2017) SimENEM scores, grade level, number of people living in the household and years attending the same school.
There were no significant differences in performance between COC students and non-COC private school students in Language, Math, Human Science and Natural Science.
How we did it
Using students’ scores on the 2018 SimENEM exam, this quasi-experimental study examined how Sistema COC high school students’ achievement compared to the achievements of similar non-sistema COC students.
The study compared students in private high schools using Sistema COC with two other student groups:
- students in private high schools not using Sistema COC
- students in public high schools
From the overall sample of students who took the 2018 SimENEM exam, the researchers first eliminated any students who were not yet 18 years old at the time of registration, because these students could not consent to their data being used in the study. They also then eliminated any students who had not taken the SimENEM in 2017, because without a 2017 SimENEM score, students could not be matched on prior achievement — a necessary step to ensure the robustness of the findings.
After these eliminations, the analytic sample consisted of 248 students:
- 163 students in private high schools using Sistema COC
- 48 students in private high schools not using Sistema COC
- 37 students in public high schools
The study compared the groups using an ordinary least squares fixed effects model.
Read the Sistema COC Product Efficacy Report
A summary of all relevant research on Sistema COC, including foundational research related to the design of the product and impact evaluation review.
Read the Sistema COC Technical Research Report
A study on the effects of Sistema COC on student achievement in Brazil.
What we learned
Efficacy statements
In the context of this study, we can make the following comparative statements about the efficacy of Sistema COC. The research team equated comparison groups by matching examinees older than 18, on the previous year's (2017) SimENEM scores, grade level, number of people living in the household, and years attending the same school2:
- Sistema COC students statistically outperformed other similar non-COC public school attendees in both language and natural science by a large margin (21 and 17 percentile points or 0.56 and 0.44 standard deviations, respectively). There were no significant differences in math and human science3.
- There were no significant differences in performance between COC students and non-COC private school students in language, math, human science and natural science4.
1, 2, 3, 4 The efficacy statements derived from the results of this study are considered comparative rather than causal for four main reasons.
- Differences in Brazilian private and public schools cannot be equated through matching.
- Half the COC examinees attended COC schools before 2017, and thus were exposed to the study treatment prior to baseline testing, precluding the matching or equating of study groups prior to treatment.
- One of the matching variables, the number of people living at home, had group mean differences larger than 0.25 standard deviations after matching for the COC to public school comparison on Language.
- Examinees who reported they used COC but were public school students were omitted from the study, because all COC schools are private schools, which makes these examinees’ reports inconsistent.