Research Blog

Latest posts

  • What does Proficient mean?

    by Katie McClarty

    Man and girl playing chess

    The end of the school year is a time for field trips, class parties, and final report cards. The iconic report card lets parents know how their student did that year and typically reflects attendance, participation, and performance in class. Parents generally understand how to interpret report card grades: A (great), C (average), or F (failing).

    The end of the year also is the time when many parents receive their child’s standardized test scores. These results, however, are not as easy to interpret. For example, in Massachusetts, a student who earns a score of 250 on the state test is considered proficient. In Washington, it takes a score of 400. Each state has its own assessments, and each defines proficiency differently.

    Now, however, nearly all the states have agreed to adopt the Common Core State Standards as an outline of what students should be taught in mathematics and English language arts. Educators will use instructional materials appropriate for teaching students the knowledge, skills, and practices laid out in these documents. That should produce less variability in instruction state to state and district to district.

    In order to monitor how well students are learning this material, most of the states also have agreed to use one of two Common Core assessments that are being developed. That will make it possible for states to report results on a common scale: a 400 in English in Tennessee, for example, would be the same as a 400 in Florida. But the question remains: is 400 good enough?

    To answer that question, states set performance standards. Typically, this is done by educators and other experts who get together and look at assessments and agree on which questions or tasks a proficient (or advanced or in need of improvement) student should be expected to answer or complete. That information is then translated into a specific score. The same process can be used to analyze the quality of examples of student work.

    More recently, it’s become possible to answer the question of what is good enough more precisely, based not just on expert judgment but also on data. If by proficiency we mean that a student has learned enough in one grade to be ready to do well in the next one, we can test that definition by tracking how students actually perform. We can look at how well a group of students performs on a 5th grade math test and then look back at how those same kids had done on the 4th grade math test. Using statistics, we can then more accurately define what it means to be proficient in the 4th grade.

    This process is called Evidence Based Standard Setting, and because scores can be linked to future performance, it can give parents confidence that if their child is proficient, he or she has not only mastered an important set of knowledge and skills, but also is likely to be successful in the next grade. It can even give students and their parents a sense of whether they’re on track to do well after high school in college or in demanding career training programs. The scores can also help identify students who need extra help before it becomes too late, and parents, using this information, can advocate on their children’s behalf to make sure they receive that help.

    The familiar report card is but one source of information about how well students are doing in school. Test results linked to important future outcomes can provide another critical piece of information to teachers, parents, and students.

    read more
  • Rigor and Readiness: Measuring the Impact of Algebra II

    by Katie McClarty

    Students in a lecture

    There has been a lot of discussion lately about the role of advanced high-school mathematics courses — in particular, Algebra II — in promoting college and career readiness. On one side of the debate, the champions of Algebra II cite research demonstrating that completing the course leads to success in higher education and to higher earnings (Adelman, 2006; Carnevale & Desrochers, 2003). Achieve has been one of the leading advocates for including advanced mathematics in required high school curricula, suggesting there are not only practical advantages (e.g., prerequisites for future study), but also benefits to students’ general academic development. Skills acquired through Algebra II (including but not limited to logical thinking, cognitive capacity, and complex problem solving) can support success in areas far beyond a day-to-day work environment.

    This isn’t to say the debate is settled. A recent report from the National Center for Education and the Economy (NCEE) found that the skills most important for succeeding in community college math courses were those introduced in middle school. By analyzing textbooks, assignments, and tests at seven community colleges, the researchers concluded that few students need to master advanced algebra to be successful. The NCEE report comes at a time when several states (e.g., Florida, Texas) are changing graduation requirements to make Algebra II optional, provide more flexible pathways toward high school graduation, and create space in students’ schedules for more vocational training.

    Isolating the causal effect of taking Algebra II on future outcomes is a serious challenge, thanks to selection bias. It is likely that students who choose to take Algebra II in high school are higher performing and more motivated than many of their peers and thus more likely to attend and do well in college. In other words, it’s something about the type of students that take Algebra II, rather than completing the course itself, that leads to better student outcomes.

    In a recent research study, my co-authors and I set about tackling this thorny issue — separating selection effects from Algebra II’s true causal effects. We will be presenting our work next week at the Association for Institutional Research Annual Forum in Long Beach, CA. We used national datasets spanning multiple decades and sophisticated econometric techniques to isolate cause-and-effect relationships between completing Algebra II in high school and subsequent college and career outcomes.

    The verdict? Algebra II seems to matter more for college outcomes (including community colleges, technical colleges, and four-year institutions) than for career outcomes. Compared to their counterparts who didn’t finish Algebra II, those who did were more likely to be admitted to selective colleges, maintain higher college GPAs, stay in school, and graduate. Conversely, for students who did not apply to college after high school, completing Algebra II was not related to finding a job immediately after high school, initial occupational prestige, earnings, or career advancement.

    This research indicates that students not planning to attend any college (two-year or four-year) may not benefit substantially from finishing Algebra II. That said, it’s important to highlight one caveat: Algebra II does not seem to negatively impact any career outcomes. In that respect, completing the course will keep doors open to college for the many students who do not solidify their postsecondary plans before enrolling in high school courses or starting their mathematics sequence. Some of our other interesting findings from this study will be the topic of future blog posts.

    read more
  • Grit, Tenacity, and Perseverance

    by Katie McClarty

    In my last blog, I discussed the importance of metacognitive learning skills—attitudes, behaviors, and beliefs about learning. These skills continue to garner attention from educational researchers and policy-makers. The Office of Education and Technology (OET) at the U.S. Department of Education recently released a report, Promoting Grit, Tenacity, and Perseverance—Critical Factors for Success in the 21st Century, which takes a closer look at defining, measuring, and developing these skills. Grit was defined as “perseverance to accomplish long-term or higher-order goals in the face of challenges and setbacks, engaging the student’s psychological resources, such as their academic mindsets, effortful control, and strategies and tactics” (p. 15).

    The task of defining and measuring grit is not simply an academic exercise; this is a trait associated with important student outcomes, including success in college. Angela Duckworth’s research shows that people with a college degree (Associate’s or higher) tend to be grittier than people without a degree. Moreover, and perhaps not surprisingly, grit seems to be associated with success in particularly challenging postsecondary environments. It is associated with retention at West Point, and research by Terrell Strayhorn has shown grit is a significant predictor of college grades for black males attending predominantly while institutions.

    Because grit may play a key role in overcoming adversity, it is encouraging that grit, tenacity, and perseverance are skills that can be developed with the right supports. For example, the OET report recommends designing learning environments that provide students opportunities to take on long-term, higher-order goals aligned with their interests. These goals are optimally challenging and intrinsically motivating. Meeting them takes perseverance. By developing such skills early, students may be more likely to persevere through challenges that are bound arise along their college and career paths.

    The central tenets of personalized learning echo these themes. First, we must identify where each student is on a learning trajectory. We use that information to provide each student with a challenging, but attainable next step. Technology and digital learning environments can facilitate the personalization process. With these tools we can collect information about students’ strengths, weaknesses, and behaviors, and then adapt learning systems to set reasonable goals for every student. By creating personalized learning solutions, we can do more than just deliver the appropriate academic content. We can set students on a path to increase their grit.

    read more