What are efficacy trials and in-class pilot studies?

However agile your product development process, developing new learning products can be both expensive and time consuming.  

In 2016 we’ve added efficacy trials and in-class pilot studies to our suite of impact evaluation services. These low cost, short cycle trials are designed to bring further efficiency to how we evaluate the impact of our learning products and enable us to do this at earlier points in a the product’s lifecycle. To make this more tangible we want to briefly describe what efficacy trials and in-class pilot studies are and how we conduct these studies in our Impact Evaluation team.

At Pearson, we’ve been working with SRI International to set up a procedure for evaluating the impact of products at an earlier stage of their development than we have done previously in our efficacy work.

Efficacy trials and in-class pilot studies are what we call our approaches to testing the impact of early stage products - we’re trying to measure the learning our product helps generate, fast. Using these approaches, we’re able to quickly identify how the product is working, if it is working as designed, and whether it has a positive impact on learning. These early results then allow us to make real product improvements, and to better measure our efficacy once the product is launched.

Efficacy Trial

An investigation that tests the performance of a product or learning solution under controlled conditions, usually in a lab or other setting outside of functioning classrooms.

Typically, the impact evaluation studies that form the basis of our efficacy work will measure how impactful a product is in its everyday use in the classroom. Efficacy Trials are different because they are conducted in a more controlled setting (like the difference between lab and field experiments in science class), so that the learning being measured isn’t influenced by things like the students’ background knowledge and demographics. They also tend to be much shorter than regular impact evaluation studies, and will look at specific product components (such as an early alerts system in our Mastering products) rather than the entire thing.

A good example of efficacy trials are A/B tests - famously used by Google and other big tech companies - which randomly assign some people product feature A, and others B, before comparing the learning gains of the two groups.

In-Class Pilot Study

An investigation that tests the performance of a learning solution under conditions that predominantly resemble the intended use setting.

As the name suggests, In-Class Pilot Studies look at the impact of products in actual learning settings. Again they differ from our typical impact evaluation studies because they are specifically investigating whether individual features, like adaptive homework or dynamic modules, are working.

Both efficacy trials and in-class pilot studies are designed to focus only on the impact of Pearson’s products, by constraining the influence of other factors. The early information they provide about the efficacy of our products allows us to evaluate their impact before the full product is even sold.