Ahead of the first tests being released, Ofqual outlined their expectations of how to maintain standards from the legacy to the reformed qualifications in a letter (Ofqual letter to AOs) sent to all awarding organisations. Ofqual outlined that a comparable outcomes approach should be used so that no learners sitting the reformed qualifications for the first time are disadvantaged by the Sawtooth effect.

What is the Sawtooth Effect?

It is generally recognised that as a reformed qualification becomes more established over time, factors such as familiarity, greater availability of professional development and support resources, increased teacher confidence, greater numbers of past papers and so on, can all contribute to improvements in cohort performance. The converse of this is that when new qualifications are introduced, there can often be a dip in cohort performance. This is often referred to as the ‘Sawtooth Effect’. Ofqual published a study into patterns of performance seen after the 2010 and 2011 reform on GQ qualifications which suggest that students and teachers took around three years on average (for yearly based assessment qualifications) to become familiar with the content and style of the new qualifications. 

Functional Skills qualifications are competency based and therefore to achieve a pass, a learner has to demonstrate a certain level of competence of key attributes/qualities within the subject. This would indicate that the level of competence to be demonstrated throughout the lifetime of the qualification should remain the same. Whilst in an ideal scenario where the content, structure of the qualifications have not changed this should be the case, however for the reformed Level 1 & Level 2 Functional Skills qualifications, if this were applied it would disadvantage those learners in the first few sittings as the Sawtooth Effect would not be taken into consideration. To take into consideration the Sawtooth Effect and to ensure learners are not disadvantaged, it might mean that grade boundaries in the first few tests might be set lower than in subsequent tests.

How did we set the standard for the reformed qualifications?

Results for the first set of paper based tests were released in October. The pass threshold boundaries are determined during what we term the ‘awarding’ process. 

During the awarding process, subject experts balance a range of evidence to determine where the pass threshold boundary should be set.

This evidence includes:

• various pieces of statistical information on how learners have performed on the test
• information relating to the entry pattern 
• a report from the Principal Examiner (author of the test) 
• script inspection by subject experts, which includes first looking at archive materials to ensure comparability of standards  
• any other external information which may be given, such as the guidance from Ofqual. 

Senior examiners play a crucial role in providing their expert judgments about the quality of work and this insight also helps ensure that the boundaries are set in the right place.

Our subject experts are trained thoroughly in the process of writing question papers and assessments that are consistent in difficulty. It is also important to ensure that assessments are valid and that they are not predictable for learners. Because of this, tests can be slightly more or less difficult than previous versions because of the content being tested and the questions that are asked. In order to ensure fairness to all candidates and comparability of standards over time, boundaries may shift to ensure that variation in difficulty is taken into account.

This is particularly important during the transition to new or changed qualifications, such as the newly reformed Functional Skills qualifications.

For further information on reformed Functional Skills standard setting read Ofqual’s blog. 

Chris Briggs, Sector Manager Post 16 English and Maths