Making sense of scores

Facility Value

The facility value of an item is the proportion of candidates that got the question right, used as an indicator of item difficulty.

This is a measure of a question’s difficulty, generally expressed as a decimal between 0 and 1 but can also be shown as a percentage. The closer the facility value is to 1, the more people got the question right. A ‘good’ facility value is typically between 0.3 and 0.8 as an exam works best with a spread of items with different facility values to help differentiate between learners.

Items with a facility value less than 0.3 are normally seen as being too difficult, and items with a facility value of greater than 0.8 are considered too easy and don’t provide much information on a candidate’s ability. However, this is not a strict rule as it is sometimes required to test essential knowledge that every learner should get correct.

Facility values are strongly affected by cohort ability so should be used in conjunction with other item statistics, such as discrimination and alpha, to paint a full picture.

Item Response Theory

Another approach to testing is Item Response Theory (IRT). This estimates the learner’s underlying ability. This is the concept that ability is something real but hidden — we can’t observe it directly — so we use someone’s pattern of answers on a test to make an informed estimate of it.

Example:

You want to judge the spelling ability of two people, Learner A and Learner B, so you give them three words to spell aloud with the following results.

Even though both learners received the same score on the spelling test- 2/3 correct, it’s clear who is the better speller based on the difficulty of the words they were being asked to spell. This is how IRT works - it considers how challenging each item is and a learner’s pattern of answers to make an estimation of their level of ability.

IRT differs from Classical Test Theory, as it is sample independent and not influenced by how able the cohort is or how difficult the item is. Classical Test Theory would say these candidates are of the same spelling proficiency.

IRT features

Item difficulty and person ability are measured on the same scale. This means that, in the simplest IRT models, when a person’s proficiency level matches an item’s difficulty level, the model predicts a 50% probability of a correct response. If an Item Response Function is closer to the y axis, it means the item is easier​.

This graph shows three different Item Response Functions, each one for a different item of different difficulties​. A person with -1 ability will have a 50% chance of getting a correct answer for the item of difficulty -1, about a 26% chance of correctly answering the item with difficulty 0 and about a 15% probability of answering the item with difficulty 1 correctly.

Requirements for IRT

There are some constraints to the use of IRT:

• Only one underlying skill can be measured at a time (e.g., only mathematical competence or literacy).
  • So don’t measure other assessments on the same scale e.g. don’t measure maths and English at once.​
• ​Items should be independent of one another- getting Q1 right shouldn’t affect if you get Q2 right​
  • “Continuation marks” in maths, or one item giving the answer to others away violate this​.
• There should only be two outcomes from an item- a right answer or a wrong answer​
  • This means something like a 25-mark essay poses a problem and would need a more complex model.

Uses of IRT

IRT requires a large item bank and lots of pre-testing to determine the item difficulties in order to judge learner ability. This is unrealistic for exams such as GCSE’s, as pre-testing isn’t allowed due to security, so CTT is required. CTT is useful for understanding how assessments perform and to inform future assessment and item design and can provide insights for samples of learners as small as 20.

Depending on the model, IRT would need a minimum of 100 to 1,000 responses per item to provide useful insights. IRT is useful for standard setting for certification exams as it produces a more reliable pass mark by taking into account candidate ability and item difficulties than when CTT is used. It is also used in Computerised Adaptive Testing (CAT) which is currently used in schools in Wales to create personalised assessments and better understand learner performance. These are formative exams taken by learners each academic year to identify areas of focus for learners and teachers. See our Wales case study here or learn more about CAT here.