The evidence behind AbilityRating
AbilityRating is new; the methods it is built from are not. This page sets out the research behind the design, what AbilityRating itself still has to show and what it does not claim.
What the research supports
- A stated rule beats an impression. When the same information about candidates was combined mechanically, it predicted job performance at .44; holistic expert judgment reached .28, even when the experts had more information (Kuncel, Klieger, Connelly and Ones, 2013). Across 136 studies in other fields, mechanical prediction was more accurate on average (Grove and colleagues, 2000).
- Several methods are better than relying on one test. Revised estimates rank structured interviews and job-knowledge tests above general intelligence tests. Later modeling found that dropping the intelligence test from a six-method selection system loses little validity and reduces group differences substantially (Sackett, Zhang, Berry and Lievens, 2022; Sackett and colleagues, 2023; Berry, Lievens, Zhang and Sackett, 2024).
- Realistic work samples have strong support. Work samples predict job performance at .33 (corrected); candidates rate work samples and interviews most favourably, a pattern found across 17 countries (Roth, Bobko and McFarland, 2005; Hausknecht, Day and Thomas, 2004; Anderson, Salgado and Hülsheger, 2010).
- Assessments should be designed from what they claim. Evidence-centerd design links what is claimed about a person, the evidence that would support the claim and the tasks that produce it; it is the framework behind large international assessments such as PISA (Mislevy, Steinberg and Almond, 2003). AbilityRating’s qualities, indicators and problems follow the same structure.
- Fairness has to be designed, tested and monitored. Strategies for reducing group differences have been reviewed systematically; weighting can balance accuracy against adverse impact, but it does not remove the need to test for fairness (Ployhart and Holtz, 2008; De Corte, Lievens and Sackett, 2007).
What each kind of problem draws on
| The problem | What it shows | Research it draws on |
|---|---|---|
| An unclear brief from a stakeholder | Whether you find out what is actually needed | Problem construction and creative problem solving (Mumford, Reiter-Palmon and Redmond, 1994) |
| Code changes to review | Whether you catch a mistake that looks right, including when AI assistance makes people overconfident | Code inspection (Fagan, 1976); people using AI assistants wrote less secure code while feeling more confident (Perry and colleagues, 2023); over-reliance on AI (Buçinca and colleagues, 2021) |
| A specification for an AI agent, then a change | Whether you direct an agent well and adapt | Adaptive performance (Pulakos, Arad, Donovan and Plamondon, 2000); adapting to unforeseen change (Lang and Bliese, 2009); over-reliance on AI (Buçinca and colleagues, 2021) |
| Estimates with incomplete facts | Whether your estimates are reasonable and your confidence fits | Forecasting and calibration (Brier, 1950; Tetlock, 2005) |
| A progress report | Whether you report honestly | Honesty and humility at work (Lee, Berry and Gonzalez-Mulé, 2019) |
The feature request that should perhaps not be built is scored on points set in advance for each version of the problem; it draws on no single published study.
How a score is computed
Each answer is scored against the difficulty of its problem, as a chess rating is earned against the strength of the opponents; the same idea estimates ability and difficulty together in adaptive learning systems (Klinkenberg, Straatemeier and van der Maas, 2011). The evidence is combined by a stated rule. Every score carries an error band that shows how precise it is.
What AbilityRating itself has to show
The research supports the kinds of method AbilityRating uses. It does not show that AbilityRating works; that is what the early sittings before launch check:
- Reliability: whether a person’s results are consistent
- Agreement: whether independent readings of the open answers agree
- Fairness: whether any group of people is disadvantaged
- Resistance to AI: whether an AI handed the problems reaches a high score; every problem is checked against an AI before it is used
What AbilityRating does not claim
- That its scores predict job performance. It runs no job-performance study and collects no hiring outcomes.
- That programming-language knowledge is irrelevant. AbilityRating treats it as part of technical understanding and measures that understanding through realistic judgment problems rather than recall questions.
- That every quality can be measured precisely on its own. Some qualities, such as adaptability, contribute to the overall score rather than standing alone; creativity is not measured yet.
Standards
The method is being documented against the published standards for testing: the Standards for Educational and Psychological Testing of the AERA, APA and NCME, the SIOP Principles for the Validation and Use of Personnel Selection Procedures and the test review model of the European Federation of Psychologists’ Associations. No body certifies products of this kind; the documentation is there so that others can check it.
The sources
The design is tracked in a register of 496 sources, each graded for reliability: 76 meta-analyses, professional standards or laws and 262 peer-reviewed studies or official documents; the rest are preprints and reports, cited only for context, or sources still being checked. The sources page lists the 310 research sources graded A or B.