It is almost an act of modern-day sacrilege to say anything that suggests data is fallible. In the increasingly intertwined worlds of business and technology, data rules supreme.
For organisations that learn to harvest the rivers of data generated by digitalisation for meaningful intelligence, we more or less accept uncritically that data equals success. It has been claimed that companies which adopt Big Data analytics see 5% to 10% gains in productivity compared to those that don’t, and the EU has even suggested that data has been responsible for a 1.9% uplift in GDP in the past decade.
Data means better decision making, clearer understanding of processes and opportunities, increased profitability and growth.
And yet, when we are able to pull ourselves away from such digital idolatry and look at things objectively and dispassionately (one of the things data is meant to help us do, ironically), there is more colour to be seen than the black-and-white acceptance of data’s omnipotent greatness. Data is just another tool for us to use. And like all tools, it is only as clever as whoever is using it.
In recent weeks, we’ve seen a stark lesson in the dangers of an uncritical belief that data will always get things right. The UK exam results fiasco was ultimately a failure of data science – a poorly designed or misapplied algorithm which, in the absence of actual examinations taking place due to COVID-19, downgraded the results of a quarter of a million students from what they had been predicted.
An object lesson in statistical bias
The problem was not that results were downgraded – had students been able to sit their exams, no doubt many would not have got the results they were expected to achieve. The problem was how the statistical model Ofqual adopted to do this worked, and it doesn’t take a data specialist to understand its fatal flaw.
Put simply, the A-Level results algorithm weighted the historical performance of schools and locations far too heavily compared to actual pupil performance. So what you got was a classic ‘postcode lottery’ where bright pupils in lower performing schools or areas got penalised while average pupils in better performing schools came out on top. Another way of putting it would be to say that the model more or less guessed that some pupils would underperform based on what had happened in their previous schools in the past, and accordingly penalised individuals at random.
The political outcry over all of this has centred on how Ofqual’s blundering algorithm has repeated the old biases about educational attainment and social background. But there is a deeper point within this about the nature of data analytics and algorithms. Contrary to popular perception, statistical models and other forms of data analysis are not inherently objective at all. They are designed by people, who in designing an algorithm, determine which data to look at, how it should be weighted and so on. Because of that, data is vulnerable to repeating and entrenching biases that we all to easily believe it eradicates.
Why recruitment needs to take note
Recruitment and HR would do well to look carefully at how and why the exams fiasco unfolded. We have written previously about the growing role that AI is playing in talent acquisition, and also about the potential for Big Data to help tackle the tech industry’s diversity problem.
There is no doubt that ‘smart’ automation using algorithms and advanced statistical modelling can help employers identify ways to make their recruitment practices more inclusive. But they have to be treated as tools, not as solutions in their own right. Care must be taken to utilise such tools appropriately for the tasks at hand. Simply designing a set of candidate selection algorithms, for example, and trusting them to come up with a diverse, representative set of new recruits risks making the same mistake Ofqual made. Without rigorous oversight, you could end up dialling in the same biases you are trying to avoid.
This article in HR Magazine makes another interesting point, and argues that, because of the risks of unconscious bias, organisations should run risk and quality impact assessments for all use of AI and algorithms in recruitment and performance management. The rationale is quite clear – if a potential candidate or member of staff feel that they have been unfairly penalised by an automated assessment system and bias can be demonstrated, the company is highly exposed to litigation. We can expect the UK government to end up in court facing a highly embarrassing class action from A-Level students who miss out on university places because of their botched downgrades.
Employers adopted AI-drive selection processes would do well to ensure they do not follow suit.