Can Big Data Help to Solve Tech's Diversity Problem?

Can Big Data Help to Solve Tech’s Diversity Problem?

Apr 2, 2020

It is no secret that the tech industry has an issue with diversity. According to Tech Nation, women makeup just 19% of workers in the UK tech sector – improving slightly to 23% of directors at tech companies, but slumping to just 13% of directors in the UK’s vibrant gaming industry.

These figures suggest that, when it comes to senior positions (outside gaming, at least), the tech industry is not much worse off than the wider economy when it comes to gender inequality – just 29% of directors in non-tech companies are women. And the tech industry actually has a higher proportion of black, Asian and minority ethnic (BAME) employees than the UK workforce at large, with 15% of employees from a BAME background compared to 13%.

But that shouldn’t disguise the scale of the problem facing the industry. The fact that just 19% of its employees are female in this day and age is shocking – a sign that something, somewhere is still going badly wrong in recruitment and talent acquisition, not to mention a colossal oversight of potential knowledge, skill and expertise. Not resources any sector can afford to waste these days.

So what is the solution? The issue of ‘unconscious bias’ in tech recruitment has long been discussed – the idea that, without even being aware of it, many decision-makers at tech companies still carry around with them the assumption that males have better tech skills than their female counterparts, or at least a better aptitude for the kind of work many technical roles involve.

According to this 2019 paper by American academic Prof Kimberley Houser, executives initially responded to criticism of the chronic under-representation of women at US tech firms by investing in a spate of ‘unconscious bias training.’ But, Houser argues, there has been no discernible improvement resulting from these efforts. She points to research that, for her, confirms why – 80% of leaders still admit to using ‘gut feeling and personal opinion’ when it comes to making recruitment decisions.

For Houser, as for many other analysts, the only solution is to significantly reduce the weight of subjective decision-making in tech recruitment. And a growing body of opinion would have it that the only way to do that is to put more trust in data.

Data to Beat Discrimination

Big Data is a phrase that has been coined to describe a step-change in data science and analytics that has taken place over the past decade or so. It basically refers to the evolution of analytics software – the first ‘Big Data’ solutions were capable of interrogating data sets on a scale far larger than previous platforms had been able to manage.

Big Data is closely related to, and easily confused with, artificial intelligence (AI). In practice, you might think of them as two sides of the same coin – Big Data solutions focus on processing massive unwieldy data sets into a structured form of useful intelligence,, AI takes things a step forward and is able to automate actions based on such intelligence.

Back to HR – Big Data and AI tools are already finding favour across the human capital management chain, from recruitment to skills development. Let’s consider the potential of each in turn for tackling the diversity problem in the tech industry.

What you might hear referred to as HR analytics or workforce analytics is the application of Big Data techniques to make a deep dive evaluation of how a company is using its most valuable resource – its people. Big Data is all about insight, and HR analytics is all about gaining insight into how factors relating to people are influencing business outcomes. Once you have that kind of intelligence, of course, you can look to make improvements.

So HR analytics is commonly used to assess the relationship between skills and productivity, to help businesses target CPD programmes at a granular level so they develop the skills that will have the biggest impact on performance. It is also already widely used to assess employee retention levels and to understand why people leave, because all companies recognise that staff turnover is an expensive business.

This is where Big Data’s role in tackling a lack of diversity becomes apparent. Rather than asking the relatively narrow question “why can’t we recruit more women?”, Big Data gives tech companies the opportunity to interrogate some more fundamental issues – namely, what is it about the way their organisation is set up, about the way it treats and manages its people, that results in a male-dominated workforce?

Big Data can lift the lid on a huge range of variables that might lead to women not only being hired less by tech companies, but also more likely to move on in a shorter time frame compared to male counterparts. By integrating data from across your HR processes, from payroll to CPD to employee sentiment, analytics can reveal otherwise hidden patterns that reveal inequalities in pay and compensation, promotion opportunities, performance measurement and employee satisfaction. In short, by helping employers understand how the deep structures of the business impact on employment patterns, HR analytics can reveal the underlying factors that make an organisation a less comfortable, less attractive place for women to work.

On the AI front, Professor Houser argues in her paper that technology is a much better tool than awareness training for eliminating unconscious bias from recruitment processes. However, she points out that it is not enough to simply trust in AI because data and computerised processes are by their nature neutral and objective – the algorithms on which AI and analytics are based are created by people, and are therefore also prone to having unrecognised bias “baked into” them.

What Houser advocates, rather, is using AI in more or less the same way we have described Big Data being used above – to identify, analyse and reveal the root causes of bias in recruitment processes, and then acting on them. Action is key with AI – examples might include stripping gender-biased language out of job adverts and replacing it with phrases that are assessed to be most inclusive and appealing to all potential candidates. Similar AI tools can also be used to strip identifying information out of CV’s and applications so everyone starts on a level playing field.

In summary, the lack of diversity and striking gender imbalance in the tech workforce are likely to be stubbornly persisting because the industry has struggled to understand the root causes. The reasons are complex, multi-faceted and run deep into the murky psycho-social underbelly of our communal shared experiences. Big Data and AI offer an opportunity to shed much-needed light on these hidden recesses because they are, by their nature, very good at extrapolating statistical insight from complex, multi-faceted, untidy data sources.

Once we understand the root causes of the imbalance in the tech workforce, then we can all address it. We just need the right tools to dig deep enough.