Sources of bias and other challenges

Tackling algorithmic discrimination in hiring is a tricky task, as automation can be used at many points along the recruitment pipeline. Hiring processes can differ across organisations, but generally they can be divided into four stages:

1. First, the organisation posts a job description.

Next, the hiring process is handled in the following stages:

1. sourcing (internally and/or externally)
2. screening and interviewing
3. evaluating and selecting applicants.

Pipeline of the hiring process

Automation can be added at any of the stages of the hiring process (though it is least common at the final stage, where the recruiter usually has the last word about whom to hire). At each stage of the recruitment process, discrimination can occur.

Discrimination can happen before candidates even apply – the job advertisement itself may drive away some candidates. Where the job ad is placed, how it is written and even the design and imagery used on it can shape who feels encouraged to apply.

«Bias» has become an umbrella term for systematic disadvantages for a group of people based on a protected characteristic such as, for example, gender, age, or national origin. Different types of bias can emerge at different times during an AI system’s life cycle. A conventional classification¹ considers:

• Pre-existing biases can stem from data sets that lack representativeness, contain outdated or flawed data, or reflect historical biases
• Technical biases are introduced by algorithmic design choices² and at the technical level (e.g. which parameters are taken into account in the model and which are not).
• Emergent biases stem from the purpose for which a system is used, the way it is used, or the societal context in which it is used (e.g. what purpose the system is intended to serve and how it is actually applied in practice).

→ If you want to learn more about different kinds of biases, read our Impact Assessment and Auditing Framework and Software Development Guide.

Thus, even if an algorithmic system has been tested and validated before market launch, its use in a specific context can trigger additional risks of discrimination. These risks can emerge from the data that a system interacts with when it is used in practice, from the way it is being used, the (lack of) processes around it, or potential biases of people using the tool. These and other factors can significantly contribute to the emergence of new, unexpected forms of biases. Moreover, intersectional forms of discrimination, where various personal characteristics interplay and lead to or reinforce discrimination, might further add to this complexity.

Addressing discrimination in algorithmic hiring requires considering the entire process — from designing the job advertisement to finalising and documenting the selection process.

The impact of a hiring pipeline is shaped by the interplay of all its components, including drafting a job advert, disseminating it, pre-selecting applications, conducting interviews, evaluating them and finally selecting an applicant and making an offer. Discrimination is already present in traditional, offline hiring processes. Often, the hope is that automating steps in the hiring pipeline will make the entire process more transparent, traceable, and hence fairer. However, existing biases in offline processes will not automatically be mitigated by moving the process online. The AI system might perpetuate or exacerbate the existing biases from the offline setting to the online setting. Any attempt to prevent and fight these biases must take into account the broader socio-technical environment in which these systems operate. While technical approaches to testing AI systems for obvious biases are essential, equally important are practical, socio-political, organisational, educational and legal approaches to effectively address algorithmic discrimination.