In the employment context, EU non-discrimination law prohibits discrimination based on the protected characteristics of age, gender, race or ethnic origin, religious or philosophical beliefs, disability and sexual preferences.

To monitor and evaluate an AI system’s performance to detect whether job candidates were treated unfairly based on characteristics protected under EU non-discrimination law, it may be necessary to analyse personal data of job candidates based on these characteristics. However, data on racial or ethnic origin, religion, philosophical beliefs, health or sexual orientation are sensitive data (special categories of personal data) under the GDPR (in addition to political opinions, trade union membership or genetic or biometric data — when used to uniquely identify someone). Age and gender are protected characteristics in non-discrimination law, but are not sensitive data under the GDPR. This means that processing of age and gender data is governed by the general GDPR rules, not by the extra strict GDPR rules designed for sensitive data (special categories of data).

The overlap between protected characteristics and special categories of personal data.¹

As a general rule, the GDPR prohibits processing of such sensitive data (Art. 9(1) GDPR), but it provides limited conditions when processing may be necessary and exceptionally allowed (Art. 9(2) GDPR).

Some organisations may rely on ‘explicit consent’ of an individual to process their sensitive data, but in the employment context, due to the power imbalance between a job candidate and a hiring company, it is unlikely for consent to be truly freely given (fully voluntarily).² Job applicants as future employees are usually in a vulnerable situation and there is a significant power imbalance between a job applicant and the hiring company.

→ Learn more: FINDHR Equality Monitoring Protocol, Section 7.8

None of the exceptions fully covers monitoring the outputs of AI systems. However, processing sensitive personal data of job applicants while monitoring the outcomes of an AI system might be helpful in detecting and preventing discrimination.This type of processing is strictly regulated under the General Data Protection Regulation (GDPR).

The AI Act introduces a new exception for processing sensitive data for AI providers if this is strictly necessary for detecting and correcting bias in training, validation or testing datasets. This exception is limited to the development stage and does not allow processing of sensitive data for monitoring systems’ outputs when in use.

Importantly, if an organisation wants to process derived, inferred or estimated sensitive data, it must meet conditions for processing sensitive data described in Art. 9(2) GDPR and other GDPR principles (see above).

→ See more: FINDHR Equality Monitoring Protocol

Taking these limitations into account, FINDHR research proposes monitoring the AI system’s outcomes in the post-deployment stage through secure multi-party computation (SMPC) and the important role of a trusted third party.

→ See more on SMPC here in chapter #14 and in FINDHR Equality Monitoring Protocol

→ See further recommendations and more on challenges of processing recruitment data in: FINDHR Software Development Guide, section 2.4.1.