TowerDebias: A Novel Debiasing Method Based on the Tower Property

Authors

Norm Matloff and Aditya Mittal

Overview

Details of the paper can be found here.

Abstract: Decision-making processes have increasingly come to rely on sophisticated machine learning tools, raising critical concerns about the fairness of their predictions with respect to sensitive groups. The widespread adoption of commercial "black-box" models necessitates careful consideration of their legal and ethical implications for consumers. When users interact with such black-box models, a key challenge arises: how can the influence of sensitive attributes, such as race or gender, be mitigated or removed from its predictions? We propose towerDebias (tDB), a novel post-processing method designed to reduce the influence of sensitive attributes in predictions made by black-box models. Our tDB approach leverages the Tower Property from probability theory to improve prediction fairness without requiring retraining of the original model. This method is highly versatile, as it requires no prior knowledge of the original algorithm's internal structure and is adaptable to a diverse range of applications. We present a formal fairness improvement theorem for tDB and showcase its effectiveness in both regression and classification tasks using multiple real-world datasets.

Keywords: Algorithmic Fairness, Tower Property, Post-Processing Fairness Techniques, Sensitive Variables, Machine Unlearning.

Installation:

As of now, the software may be installed using the devtools package:

library(devtools)
install_github("matloff/towerDebias", force = TRUE)

Dataset Information

The following datasets are presented and discussed in the paper:

1. Svcensus (svc): The goal is to predict wage income ($Y$) with gender as the sensitive variable ($S$). This is a regression task with binary $S$.
2. Law School Admissions (lsa): The goal is to predict LSAT scores ($Y$) with race as the sensitive variable ($S$). This is a regression task with categorical $S$.
3. COMPAS (cmp): The goal is to predict two-year recidivism risk ($Y$) with race as the sensitive variable ($S$). This is a classification task with categorical $S$.
4. IranianChurn (iranChurn): The goal is to predict exit status ($Y$) with age and gender as the sensitive variables ($S$). This is a classification task with continuous and binary $S$.
5. Dutch Census (dutchCensus): The goal is to predict occupation ($Y$) with gender as the sensitive variable ($S$). This is a classification task with binary $S$.

How to Use

TowerDebias improves fairness by modifying black-box model predictions. We provide a hyperparameter $k$ to control the fairness-utility trade-off.

Example: We first train a logistic regression model on to predict recidivism rates using Matloff's qeML and establish baseline probability results. See documentation for details.

data(compas1)

# original model prediction - logistic regression via qeML
logOut <- qeLogit(compas1, 'two_year_recid', yesYVal = "Yes")
lr_preds <- predict(logOut, compas1[100,-8])$probs
print(lr_preds) 

# original probability of recidivism is 0.25
#      [,1]
# [1,] 0.2024764

# towerDebias prediction - Note: k = 25
dataXS <- compas1[logOut$holdIdxs,-8]
tb_preds <- towerDebias(dataXS, 'race', logOut$holdoutPreds$probs, compas1[100,-8], k = 25)
print(tb_preds)

# the new probability of recidivism jumps to 0.25 
# [1] 0.2512924
# attr(,"stdErrs")
# [1] 0.02318267

Additionally, suppose the client has access to a test dataset that they wish to debias and compare any fairness-utility trade-offs. In this case, the compareBias function can be used to analyze the effects of fairness versus utility trade-offs.

# create test/train split
smp <- sample(1:nrow(compas1), nrow(compas1) * 0.9)
trn <- compas1[smp,]
tst <- compas1[-smp,]
logOut <- qeLogit(trn, 'two_year_recid', yesYVal = "Yes", holdout = NULL)              # logistic regression                        
lr_preds <- predict(logOut, tst[,-8])$probs; lr_preds                                  # generate original model predictions on test data
compareBias(tst, 'two_year_recid', 'race', lr_preds, k = 25, yesYval = "Yes")            # k = 25

# The result below shows the fairness-accuracy trade-offs, improvements in fairness (correlation reductions) can be achieved with minimal loss in misclassification rate.
# $debias_comparisons
#                          blackBox towerDebias
# Misclassification Rate 0.26279863 0.276450512
# race.African-American  0.25985302 0.211483025
# race.Asian             0.04962804 0.013732859
# race.Caucasian         0.14129560 0.128081437
# race.Hispanic          0.11902078 0.095376784
# race.Native American   0.01497951 0.006372373
# race.Other             0.10758452 0.084297627

Run the experiments

To replicate the experiments from the paper, run the following scripts:

cd towerDebias
Rscript experiments/{dataset_name}.R 

The final graphs will be stored in results/{dataset}.

Additional Notes

XXX

Contact

For any questions or feedback, feel free to contact Aditya Mittal.