Leiden University Student Repository

Schneider, Leona (2024)

Exploring the Impact of Engineering Choices on Fairness in Machine Learning Classifiers Across Intersectional Subpopulations

Master thesis | Psychology (MSc)

Using machine learning (ML) classifiers in high-stakes situations, such as financial and health decisions, is becoming increasingly popular. Those classifiers can impact the outcome of essential matters for individuals and carry a social responsibility. There has been a demonstrated potential for classifiers showing disparate treatment for more vulnerable subpopulations. However, the classifier pipeline's design process often focuses on optimising prediction accuracy, disregarding the potential influence of engineering choices on the fairness outcome. This research scrutinizes the influence of ML engineering choices on the fairness outcome of different subpopulations. For this, engineering choices and their impact on fairness were investigated in five decision points (missing data imputation, scarce outcome intervention, algorithm choice, hyperparameter tuning, and threshold setting). Furthermore, a simulated society modelling social benefit fraud detection was utilized and tested...Show moreUsing machine learning (ML) classifiers in high-stakes situations, such as financial and health decisions, is becoming increasingly popular. Those classifiers can impact the outcome of essential matters for individuals and carry a social responsibility. There has been a demonstrated potential for classifiers showing disparate treatment for more vulnerable subpopulations. However, the classifier pipeline's design process often focuses on optimising prediction accuracy, disregarding the potential influence of engineering choices on the fairness outcome. This research scrutinizes the influence of ML engineering choices on the fairness outcome of different subpopulations. For this, engineering choices and their impact on fairness were investigated in five decision points (missing data imputation, scarce outcome intervention, algorithm choice, hyperparameter tuning, and threshold setting). Furthermore, a simulated society modelling social benefit fraud detection was utilized and tested on various pipeline combinations. The pipeline fairness performances were tested under different data challenges characterized by scarcely distributed outcomes and non-linear decision boundaries. The analysis demonstrated that the selected choices in the decision points influences the fairness outcome, often showing the potential to benefit advantaged subpopulations and harm disadvantaged subpopulations. The decision point of chosen scarcity intervention portrayed the overall highest impact on the fairness outcome. Conclusively, it highlights how each step along the ML classifier pipeline should be evaluated for its fairness impact, taking a fairness-aware approach that does not solely focus on raising a model's accuracy but also incorporates ethical considerations.Show less

High-Stakes Algorithmic Decision-Making, Machine Learning Fairness, Bias in Machine Learning, Fairness Metrics, Intersectional Fairness, Disparate Impact and Treatment, Fairness-Accuracy Trade-off, Pipeline Design Choices, Hyperparameter Tuning and Fairness, Scarcity Intervention

Author

Schneider, Leona

Faculty

Faculty of Social & Behavioural Sciences

Specialisation

Methodology and Statistics

Supervisors

Rüya Kocer

ECTS Credits

20

Language

en