In this work, we present a framework to measure and mitigate intrinsic biases with respect to protected variables --such as gender-- in visual recognition tasks. We show that trained models significantly amplify the association of target labels with gender beyond what one would expect from biased datasets. Surprisingly, we show that even when datasets are balanced such that each label co-occurs equally with each gender, learned models amplify the association between labels and gender, as much as if data had not been balanced! To mitigate this, we adopt an adversarial approach to remove unwanted features corresponding to protected variables from intermediate representations in a deep neural network -- and provide a detailed analysis of its effectiveness. Experiments on two datasets: the COCO dataset (objects), and the imSitu dataset (actions), show reductions in gender bias amplification while maintaining most of the accuracy of the original models.


pull figure
On the top we illustrate our newly introduced concept of Dataset Leakage which measures the extent to which gender --or more generally a protected variable-- can be inferred from randomly perturbed ground-truth labels. On the bottom we illustrate our concept of Model Leakage which measures the extent to which gender can be inferred from the outputs of a model. A model amplifies bias if model leakage exceeds dataset leakage.


pull figure
Bias amplification as a function of F1 score on COCO object classification and imSitu action recognition. Models in the top left have low leakage and high F1 score. The blue dashed line indicates bias and performance of adding progressively more noise to the original model representation. Our adversarial methods (circled) are the ones which make a better trade-off between performance and bias amplification than randomization and other baselines.

Example Results

pull figure
Images after adversarial removal of gender in image space by using a U-Net based autoencoder as inputs to the recognition model. While people are clearly being obscured from the image, the model selectively chooses to obscure only parts that would reveal gender such as faces but tries to keep information that is useful to recognize objects or verbs. 1st row: WWWM MMWW; 2nd row: MWWW WMWW; 3rd row: MMMW MMWM; 4th row: MMMW WWMM. W: woman; M: man.



  title = {Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations},
  author = {Wang, Tianlu and Zhao, Jieyu and Yatskar, Mark and Chang, Kai-Wei and Ordonez, Vicente},
  booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  month = {October},
  year = {2019}