Dataset Bias in Computer Vision: How to Audit Human Image Data
Dataset bias in computer vision cannot be evaluated from one demographic percentage. The distribution available to a model is shaped by where images came from, how subjects entered the collection, which examples were retained, how labels were defined, what visual conditions were represented and how evaluation data was constructed. A useful dataset bias audit therefore examines the complete data pipeline.