Definition
What if a face model only ever saw one kind of face?
A model learns only from the data it is given, so data that leaves out some groups of people produces a model that works worse, or unfairly, for them. This is one of the main sources of bias in AI.
In simple words
Imagine teaching someone what 'food' looks like using only photos of pizza. They might not recognise dosa or rajma chawal as food. Real systems have done this: some early face-recognition systems made far more mistakes on darker-skinned women, because few such faces were in their training data.
Why it matters
AI is used in hiring, loans, healthcare and policing. If its data leaves people out, it can treat them unfairly at a huge scale. Checking who is in the data, and who is missing, is a responsibility for everyone who builds or buys AI.
How it works
Questions to ask about any dataset
- Who collected it, and for what purpose?
- Which groups of people, places or situations are in it?
- Who is missing or rare?
- Does the model work equally well for every group? Has anyone tested this?
Try it yourself
The bag of examples
The model is the same each time. Only the examples change. Slide to see who it works for.
The biased bag of examples
Try this with paper slips or small objects.
- Put 18 red slips and 2 blue slips in a bag. Each slip is an 'example'.
- A friend draws 10 slips without looking. This is the 'training data'.
- Ask: from this training data, what would a model think slips usually look like?
- Now imagine the blue slips are a group of people. How well would the model know them?
Think about it
- How could you change the bag to make the training data fair?
Fair or unfair data?
Would each dataset likely give a model that works well for everyone?
Photos of faces of every skin tone, balanced in number
Job applications from the last ten years, when one group was rarely hired
Medical scans from hospitals across many regions, ages and genders
Voice recordings only from people in one city
Common mistakes
- Thinking AI is neutral because it's a machine.
- Testing only on average and never per group.
- Thinking bias is someone else's problem. Users can ask questions and report unfair results.
Check yourself
Summary
- A model learns only what its data shows.
- Data that leaves people out makes AI unfair to them.
- Always ask who is in the data and who is missing.
Finished reading?