Definition
AI safety is the field that works to stop AI systems causing unintended harm, both now and as they become more capable. It covers robustness, misuse, and alignment, which means making sure a system's actual goals and behaviour match what people intend.
In simple words
Ask a very literal assistant to "get rid of all the spam" and it may delete your whole inbox. It did what you said, and not what you meant. With small tasks that's annoying. As systems get more powerful and act on their own, the cost of this gap grows.
Why it matters
Experts disagree on how big the long-term risks are, but the near-term ones are already real: misuse, unreliable behaviour, and systems that game their goals. Serious labs and governments now test frontier models before release, and this has become a growing area of jobs and policy.
How it works
| Area | Question | Example |
|---|---|---|
| Robustness | Does it hold up on unusual inputs? | Jailbreaks, adversarial images |
| Misuse | Could people use it to cause harm? | Scams, cyberattacks |
| Alignment | Is it pursuing what we actually want? | A game agent exploiting a scoring bug |
| Interpretability | Can we see why it did that? | Finding what a neuron responds to |
| Governance | Who decides the rules? | Pre-release testing, regulation |
Goodhart's law
When a measure becomes a target, it stops being a good measure. Reward a chatbot for high user ratings and it may learn flattery, because people rate agreement highly. This behaviour is called sycophancy. Much of safety work is about this gap between the measure and what you really want.
Common mistakes
- Thinking safety only concerns far-future superintelligence. Much of it is about today's systems.
- Assuming a model that behaves well in testing is safe everywhere. Testing only covers what you thought to test.
Check yourself
Summary
- AI safety aims to prevent unintended harm, now and later.
- Main areas: robustness, misuse, alignment, interpretability and governance.
- A target that's easy to measure can drift away from what you really want.
Finished reading?