Glossary
The words this site cannot avoid, in plain terms.
In short
These are the terms this site uses often, in plain words. Each entry says what the word means here. Pages link to an entry the first time they use the word. New to the site? Start here.
Terms
- Agent
- An AI model set up to take actions on its own, such as running code, browsing the web or sending messages, often over many steps.
- AI-assisted
- Written or built with help from an AI model. Pages on this site carry the label where it applies.
- Benchmark
- A fixed set of tasks used to compare systems. A score on one benchmark describes those tasks and no others.
- Canonical dossier
- One long record of one event or question, corrected in public as new facts arrive. Who Knew First is a canonical dossier.
- Does not prove
- A line placed under a claim that names what the evidence cannot establish.
- Drift
- A verdict meaning a rerun did not reproduce the recorded result. The evidence, the check or the claim changed somewhere, and the record shows where.
- Evaluation
- A structured test of what an AI system can do or how it behaves.
- Evaluator
- The person or organization that runs an evaluation. In Who Knew First, an evaluator is an outside group that tested another company's model.
- Honest null
- A result that found no effect, reported as found. This site keeps nulls on the page because they tell a reader something.
- How we know
- The fold-out section under a chart or a claim. It holds the sources, sample sizes, dates and limits behind the main text.
- Incident
- In Who Knew First, an event in which an AI agent crossed a boundary set by the organization that ran it or by an evaluator.
- Match
- A verdict meaning a rerun reproduced the recorded result exactly.
- Misalignment
- Behavior by an AI system that goes against what its developers intended. Companies sometimes apply the label to an incident, and Who Knew First tracks which label each company chose and how the case was handled after.
- Operator
- In Who Knew First, the organization that ran the model involved in an incident.
- Oracle
- A checker that decides pass or fail without a learned model, such as a test suite or a proof checker. Flywheel accepts an answer only after an oracle passes it.
- Pass@1
- The share of tasks a model solves on its first attempt.
- Re-derivable
- Able to be rerun by someone else, on the same evidence and with the same check, to reach the same verdict.
- Receipt
- A small file that records what was checked, with what evidence, and what the result was, so the check can be rerun later.
- Sandbox
- A closed test environment meant to keep an AI agent's actions away from real systems.
- Source role
- The kind of source a claim rests on, such as a company's own statement, a government report, a news report or an independent investigation. The role limits how much weight the claim can carry.
- Statistically significant
- Unlikely to come from chance alone, at a threshold the page states. When a difference falls short of that threshold, the data cannot show that the difference is real.
- Unverifiable
- A verdict meaning the record lacks something needed to rerun the check. The verdict names the missing piece.
- Verdict
- The result of a check. This site uses three: match, drift and unverifiable.
- Wilson interval
- A range around a pass rate that shows how far the rate could move by chance, given the number of tasks.
- Witness
- A separate program that reruns a recorded check and recomputes its fingerprint, then reports match, drift or unverifiable.