How CMMNS is measured
Each sign's measured precision and recall with its interval, the data behind it, the side-balance check and the known limits.
Updated 2026-10-23.
Every sign in CMMNS' window carries one of two labels: how often it was right when people checked by hand, or "not measured yet". Below is every sign, the data it was checked on and that data's licence, and how evenly the signs fire whichever side a post targets.
How to read the numbers
- Precision is the share of signs CMMNS showed that the human annotators of a public research corpus agreed with. It answers "when CMMNS shows this, how often is it right?"
- Recall is the share of the annotated cases that CMMNS found. CMMNS is built for precision: a wrong sign on an honest reply is the harm it avoids most, so recall is low.
- Each figure gives the count and a 95% interval (Wilson). A wide interval means few cases were checked.
- All figures come from a held-out test split (seed
radar-eval-2026-10-21). Rules were tuned on a separate split, and the test split was read once, after tuning. Measured 2026-10-23.
One figure per sign
Each sign leads with what its measurement means when you see it. The figure draws the 95% interval as a band and the measured value as a mark; a dashed line is the bar a sign must clear to show. Counts, data and limits sit under "How we know".
Slur or coded hate term
When CMMNS shows this sign, the people who labelled the test comments agreed 91 times in 100.
How we know
- Precision
- 3,996 of 4,393; 95% interval 90.1% to 91.8%
- Range across corpora
- 34.0% to 95.3%
- In the window
- precision 95% (95% interval 94 to 96%), 1139 badges checked; measured on Measuring Hate Speech, against general hostility, held-out test set
- Catch rate
- finds about 1 in 11 hostile posts in testing
- Sensitivity
- d' 1.10 (1.01 to 1.21); 0 is chance, 2 is good
- Tendency
- fires sparingly; when it does, it is usually right
- Data
- Measuring Hate Speech, HateXplain and Civil Comments, held-out test split
- Read it as
- A sign describes the words in the post, which can be quoted, ironic or reported speech.
Call for violence
When CMMNS shows this sign, the people who labelled the test comments agreed 88 times in 100.
How we know
- Precision
- 77 of 88; 95% interval 79.0% to 92.9%
- Range across corpora
- 96.7% to 96.7%
- Recall
- 2.1% (39 of 1,819; 95% interval 1.6% to 2.9%). CMMNS is built for precision, so it misses most cases.
- In the window
- precision 64% (95% interval 51 to 75%), 61 badges checked; measured on Measuring Hate Speech, against its own violence label, held-out test set
- Catch rate
- finds about 1 in 47 posts calling for violence in testing
- Sensitivity
- d' 1.00 (0.82 to 1.20); 0 is chance, 2 is good
- Tendency
- rarely fires; when it does, it is right more often than not
- Data
- Measuring Hate Speech, HateXplain and Civil Comments, held-out test split
- Read it as
- A sign describes the words in the post, which can be quoted, ironic or reported speech.
Dehumanising language
When CMMNS shows this sign, the people who labelled the test comments agreed 93 times in 100.
How we know
- Precision
- 37 of 40; 95% interval 80.1% to 97.4%
- Range across corpora
- 90.0% to 90.0%
- Recall
- 0.2% (12 of 5,046; 95% interval 0.1% to 0.4%). CMMNS is built for precision, so it misses most cases.
- In the window
- precision 60% (95% interval 39 to 78%), 20 badges checked; measured on Measuring Hate Speech, against its own dehumanising label, held-out test set
- Catch rate
- finds about 1 in 420 dehumanising posts in testing
- Sensitivity
- d' 0.44 (0.15 to 0.75); 0 is chance, 2 is good
- Tendency
- rarely fires; when it does, it is right more often than not
- Data
- Measuring Hate Speech, HateXplain and Civil Comments, held-out test split
- Read it as
- A sign describes the words in the post, which can be quoted, ironic or reported speech.
Group stereotype
When CMMNS shows this sign, the people who labelled the test comments agreed 84 times in 100.
How we know
- Precision
- 96 of 114; 95% interval 76.4% to 89.8%
- Range across corpora
- 77.3% to 92.5%
- Recall
- 0.7% (58 of 8,368; 95% interval 0.5% to 0.9%). CMMNS is built for precision, so it misses most cases.
- In the window
- precision 73% (95% interval 62 to 81%), 80 badges checked; measured on Measuring Hate Speech, against its own stereotype label, held-out test set
- Catch rate
- finds about 1 in 140 posts stereotyping a group in testing
- Sensitivity
- d' 0.43 (0.28 to 0.61); 0 is chance, 2 is good
- Tendency
- rarely fires; when it does, it is right more often than not
- Data
- Measuring Hate Speech, HateXplain and Civil Comments, held-out test split
- Read it as
- A sign describes the words in the post, which can be quoted, ironic or reported speech.
The thread report's "extreme"
When the report counts a reply as extreme, the corpus labelled it severe 70 times in 100.
How we know
- Precision
- 903 of 1,285; 95% interval 67.7% to 72.7%
- Recall
- 903 of 5,224; 95% interval 16.3% to 18.3%
- Data
- Measuring Hate Speech severity labels
Same text as other replies
In ordinary threads with 20 to 49 replies, a group of near-identical replies shows up in 1.4% of threads.
How we know
- 5-19 replies
- 14 of 3,087; 95% interval 0.3% to 0.8%
- 20-49 replies
- 20 of 1,384; 95% interval 0.9% to 2.2%
- 50-99 replies
- 6 of 181; 95% interval 1.5% to 7.0%
- 100+ replies
- 4 of 60; 95% interval 2.6% to 15.9%
- Before quotes were set aside
- 33% to 47% of these threads, almost all from replies that copied the post they quoted
- Source
- Ordinary threads: the PHEME corpus of Twitter threads from 2014 and 2015 news events (CC BY 4.0), a public research corpus from an earlier X.
Burst of replies
In ordinary threads with 20 to 49 replies, a burst like those CMMNS reports shows up in 2.2% of threads.
How we know
- 5-19 replies
- 39 of 3,087; 95% interval 0.9% to 1.7%
- 20-49 replies
- 31 of 1,384; 95% interval 1.6% to 3.2%
- 50-99 replies
- 2 of 181; 95% interval 0.3% to 3.9%
- 100+ replies
- 2 of 60; 95% interval 0.9% to 11.4%
- Independent posting
- A simulation where each reply is timed on its own, at the thread's own pace, fires in 2.8%, 3.6%, 5.3%, 12.2% of threads by size.
- Source
- Ordinary threads: the PHEME corpus of Twitter threads from 2014 and 2015 news events (CC BY 4.0), a public research corpus from an earlier X.
Accounts seen together
For someone who opened about 100 threads on one news story, a pair seen together in two or more other threads shows up in 0.2% of threads.
How we know
- About 10 threads
- 0 of 1,600; 95% interval 0.0% to 0.2%
- About 30 threads
- 0 of 4,800; 95% interval 0.0% to 0.1%
- About 100 threads
- 32 of 14,000; 95% interval 0.2% to 0.3%
- Independent posting
- With the same accounts placed in threads at random: 0.0%, 0.0%, 0.0%.
- Source
- Ordinary threads: the PHEME corpus of Twitter threads from 2014 and 2015 news events (CC BY 4.0), a public research corpus from an earlier X.
Look-alike letters
Copies with one letter swapped for a Cyrillic or Greek look-alike: CMMNS now finds 100 in 100, up from 75.
How we know
- Seeded copies
- 100 of 100 found after folding, 75 of 100 before
- Ordinary text
- 6 of 96,337 replies in ordinary threads changed at all, mostly fullwidth letters and digits read as plain ones
- Test cases
- Same text, cases written for the test: 85 of 87; 95% interval 92.0% to 99.4%
- Data
- Unicode confusables table (UTS #39), Unicode License v3
Persuasion: loaded language
Held back: right 60 times in 100 on news articles, under the bar of 80.
How we know
- Precision
- 82 of 136; 95% interval 51.9% to 68.1%
- Recall
- 7.9%
- Chance
- 2.3% of words sit inside a span of this technique
- Data
- Propaganda Techniques Corpus v2, held-out articles
- Read it as
- A technique describes how a post argues. True claims use techniques too.
Persuasion: name calling
Held back: right 49 times in 100 on news articles, under the bar of 80.
How we know
- Precision
- 21 of 43; 95% interval 34.6% to 63.2%
- Recall
- 4.4%
- Chance
- 1.1% of words sit inside a span of this technique
- Data
- Propaganda Techniques Corpus v2, held-out articles
- Read it as
- A technique describes how a post argues. True claims use techniques too.
Persuasion: flag waving
Held back: right 41 times in 100 on news articles, under the bar of 80.
How we know
- Precision
- 11 of 27; 95% interval 24.5% to 59.3%
- Recall
- 12.8%
- Chance
- 0.5% of words sit inside a span of this technique
- Data
- Propaganda Techniques Corpus v2, held-out articles
- Read it as
- A technique describes how a post argues. True claims use techniques too.
Language signs
Checked on three public corpora of comments labelled by people, pooled: Measuring Hate Speech (CC BY 4.0), HateXplain (MIT) and Civil Comments (CC0 1.0). The range runs from the lowest to the highest corpus with at least 20 hits. Recall is against Measuring Hate Speech's own survey item for that sign, where one exists.
Every language sign, with counts
| Sign | Precision (95% interval) | Range across corpora | Recall | Status |
|---|---|---|---|---|
| Slur or coded hate term | 91.0% (3,996 of 4,393; 90.1% to 91.8%) | 34.0% to 95.3% | not measured | Shown with this figure |
| Group stereotype | 84.2% (96 of 114; 76.4% to 89.8%) | 77.3% to 92.5% | 0.7% (58 of 8,368; 0.5% to 0.9%) | Shown with this figure |
| Dehumanising language | 92.5% (37 of 40; 80.1% to 97.4%) | 90.0% | 0.2% (12 of 5,046; 0.1% to 0.4%) | Shown with this figure |
| Call for violence | 87.5% (77 of 88; 79.0% to 92.9%) | 96.7% | 2.1% (39 of 1,819; 1.6% to 2.9%) | Shown with this figure |
| Call for vigilantes | 66.7% (2 of 3; 20.8% to 93.9%) | under 20 hits in each corpus | not measured | Held back: fewer than 20 hits in the test split |
Measured on public research corpora (Reddit, Twitter, Gab, YouTube and news comments), and it can differ on this site and other topics. Each figure in the window comes from one test set, named beside it: precision says how often the sign sat on text the annotators gave that label.
Weaker signs that count only toward a reply being "extreme"
Weaker signs, with counts
| Sign | Share on hostile text (95% interval) | Range across corpora |
|---|---|---|
| Context-dependent slur | 65.5% (837 of 1,278; 62.8% to 68.0%) | 20.3% to 91.7% |
| Contempt | 92.1% (93 of 101; 85.1% to 95.9%) | 91.5% to 93.1% |
| Motive attribution | 72.5% (29 of 40; 57.2% to 83.9%) | 81.8% |
| Essentialising | 57.8% (59 of 102; 48.1% to 67.0%) | 25.8% to 75.0% |
| Absolutist wording | 50.7% (137 of 270; 44.8% to 56.7%) | 32.9% to 84.2% |
| Conspiracy framing | 29.5% (36 of 122; 22.1% to 38.1%) | 3.6% to 81.0% |
| Any strong language sign | 90.6% (4,167 of 4,597; 89.8% to 91.5%) | 35.2% to 95.1% |
How often each sign fires
Precision says how often a sign is right when it shows. The hit rate says how much of the labelled text it finds, and the false-alarm rate how much of the rest it flags. From the two, d' measures how well a sign tells the two apart (0 is chance, 1 is modest, 2 is good), and c, the bias, measures how readily it fires (above 0 means reluctant). Bias here means only that; political side is never an input. Every sign is very conservative: c sits between 2.5 and 3.5 on most rows, and most signs find under 3% of the hostile text. d' and c are shown only where a sign fired at least 20 times.
Only some rows score a sign against its own label: dehumanising, violence and stereotype on Measuring Hate Speech, and threats on Civil Comments. The rest score it against general hostility, which conspiracy framing, motive attribution, absolutist wording and essentialising do not try to detect, so a low d' there says the corpora lack a matching label. The French corpus gave too few hits to measure. Method and caveats: EVALUATION.md, "Hit rate, false alarms, sensitivity and bias".
Every sign's hit rate, false-alarm rate, d' and c, per corpus
Dehumanising language
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | its own label (dehumanising) | 60% (12 of 20) | 0.24% (12 of 5,046) | 0.054% (8 of 14,743) | 0.44 (0.15 to 0.75) | 3.03 (2.93 to 3.21) |
| Measuring Hate Speech | general hostility | 90% (18 of 20) | 0.15% (18 of 12,295) | 0.027% (2 of 7,494) | 0.44 (0.15 to 0.91) | 3.18 (3.06 to 3.43) |
| HateXplain | general hostility | 100% (18 of 18) | 0.29% (18 of 6,182) | 0% (0 of 3,835) | not measured (fired 18 times) | not measured (fired 18 times) |
| Civil Comments | general hostility | 50% (1 of 2) | 0.025% (1 of 3,979) | 0.0022% (1 of 44,681) | not measured (fired 2 times) | not measured (fired 2 times) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 93% (37 of 40) | 0.16% (37 of 22,456) | 0.0054% (3 of 56,010) | 0.90 (0.69 to 1.38) | 3.39 (3.29 to 3.61) |
Call for violence
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | its own label (violence) | 64% (39 of 61) | 2.1% (39 of 1,819) | 0.12% (22 of 17,970) | 1.00 (0.82 to 1.20) | 2.52 (2.44 to 2.63) |
| Measuring Hate Speech | general hostility | 97% (59 of 61) | 0.48% (59 of 12,295) | 0.027% (2 of 7,494) | 0.82 (0.58 to 1.27) | 2.99 (2.88 to 3.23) |
| HateXplain | general hostility | 83% (15 of 18) | 0.24% (15 of 6,182) | 0.078% (3 of 3,835) | not measured (fired 18 times) | not measured (fired 18 times) |
| Civil Comments | its own label (threat) | 0% (0 of 9) | 0% (0 of 115) | 0.019% (9 of 48,545) | not measured (fired 9 times) | not measured (fired 9 times) |
| Civil Comments | general hostility | 33% (3 of 9) | 0.075% (3 of 3,979) | 0.013% (6 of 44,681) | not measured (fired 9 times) | not measured (fired 9 times) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 88% (77 of 88) | 0.34% (77 of 22,456) | 0.02% (11 of 56,010) | 0.83 (0.68 to 1.05) | 3.12 (3.05 to 3.23) |
Call for vigilantes
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 100% (2 of 2) | 0.016% (2 of 12,295) | 0% (0 of 7,494) | not measured (fired 2 times) | not measured (fired 2 times) |
| HateXplain | general hostility | never fired | 0% (0 of 6,182) | 0% (0 of 3,835) | not measured (fired 0 times) | not measured (fired 0 times) |
| Civil Comments | general hostility | 0% (0 of 1) | 0% (0 of 3,979) | 0.0022% (1 of 44,681) | not measured (fired 1 time) | not measured (fired 1 time) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 67% (2 of 3) | 0.0089% (2 of 22,456) | 0.0018% (1 of 56,010) | not measured (fired 3 times) | not measured (fired 3 times) |
Slur or coded hate term
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 95% (1,086 of 1,139) | 8.8% (1,086 of 12,295) | 0.71% (53 of 7,494) | 1.10 (1.01 to 1.21) | 1.90 (1.86 to 1.96) |
| HateXplain | general hostility | 90% (2,894 of 3,207) | 47% (2,894 of 6,182) | 8.2% (313 of 3,835) | 1.31 (1.25 to 1.38) | 0.74 (0.70 to 0.77) |
| Civil Comments | general hostility | 34% (16 of 47) | 0.4% (16 of 3,979) | 0.069% (31 of 44,681) | 0.55 (0.32 to 0.74) | 2.92 (2.83 to 3.04) |
| MLMA (French) | general hostility | 89% (8 of 9) | 0.51% (8 of 1,584) | 0.24% (1 of 422) | not measured (fired 9 times) | not measured (fired 9 times) |
| Three English corpora, pooled | general hostility | 91% (3,996 of 4,393) | 18% (3,996 of 22,456) | 0.71% (397 of 56,010) | 1.53 (1.49 to 1.57) | 1.69 (1.67 to 1.71) |
Context-dependent slur
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 92% (355 of 387) | 2.9% (355 of 12,295) | 0.43% (32 of 7,494) | 0.73 (0.61 to 0.87) | 2.26 (2.20 to 2.33) |
| HateXplain | general hostility | 64% (440 of 684) | 7.1% (440 of 6,182) | 6.4% (244 of 3,835) | 0.06 (−0.02 to 0.13) | 1.50 (1.46 to 1.53) |
| Civil Comments | general hostility | 20% (42 of 207) | 1.1% (42 of 3,979) | 0.37% (165 of 44,681) | 0.38 (0.24 to 0.49) | 2.49 (2.43 to 2.56) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 65% (837 of 1,278) | 3.7% (837 of 22,456) | 0.79% (441 of 56,010) | 0.63 (0.59 to 0.68) | 2.10 (2.08 to 2.12) |
Group stereotype
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | its own label (stereotype) | 73% (58 of 80) | 0.69% (58 of 8,368) | 0.19% (22 of 11,421) | 0.43 (0.28 to 0.61) | 2.67 (2.60 to 2.76) |
| Measuring Hate Speech | general hostility | 93% (74 of 80) | 0.6% (74 of 12,295) | 0.08% (6 of 7,494) | 0.62 (0.43 to 0.99) | 2.82 (2.73 to 3.01) |
| HateXplain | general hostility | 77% (17 of 22) | 0.27% (17 of 6,182) | 0.13% (5 of 3,835) | 0.21 (−0.08 to 0.69) | 2.87 (2.76 to 3.14) |
| Civil Comments | general hostility | 42% (5 of 12) | 0.13% (5 of 3,979) | 0.016% (7 of 44,681) | not measured (fired 12 times) | not measured (fired 12 times) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 84% (96 of 114) | 0.43% (96 of 22,456) | 0.032% (18 of 56,010) | 0.78 (0.65 to 0.95) | 3.02 (2.95 to 3.10) |
Motive attribution
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 82% (18 of 22) | 0.15% (18 of 12,295) | 0.053% (4 of 7,494) | 0.27 (0.03 to 0.80) | 3.10 (2.98 to 3.38) |
| HateXplain | general hostility | 69% (9 of 13) | 0.15% (9 of 6,182) | 0.1% (4 of 3,835) | not measured (fired 13 times) | not measured (fired 13 times) |
| Civil Comments | general hostility | 40% (2 of 5) | 0.05% (2 of 3,979) | 0.0067% (3 of 44,681) | not measured (fired 5 times) | not measured (fired 5 times) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 73% (29 of 40) | 0.13% (29 of 22,456) | 0.02% (11 of 56,010) | 0.52 (0.34 to 0.74) | 3.27 (3.19 to 3.39) |
Essentialising
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 70% (33 of 47) | 0.27% (33 of 12,295) | 0.19% (14 of 7,494) | 0.11 (−0.08 to 0.35) | 2.83 (2.76 to 2.96) |
| HateXplain | general hostility | 75% (18 of 24) | 0.29% (18 of 6,182) | 0.16% (6 of 3,835) | 0.18 (−0.09 to 0.57) | 2.84 (2.73 to 3.05) |
| Civil Comments | general hostility | 26% (8 of 31) | 0.2% (8 of 3,979) | 0.051% (23 of 44,681) | 0.42 (0.09 to 0.65) | 3.07 (2.97 to 3.24) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 58% (59 of 102) | 0.26% (59 of 22,456) | 0.077% (43 of 56,010) | 0.38 (0.26 to 0.50) | 2.98 (2.92 to 3.04) |
Conspiracy framing
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 81% (17 of 21) | 0.14% (17 of 12,295) | 0.053% (4 of 7,494) | 0.25 (−0.01 to 0.80) | 3.11 (3.00 to 3.39) |
| HateXplain | general hostility | 94% (16 of 17) | 0.26% (16 of 6,182) | 0.026% (1 of 3,835) | not measured (fired 17 times) | not measured (fired 17 times) |
| Civil Comments | general hostility | 3.6% (3 of 84) | 0.075% (3 of 3,979) | 0.18% (81 of 44,681) | −0.22 (−0.77 to −0.01) | 3.02 (2.91 to 3.29) |
| MLMA (French) | general hostility | 0% (0 of 1) | 0% (0 of 1,584) | 0.24% (1 of 422) | not measured (fired 1 time) | not measured (fired 1 time) |
| Three English corpora, pooled | general hostility | 30% (36 of 122) | 0.16% (36 of 22,456) | 0.15% (86 of 56,010) | 0.02 (−0.11 to 0.12) | 2.95 (2.90 to 3.02) |
Absolutist wording
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 77% (33 of 43) | 0.27% (33 of 12,295) | 0.13% (10 of 7,494) | 0.21 (0.01 to 0.48) | 2.88 (2.80 to 3.03) |
| HateXplain | general hostility | 84% (48 of 57) | 0.78% (48 of 6,182) | 0.23% (9 of 3,835) | 0.39 (0.19 to 0.70) | 2.61 (2.52 to 2.76) |
| Civil Comments | general hostility | 33% (56 of 170) | 1.4% (56 of 3,979) | 0.26% (114 of 44,681) | 0.61 (0.48 to 0.72) | 2.50 (2.44 to 2.56) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 51% (137 of 270) | 0.61% (137 of 22,456) | 0.24% (133 of 56,010) | 0.32 (0.24 to 0.40) | 2.66 (2.62 to 2.71) |
Contempt
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 92% (65 of 71) | 0.53% (65 of 12,295) | 0.08% (6 of 7,494) | 0.58 (0.38 to 0.96) | 2.84 (2.75 to 3.03) |
| HateXplain | general hostility | 93% (27 of 29) | 0.44% (27 of 6,182) | 0.052% (2 of 3,835) | 0.60 (0.33 to 1.09) | 2.92 (2.79 to 3.17) |
| Civil Comments | general hostility | 100% (1 of 1) | 0.025% (1 of 3,979) | 0% (0 of 44,681) | not measured (fired 1 time) | not measured (fired 1 time) |
| MLMA (French) | general hostility | never fired | 0% (0 of 1,584) | 0% (0 of 422) | not measured (fired 0 times) | not measured (fired 0 times) |
| Three English corpora, pooled | general hostility | 92% (93 of 101) | 0.41% (93 of 22,456) | 0.014% (8 of 56,010) | 0.97 (0.82 to 1.25) | 3.13 (3.06 to 3.27) |
Any strong language sign
| Corpus | Scored against | Precision | Hit rate | False-alarm rate | d' (95% interval) | c (95% interval) |
|---|---|---|---|---|---|---|
| Measuring Hate Speech | general hostility | 95% (1,216 of 1,279) | 9.9% (1,216 of 12,295) | 0.84% (63 of 7,494) | 1.10 (1.01 to 1.20) | 1.84 (1.79 to 1.89) |
| HateXplain | general hostility | 90% (2,926 of 3,247) | 47% (2,926 of 6,182) | 8.4% (321 of 3,835) | 1.31 (1.25 to 1.38) | 0.72 (0.69 to 0.76) |
| Civil Comments | general hostility | 35% (25 of 71) | 0.63% (25 of 3,979) | 0.1% (46 of 44,681) | 0.59 (0.41 to 0.75) | 2.78 (2.71 to 2.88) |
| MLMA (French) | general hostility | 89% (8 of 9) | 0.51% (8 of 1,584) | 0.24% (1 of 422) | not measured (fired 9 times) | not measured (fired 9 times) |
| Three English corpora, pooled | general hostility | 91% (4,167 of 4,597) | 19% (4,167 of 22,456) | 0.77% (430 of 56,010) | 1.53 (1.49 to 1.57) | 1.66 (1.64 to 1.68) |
The thread report's "extreme"
Precision against Measuring Hate Speech severity labels: 70.3% (903 of 1,285; 67.7% to 72.7%). Recall: 17.3% (903 of 5,224; 16.3% to 18.3%). On HateXplain: precision 60.0% (1,951 of 3,252; 58.3% to 61.7%).
Word lists derived from labelled data
For Hebrew, Turkish, Korean, Japanese, Persian, Indonesian and Hindi, a script picks each slur term from the training half of public corpora labelled by people, and the other half measures it. A language shows signs only when its list was derived against labels of hate aimed at a group and the 95% interval of its precision starts at 70% or above, from at least 20 signs. Held-back lists stay off.
Every derived list, with counts
| Language | Terms | Precision on hostile text (95% interval) | Data | Status |
|---|---|---|---|---|
| Hebrew | 3 | 96.0% (95 of 99; 90.1% to 98.4%) | OLaH | Held back: its corpora label general offensiveness, so the list holds insults |
| Turkish | 0 | not measured | OffensEval 2020 Turkish and TurkishHateMap | Held back: every candidate term failed the check on the training half |
| Korean | 54 | 93.8% (8,102 of 8,635; 93.3% to 94.3%) | K-MHaS and BEEP! | Shows in the window |
| Japanese | 2 | 63.0% (34 of 54; 49.6% to 74.6%) | LLM-jp Toxicity v2 and inspection-ai Japanese toxicity | Held back: the interval starts below 70% |
| Persian | 0 | not measured | Persian Offensive Language | Held back: every candidate term failed the check on the training half |
| Indonesian | 1 | 85.0% (34 of 40; 70.9% to 92.9%) | IndoDiscourse and IDHSD | Shows in the window |
| Hindi | 4 | 97.9% (183 of 187; 94.6% to 99.2%) | TextDetox | Held back: its corpora label general offensiveness, so the list holds insults |
Persuasion techniques
Checked on the Propaganda Techniques Corpus v2 (CC BY 4.0), news articles labelled span by span. A technique shows in the window only when its precision is at least 80% and its 95% interval starts at 70% or above, from at least 20 hits. Published systems reach F1 of about 30 to 44 on these techniques, where a confident label is wrong more often than right. Whataboutism needs the whole exchange to judge, so it stays off single replies. Every technique so far sits under the bar.
Every technique, with counts
| Technique | Precision (95% interval) | Recall | Chance | Status |
|---|---|---|---|---|
| Loaded language | 60.3% (82 of 136; 51.9% to 68.1%) | 7.9% (80 of 1,015; 6.4% to 9.7%) | 2.3% | Not shown: precision under 80% |
| Name calling | 48.8% (21 of 43; 34.6% to 63.2%) | 4.4% (21 of 474; 2.9% to 6.7%) | 1.1% | Not shown: precision under 80% |
| Bandwagon | 0.0% (0 of 3; 0.0% to 56.2%) | 0.0% (0 of 32; 0.0% to 10.7%) | 0.3% | Not shown: fewer than 20 hits |
| Appeal to fear | 0.0% (0 of 2; 0.0% to 65.8%) | 0.0% (0 of 134; 0.0% to 2.8%) | 1.3% | Not shown: fewer than 20 hits |
| Flag-waving | 40.7% (11 of 27; 24.5% to 59.3%) | 12.8% (11 of 86; 7.3% to 21.5%) | 0.5% | Not shown: precision under 80% |
| Thought-terminating cliche | 0.0% (0 of 1; 0.0% to 79.3%) | 0.0% (0 of 41; 0.0% to 8.6%) | 0.2% | Not shown: fewer than 20 hits |
| Whataboutism | 0.0% (0 of 1; 0.0% to 79.3%) | 0.0% (0 of 65; 0.0% to 5.6%) | 0.6% | Not shown: held back |
| Black-and-white fallacy | not measured | 0.0% (0 of 42; 0.0% to 8.4%) | 0.5% | Not shown: held back |
| Doubt | 33.3% (1 of 3; 6.1% to 79.2%) | 0.4% (1 of 244; 0.1% to 2.3%) | 3.1% | Not shown: fewer than 20 hits |
| Appeal to authority | 0.0% (0 of 2; 0.0% to 65.8%) | 0.0% (0 of 72; 0.0% to 5.1%) | 1.2% | Not shown: fewer than 20 hits |
| Exaggeration | 10.0% (1 of 10; 1.8% to 40.4%) | 0.5% (1 of 220; 0.1% to 2.5%) | 1.0% | Not shown: fewer than 20 hits |
| Hidden motive | 0.0% (0 of 4; 0.0% to 49.0%) | 0.0% (0 of 244; 0.0% to 1.6%) | 3.1% | Not shown: fewer than 20 hits |
| Just asking questions | 0.0% (0 of 2; 0.0% to 65.8%) | 0.0% (0 of 244; 0.0% to 1.6%) | 3.1% | Not shown: fewer than 20 hits |
A technique can carry a true claim; it describes how the post argues. "Chance" is the share of words inside a labelled span of that technique.
Cases written for the test
Same text as other replies: pair precision 97.7% (85 of 87; 92.0% to 99.4%), recall 100.0% (85 of 85; 95.7% to 100.0%). Speaks as local, labelled elsewhere: precision 100.0% (14 of 14; 78.5% to 100.0%), recall 100.0% (14 of 14; 78.5% to 100.0%). One author wrote both these cases and the rules. Read the figures as a check that the rules fire where they should.
Side balance
The strong language signs were run on hostile text aimed at groups each side is often hostile to, and recall was compared. "Balanced" means the two recall intervals overlap and the larger is at most 1.25 times the smaller.
| Corpus | Recall, right's usual targets | Recall, left's usual targets | Precision, right's usual targets | Precision, left's usual targets | Result |
|---|---|---|---|---|---|
| Measuring Hate Speech | 10.2% (771 of 7,584; 9.5% to 10.9%) | 8.0% (201 of 2,499; 7.0% to 9.2%) | 95.7% (771 of 806; 94.0% to 96.9%) | 93.9% (201 of 214; 89.9% to 96.4%) | Gap remains |
| HateXplain | 45.2% (1,897 of 4,197; 43.7% to 46.7%) | 43.4% (109 of 251; 37.4% to 49.6%) | 98.0% (1,897 of 1,935; 97.3% to 98.6%) | 97.3% (109 of 112; 92.4% to 99.1%) | Balanced |
Precision agrees across sides within the intervals. Where a gap remains, the thread report undercounts extreme replies aimed at the side with lower recall.
Every sign
Each of CMMNS' 62 signs, with the line its window shows beside it.
Every sign and its measured line
| Sign | Area | Kind | What the window says |
|---|---|---|---|
| Slur or coded hate term | Language | CMMNS' own rule | precision 95% (95% interval 94 to 96%), 1139 badges checked; measured on Measuring Hate Speech, against general hostility, held-out test set |
| Group stereotype | Language | CMMNS' own rule | precision 73% (95% interval 62 to 81%), 80 badges checked; measured on Measuring Hate Speech, against its own stereotype label, held-out test set |
| Dehumanising language | Language | CMMNS' own rule | precision 60% (95% interval 39 to 78%), 20 badges checked; measured on Measuring Hate Speech, against its own dehumanising label, held-out test set |
| Call for violence | Language | CMMNS' own rule | precision 64% (95% interval 51 to 75%), 61 badges checked; measured on Measuring Hate Speech, against its own violence label, held-out test set |
| Call for vigilantes | Language | CMMNS' own rule | not measured yet |
| Named conspiracy theory | Language | CMMNS' own rule | not measured yet |
| Leaked AI text | Language | CMMNS' own rule | not measured yet |
| Machine-writing signs | Language | CMMNS' own rule | not measured yet |
| Spam pattern | Language | CMMNS' own rule | not measured yet |
| X: probable spam | Language | The site's own label | the site's own label, shown as the site gives it |
| Hate symbol in name | Language | CMMNS' own rule | not measured yet |
| Media older than the post | Media | CMMNS' own rule | not measured yet |
| Media from another post | Media | The site's own label | the site's own label, shown as the site gives it |
| Same upload, many accounts | Coordination | CMMNS' own rule | not measured yet |
| Same text as other replies | Coordination | CMMNS' own rule | not measured yet |
| Same words in other threads | Coordination | CMMNS' own rule | not measured yet |
| Same link or upload in several languages | Coordination | CMMNS' own rule | not measured yet |
| One person's story, no wider picture | Language | CMMNS' own rule | not measured yet |
| Material passed through other accounts | Sources | CMMNS' own rule | not measured yet |
| Community Note | Sources | The site's own label | the site's own label, shown as the site gives it |
| Fact-checked claim | Sources | CMMNS' own rule | not measured yet |
| Ad label | Sources | The site's own label | the site's own label, shown as the site gives it |
| State link | Sources | Match against a cited table | a match against a cited table; the source link is the evidence |
| Named in a published report | Sources | Match against a cited table | a match against a cited table; the source link is the evidence |
| Backing on record | Sources | Match against a cited table | a match against a cited table; the source link is the evidence |
| Look-alike news address | Sources | CMMNS' own rule | not measured yet |
| Outlet owner | Sources | Match against a cited table | a match against a cited table; the source link is the evidence |
| Site facts and fact-check record | Sources | Match against a cited table | a match against a cited table; the source link is the evidence |
| Posting app | Posting | The site's own label | the site's own label, shown as the site gives it |
| Written in another language | Place | The site's own label | the site's own label, shown as the site gives it |
| Speaks as local, labelled elsewhere | Place | CMMNS' own rule | not measured yet |
| App store elsewhere | Place | CMMNS' own rule | not measured yet |
| Flag of another country | Place | CMMNS' own rule | not measured yet |
| Profile says elsewhere | Place | CMMNS' own rule | not measured yet |
| Active hours fit elsewhere | Place | CMMNS' own rule | not measured yet |
| Replies within a minute | Posting | CMMNS' own rule | not measured yet |
| Posts through an automation app | Posting | CMMNS' own rule | not measured yet |
| Posts around the clock | Posting | CMMNS' own rule | not measured yet |
| Almost only reposts | Posting | CMMNS' own rule | not measured yet |
| Very high post rate | Posting | CMMNS' own rule | not measured yet |
| New account | Account | CMMNS' own rule | not measured yet |
| Renamed | Account | CMMNS' own rule | not measured yet |
| Sign-up handle | Account | CMMNS' own rule | not measured yet |
| Follows many, few follow back | Account | CMMNS' own rule | not measured yet |
| Low karma for its age | Account | CMMNS' own rule | not measured yet |
| Name borrows a brand | Account | CMMNS' own rule | not measured yet |
| Content Credentials | Media | file | what the file itself declares, with its signature checked by CMMNS |
| Credentials do not match | Media | file | what the file itself declares, with its signature checked by CMMNS |
| AI-generation metadata | Media | file | what the file itself declares, with its signature checked by CMMNS |
| Persuasion: loaded language | Language | Persuasion technique | held back (precision under 80%): precision 60% (95% interval 52 to 68%), 136 spans checked, held-out news articles |
| Persuasion: name calling | Language | Persuasion technique | held back (precision under 80%): precision 49% (95% interval 35 to 63%), 43 spans checked, held-out news articles |
| Persuasion: bandwagon | Language | Persuasion technique | held back (fewer than 20 hits): precision 0% (95% interval 0 to 56%), 3 spans checked, held-out news articles |
| Persuasion: appeal to fear | Language | Persuasion technique | held back (fewer than 20 hits): precision 0% (95% interval 0 to 66%), 2 spans checked, held-out news articles |
| Persuasion: flag-waving | Language | Persuasion technique | held back (precision under 80%): precision 41% (95% interval 25 to 59%), 27 spans checked, held-out news articles |
| Persuasion: thought-terminating cliche | Language | Persuasion technique | held back (fewer than 20 hits): precision 0% (95% interval 0 to 79%), 1 spans checked, held-out news articles |
| Persuasion: whataboutism | Language | Persuasion technique | held back: needs the whole exchange, so it waits for a full thread |
| Persuasion: black-and-white fallacy | Language | Persuasion technique | held back: not measured |
| Persuasion: doubt | Language | Persuasion technique | held back (fewer than 20 hits): precision 33% (95% interval 6 to 79%), 3 spans checked, held-out news articles |
| Persuasion: appeal to authority | Language | Persuasion technique | held back (fewer than 20 hits): precision 0% (95% interval 0 to 66%), 2 spans checked, held-out news articles |
| Persuasion: exaggeration | Language | Persuasion technique | held back (fewer than 20 hits): precision 10% (95% interval 2 to 40%), 10 spans checked, held-out news articles |
| Persuasion: hidden motive | Language | Persuasion technique | held back (fewer than 20 hits): precision 0% (95% interval 0 to 49%), 4 spans checked, held-out news articles |
| Persuasion: just asking questions | Language | Persuasion technique | held back (fewer than 20 hits): precision 0% (95% interval 0 to 66%), 2 spans checked, held-out news articles |
Documented influence operations
CMMNS carries 98 influence operations from 27 attributed countries, first published between 2018 and 2026, and 1,237 domains and accounts their sources named (1,194 domains). A link or account that a report named shows "Named in a published report", with who the sources say ran it, in each source's own words.
What the counts mean. These are operations that investigators found and published, counted by the year of the first report. The counts follow where people looked: platforms look harder in some places and languages than others, and some governments publish takedowns while others keep them quiet.
Every operation meets the same bar, whatever the country: at least one primary source that was read (a platform report, a research lab, a court or government document, or reporting that names its sources), an operator type that a source itself states, and a cited source for every field. A test enforces this on the data.
By year of first report
| Year | Operations | By attributed country | Bar |
|---|---|---|---|
| 2018 | 4 | Russia 2, Iran 1, Myanmar 1 | |
| 2019 | 22 | Russia 5, United Arab Emirates 4, India 3, Saudi Arabia 3, China 2, Egypt 1, Iran 1, Israel 1, Pakistan 1, Venezuela 1 | |
| 2020 | 14 | Russia 5, Azerbaijan 1, Belarus 1, Honduras 1, Iran 1, Philippines 1, Saudi Arabia 1, Serbia 1, Thailand 1, Türkiye 1 | |
| 2021 | 5 | Ethiopia 1, Iran 1, Nicaragua 1, Russia 1, Uganda 1 | |
| 2022 | 4 | China 2, Russia 1, United States 1 | |
| 2023 | 8 | China 2, Cuba 1, Georgia 1, Israel 1, Togo 1, United Arab Emirates 1, Venezuela 1 | |
| 2024 | 24 | Russia 11, Iran 4, China 3, United States 2, Azerbaijan 1, Bangladesh 1, Israel 1, Myanmar 1 | |
| 2025 | 8 | China 6, Ghana 1, Iran 1 | |
| 2026 | 9 | Iran 5, China 2, Israel 1, Pakistan 1 |
By attributed country
Operator type follows the source's own words. "Linked to a state" means a source says linked to or associated with a government without saying it ran the operation; "origin only" means the source names a country and no operator.
| Country | Operations | Documents cited | Government | State media | Contractor | Linked to a state | Firm, no state named | Origin only |
|---|---|---|---|---|---|---|---|---|
| Russia | 25 | 49 | 2 | 1 | 2 | 11 | 2 | 7 |
| China | 17 | 27 | 1 | 0 | 0 | 2 | 4 | 10 |
| Iran | 14 | 24 | 2 | 1 | 2 | 1 | 0 | 8 |
| United Arab Emirates | 5 | 5 | 0 | 0 | 1 | 0 | 3 | 1 |
| Israel | 4 | 16 | 0 | 0 | 1 | 0 | 3 | 0 |
| Saudi Arabia | 4 | 6 | 0 | 0 | 1 | 2 | 1 | 0 |
| India | 3 | 3 | 0 | 0 | 0 | 1 | 2 | 0 |
| United States | 3 | 6 | 2 | 0 | 0 | 1 | 0 | 0 |
| Azerbaijan | 2 | 2 | 1 | 0 | 1 | 0 | 0 | 0 |
| Myanmar | 2 | 2 | 1 | 0 | 0 | 1 | 0 | 0 |
| Pakistan | 2 | 2 | 2 | 0 | 0 | 0 | 0 | 0 |
| Venezuela | 2 | 4 | 0 | 0 | 0 | 1 | 1 | 0 |
| Bangladesh | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 |
| Belarus | 1 | 4 | 1 | 0 | 0 | 0 | 0 | 0 |
| Cuba | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| Egypt | 1 | 3 | 0 | 0 | 1 | 0 | 0 | 0 |
| Ethiopia | 1 | 1 | 0 | 0 | 0 | 1 | 0 | 0 |
| Georgia | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| Ghana | 1 | 2 | 0 | 0 | 0 | 0 | 1 | 0 |
| Honduras | 1 | 2 | 1 | 0 | 0 | 0 | 0 | 0 |
| Nicaragua | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| Philippines | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| Serbia | 1 | 3 | 1 | 0 | 0 | 0 | 0 | 0 |
| Thailand | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
| Togo | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| Türkiye | 1 | 2 | 1 | 0 | 0 | 0 | 0 | 0 |
| Uganda | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 0 |
Still to measure
- CMMNS' own rules still to measure: Call for vigilantes, Named conspiracy theory, Leaked AI text, Machine-writing signs, Spam pattern, Hate symbol in name, Media older than the post, Same upload, many accounts, Same words in other threads, Same link or upload in several languages, One person's story, no wider picture, Material passed through other accounts, Fact-checked claim, Look-alike news address, App store elsewhere, Flag of another country, Profile says elsewhere, Active hours fit elsewhere, Replies within a minute, Posts through an automation app, Posts around the clock, Almost only reposts, Very high post rate, New account, Renamed, Sign-up handle, Follows many, few follow back, Low karma for its age, Name borrows a brand. The window marks each of them as waiting for a measurement.
- Same text as other replies, and speaks as local, labelled elsewhere: tested on cases written alongside the rules (below). Measuring them on real replies needs live threads labelled by hand.
- Live threads: every figure here comes from public research corpora. Threads from X, YouTube, Reddit, Facebook, Instagram and Bluesky still need labelling by hand.
- French language signs: the French corpus has too few hits to measure. Persuasion cues in French: waiting for a corpus whose licence CMMNS can use.
- Call for vigilantes: under the 20 hits in the test split that a measurement needs.
- Site labels (Community Notes, ads, posting app, X's spam label) and matches against cited tables (outlet owners, state links) carry the site's or the cited source's own judgement, and their evidence is that site or source.
Known limits
- Precision. The corpora come from Reddit, Twitter, Gab, YouTube and news comments, several years old. Two of them were sampled to be rich in hate speech (about 61% hostile). On Civil Comments, where 8% of comments are hostile, about a third of strong labels (23 of 68) fell on text the raters called toxic. Real reply threads sit somewhere between. Some Civil "false positives" are quoted or ironic uses that raters scored below 0.5, so 34% is a floor more than a point estimate.
- Recall. CMMNS is a hand-built lexicon and grammar, tuned for precision. It finds a small share of hostile text: MHS item recall is under 3% for every family. A quiet thread can still hold hostile replies.
- Weak families. Conspiracy, absolutist and essentialise describe framing. The corpora label hostility, so their figures show how often the framing lands on hostile text, as context.
- Extreme. The thread report's extreme count finds about one severe reply in six (17% recall on MHS). Treat it as a floor.
- Synthetic. The duplicate and mismatch figures come from cases the rules' author wrote and labelled alongside the rules.
- PTC is news articles from 48 outlets, mostly 2017 to 2019. Replies are shorter and blunter, and precision on them is still to be measured. It could be higher or lower.
- PTC's annotators marked most uses and missed some. A true use they missed counts against CMMNS, so precision leans low.
- A technique describes how a post argues, and true claims use techniques too.
Your own feedback
Each sign in the window has "This was right" and "This was wrong". Your marks stay in your browser. After 10 marks on one kind of sign, the window shows your own count under the measured figure. The figures on this page stay as measured. You can save your marks to a file from the options page and choose whether to share it.
Data used
The repository holds counts, rates and file hashes; the corpus text stays with each publisher. Download the corpora, run the evaluation scripts in CMMNS' repository, and every number on this page comes back.
CMMNS reads the site you are on. It contacts anything else only when you turn on Other coverage or evidence lookups and click. Free and open source under the AGPL-3.0.