Most open-source AI detectors can't hold a 0.5% false-positive rate [P]
We needed to know where the open-source AI-detection field actually stands, so we ran every notable open detector through the same protocol.
Setup:
- Public data only: Jabarian & Imas 2025 (NBER), Liang 2023 TOEFL essays, a 1,060-text frontier set (GPT-5.x, Claude Opus 5, Gemini 3.x), 5,000 pre-LLM (2018) FineWeb pages as human pool
- Every model gets its threshold set on the same 6,930 human docs to a matched 0.5% FPR
- Then measure recall per group: raw AI, humanizer-paraphrased AI, frontier models
What surprised us:
- 4 of 6 models effectively can't reach 0.5% FPR. MAGE scores >0.9999 on 26% of ordinary human web text. The old OpenAI RoBERTa detector lands at AUC 0.31 - worse than coin flip on modern generators.
- Humanizer-paraphrased text is where everything collapses: best model catches 42%, second best 4%.
- All models flag non-native essays at a higher rate than they flag native essays. This is a fundamental flaw in the entire class of models, not just one.
| model | ROC-AUC | raw AI | humanized AI | frontier models |
|---|---|---|---|---|
| tropa-mini | 0.968 | 93.2 % | 41.6 % | 33.6 % |
| desklib/ai-text-detector-v1.01 | 0.875 | 83.9 % | 4.0 % | 1.8 % |
| SuperAnnotate/ai-detector | 0.824 | 0.5 % | 1.4 % | 0.6 % |
| Hello-SimpleAI/chatgpt-detector-roberta | 0.571 | 0.8 % | 0.4 % | 0.2 % |
| yaful/MAGE | 0.507 | —* | —* | —* |
| roberta-large-openai-detector | 0.313 | 0.0 % | 0.1 % | 0.0 % |
More information and data on HF
\recall at matched 0.5% FPR; * MAGE can't reach 0.5% FPR at any threshold (flags 26% of human web text with score >0.9999)**
Disclosure: one of the six is ours - we run a hosted detector and released this one as open weights (Apache-2.0). All datasets and the methodology are in the model card, so you can rerun everything: huggingface.co/wasitaigeneratedcom/ai-text-detector-small
[link] [comments]
Want to read more?
Check out the full article on the original site