human is a linter that counts patterns typical of AI-written text. It runs on rules, with no model.
I wanted to know whether it got better combined with Jev, a model from TypeSafe that answers yes-or-no questions with a probability.
It didn't. I gave it 1,779 of human's flags and asked whether each one was a real case. It said yes to 37% of the flags in human text and 38% of the flags in model text, so it can't tell them apart.
How I measured it
If a text was written before ChatGPT existed, any flag human raises on it is a false alarm.
I collected text from before 2022: posts from technical blogs, articles saved in the Internet Archive, Hacker News comments and Wikipedia. Then I asked Claude (Sonnet, Haiku and Opus) to write about similar topics.
Then I counted each rule's hits on each side.
What came out
Some rules were mostly flagging people. The one for repeating the same word fired 53 times per 10,000 words in human text, three times its rate in Claude's. And the Spanish "utilizar", which was on the banned list, shows up in more than 200 human sentences and in 6 model ones. It's ordinary Spanish.
Others show up more in Claude, but don't give it away. Talking about an unnamed crowd ("what most people never check") or putting a colon mid-sentence happens 2 to 3 times more in Claude. Both also appear in human text, so a single flag proves nothing.
What I changed in human
- The repeated-word rule now needs 5 repetitions instead of 3.
- "utilizar", "hoy en día" and "a día de hoy" are no longer banned.
- Two more rules got a little wider. They catch much more model text, and a little more human text too.
Try it with your own data
human is open source, MIT licensed, and installs with npm:
npm install -g @e1i0/human
Put together two folders. In one, text written before 2022: your blog, your team's docs, whatever you have. In the other, text a model wrote about similar topics. Then run:
human-calibrate --human old/ --ai model/
It tells you, rule by rule, how many times it fires per 10,000 words in each folder. This is part of what I got:
| Rule | People | AI | Difference | Verdict |
|---|---|---|---|---|
mid-sentence-colon |
17.3 | 63.0 | 3.6× | useful |
unnamed-crowd |
1.9 | 7.4 | 3.2× | useful |
aphorism-closer |
17.7 | 23.8 | 1.3× | no real difference |
semicolon |
11.1 | 5.8 | 0.5× | flags people |
polyptoton |
5.1 | 0.5 | 0.2× | flags people |
Times per 10,000 words.
The polyptoton row already uses the new setting. It fires far less than before, but still more on human text.
Your texts may give different numbers.
What I don't know
I only measured Claude models. GPT or Gemini may write differently. And the human text is technical, so for other kinds of text the numbers may change.
The first time, the AI text came out contaminated. I have a hook that runs human on every file Claude writes, and Claude was fixing its own patterns by itself, without me checking anything by hand.
If you try it with another model, tell me what you get in an issue.