hckrnws

I also uploaded my PhD dissertation from 2012 and got back 40% machine, which is just barely below the 42% threshold.

I don't publish anymore... but does this mean I wrote like a LLM or did LLMs learn from me? :p

Update: I also uploaded a IEEE CLUSTERS paper I wrote in 2015 and it came back 74% machine written :|

The biggest results: in Jan of 2026 about 39% of papers got flagged as AI written. In computer science speicifcally the peak was at 65%. Mathematics barely moved away from 0.7%, though the proof heavy math texts might just not get picked up by the detector properly.

All this is a detector estimate of a statistical signal and not a proof any given author used AI. Machine written can also mean heavy AI-assisted editing.

It's hard to say if this code is structurally better or worse than before, but it's certainly voluminous and as far as leadership could ever tell, with their flawed metrics, that is all that matters. It will be years before we figure out if this is a good idea and worth the cognitive atrophy.

Anyone not using LLMs all day is just not going to be as prolific. I can't imagine that the same factors aren't at play in the scientific research community where it's all about how much you can publish.

I've also uploaded text samples from my own (unreleased) research from pre-LLM era, and it's seemingly scoring pretty high on the LLM-detection scores. On other papers, nearly every sentence is highlighted as red "machine-leaning," but that does not impact the score? Additionally, there are dramatic differences between the scores for identical text with and without LaTeX formatting, despite the fact that it should not matter.

The takeaway from this should be "it is difficult to detect generated text and we should be careful about accepting results simply because they confirm a hypothesis."

--

Relatedly, the text above scores as highly machine-written, despite the fact that I just wrote it with my human hands, I promise :)

It is not because we ask LLM to write us a paper about X. It is because we are bad at writing in a scientific style, and american editors expect us to do it. With LLMs, we can write in basic sentences and tell the LLM the idea and it converts that to nice writing.

If you write each paragraph and have an LLM make that paragraph more scientific, it is entirely your paper, but it is flagged as LLM generated. If you have an LLM write the paper but speak good enough English, you can make it look human even though it is not human.

When 65% of the papers you read have the characteristics of being AI written, whether or not you use AI to write, your writing will be influenced by the AI style. I imagine this must be particularly the case for newbie researchers who are still developing their writing style

A more interesting question, to me, is looking at a corpus of essays and analyzing how writing has changed with the introduction of LLMs. We can look at changes in vocabulary, linguistic features, style embeddings, regular embeddings, typos, errors, and references over time. When looked at in this way it is clear that academic writing has changed at the population level but what has led the change is harder to track down.

Perhaps that’s what’s happening.

We were discussing research in general and I asked them: "Do you prefer the writing of the papers or the research?"

They, almost unanimously, agreed that they preferred the research. This makes sense as if they preferred writing they probably would have chosen another profession.

I say this b/c having LLMs available to turn research diagrams, code etc into a paper (or at least the starting point of a paper) will probably lead to MORE quality research papers. This is b/c I'm sure there was some friction in a researcher's mind of "I would love to do the research on this but don't want the trouble of writing the paper".

Put another way: on a 2D plot with one axis being the skills as a researcher and the other being hatred of writing, LLMs may "unlock" the people high on both axes to get more papers out.

Post Script: I agree that this could also lead to more BAD papers but the net may turn out to be positive in the long run.

For example, LLMs love to talk about LLMs (and the people who write with LLMs love to write about LLMs). Could "large language model" itself therefore be flagged as an AI-like phrase by this approach? It didn't exist much in the literature before 2022, does now, and certainly does more in AI-generated text: but, it is not actually a great way to distinguish modern AI generated text from human written text.

A helpful control would be to show that on some cohort of papers that can be declared reasonably clean of LLM generated text post 2023 there are very low rates compared to the arxiv.

For example, while papers in the journals Nature and Science are unlikely to be entirely LLM free at this point, if those were tested through 2026, we should see a line significantly lower than the arxiv's growth.

The question is just how to organize these outputs and conclusions in a way that is consistently reproducible and also how to correct errors or remove LLM nonsense where it refuses to take a position on something.

Before it made sense to do this in papers but it feels like we need something like a paper format.. that is fully reproducible ideally and optimized for aggregating knowledge in a better way. I.e. before a person spent months on one of these and there was just more filtering, and the output itself was a clear signal of time spent and effort that no longer exists.

Its not clear from your writeup what threshold needs to be reached to be classified as "machine written". A preprint where half the text is human and half is 100% AI should be a different category than a preprint where 100% of the text is AI-assisted.

Also its cool that you're making the detector available. When you say "cheap to run", do you know how this compares to pricing for a commercial detector pangram or GPTZero?

I don't think the problem is as bad as a naive reading of this article suggests. I'm highly skeptical that anywhere near 65% of recent CS papers that I've read (mostly systems papers) are substantially AI-written. I threw some recent papers I've read into the system and they come back as 0-7%.

Admittedly, a Stack Overflow answer is somewhat outside the realm of scientific writing, so it's still possible that the detector may be accurate within that domain. That said, it's a cautionary tale on the hazards of applying classifiers like this outside of the domain that they were trained on.

This paper was written using AI, to be honest

One thing I see a lot is papers flagged as AI because they include llm rollouts in the paper as examples.

How is the author so sure that people or companies weren’t already using early tools at that time?

Gotta be trolling :-D

but this post makes me wonder, if more papers' are written with AI, or the shape of knowledge of converging?

I asked Codex to generate an article with a high score and then asked Codex to (reverse?) hill climb that score. The original generation scored 97% and then the optimized one scored 1%. Both are pretty bad and read like slop.

https://gist.github.com/wbew/8a2bd6686bf875210f2244ac8ea65bf...

consider the fraudster that went around suing people on the basis of his absurd claims of being bitcoin's creator. He's now transitioned to using AI to gather graduate degrees and is obtaining masters and doctoral degrees at a regular place and writing multiple 'papers' per day that are all quite obviously AI slop.

People report his cheating and publications and simply no one cares... and this is someone court adjudicated to have fabricated evidence in court on a massive scale, including through the use of AI.

But when it comes to the degrees and publication everyone involved that wanted paid got paid, and apparently that's all that matters.

So what?

Genuinely. I get that there may be some visceral reaction against this, but when I break it down, I mostly fail to see the problem. Seems like what is actually important is:

Compared to before, when a human reads it, do they -- or society -- get something good out of it? Is it worth it to add this to the "pantheon?"

If that's not what's happening enough, and if this doesn't describe the process -- then the problem lies elsewhere, no?

I don't want to read slop generated by AI. AI written articles are generally low effort.

Pot meet kettle?

The article doesn't seem to mention consideration of AI for polishing human work.