detectai.media

Why schools shouldn't use AI detectors on students

Post-hoc AI detectors are unreliable, biased against identifiable innocent students, and using them on minors inverts due process. The vendors and major institutions already concede the score is not proof.

By The DetectAI team
• 6 min read
Contents

Schools should not use post-hoc AI detectors to assess students or treat a flag as evidence of misconduct. The tools are unreliable, they are biased against identifiable groups of innocent students, and using them on minors inverts due process: a child ends up having to prove a negative against a black box. This is not a fringe position. The companies that build these tools and the universities that once deployed them have already conceded that a detector score is not, by itself, proof.

The students most likely to be wrongly flagged are the youngest

A detector’s core signal is low perplexity: text a language model finds unsurprising reads as machine-written. Developing writers produce exactly that. They use simpler vocabulary and more formulaic structure, which is the same statistical signature the tool treats as a tell. The direction is causal, not a coincidence. Liang et al. (Patterns 2023) took native English essays and prompted a model to “Simplify word choices as if written by a non-native speaker,” and the essays’ misclassification rate rose from 5.19% to 56.65%. Making writing look less developed, without changing its authorship, is enough to spike the false-positive rate. Liang did not measure a false-positive rate for schoolchildren directly, so the precise claim is this: the mechanism that flags plain, simple prose, together with that experiment, is why the least experienced writers sit at the highest structural risk.

The bias compounds for whole groups of students

The same mechanism does not fall evenly. Liang et al. (Patterns 2023) ran seven detectors over 91 human-written TOEFL essays and found an average false-positive rate of 61.22%, with individual tools ranging from 48% to 76%, while the same detectors were near-perfect on native-speaker essays. Weber-Wulff et al. (International Journal for Educational Integrity 2023) found false accusations rose to 11.1% once genuine human writing had been machine-translated. Neurodivergent students, whose writing can be more repetitive or more formulaic, fall in the same low-perplexity zone by the same logic, though a measured rate for that group is a documented gap rather than a number we can cite. That gap is not a reason to deploy the tools on those students; it is a reason to keep them out of a disciplinary process. A school running a detector across a whole cohort is not applying a uniform error. It is loading that error onto the students who can least afford it.

Using a detector on a minor inverts due process

Set reliability aside for a moment and look at the process. A flagged child is told a machine says the work is not their own, and asked to prove otherwise, against a tool whose version, threshold, training distribution and error rate they may never see, and whose report they may not even know to be document-level or passage-level. That is an inversion an adult in a formal tribunal would not accept, applied to a minor with far less standing to contest it. And it does not stay rare. Base rates make repeats a near-certainty at scale: even a 1% per-document false-positive rate is roughly 500 wrongly flagged essays across a 50,000-essay term. Vanderbilt University (2023) made the same arithmetic concrete for its own campus, noting that a 1% rate across its roughly 75,000 annual papers would wrongly flag around 750 of them. These figures are illustrative rather than a single measured corpus, but the shape holds: run the tool widely enough and it will accuse innocent students on a schedule.

The people closest to the tools have already walked this back

A school does not have to take this on faith, because the people closest to the tools have already reached the conclusion. Perkins et al. (2024), testing six major detectors across 805 documents, state that the tools “cannot currently be recommended for determining whether violations of academic integrity have occurred.” Weber-Wulff et al. (International Journal for Educational Integrity 2023) judged a batch of public and commercial tools “neither accurate nor reliable,” and observed that a detector report is “a simple claim without verifiable evidence,” so a student accused on it alone “would have no possibility for a defence.”

The institutional record points the same way. Vanderbilt University disabled Turnitin’s AI detector and concluded it is not “an effective tool that should be used.” The University of Pittsburgh recommends against the tools and disabled the same detector. Rice University’s Honor Council “will not use detector software as the sole or primary evidence in an adjudication.” MIT Sloan publishes guidance titled “AI Detectors Don’t Work.” The University of North Florida cites both insufficient accuracy and a FERPA privacy risk in submitting student work to an outside detector. The Joint Council for Qualifications, which oversees the UK exam boards, requires detection to form part of a “holistic approach” rather than a sole judgement. Even Turnitin, the leading vendor, states that its score “does not make a determination of misconduct,” while OpenAI withdrew its own classifier “due to its low rate of accuracy.” When the vendors and the institutions both decline to treat the score as proof, a school that treats it as proof is standing on ground its own suppliers have vacated.

The ruling for schools: do not use detectors on students

Put the three failures together and the conclusion is not close. The tools are unreliable, they are biased against innocent and identifiable students, and using them on minors reverses the burden of proof. This is a reliability and due-process ruling, not a claim that cheating never happens and not a method for evading a detector. Post-hoc AI detectors should not be used to assess students or minors, or treated as evidence of misconduct. If an institution has a genuine authorship concern, the answer is ordinary evidence: the drafting trail and version history, notes and sources, a conversation about the work where appropriate, and assignment design that makes the process visible, all judged against the course’s stated AI policy. When both the makers and the institutions concede the score is not sole evidence, refusing to build integrity cases on it is the earned conclusion, not an activist one.

For the underlying mechanism, see why AI text detectors falsely accuse real writers; for what a wrongly accused writer can do, see falsely accused of using AI? what to do.

Sources

#text#students#detection
Last updated
30 June 2026
Category
Reliability