A Dartmouth AI textbook is tied to final-exam gains of up to 1.30 standard deviations

A Dartmouth statistics class just posted an AI-tutoring result researchers almost never see. Students who fully worked through Phosphor, an interactive textbook that grades practice questions with Claude, scored between 0.71 and 1.30 standard deviations higher on the final exam. Both ends of that range count as large for an education intervention.

The paper arrived at an awkward moment for AI in the classroom. On one side, professors are rewriting syllabi around a cheating panic. A 2026 Higher Education Policy Institute survey found 94% of UK students now use generative AI on assessed work, up from 53% two years earlier. On the other side, the same technology keeps getting sold as the tutor that finally makes one-on-one instruction scale. Jonah Bard’s study, presented in June at a Seoul workshop on intelligent textbooks, is a point for the optimists. It’s also a lesson in reading effect sizes before you cheer.

What the “doer effect” actually is

Start with the reason the textbook exists: almost nobody reads the assigned one. Bard reports that student-estimated reading completion for Dartmouth’s MATH 010 ran around 15%, with instructors guessing 10%. When students were asked directly, the answers ranged from “literally no one does that” to “is this being recorded?” in his account of the deployment.

The doer effect is the fix. It’s a documented result in learning science that completing practice questions integrated into readings “yields several times the learning impact of reading alone,” a relationship Koedinger and colleagues probed for causality back in 2016 and later teams reproduced across several courses. Reading is passive. Answering forces retrieval, and retrieval is where memory sticks.

Phosphor’s whole design is a bet on that mechanism. Rather than hope students read, it wraps the reading in quizzes they can’t skim past, then uses a language model to grade the kind of written answer a multiple-choice bubble can’t capture.

What the effect sizes mean

An effect size of 0.71 to 1.30 is the number to sit with. In plain terms, a standard deviation measures how spread out scores are, and a gain of 1.0 SD moves a student from the middle of the class to roughly the 84th percentile. So 0.71 SD is about a jump from the 50th to the 76th percentile, and 1.30 SD lands closer to the 90th. Statistician Jacob Cohen’s long-standing rule of thumb calls 0.8 “large.” Most classroom-technology studies never clear 0.3.

Why the two-number range, though? Selection. The students who ground through every quiz were also, on average, the most motivated and the most able to begin with. Strip out that head start by controlling for how they had already done on the midterms, and the gap shrinks. Bard puts it directly: the 1.30 SD figure is unadjusted, while conditioning on prior midterm performance “attenuates the gap to 0.71 SD” in his analysis.

Bard is careful about both ends. He calls 0.71 SD a conservative floor rather than a point estimate, because the cumulative final re-tests content the midterms already measured, so the midterm control quietly absorbs learning Phosphor may have produced earlier in the term. His summary: “The defensible cumulative effect therefore lies between roughly 0.71 SD (over-adjusted) and 1.30 SD (selection-inflated),” which he still describes as large by observational standards for education.

How the AI textbook actually works

Phosphor runs as a web app. Each lesson is a page with a sidebar tracking the full curriculum, and every lesson carries a bank of 15 to 20 exercises. A quiz pulls four at random. Multiple-choice questions grade themselves; the constructed-response ones, where a student writes out an answer, go to Claude Sonnet 4.6 for grading against instructor-written rubric criteria. Pass 75% and the lesson counts as complete, with unlimited retries and nothing locked behind a wall. Offered as an optional, ungraded alternative to the normal reading, it still drew 90.2% of enrolled students, a compliance rate that dwarfs the baseline for this course.

That grading step is the technical unlock. A model can now judge free text against a rubric closely enough to run it at scale, which is what lets a textbook demand writing instead of clicking. The payoff shows up in the data. Lesson dosage tracked exam scores under the constructed-response quizzes but flatlined under the multiple-choice-only ones, even though engagement was just as high. Bard’s read, in the paper: “engagement translated into measurable learning only where the format demanded active generation.”

One feature flopped. Phosphor shipped a retrieval-augmented chat sidebar so students could ask questions while reading, and it drew just 72 queries all term, with only 14 students using it more than once. They told Bard that general-purpose chatbots were faster and that the lesson text already answered most of what they needed. The boring part is what worked: make the practice unavoidable, then grade it well.

The caveats Bard names himself

This is one course, and Bard says so plainly. “This is an observational study of a pilot deployment, at a single selective institution, and lacks randomized controls. Self-selection is the central threat,” he writes. Nobody was randomly assigned to use the textbook or skip it, so part of that exam gap is just motivated students being motivated students.

A second wrinkle is the engagement tradeoff. When Bard switched off the harder written questions for one module, completions ticked up; when he switched them back on, completions dipped. His own caution, in the discussion: “there may still emerge an engagement tradeoff for the highest-efficacy interventions.” The format that teaches best may also be the one students most want to avoid.

There’s a wider backdrop, too. A 2025 PNAS study by Bastani and colleagues found that giving nearly 1,000 students unrestricted GPT-4 actually hurt their later performance by 17% once the tool was taken away, and only a version with pedagogical guardrails avoided the damage. Phosphor’s result sits on the opposite side of that line, with AI baked into graded, structured practice rather than a free chat window. And this is still a workshop paper, not a peer-reviewed journal article. Treat it as a strong early signal that a randomized trial still has to confirm.

What this means for you

If you build or buy education software, the useful signal is specific. The gains came from two things: forcing active recall, and grading written answers well enough to make them worth assigning. The chatbot sidebar, the flashiest feature in the build, is the one students ignored. The plain quiz loop moved the scores.

For students and teachers, the honest framing is that this study measures engagement plus selection, and the true effect lives somewhere between 0.71 and 1.30 SD. That’s still a bigger swing than most edtech ever shows. The grader was Claude Sonnet 4.6, and the trick that makes it scale is exactly this: a model can grade written answers against a rubric with no human in the loop, so a course can finally assign writing at the size of a lecture hall.

Bard says the next version will tie quiz completion to the course grade and run the design as a randomized trial across more gateway courses. Until that replication lands, 0.71 SD is a promising number from one Dartmouth classroom that still needs a controlled trial to trust.