Every message the AI tutor sent to a child in Google’s UK trial was read by a human expert first, and 5.3% of them were changed or thrown away before any student saw them.

That sentence is from the technical report behind the research Google cites when it says its classroom AI works. On 3 November, Google announced that access to Gemini, Gemini Notebook, Gemini in Classroom and Google Search “will be entirely off for K-12 student accounts until school administrators explicitly turn them on” (Designing responsible AI for learning). The announcement links the word “research” to two posts as evidence that “Guided Learning leads to measurable learning gains for students”.

Both linked studies are real, pre-registered, and more candid than the announcement that cites them. They also share a feature the product rollout does not: in each one, a trained adult stood between the model and the student.

The condition the UK trial ran under

The UK technical report describes a 10-week randomised trial with N = 1179 students, run on Eedi’s platform with 58 recruited mathematics educators. Students who answered a diagnostic question wrong got one of four things: a static hint, a human tutor, or one of two AI tutoring variants.

The supervision was not a spot check. Gemini generated 114,935 draft messages; supervising tutors reviewed 94,385 of them (the rest were superseded mid-conversation) and “approved the vast majority of these without any changes: 89,406, or 94.7% of the non-superseded drafts”. The remaining 4,979 drafts were regenerated or rewritten, and when a tutor rewrote one they changed “a median of 69% of the original text”.

The report is explicit that this review is what kept errors away from children. Six drafts across the whole corpus were flagged for a mathematical error or a tone problem, and “None of these six drafts reached a student. Supervising tutors replaced five of them during live review, and a new student message superseded the sixth before the supervising tutor could review it.”

The authors also anticipated the obvious objection to a 94.7% approval rate, that the reviewers were rubber-stamping, and tested it: they audited delivered transcripts against a rubric independently of the live decisions, and found weekly approval rates sat in “a narrow range between 92.8% and 96.4%” with no drift as tutors got familiar with the tool. That is a well-designed check, and it earns the result.

The result is split across three measures, and only two of them are positive

Here the report is harder on itself than any critic would be. It measured learning at three levels, and I counted all three rather than the one the announcement implies.

On the proximate measure, whether a student corrects the mistake they just made, AI tutoring matched human experts: 56.9% and 56.5% for the two AI variants, 57.7% for expert tutors, against 43.4% for standard remediation. On near transfer, the next unit’s first question: 58.4% and 58.3% for AI, 59.1% expert, 50.5% standard. On the intermediate curriculum quiz the gaps narrow to 62.1% and 63.5% for AI against 59.6% for standard remediation.

On the broad measure, the independent Renaissance Star Maths test, the effect vanishes. The report states it plainly: “we did not see an overall advantage on Star Maths for tutoring over standard remediation.” AI tutoring with question context came in at −0.01 SD, the personalised variant at +0.02 SD.

And the result that no vendor would volunteer: the human tutors did worst of all on that test, at −0.09 SD, with the authors estimating “an 87.1% probability that expert tutoring underperformed standard remediation”. The report’s own summary of its pre-registered analysis is “two surprising patterns: no overall effect of AI tutoring, and an apparent dip for expert tutoring.”

Their explanation is coverage, and it is checkable rather than hand-waved. Tutored students moved slower, so they covered less: a median of 99 study units for AI tutoring and 77 with a human tutor, against 132 with a static hint. Controlling for condition, each +1 SD in units completed was worth +0.09 SD on Star Maths. The one finding that cuts the other way is also reported: standard remediation left boys ahead of girls by +0.31 SD, and all three tutoring conditions cut that gap to near zero.

The Sierra Leone trial is positive, and conditional in the same way

The Sierra Leone report is the stronger result: 1,763 students across 48 classrooms, an intent-to-treat effect of +0.258 SD (p = 0.029), rising to +0.380 SD for students who hit the requested 12 hours. The “1.2 to 1.7 years” figure the announcement’s lineage traces back to is in there, benchmarked against typical progress in low- and middle-income countries.

It also contains this, in the authors’ own voice: “Crucially, the intervention in this trial is not a technological fix. It is an integrated approach that weaves together pedagogical structure, teacher design and direction, peer interactions between students, and pedagogical AI. The teachers played an especially key role.”

And a limitation they raise unprompted, which runs directly against the marketing use of their work: “studying with Gemini conferred the greatest benefits to students who started with strong mathematics skills. This is a common pattern in educational technology: new tools frequently widen achievement gaps rather than close them.”

Where the gap is

Nothing above says Google lied. The announcement’s claim, that Guided Learning “leads to measurable learning gains”, is supported: the Sierra Leone effect is real and significant, and the UK proximate gains are large and replicated across two AI variants.

The gap is between what was measured and what is being shipped. Both studies measured a teacher-designed activity with an expert in the loop, and the UK one measured a configuration where a human read 94,385 drafts before delivery and intervened on 4,979 of them. The product being switched on in classrooms is Search and a chatbot, used by a student at a screen, with no supervising tutor reading the drafts. The trials are not evidence about that configuration, and they do not claim to be.

Two things the announcement does not say, both verifiable by reading it: the word “Search” appears in it as a product getting Guided Learning capabilities, and the page nowhere states that declining AI also costs a school Google Search. That coupling comes from Google’s education VP Chris Phillips speaking to reporters, who said “Search and AI are deeply integrated” and that students at an opted-out school “will not have access to search” (Ars Technica, 3 November). An administrator reading only the blog post would not learn the price of saying no.

What I could not check

The New York Times piece that broke the coupling is paywalled to an automated fetcher (HTTP 403), so the Phillips quotes here are taken from Ars Technica’s account of that interview rather than from the original, and are labelled as such. I did not open the Don Bosco implementation report behind the Italy figures (79% creativity, 38% drop in front-of-class lecturing, 70% reduction in administrative time), so those numbers are not used above beyond naming them as unverified here. The Star Maths effects are posterior estimates with credible intervals that cross zero, which is the authors’ own framing and not a null hypothesis test. And no document I opened states what supervision, if any, applies to Guided Learning in Search, which is the question the rollout actually turns on.