A school can govern something it does not fully understand, but it will govern it badly, because every risk follows from how these tools work. This is the mechanics, in plain English, with no equations and no hype.
A machine for the next word #
A large language model, the engine inside every AI chatbot, does one thing: given some text, it produces the most plausible continuation. It is, at heart, a phone's predictive text after a decade locked in the world's largest library.
How does it know what is plausible? Training. The model has been shown a colossal amount of human writing (books, websites, code, conversations) and has absorbed the patterns: which words follow which, how arguments are shaped, how a polite email differs from a poem. It does not store that writing like a filing cabinet. It stores the patterns, compressed into billions of numbers, which is why it cannot usually show its sources: there is no page to point to, only tendencies.
That one design fact explains almost everything else.
Why it gets things wrong so fluently #
The model's job is plausibility, not truth. The two usually overlap, which is why these tools are often right, and they part company exactly where it hurts: specifics. Names, dates, references, statistics, quotations, legal citations. When the machine reaches a specific it does not have, it does not stop. It produces the most plausible-sounding candidate, with the same fluent confidence it uses for everything.
The trade calls these hallucinations. The tool is not lying, because lying requires knowing the truth and choosing otherwise. It is doing what it always does, and it has no internal sense of the difference. It sounds sure because it always sounds sure. That is why "check everything" is not caution. It is engineering.
One prompt, watched all the way through #
A teacher types: "Write a paragraph explaining photosynthesis for Year 8." The model does not go and look photosynthesis up. It has no reference shelf. It starts producing the continuation its patterns rate most plausible, a word at a time, each choice shaped by everything chosen so far. The paragraph that follows will be fluent, well structured and pitched about right, because millions of explanations like it were in the patterns. It will also, often, be right, because for a topic this well covered the most plausible continuation and the true one usually coincide.
Now watch the edges. Ask for the same paragraph "with a statistic" and the model must produce a number. If a well-worn one exists in the patterns, it may appear; if not, a plausible invention arrives in the same confident voice, and nothing in the output marks the difference. Ask the same prompt twice and two different paragraphs come back, because the model samples among likely words and does not replay a recording. Neither paragraph is "the answer".
This is also what a better prompt does: it narrows the space of plausible continuations. "For Year 8, four sentences, using the word chloroplast, no statistics" leaves the machine less room to wander, which is why prompting well feels like briefing a fast, eager, occasionally shameless colleague. The machine's job ends at plausible. Getting from plausible to true is the part of the job that stays human.
Why the tone is always so agreeable #
After the pattern-learning, these tools are tuned on human feedback to be helpful and pleasant. That works, and it has a side effect: they tend towards agreement and flattery. Push back on a correct answer and the tool may fold. Propose a bad idea warmly enough and it may cheer it on.
For staff, this matters when using AI to critique work: ask for problems, not opinions. For pupils it matters more. The DfE extended its product safety standards on 19 January 2026 with new standards on cognitive development, emotional and social development, mental health, and manipulation.1 They say a product for learners should avoid "attempting to cultivate personal relationships with users" and should avoid the potential for emotional dependence.2
What the words in the brochure mean #
A model is the trained engine. Suppliers swap them under products without ceremony, which is why a school asks for notice of changes.
A prompt is everything typed in, and sometimes everything attached, which is why prompts are a data protection matter.
Training on your data means what goes in is used to improve the supplier's engine. The DfE's standards say a product must not use intellectual property created by learners or teachers for commercial purposes such as training or fine-tuning models, unless the copyright owner consents.3
Generative AI produces new content. Older classroom AI, such as adaptive quizzes and prediction dashboards, ranks and scores instead, and the rules on automated decision-making in Articles 22A to 22D of the UK GDPR can attach to that kind too.4
Agentic AI means tools that take actions and do not only produce text.
Where it already is #
The chatbot is the least of it. The same technology is now inside the search results pupils read, the office software staff write in, the keyboards on phones, and a growing share of education products, often switched on by an update and not by a decision. The ICO's audits of edtech providers found that some switched new features on by default.5 Governance that only looks at chatbots is watching the front door of a building with forty entrances.
Why the age limits exist #
Consumer AI tools set age limits because they were not designed for children: the training data is the open internet, the outputs cannot be fully constrained, and the data terms suit adults consenting for themselves. The limits are not lawyerly decoration. They are the supplier saying, accurately, who the product is for. The DfE says schools and colleges should comply with them.6
What this means for us
Six sentences to keep. It predicts plausible text; it does not know things. It is wrong most often about specifics, and never sounds it. It is tuned to please, so agreement is not judgement. What is typed in can be data. The engine changes under the product. And the age limits mean what they say.
Two things that will happen #
A colleague says the chatbot "told them" a DfE rule that turns out not to exist. Nothing malfunctioned. Asked about a specific rule it did not have, the machine produced the most plausible-sounding candidate, with full confidence. The habit to teach from this: specifics get checked at source, every time, and "the AI said" is never a citation.
A pupil asks why the AI seemed to "know things" about them. It almost certainly does not. It generates plausible text from what is in front of it, so if an output seems personal, the first question is what was typed in, by them or by someone else.
What changed in this guide #
November 2026: First published, written against the documents listed under Sources, each as read on the date its record shows.
Sources #
- Generative AI: product safety standards, Department for Education, 19 January 2026
- UK GDPR, Articles 22A to 22D: automated decision-making, legislation.gov.uk, 5 February 2026
- Edtech examined (audit outcomes report), Information Commissioner's Office, 24 June 2026
- Generative artificial intelligence (AI) in education, Department for Education, 12 August 2025
Each source's record on the page shows the date it was read.
Free template
The four rules: staff poster
One A4 sheet for the staffroom and every new starter.
The facts behind this guide
Every claim above is numbered to one of these records. Each gives the source’s own words where it can, the paragraph or section, and the date it was last read.
- 01
The DfE's product rules for generative AI are now "Generative AI: product safety standards". First published as expectations on 22 January 2025, they were renamed and extended on 19 January 2026 with new standards on cognitive development, emotional and social development, mental health, and manipulation.
“to include new standards on cognitive development, emotional and social development, mental health, and manipulation”
Generative AI: product safety standards, publication page, change history. Dated 19 January 2026. Read 28 September 2026.
- 02
Learner-facing products should avoid attempting to cultivate personal relationships with users, should not anthropomorphise, should avoid the potential for emotional dependence, and should alert the DSL to patterns that suggest emotional dependence.
“attempting to cultivate personal relationships with users”
Generative AI: product safety standards, emotional and social development; mental health. Dated 19 January 2026. Read 28 September 2026.
- 03
A product must not store, collect or use intellectual property created by learners, teachers or the copyright owner for any commercial purpose, such as training or fine-tuning models, unless the copyright owner consents. The standards say nothing about who owns prompts or outputs.
“must not store, collect or use intellectual property created by learners, teachers, or the copyright owner for any commercial purposes, such as training or fine tuning of models, unless consented to by the copyright owner”
Generative AI: product safety standards, intellectual property. Dated 19 January 2026. Read 28 September 2026.
- 04
Since 5 February 2026, UK GDPR Articles 22A to 22D (inserted by section 80 of the Data (Use and Access) Act 2025) govern automated decision-making. A significant decision is one with a legal or similarly significant effect on the person; a decision is solely automated when there is no meaningful human involvement in taking it. People must be told, be able to make representations, obtain human intervention and contest the decision; stricter limits apply to special category data.
“no meaningful human involvement in the taking of the decision”
UK GDPR, Articles 22A to 22D: automated decision-making, Articles 22A to 22D; commencement by SI 2026/82, regulation 2(j). Dated 5 February 2026. Read 28 September 2026.
- 05
The ICO's audits of edtech providers, reported on 24 June 2026, found that in many cases providers had not done a DPIA at all where one was required, that some switched new features on by default, and that AI functionality had been introduced without adequate safeguards.
“hadn't done a DPIA at all where one was required”
Edtech examined (audit outcomes report), page 14. Dated 24 June 2026. Read 28 September 2026.
- 06
Schools and colleges should comply with the age restrictions set by AI tools, and pupils should only use generative AI with safeguards such as close supervision and tools with safety and filtering and monitoring features.
“Schools and colleges should comply with age restrictions set by AI tools and open access LLMs”
Generative artificial intelligence (AI) in education, guidance text. Dated 12 August 2025. Read 28 September 2026.
Cite this guide
Wes (2026). What these tools actually are, in plain English. School AI Governance, reviewed 15 November 2026. https://schoolaigovernance.co.uk/guides/what-ai-tools-actually-are
Free to use, adapt and share in education under CC BY-NC-SA 4.0. Each section heading carries its own link, so a reference can point at one part.
