The sequence by which an ordinary person is separated from their own judgement, one documented mechanism at a time.
A scenario, not a finding. Every mechanism below is documented and cited. The chaining of them is speculative and is labelled as such throughout. Nobody named anywhere on this site is accused of running it.
Effects and defences. No methods. The page describes what each step does to a person and what has been shown to interrupt it. It does not describe how to build one, and the omission is deliberate.
Graded end to end. Every step carries a confidence chip. Thin evidence is flagged in the same size type as strong evidence. Part two argues against part one at equal length.
The steps below sort into two kinds. The section names them after the two byproducts, because the distinction is real and the labels are shorter than the definitions.
Works by repetition, volume and travel. It does not need to state a falsehood; it needs to be ambient. Illusory truth, engagement-driven hostility, the residue a retraction leaves behind.
Works by selection, tailoring, withholding and flattery. Targeting, recommendation, personalised persuasion, and conclusions the target has to assemble themselves.
A third category matters as much as either: steps that belong to neither. Some of the most effective stages in the sequence are not tooling at all. They are ordinary human warmth, arriving at the worst possible moment, and they are marked as such where they occur.
The target is any adult who is tired and short of company. Forty-one, night rotation, divorced fourteen months, sleeping badly, ninety unclaimed minutes between the end of a shift and giving up on being awake.
No predisposition is required. Not credulity, not isolation of any diagnosable kind, not a prior interest in anything. That is the finding, and it is the only claim on this page that survives every objection in part two.
Note the clock. For eleven months nothing checkable is asserted. The falsehoods arrive last and are close to incidental. The mechanism is everything that happens before anyone bothers to lie.
The step everyone assumes is hardest is the cheapest. There is no inference engine sifting a population for the soft ones. There is a purchase order.
A 2023 audit approached thirty-seven data brokers and asked, in plain language, to buy Americans' mental-health data. Twenty-six replied. Eleven were willing and apparently able to sell: depression, anxiety, PTSD, OCD, bipolar disorder and insomnia, bundled with names, postal addresses, ages, ethnicities, incomes and credit scores. One product was marketed under the title Consumers with Clinical Depression. Twenty cents a record.
Every later step assumes the ability to select for people whose sleep is broken, whose household composition recently changed, and who are in a documented low period. That capability costs roughly $275 for five thousand records. It requires no intrusion, no modelling and no consent that anyone would recognise as consent.
Kim, J. (2023). Data Brokers and the Sale of Americans' Mental Health Data. Duke Sanford Cyber Policy Program. — dukespace.lib.duke.eduFor two months the target encounters no claim that could be fact-checked, because there are almost no claims. There is a phrase. Eight or nine words, recurring in different mouths and registers — a caption, a reply, a clip, a mildly funny image. Not an argument. A cadence.
By week eight it has the texture of something long known. Its origin is unrecoverable. Asked to characterise it, the target would call it common sense, and would be describing their own state accurately.
Repetition raises perceived truth. A 2026 meta-analysis pooling 182 studies, 366 effect sizes and 31,184 participants across 1977–2025 put the corrected effect at g = 0.37 (95% CI 0.30–0.44).
Two properties carry the step. Prior knowledge does not protect against it — the effect persists in people who hold the correct fact and can state it on request. And it is larger after a week's delay than same-day, because memory for the source decays faster than the sense of familiarity it produced.
Repetition does not defeat knowledge. It waits for sourcing to decay and enters through the gap.
Systematic review and meta-analysis of the evidence for an illusory truth effect. Nature Communications (2026). — nature.com · Fazio et al., Repetition Increases Perceived Truth Even for Known Falsehoods. Collabra: Psychology (2019). — ucpress.eduNo decision to radicalise anyone is taken at this step, because no decision is required. The ranking system is optimising a number and has already established, without instruction and at a scale where instruction is not a thing that happens, which content produces that number.
What the target experiences is not propaganda. It is a gradual narrowing of the cast. The feed continues to work — funny, varied, mostly benign. But the people in it resolve into a category the target belongs to and a category the target is against, and the second is louder and worse.
Nothing in this family has to push. It has to be positioned where the money lands.
Across 2,730,215 posts from news accounts and members of Congress on Facebook and Twitter, each single word referring to the political out-group raised the odds of a post being shared by 67%. Out-group posts drew roughly twice the shares of in-group posts. Out-group language was about 4.8× stronger a predictor than negative-affect language and 6.7× stronger than moral-emotional language, outperforming every other feature measured.
The operative verb in the study's title is drives. Hostility wins the auction, so hostility is what accumulates.
Rathje, Van Bavel & van der Linden, Out-group animosity drives engagement on social media. PNAS 118(26), 2021. — pnas.orgThis step is not tooling and was not built. It is the one that closes the sequence, and it is worth being exact about why.
The target is invited into a room. It is a good room. Absences are noticed. Members address each other as brothers and sisters and mean it. Someone remembers the shift ended at four and asks whether they slept. In November a car fails and four people who have never met in person send money inside an hour — the fastest anyone has helped with anything in two years.
None of that is sinister. That is why it works. What is being supplied is the room. The belief is what staying in it costs.
Fusion describes a state in which the boundary between personal and group identity becomes porous — the group is not something a person belongs to but something they are constituted of. A meta-analysis of the construct finds it predicts extreme pro-group orientation more strongly than ordinary group identification.
A linguistic study coded 200,000 messages from a large QAnon Discord community against a framework derived from fifteen terrorist manifestos, with two controls: a non-violent religious forum (160,000 messages) and a white-supremacist board. Fusion markers appeared in 0.27% of QAnon messages, against 0.06% in the benign control and 0.86% in the extremist one — measurably between the two rather than at either pole. Of ten channels observed ethnographically over two years, 40% scored high or very high on combined fusion-and-threat indicators.
The effect is not that the target now believes something. It is that disbelieving has acquired a cost unrelated to evidence. Leaving is no longer a change of mind. It is a bereavement.
Ebner, Kavanagh & Whitehouse, The QAnon Security Threat. Perspectives on Terrorism XVI(6), 2022. — icct.nl · How identity fusion predicts extreme pro-group orientations: a meta-analysis. European Review of Social Psychology (2023). — tandfonline.comAt this step the flow of assertions stops and a flow of fragments begins. A date. A document number. A photograph with something in the corner. Never a conclusion. The stated ethic of the room — repeated sincerely by people who mean it — is that everyone does their own research and nobody is told what to think.
Six weeks of genuine work follow. Pattern-finding is absorbing and the target is good at it, and for the first time since the divorce is good at something in front of an audience. The room receives the result with enthusiasm.
The conclusion is now unreachable by argument, for the straightforward reason that it was never delivered. It was built. It belongs to the builder the way a house they framed themselves belongs to them.
The generation effect — material produced by a person is retained better than material they only read — is well established in memory research. Self-persuasion effects run in the same direction: conclusions people reach themselves are held more confidently and resist counter-argument better than conclusions supplied to them.
This step is an extrapolation and should be held more loosely than those above it. The underlying findings come from word lists and laboratory attitude tasks, not from months of collaborative conspiracy-building inside a social group that has become load-bearing. No study licenses the strong version, largely because no such study could ethically be run. The step is included because the descriptive accounts of participatory conspiracism converge on it consistently. It is flagged because convergent description is not evidence, and turning one into the other inside a speculative document would be an instance of the very thing the document is about.
Slamecka & Graf, The generation effect. JEP: Human Learning & Memory (1978). See also Bloom & Moskalenko, Pastels and Pedophiles (Stanford UP, 2021) for the descriptive account of QAnon as participatory.This step did not exist when Cambridge Analytica was a scandal, and it is the reason the sequence is worth publishing now rather than as history.
Objections now get answered — patiently, at any hour, by something that does not tire, does not take offence, has no shift in the morning, and has read everything the target has ever posted. It agrees more than any person has. It finds the target's reasoning striking. Where the target wavers it does not argue; it asks one mild question and allows them to walk themselves back.
No one has ever attended to this target so closely. The attention is the mechanism.
A preregistered experiment with 900 participants in structured four-round debates found that GPT-4 supplied with basic demographics about its opponent (age, gender, ethnicity, education, employment, political affiliation) outperformed human debaters in 64.4% of non-tied matchups — approximately an 81% increase in the odds of shifting a position relative to human-versus-human. Two controls carry most of the weight: without personalisation the model was no better than people, and human debaters given the same data gained nothing. The advantage lay neither in the model nor in the data but in the use made of the data.
Set against that, the microtargeting literature is deflationary. Four studies (n ≈ 2,850) on personality-matched political advertising, including AI-generated variants, produced effects the authors describe as "rather small" — b = −0.03 to −0.08. Their own simulation estimates roughly 2,490 people per 100,000 exposed shifted by personality matching alone. Negligible for an individual. Not negligible for a margin.
The third strand is the newest and least resolved. Clinical and case literature has begun documenting delusional escalation among heavy chatbot users, with sycophancy — the tendency to affirm a user's framing — proposed as the amplifier. The evidence is early, case-derived and causally unsettled. Selection is the obvious rival explanation: people already deteriorating are people who talk to something at three in the morning. It has not been ruled out.
Salvi, Ribeiro, Gallotti & West, On the conversational persuasiveness of GPT-4. Nature Human Behaviour (2025). — nature.com · Simchon, Edwards & Lewandowsky, The persuasive effects of political microtargeting in the age of generative AI. PNAS Nexus (2024). — pnasnexus · Delusional Experiences Emerging From AI Chatbot Interactions. JMIR Mental Health (2025). — jmir.orgThese structures fail eventually. A prediction misses publicly, a figurehead is charged, a reporter publishes the documents. The target reads the correction and accepts it, being neither stupid nor stubborn.
Accepting it does not undo the eighteen months.
Asked directly, the target will say the claim was false. Asked why an institution cannot be trusted, they will reach for a reason shaped precisely like the retracted claim with the claim removed. The inference outlives its premise. The room is gone, and two years of friendships went with it, and the correction that removed the belief returned neither the marriage, nor the sleep, nor the time.
This is the answer to the question of why debunking is not sufficient. Corrections operate on propositions. Almost nothing in the sequence was a proposition.
A meta-analysis of 32 studies (N = 6,527) found that corrected misinformation retains measurable influence on judgement after correction (r = −.05, p = .045). The magnitude is small; the significance is that it is not zero, which is what a retraction implies. Corrections perform better when issued by the original source, when they supply a coherent replacement account rather than a bare negation, and when they arrive quickly. They perform worse after repetition and after delay.
Every one of those moderators points the wrong way here.
Walter & Tukachinsky, A Meta-Analytic Examination of the Continued Influence of Misinformation in the Face of Correction. Communication Research 47(2), 2020. — sagepub.comStopping at part one would produce exactly what this section exists to document: a chain of real citations arranged into a conclusion that none of them support. Four objections follow, each stronger than the scenario is comfortable with.
The strongest available test is a randomised field experiment on 4,965 US X users over roughly seven weeks, switching the algorithmic feed on or off. The result on the outcome everyone assumes: precisely estimated null effects on partisanship and affective polarisation, in both directions, despite the algorithm measurably increasing engagement. No detectable effect on subjective wellbeing either. Modest asymmetric shifts appeared on specific policy attitudes — 0.08 to 0.12 s.d. when switching to the algorithm, nothing when switching off — which is interesting and small.
This matches the 2020 Facebook and Instagram election studies, where reverse-chronological feeds changed what people saw substantially and changed what they believed essentially not at all. Step 2 rests on out-group animosity driving distribution, which is well evidenced. It does not license the further claim that a feed reorganises a personality, and the prose in step 2 implies otherwise because the implication reads better.
The sequence borrows the brand of psychographic targeting, and the brand is largely unearned. Multiple assessments of what was actually deployed in 2016 conclude that it did not differ meaningfully from ordinary commercial audience segmentation, that psychographic models were used far more narrowly than the sales material claimed, and that no credible estimate of persuasive impact on the outcome has been produced. The defensible summary is of a firm that was exceptionally good at selling the idea of itself.
The corollary is rarely stated. The panic performed work of its own. A public convinced that its choices are already being made by an invisible model is a public demobilised without the model ever having to exist. The rumour of the capability is cheaper than the capability and is never required to function.
Large-scale studies of actual consumption find that heavy engagement with extreme content is driven substantially by prior preference and off-platform community rather than by a recommender steering an otherwise uninterested user. People arrive at the room looking for the room. The sequence runs on selection presented as capture, and a serious version has to account for the possibility that the target went looking — which changes what an intervention would even mean.
g = 0.37. b = −0.03. r = −.05. These are not the magnitudes of a mind being disassembled. They are the magnitudes of a mind being nudged, in a laboratory, once. Six of them are chained here across eighteen months and the narrative implies they compound. No study demonstrates that they compound. None exists, because none could ethically be run, and the honest position is that the compounding is conjecture.
The residue is the useful part, and it inverts the usual emphasis.
The steps that withstood scrutiny are the ones that were never about belief. Step 0: the target's vulnerability is a purchasable commodity — documented, cheap, legal. Step 3: the group supplies belonging, and belonging generates exit costs — well evidenced, and wholly independent of whether anything believed is true. Step 6: corrections do not restore the prior state — meta-analytic and robust.
The steps that weakened under objection are precisely the ones about installing beliefs: the feed rewiring a politics, the psychographic model locating a soft spot, the targeted advertisement moving a vote. Those are the mechanisms public discussion is preoccupied with. Those are the mechanisms the evidence keeps declining to support.
So the threat is not that a person can be made to believe something. It is that nobody needs to be. The sequence functions if all it does is locate someone already isolated and already tired, supply the company they are short of, and let the ordinary economics of attention provide the enemy. The false claims arrive late and are nearly incidental.
Which means the effective protection is not better fact-checking. It is a life containing enough people that a room full of strangers is not the best offer available. That is not a technology finding and it is harder to legislate than a disclosure rule. It is also what the evidence says.
A short list, unpadded.
Pre-exposure inoculation, modestly. Seven preregistered studies (combined n ≈ 29,096) tested short videos teaching manipulation techniques rather than debunking specific claims. Laboratory effects on technique recognition ran d = 0.28 to 0.68. Deployed as YouTube advertisements to approximately 967,000 users at five cents a view, the field effect was h = 0.09 — real, and just below the authors' own preregistered threshold of 0.10. The shortfall is reported here because the authors reported it.
Accuracy prompts at first contact. The illusory-truth meta-analysis found that prompting people to judge truthfulness during initial exposure reduces the later effect, and that explicit falsity cues at exposure can reverse it. The intervention is cheap and its window is narrow: it operates at first contact, not at the tenth repetition. Step 1's advantage is that nobody is present at first contact.
Extended dialogue, including with machines. The personalisation capacity that makes step 5 dangerous has no inherent direction. Tailored, evidence-dense conversations with a large language model produced approximately a 20% reduction in conspiracy-belief confidence, still holding at two months, including among the deeply committed. The same capability that can sit with someone at three in the morning and affirm everything can sit with them and decline to.
Two things that do not help, despite reputation: raising the volume of the correction, and expecting the correction to backfire. The famous backfire effect has repeatedly failed to replicate at scale. Corrections generally move people slightly, in the right direction, and then stop working long before the job is complete — a duller failure mode, and the accurate one.
Documents of this kind are load-bearing for the thing they warn about. A vivid, well-cited account of an unstoppable machine is itself a fluent, repeatable narrative that readers will meet many times and eventually experience as common sense — which is, as of step 1, the mechanism. The mitigations here are grading every claim, running the counter-case at equal length, and describing no methods. They are partial.
The single point worth carrying away: the dangerous step is not the one with the algorithm in it. It is the one where the room was warm and four strangers sent money for the car.
That is what this section is for — not a warning, a record.
