Past the forum stunts, real damage turned up fast — and this is the part that justified the clampdown.
- Confident nonsense. Models make things up convincingly — the hallucination problem again. Early users trusted whole paragraphs about court cases that never existed, citations that were invented, even medical doses that were genuinely dangerous. It would also fail laughably easy things — "how many R's in strawberry?" became a meme precisely because the model kept getting it wrong with total confidence. Rarer now. Not gone.
- Cheating and coursework. Students handed in AI essays. Schools banned the tools, then un-banned them with rules, then bought official licences. That fight isn't settled — and it's really about how people learn, not just catching cheats.
- Misinformation at scale. Cheap drafts for propaganda, spam blogs, fake reviews. AI didn't invent lying; it just slashed the cost of sounding official while doing it.
- Sensitive instructions. The press documented real cases where jailbroken models handed over harmful how-to content. That's the direct reason refusal messages got blunter — and why a lot of people now grumble that the apps are "nanny bots."
A limit I learned the hard way
I didn't know the words tokens or context window back then. I just noticed that after roughly twenty messages in one chat, the model went odd — forgetting what we'd agreed, hallucinating more, drifting off the point. My scrappy fix: ask it to summarise the whole session, open a fresh chat, paste the summary in, and carry on. Crude, but it worked. Today's models hold far more in their heads, but the limit hasn't vanished — it's just further out. Worth knowing before it bites you.
Strip it back and the picture is this: enormously powerful pattern-matching with no built-in conscience — and, early on, not much memory either. For a long time the only safety system was the vendor's policies plus your own judgement. That second half is still on you.
Continue — the privacy and legal fights.