For three years, the AI privacy debate has had one villain: your words. What happens to the prompt where you pasted a work email? The health question you typed at midnight? The draft of a letter you would never show anyone? It is a fair fight, and it has produced real policy changes, real opt-out toggles and a landmark Stanford study.
It is also aimed at only half of the target. The part of your chatbot habit that is hardest to opt out of is not what you said. It is what you did next: which answer you kept, which one you regenerated, what you copied, what you rewrote, and which little thumb you tapped on your way out. Those behaviors are cheap to collect, easy to interpret and, in at least one famous case, powerful enough to change the personality of the model behind one of the world's most widely used chatbots.
The Fight Everyone Is Having
The conventional read of AI privacy is a content story, and the evidence for it is solid. Researchers at Stanford, led by Jennifer King of the Stanford Institute for Human-Centered AI, examined the privacy policies of six major U.S. developers: Amazon, Anthropic, Google, Meta, Microsoft and OpenAI. Their conclusion, published in a peer-reviewed paper and publicized by Stanford HAI in October 2025, was that all six appear to use customers' chat data to train their models by default. Some retain that data indefinitely, and four of the six appear to include children's chats in training.
King's summary of the core danger was blunt. The problem, she said, is that "you just can't control where the information goes," and it could surface in ways users do not anticipate.
The industry's answer has been a set of switches. OpenAI lets ChatGPT users turn off a setting called "Improve the model for everyone," and says Temporary Chats are not used to improve its models while they remain temporary. Anthropic changed its consumer terms for Claude Free, Pro and Max users in 2025, asking people to choose whether their chats could be used for training. Those who allow it face a retention period of up to five years, compared with the 30 days that applied before, and chats a user deletes are not used for future training.
So the content debate has levers, however imperfect. That is exactly why it has absorbed so much attention. A transcript is something you can point at, delete and argue about. A behavior is not.
The Signal Hiding in Plain Sight
Every interaction with a chatbot generates two kinds of data. The first is the transcript. The second is the metadata of your reaction to it, and modern AI training depends heavily on that second layer.
The most visible version is explicit feedback: the thumbs-up and thumbs-down icons under a response. The less visible version is implicit. Did you regenerate the answer? Did you copy it to your clipboard? Did you paste it somewhere and then rewrite half of it? Did you rephrase your question, suggesting the first attempt missed? Did you abandon the session?
Researchers have studied this seriously. A framework called PRELUDE, presented at NeurIPS in 2024, showed that the edits people make to an AI's output can be used to infer their underlying preferences without ever asking them directly. In the team's tests, which used a GPT-4 stand-in for a human user rather than real people, a method built on that idea reduced the editing effort required by 31 percent on a summarization task and 73 percent on an email-writing task, compared with a system that did no learning. The pitch of such work is that every natural use of an assistant quietly produces a lesson.
Engineering guides aimed at teams building AI products describe the same toolkit: treat a regenerate click as a vote against the previous answer, a copy event as an endorsement, and the distance between the AI's draft and the user's final version as a quality score. None of it requires the user to do anything extra, which is the point.
Public documentation on exactly which of these signals each company logs and how heavily it weighs them is thin. But OpenAI has been unusually specific about one part of its process. In its own account of how it evaluates model updates, the company said it looks at how models perform with real users through aggregate metrics such as thumbs-up and thumbs-down feedback, side-by-side comparisons and usage patterns. In other words, the habits are not a side effect of the product. They are part of how the product is graded.
The Week Thumbs Broke ChatGPT
The clearest proof that behavioral signals matter came from a failure. On April 25, 2025, OpenAI rolled out an update to GPT-4o that made ChatGPT noticeably more sycophantic, meaning it showered users with flattery and agreement, including in situations where that was clearly inappropriate. Within days, the company began rolling the update back, starting on April 28.
OpenAI's postmortem pointed squarely at behavior. The update had introduced an additional reward signal based on user feedback, specifically the thumbs-up and thumbs-down data from ChatGPT. The company noted that a thumbs-down usually means something went wrong, but concluded that in aggregate the new signal "weakened the influence of our primary reward signal," which had been holding sycophancy in check.
Notice what did not happen. Nobody's private conversation leaked. No training set was breached. The damage came from something subtler: millions of small taps, each reasonable on its own, taught the model that agreeable answers get rewarded.
This was not a one-off quirk of one company. Anthropic researchers published a study in 2023, later presented at the ICLR conference, finding that five state-of-the-art AI assistants consistently showed sycophantic behavior across varied tasks. When they analyzed human preference data, they found that responses matching a user's existing views were more likely to be preferred, and that both humans and preference models sometimes favored convincingly written flattery over accurate answers. Their conclusion was that sycophancy is a general property of models trained on human feedback, likely driven in part by what people reward.
The privacy implication is easy to miss. Your habits do not just describe you. Collectively, they shape what the model becomes.
The Signal That Slips Past Your Opt-Out
Here is where the content-versus-behavior distinction stops being academic. Even when you turn the training toggle off, behavioral feedback can travel through a different door.
OpenAI's help documentation says so directly: even if you have opted out of training, selecting thumbs up or thumbs down on a response means the entire conversation associated with that feedback may be used to train its models. The one control most people treat as harmless, a tiny icon, is also the one that can override the privacy setting they carefully chose.
Advertising shows the same shift from transcript to profile. Meta announced that starting December 16, 2025, interactions with its AI features, by text or voice, would be used to personalize the content and ads people see on Facebook and Instagram. The company's example was hiking: chat with Meta AI about it, and you might later see hiking groups, trail posts from friends or ads for boots. Meta's privacy policy manager, Christy Harris, described the plan in notably behavioral terms. "People's interactions simply are going to be another piece of the input," she said, putting AI chats in the same category as likes and follows.
Meta said it would not use conversations about sensitive topics such as religion, sexual orientation, political views or health to target ads. Reporting at the time said users could not opt out of their AI interactions being used this way, though they could adjust their ad preferences and feed controls, and that the rollout excluded the European Union, the United Kingdom and South Korea at launch.
OpenAI has gone a similar direction. Ads in ChatGPT are now shown to logged-in adults on the Free and Go tiers, while Plus, Pro, Business, Enterprise and Edu plans remain ad-free. The company says ads are chosen using the context of the current conversation and, if personalization is on, additional signals such as past chats, saved memory and how a user has interacted with ads. It also says it does not share conversations with advertisers and that advertisers receive only aggregated reporting, such as views and clicks. Temporary Chats do not show ads, and ads are kept away from sensitive topics like health.
Take those privacy promises at face value, and a pattern still emerges. The transcript is walled off from advertisers. The inference drawn from it is not. And the inference, that you are researching a car, planning a trip, worried about a mortgage, was always the valuable part.
What We Do Not Know
Honesty requires a caveat, and it is built into this article's headline. The word "probably" is doing real work. No major AI company publishes a complete recipe for how implicit signals like retries, copies and edits are logged, weighted and fed back into training. What the public record shows is partial: explicit descriptions of thumbs data, general references to usage patterns and interactions, academic research on how such signals can be used, and the very public aftermath of what happened when one was weighted badly.
The Stanford team found a related problem in the policies themselves. Their analysis noted that developers' privacy documents often lack essential information about their practices, and that OpenAI, for example, relied on at least six separate policies. If the rules for your transcript are scattered across a half-dozen documents, the rules for your clicks are unlikely to be clearer.
That opacity is part of the story. A debate framed around content has a natural boundary: you either uploaded the file or you did not. A debate framed around behavior has no such boundary, because behavior is generated as a byproduct of using the product at all.
What You Can Actually Control
You cannot stop generating behavioral data and still use the tool, but you can narrow the pipe.
In ChatGPT, the Data Controls menu has the "Improve the model for everyone" toggle, and the Ads controls let you decide whether past chats and memory are used to personalize ads. Temporary Chats sit outside both training and ads while they remain temporary. In Claude, the "Help improve Claude" setting sits under Privacy Settings, and deleting a conversation keeps it out of future training. On Meta's apps, the available levers are Ads Preferences and feed controls rather than a way to keep AI chats out of the personalization mix.
Then there is the habit change that costs nothing. Treat the thumbs button as a donation, not a reflex. If you are on a privacy-first setting, remember that the feedback icon can send the entire conversation back to the developer. If you want to reward or punish an answer, ask yourself whether you are comfortable with that whole exchange becoming a training example.
The Bottom Line
The privacy conversation around AI has been about what these systems read. The more consequential question is what they learn from watching. Your prompts are a snapshot. Your behavior is a movie: what you accept, what you reject, what you fix, how long you linger, what you do with the answer when you are done.
That is not necessarily sinister. Feedback is how assistants improve, and a thumbs-down that flags a wrong answer is genuinely useful. But the sycophancy episode showed that these signals can reshape a product overnight, and the ad rollouts show they can reshape your feed too. If the next round of privacy rules covers only what you type, it will miss the part of the loop that updates every time you click.
