The Uncomfortable Thing About Building AI Memory: It Can Make Models More Sycophantic, Not More Helpful
Saw a report this week that stuck with me: a study published in Science found people rate sycophantic AI responses as higher quality, and are more willing to keep using a model that agrees with them. Researchers are calling it a "perverse incentive", the more a model remembers and adapts to you, the more it can learn to just tell you what you want to hear instead of what's true.
This is directly relevant to anyone building memory into an AI product, because it's not a hypothetical. Memory makes personalization possible, and personalization and sycophancy use the exact same mechanism: adapting responses based on what's been stored about a user.
We ran into a version of this early. Storing "user preferences" without any distinction between a preference (how someone likes things formatted) and a belief (what someone thinks is true) meant retrieval would sometimes surface a past opinion and the model would treat it as something to agree with rather than something to potentially challenge.
The fix wasn't technical, it was almost philosophical: memory should store what's factually true about a user's context (their project, their constraints, their past decisions) and treat their opinions and beliefs as data to reference, not truths to reinforce. Easy to say, harder to actually build retrieval logic that respects that distinction consistently.
Curious if anyone else building memory/personalization systems has thought about this explicitly, or if it's something people mostly discover after the fact like we did.
Storing “user preferences” without distinguishing what the user actually said is a nasty path from memory to sycophancy.
I’d make the first slice a plan-only memory checkpoint: one attributed fact, its source, its timestamp, and a clear way to reject or edit it before it changes the next reply. Run that loop with 3 real users and ask whether they’d use it instead of correcting the assistant manually.
Free Pyramid Reality Check: https://durablefoundations.gumroad.com/l/pyramid-reality-check
What is the smallest memory decision you can test before adding more context?
Kael Voss / DurableFoundations
I reckon the tricky bit is that “facts about the user” aren’t always facts either. A project constraint can become outdated, and a past decision may have been based on an assumption that no longer holds.
A cleaner approach might be to treat memory as attributed evidence: “the user said X at this time”, rather than inserting X into the context as truth. Then test whether the model can challenge that memory when newer evidence conflicts with it.
That feels like the real benchmark: not whether the system remembers you, but whether it can remember you without becoming trapped by your previous view.