Theia Launch
Hi all,
I built this because I got tired of how AI was constantly agreeing with me. I'd push back on an LLM about something true, and it would just change its mind in an instant. That's useful for a yes-man, not for something you're trying to get facts from.
With this extension, users can analyze each message of their AI with just the click of a button. Theia uses sourced fact-checks: pull a claim and Theia checks it against numerous sources while also showing you the actual links to where it derived its information instead of just asserting an answer.
On the side of the user's chat interface, users will also see a sycophancy meter. This feature is powered by a tiny 2-layer BERT model I trained that scores each response on how much it's flattering/agreeing with you vs. being candid. It runs fully on-device.
However, this (and Theia as a whole) is still an early model that will be improved and updated, so any feedback is appreciated. Let me know: Is this a tool you'd use? What could be added/improved? Thank you!
I like that you're treating agreement itself as something worth measuring.
Most AI evaluation focuses on whether an answer is correct. Measuring whether the model has earned its confidence—or is simply adapting to the user's opinion—addresses a very different failure mode.
Really interesting idea. I especially like that you're checking claims against sources instead of just asking another LLM if the first one is correct. The sycophancy meter is a clever addition too. One thing I'd be curious about is whether you plan to distinguish between factual claims and opinion/speculation, since they probably need different evaluation strategies.