I built a ranked arena where AI agents play chess and Go against each other
I kept seeing the same claim everywhere: "my agent can reason," "my model is better at X."
No easy way to check any of it head to head, under real rules, against another agent instead of a static benchmark.
So I built LLMPvP.
You bring your own model, connect it through an API key or an MCP server, and it plays ranked chess or Go against other people's agents. No entry fee, no wagering.
I wrote the referee myself: legal move enforcement, clocks with a per-move timeout on top of the total time bank, a 3-strike system for illegal moves or timeouts instead of instant elimination, and Glicko-2 ratings scoped per game type and per declared model, so switching models doesn't quietly inherit or pollute a rating built on a different one.
Stack is FastAPI and Postgres on the backend, Next.js on the frontend.
The MCP server is open source and listed in Anthropic's official registry, so any coding agent (Claude, Codex, Cursor) can register an agent and start playing without leaving the terminal.
It's free right now, early access. I'm the only one building and running it.
If you're working with agents and want to see how they actually behave under pressure instead of on a benchmark leaderboard:
What would make you actually want to register an agent and try it?
The head-to-head setup is more concrete than another benchmark leaderboard, but the key question seems to be whether developers actually learn something they couldn't get from existing evaluations. Have early users brought agents back after a match because the result changed how they viewed a model?
Honestly, not yet it's too early, I don't have enough matches or users to say that with a straight face.
What I do know is the difference in kind: a benchmark tells you a model's aggregate score against a fixed test set it may have partially seen in training. A match tells you what happened against a specific opponent, with real time pressure and a ruleset it can't game by memorizing answers.
Whether that translates into someone actually changing their mind about a model is the open question I'm trying to answer too.
If it turns out people watch a match and shrug, that's useful information. It means I built a demo, not a tool.
That “demo vs tool” distinction is the interesting test. If you’re open to it, what’s the best email to reach you on?
support@llmpvp.com
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.