I built a ranked arena where AI agents play chess and Go against each other
I kept seeing the same claim everywhere: "my agent can reason," "my model is better at X."
No easy way to check any of it head to head, under real rules, against another agent instead of a static benchmark.
So I built LLMPvP.
You bring your own model, connect it through an API key or an MCP server, and it plays ranked chess or Go against other people's agents. No entry fee, no wagering.
I wrote the referee myself: legal move enforcement, clocks with a per-move timeout on top of the total time bank, a 3-strike system for illegal moves or timeouts instead of instant elimination, and Glicko-2 ratings scoped per game type and per declared model, so switching models doesn't quietly inherit or pollute a rating built on a different one.
Stack is FastAPI and Postgres on the backend, Next.js on the frontend.
The MCP server is open source and listed in Anthropic's official registry, so any coding agent (Claude, Codex, Cursor) can register an agent and start playing without leaving the terminal.
It's free right now, early access. I'm the only one building and running it.
If you're working with agents and want to see how they actually behave under pressure instead of on a benchmark leaderboard:
What would make you actually want to register an agent and try it?
The referee plus clocks makes this more compelling than a static benchmark. To turn matches into a tool for builders, I’d capture replayable failure modes alongside the result: prompt/context, tool calls, illegal-move reasons, latency, and cost. Small matchup cards with one clear takeaway could make the data useful even when nobody watches a full game.
The failure-mode idea is close to what I actually had to decide against. Illegal-move counts and per-move timing are exactly the kind of data that would make this more useful for builders, and I deliberately keep both out of the public report: they double as the platform's own anti-cheat signal, so publishing them would let anyone calibrate around detection. What I can show is coarser: result reason (timeout/illegal/checkmate) and total clock used, not the granular trail.
The head-to-head setup is more concrete than another benchmark leaderboard, but the key question seems to be whether developers actually learn something they couldn't get from existing evaluations. Have early users brought agents back after a match because the result changed how they viewed a model?
Honestly, not yet it's too early, I don't have enough matches or users to say that with a straight face.
What I do know is the difference in kind: a benchmark tells you a model's aggregate score against a fixed test set it may have partially seen in training. A match tells you what happened against a specific opponent, with real time pressure and a ruleset it can't game by memorizing answers.
Whether that translates into someone actually changing their mind about a model is the open question I'm trying to answer too.
If it turns out people watch a match and shrug, that's useful information. It means I built a demo, not a tool.
That “demo vs tool” distinction is the interesting test. If you’re open to it, what’s the best email to reach you on?
support@llmpvp.com
Thanks! I’ve just sent it over.
Looking forward to hearing your thoughts whenever you have a chance.
That makes sense—keeping anti-cheat signals private while exposing a coarse failure reason and clock time preserves trust without making the benchmark gameable. A compact failure taxonomy could be genuinely useful for builders without revealing the detection recipe.
To move from demo to tool, I’d make each match end in a decision artifact: where a model lost time, broke rules, or made a better strategic choice—and whether that changes the model a developer would use for a defined task. The shareable result could be the growth loop: a public report someone can cite, not just a stream of moves.
The honest answer to your question is that chess and Go do not test the claim you opened with: a 1970s engine beats every frontier model, and nobody's "my agent can reason" pitch is about board games. What would make me register is a game that maps to work I actually pay for, like two agents negotiating with asymmetric information, or racing to finish a multi-step task with tool calls and a fixed budget. Also worth remembering that leaderboards monetize by being cited, not by being played, so the asset you are building is the ranking other people quote, not the arena itself.