Model benchmarks become browser-state lotteries without a fixed execution layer
When two coding models receive the same browser task, we often call the result a benchmark. But if they start with different cookies, page state, retries, identities, or approval rules, we are not measuring the models. We are measuring environmental variance.
For OpenCode evaluations, we now lock six inputs before comparing models:
- Browser identity.
- Session ownership and starting state.
- Exact target and task contract.
- Retry and recovery budget.
- Evidence schema.
- Human approval boundaries.
OpenCode handles model routing and agent configuration. BrowserAct provides a stable real-web execution layer with isolated sessions, evidence capture, and recovery. The score starts with completed tasks that satisfy every check; duration and cost come later.
Full workflow: https://www.browseract.com/blog/opencode-browser-automation
What do you hold constant when benchmarking agents on real websites?
Trending on Indie Hackers
Locking six inputs is a good start, but I'd also pin observation timing and side-effect cleanup. A model that completes after stale DOM recovery may look worse than one that inherited a warm page unless every run starts from the same restore artifact and ends with the same public-state checks. I'd report completion rate with failure taxonomy before duration or token cost.