I would not scale Selenium before fixing the browser layer

We have been debugging browser automation workflows long enough to see the same pattern repeat: the first instinct is to scale the scraper before fixing the browser layer.

More workers. More proxies. More retry loops. More fallback selectors.

Then the same failures come back, just across more machines:

• headless got detected faster than visible Chrome
• navigator.webdriver needed patching
• 429s did not show up as clean HTTP errors, just a "Reload" button in the page
• login modals intercepted clicks on almost every load
• VPN endpoints still got flagged
• UI A/B tests forced selector fallbacks

That is a lot of machinery before the scraper even gets to the business logic.

I think a lot of us over-scale too early. We add more nodes, more proxies, more retries, more modal hacks. Then the real issue is still sitting underneath: the browser layer is leaking automation signals and the interaction layer is brittle.

This is the exact reason we built browser-act.

The CLI gives an agent or scraper a persistent browser profile, stealth browser mode, real Chrome control when you need aged cookies, built-in captcha solving, and an indexed state command so the script clicks element numbers instead of trying to invent CSS selectors.

The workflow becomes closer to:

browser-act --session jobs browser open <browser_id> https://example.com
browser-act --session jobs wait stable
browser-act --session jobs state
browser-act --session jobs click 5
browser-act --session jobs get markdown

That does not remove every hard problem. If a site is doing serious A/B testing, you still need detection around page variants. If the target is rate-limiting account behavior, IP and timing still matter.

But it moves the first layer of pain out of every scraper project.

Instead of each project maintaining its own Puppeteer/Selenium + stealth plugin + captcha API + cookie jar + selector fallback stack, the browser behavior becomes a reusable CLI boundary. That is especially useful for AI agents, because the agent can inspect a page through state and get markdown instead of burning tokens on raw HTML or hallucinating selectors.

One concrete number from our own runs: Product Hunt's AI page was 680,193 chars as raw HTML and 49,272 chars as markdown through browser-act get markdown - about a 93% reduction before the model sees it.

If I were rebuilding a Selenium-heavy scraping stack today, I would not start by buying better proxies or adding more boxes.

I would first ask:

1. Can the browser profile persist cookies and real session history?
2. Can the automation avoid obvious headless fingerprints?
3. Can the agent interact without brittle selectors?
4. Can I inspect network/markdown when I do not need DOM control?
5. Can captcha/human handoff be one command instead of a separate subsystem?

Scale still matters. But scaling a fragile browser layer just gives you 50 fragile browsers.

Install command if you want to test it:

npx skills add browser-act/skills --skill browser-act

Curious how people here split the stack now: do you still keep Selenium/Playwright as the main runtime, or do you try to reverse-engineer requests first and only use a browser for the protected steps?