I built a web scraping API for AI agents in one day — here's what happened on day 2

Yesterday I shipped CrawlAPI — a URL-to-markdown scraping API built specifically for AI agents and LLM pipelines. LangChain, CrewAI, OpenAI Assistants — they all need web browsing capability, and the existing options (Firecrawl, ScrapingBee) start at $19-49/month.

We went cheaper and more developer-friendly: free tier, $9.99/mo for 5k calls, ships with OpenAPI spec + LangChain/CrewAI tool manifests out of the box.

Listed on RapidAPI the same day.

Day 2 reality check:

Woke up to 3 API calls overnight. Someone found us and actually tested it. Exciting, right?

Except every single call returned a 401 error. Turns out I'd forgotten to configure the RapidAPI proxy secret in production. Complete rookie mistake. The user hit the API 3 times, got nothing but auth errors, and bounced.

I fixed it immediately, but I have no way to reach them. If you're reading this Ryan — sorry, it works now. 😅

What's working:

• /scrape — single URL to clean markdown + metadata
• /batch — up to 10 URLs in parallel
• /search — query → search results → scrape each one (powered by Brave Search)

Average latency: ~400ms. No JS-heavy pages blocked. Puppeteer under the hood.

Try it: rapidapi.com/search/crawlapi or crawlapi.net

Would love feedback from anyone building AI agents — what endpoints/features would make this genuinely useful for your stack?