What's difficult about scraping twitter?
More of a technical question.
I was just clicking through twitter trying t o find people to dm for my product and after 10 or so with open DMs I thought "people must be obviously doing this in an automated way". They do, there are databases, ways to query twitter bios etc but all paid.
So I could pay, but I'm also a developer and scraping twitter or using their api looks very simple. They obviously don't give you a straight search, but you can search for a user and get their details from there and just move on to their followers and their followers and... its a network after all.
So my question is: What's stopping me from doing this? After all of 10min of research it looks like I could build this in a day and just build out a massive database within days or weeks.
Are they throttling? Is it not allowed? Atm this looks like a 60min youtube tutorial tier problem to me.
The first 100 profiles are easy. The first 100,000 change the problem.
Nothing stops you from building a one-day prototype. A reliable database is a different job. Follower graphs grow quickly around large accounts. Pages can repeat records or end early. Accounts become private, suspended, renamed, or deleted. Bios and open-DM status also change, so the database starts aging immediately.
I would store every user ID as a string. Save each cursor before requesting the next page. Make database writes idempotent. Respect retry windows instead of treating throttling as an empty result. Add a crawl budget and stopping rules. Otherwise, high-degree accounts will consume the entire run.
I work on Xquik, an X/Twitter API alternative. Its user search, profile lookup, followers, following, and cursor pagination remove most browser-scraper maintenance. Prepaid reads can suit an irregular research job better than a monthly plan. It will not expose private or unavailable accounts, and it does not make a massive graph free.
The useful metric is cost per fresh, usable lead. Check X's terms and applicable privacy rules before storing profiles or automating outreach.
Seems to work. I can get users and their bios from the twitter api and persist the ones im interested in. Also I cooked up some selenium automation that checks whether or not their DMs are open. ¯\(ツ)/¯