💀First SaaS tool can’t keep up with growth, should I rebuild from scratch or continue to patch?
Hey friends,
6 months ago I asked a friend to build me a Ruby-based scraper & automation tool for Kickstarter to automate some repetitive tasks on the platform.
Since then a lot of other creators have asked me if they can use it too, and we opened it up, but as the use increases we keep running into crashes and errors because we never built this for scale (or even multiple users!)
I want to grow this as a SaaS as there is strong demand, but we’re stuck deciding if our current codebase needs to be set on fire and rebuilt from the ground up, or if it’s upgradable.
Anyone willing to take a look or offer advice? It much appreciated!
Cheers
It's hard to say with such a vague description of the problems. What types of crashes and errors are you getting? What is the broad architecture of the system? Generally speaking iterating on a codebase is preferable to trying to rebuild the whole thing.
Update: to get more user feedback while reducing the need for infrastructure, we’re building a toned down version of the product as a Chrome extension, so we don’t have to manage headless browsers, but our users can use their own browsers.
Kickstarter doesn’t have an API so in addition to the actual functionality we’re having to do a lot on CAPTCHA solving, IP rotation etc. So instead of solving those problems at scale now, we’re pushing that back to the individual user where they are less likely to be a problem and we can still offer 90% of the functionality to them that we built. For example: its not a big problem for a user to solve a CAPTCHA once every 24 hours, but it requires a lot of code to make sure hundreds of our bots don’t get blocked.
We’re not giving up on the full version but this is an intermediate step in our bootstrapped, first-time SaaS venture as we learn the ropes.
I’ll document the full journey once we have the Chrome extension up and a reflection on this decision.
Penny for your thoughts!
I have a strong background in this area. If you want, I am willing to take a look.
Wow I would seriously appreciate that, both myself and our developer could use a fresh pair of eyes and/or a different perspective. What’s the best way to reach out to you? Thanks so much for your kindness.
Same as my handle here, but on google's well-known email service. So, couple things--I will just look at code/architecture not really planning to debug. I can give my thoughts. If you can give any errors or description of what is going wrong, that would be helpful and perhaps I can offer some direction. A good description of the problems you're seeing (or stack trace), and when they happen. Not to down-sell, but probably my drive-by input won't be more than generally useful, but it's free, and I'm not looking for anything out of it.
Test different pricing models and see if people still buying.
Raise your price to slow demand to buy time to improve quality.
You can take your existing scraper and turn it into a docker image. After that, you would be able to create an infinite number of containers that would perform the web scraping at scale. To manage all those containers you can use Kubernetes. Take a look at this article I wrote for DigitalOcean https://www.digitalocean.com/community/tutorials/how-to-build-a-concurrent-web-scraper-with-puppeteer-node-js-docker-and-kubernetes . I am using node.js in the tutorial but the logic can be used for any programming language and scraping framework.
I wouldn't recommend diving into Docker and Kubernetes right away. It will add too much complexity to the system which is the last thing a very early stage startup needs. You can get very far with just a few vms and some simple bash scripts. My company ingests millions of rows of data per day and we run the whole thing on 5 vms
Thanks Mike that's a great alternative approach to a scraper setup. Do you use AWS VMS or another service? Cheers
Thank you, super helpful and much appreciated! Going to dive into that.