Does my idea work on Heroku?
Hi guys,
I want to make an analytics app that presents some nice interactive graphs base on a database.
For the tech I decided on:
- Python for scraping
- Pandas or Dask for the calculations
- Azure Document-db for storing (free tier)
- Metabase for vizualization (since I can not design well with Dash)
- deploy everything on Heroku (Hobby).
Do you know if it is possible to deploy all that to one heroku machine? Or what can I do to make the backend working on a schedule while the frontend works all the time?
Sorry for such a noob question, but normaly i just do db backend stuff, and have no expierience with the prior toolset altogether.
I would suggest pretty much you can all those with Firebase. One platform.
You can use Firestore for db.
This comment was deleted 5 years ago
Honestly, anything is possible.
I've scraped on Python previously both using simple HTTP requests, and also using a headless Chrome browser, both on Heroku. It's doable.
I imagine you can use Pandas (or Dask, idk what that is) for calculations. I don't see why there would be a problem there.
I'm not familiar with Azure Document-db. Is that the database you plan on using? Does it have specific benefits over something like MongoDB or PSQL for your use case?
If you Google "metabase heroku" you see some articles indicating you can do it.
For scheduling, you have many options. If you're doing a simple 5, 10, 15, etc minute intervals, Heroku Scheduler would suffice. I prefer keeping my scheduling not dependent on where I'm hosting my app. If you go this route, you can use RQ, Celery, or maybe even APScheduler, depending on your needs.
I just thought for Azure Document-DB that it is almost like Mongo-DB but does not cost anything and the access times should be good. I´m planning on millions of rows. If I want to keep the user expierience high I think psql should not be enough if there are a lot of users at the same time (because of logging issues)
You should be aware of that each process runs int its own individual dyno (https://www.heroku.com/dynos) on Heroku.
The processes that need to be run for an app are defined in the Procfile: https://devcenter.heroku.com/articles/procfile
Do I have to set up 2 projects than with 2 different proc files?
one Proc file can have many dynos, but you have to pay per dyno. - thus simplified per process you run.
But you can only have one HTTP endpoint per app. So for example the Metabase app, if the other parts also need to expose an HTTP interface you would need to move those into a separate heroku app.
But in either way I'd recommend using individual heroku apps for your components and not try to hack around there (this will cause you more trouble in the end)
or if for that pricing is an issue, use a virtual server (e.g. digitalocean) and deploy everything there.
I'm not a Python expert so I don't know for sure, but you shouldn't have any problems. Heroku also has an in-built scheduler so you can set it to run a script at an interval, it will automatically run it on a separate dyno for you.
Hello frieddutchman! First of all, welcome to IH!
I don't know much about Pandas/Dask, Azure Document and Metabase but I've been using the free plan of Heroku for quite a bit and I'm happy with it. However, I don't know how it will behave if eventually my projects reach a wider audience (let's hope that happens some day!).
For instance, for http://spotista.xyz I'm running Python with Flask and I have the online form which is always active. In the backend I use APScheduler to schedule some tasks. Also, to prevent the dynos from "sleeping" I use uptimerobot.com. Perhaps it's not the optimal solution but it works for me so far, at least for testing my ideas.
Good luck with the project!
Thank you so much for your ideas!
No problem :)