What do you often run as background services? (eg: on rabbitmq)
At my previous company, we used to do a lot of processing on background services, like sending email, generating PDFs, converting images to icons etc.
I'm curious to see if there are a high number of these activities that all SAAS companies end up building.
Trending on Indie Hackers
At a company with moderate focus on user experience, anything that is user-initiated and takes longer than about a second will be done in a background job so that the UI can update the user that the work is in progress. A medium-sized business will have dozens or hundreds of different types of jobs.
I wrote about this recently (link below), but the take-away is that using the database (Postgresql) as your queue is quite effective. Workers run async to your web processes and poll the database for new things.
https://www.simplecto.com/djang-async-task-postgres-not-kafka-celery-redis/
hey thanks Sam, that's an interesting article.
Follow up question: what kind of tasks do you normally perform in the workers?
thanks!
Following up -- I don't normally send emails because the volume is low. I keep those the same thread as the request.
For tasks I take async:
database cleanups (eg - I clean out "free users" who have not confirmed emails for more than 7 days)
remote endpoint heartbeat - I run dummy /test/ roudtrip transactions on third party APIs continuously so that I can see failure before my users see it. Foe example, if the payment gateway goes down then I can alert users before they attempt transaction. Less of an issue with Stripe, but other payment gateways, watch out!
Cache warming for thumbnails (resizing or pulling from s3 buckets)
web page crawling
making high fidelity screenshots
we had also done email & sms notifications as background services
since we had an existing base (from a previous saas app), it was fairly easy to get running
My company runs all our major data processing on rabbitmq & celery services. For some context, we're in the data extraction space, so lots of processing documents to extract useful data from them.
that sounds awesome - do you perform any other operations in your workers other than data processing?
Currently, using bee-queue and redis for small work loads and works well, high volume used rabbit, and google pub/sub
We often run AI model training as background services
For us it was really easy to handle this via Zapier. It was really easy to expose different API calls with a key that we give Zapier. Some endpoints we call nightly, some get called hourly, some our triggered based on Zaps.
I know it seems a bit amateurish compared to other solutions, but it was easy to setup and it works.
My sites are running on Phoenix Framework (Elixir lang) and run on the same Erlang VM that RabbitMQ does, so for me, it's very light-weight. I just create a new module, write a few functions and the new service is up.
As a result, I don't need Redis, RabbitMQ or similar external tools.
Pretty much anything that is going to take 1 second or more to run, including emails, calculating leave accruals, sending messages to/from Slack, handling email bounces etc.
However we don't use a Ruby gem/extension for that, but rather we do it via the AWS SQS service, which sends queued messages to a specific background worker server.
IMO functions (AWS Lambdas or Azure Functions) goes as most cost-efficient and easy to maintain way.
This comment was deleted 2 years ago.
First of all, big fan of your blog - you should write more or at least enable newsletter subscriptions :)
Since the nature of a service Panelbear gets bombarded with data in the form of client requests, I'm curious what your k8s instance size is vs your MAUs (monthly active users) - only if you're comfortable sharing this :)
This comment was deleted 2 years ago.