March 10, 2019

How to design a task automation SaaS app?

Yo Folks!
I'm relatively new to programming, about a year or so.

Just wanting to make a small web app that will allow registered users to schedule automated tasks - for example, send an email every tuesday to X, or send an SMS/Slack message on the last wednesday of each month to the team etc.

Each user can create their own automations ( and yes i know IFTT does this, but humor me here!) and I need to design a back end that tracks each user, their scheduled automations, and somehow make sure those automations get run.

Anyone has any experience with designing a system like this?

Cheers, Z

#tech-questions

  1. 1

    Hey Z,

    Developers from www.designpac.net are capable to do it. You can rent them at low monthly flat fee.

  2. 1

    The challenge of a task automation SaaS isn't executing tasks, but executing them consistently and keeping a consistent state throughout the execution (including properly handling failure).

    It's too much to write about in one comment, but here are some key tips:

    • Tasks as a queue (take a look at AMQP / RabbitMQ).

    • Separate workers from scheduling.

    • Handle failure perfectly (e.g. their task includes 20 steps, and one "send a mail", but one thing goes wrong after you've already sent the mail - what now?). Also related: prepare the execution environment, test, and only then execute.

    • Similar to AMQP's message delivery guarantee, build an execution guarantee. Make sure that scheduled tasks get executed, with back-up workers to pick up any task that was left behind because of any reason.

    • Related to the previous point: Don't execute each scheduled task more than once. If your internal communication fails to flag a task as "picked up" or "executed" and a second worker picks it up again, you get issues like sending an email to a person twice (which can harm someone's business).

    • Sandbox each execution in its own clean environment.

    • Clean input, clean input, clean input. Make one mistake, and you'll find someone's email body suddenly escaping your execution sandbox to take over your server.

    • Don't build your own monitoring. Use something like Prometheus instead.

    • Distribute your workers and schedule over the whole world. Some countries will go offline sometimes, your network should be capable of handling that.

    If this hasn't discouraged you yet, and you're thinking "I really just want my simple task thing to work", you can build a minimal version that has none of the above and "works".

    Just don't sell it to anyone or give anyone access and you're good. The moment you do, you're liable for the processed data and the bad things explained above that will inevitably happen to it.

    If you like automation and want to build some projects around it, build a single-task-as-a-service instead of tasks-as-a-service: Find one popular task, find a popular library/framework that has already done all the hard work, build a user-friendly web interface for it with some kind of basic scheduling (and no execution guarantees), and ship it.

    1. 1

      definitely just an MVP at the moment, and would ship a basic test function to 10 paying customers and learn.

      the concept is that registered users on my app can create (say) 5 sms or email automations. each of those "tasks" will be run at a time specified by the user. So if 100 users sign up, each with 5 tasks, there will be a list of 500 tasks each with their own intervals and frequencies.

      if i understand you right:

      I need to build the back end that presumably does the following right?

      store each "task" with its properties in a DB

      write a node/python script that parses the list of tasks, and works out which ones need to be done right away

      create a cron job that runs periodically and invokes the node python script

      does this sound right? any ideas on how to write a script that knows which tasks are due next etc, out of the list of 500 tasks?

      Thanks!

    2. 1

      This comment was deleted a month ago.

  3. 1

    Like others, I would recommend a strong scripting language like Python for automation are also even just using javascript and node. Take a look at Google's Firebase. They have a whole suite of cloud services that are easy to use. You could set up your backend database and start running automatic tasks within the day if you put your mind to it.

  4. 1

    I do a lot of automation stuff and my go to language would be Python. I suggest you look at reading "Automate the Boring Stuff with Python". Also like others have said, you will need a database to track the tasks.

    1. 1

      thank you!

      the concept is that registered users on my app can create (say) 5 sms or email automations. each of those "tasks" will be run at a time specified by the user. So if 100 users sign up, each with 5 tasks, there will be a list of 500 tasks each with their own intervals and frequencies.

      I need to build the back end that presumably does the following right?

      1. store each "task" with its properties in a DB

      2. write a node/python script that parses the list of tasks, and works out which ones need to be done right away

      3. create a cron job that runs periodically and invokes the node python script

      does this sound right? any ideas on how to write a script that knows which tasks are due next etc, out of the list of 500 tasks?

      Thanks!

      1. 1

        Depending on how fast each task can be executed. In your case it will depend on how fast the platform can send sms. The steps you outline should work. But why not just have a script that run infinity and check the database every 1 minute increment. Pull out task that has current minute time and execute those task? For example the time is 12:40pm. Your script will pull out all task that needs to execute at the 40 minute mark and only execute those task. Then the script will repeat at 12:41pm.

        1. 1

          thats what I'm leaning to now too. But (this is embarrassing!) not sure how to set up the database fields to show the next_due time and then extract all this that are 1 min overdue etc :(

          any samples you have/know of?

          1. 1

            Google would be your best friend. Here is a example for postgresql (source : https://www.postgresql.org/docs/current/functions-datetime.html#FUNCTIONS-DATETIME-EXTRACT ) Or asking on stackoverflow will get you best result.

            1. 1

              thank you! yes had posted to SO a few days ago. Still waiting. Appreciate this link though!

  5. 1

    Seems fairly straight forward. Mvp would be store tasks in the db along with data about how the recur. Every minute get the tasks and build the tasks for that minute then execute them.

    What are you writing this in? Something like Node would make it easier since you can depend on 3rd party libraries for connections to other services (twilio for sms, slack for slack, etc).

    1. 1

      Thanks mate! its straight forward cause you know what you're doing ;-P im still learning!

      what you describe seems intuitively right, and yes i will build in node/python.

      the concept is that registered users on my app can create (say) 5 sms or email automations. each of those "tasks" will be run at a time specified by the user. So if 100 users sign up, each with 5 tasks, there will be a list of 500 tasks each with their own intervals and frequencies.

      if i understand you right:

      I need to build the back end that presumably does the following right?

      1. store each "task" with its properties in a DB

      2. write a node/python script that parses the list of tasks, and works out which ones need to be done right away

      3. create a cron job that runs periodically and invokes the node python script

      does this sound right? any ideas on how to write a script that knows which tasks are due next etc, out of the list of 500 tasks?

      Thanks!

      1. 1

        You are directionally correct.

        I think you need to come up with some kind of a scheme for indicating how often tasks can run (like CRONs * * * * ) then when someone creates a task you just save the task and the scheme.

        Then you have a cron that wakes up every minute and looks for things that fit into the scheme in the db, then executed the action.

        For the first version I would do a simple static action like send an email or SMS, then level it up from there.

        Shoot me an email and we can chat about this offline. Email is at www.jsfour.com