Managing time-based workflow in code

Usually for a task like this, don't over engineer it

TASK-A stores some sort of row (in an SQL table) with a date
TASK-B is triggered once every 24 hours via cloud cron solution (https://cloud.google.com/scheduler/, kubernetes, https://www.easycron.com/, etc.)
TASK-B searches through all the rows that haven't been marked completed, and are at least 7 days old, and for each of those rows executes its task, and marks the row as completed (maybe even delete the row if you don't need it)

A simple solution like this can scale to 100k, maybe even 1M rows on a nice db server

In the Ruby land - which is where I write most of my code, we have something called Sidekiq which supports this. So, you'd simply schedule your future job as:

MyWorker.perform_at(7.days.from_now, arg1, arg2)

See: https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs

I'm sure there's similar functionality available in other languages too.

If not, one way of achieving the same functionality might be to store a DB timestamp record along with the job details/arguments with a execute_at timestamp and have a periodic polling cronjob that monitors the records say every minute and executes all jobs for which the execute_at timestamp is in the past. Not the most versatile and ideal solution - but simple and good enough for a side project.

If you're using AWS you can, for instance, set a TTL of 7 days on a DynamoDB row. Then you can trigger a Lambda function when that TTL expires.

If I'm understanding you correctly this sounds like https://www.hangfire.io/ for the .net ecosystem (although I haven't used it myself) - might be worth a look.

Cool little service but honestly I wouldn’t pay for it. Rather just run a cronjob and deal with the minor inaccuracies. Do you have an example of a feature that needs to wait 7 days and be executed perfectly to the millisecond?

Yes, thanks for your answers!

@Earth256
Yes, you do it asynchronously and yes you'll probably use a background job scheduler.

@peterj1994
Yes, most of the people will use a cron job.

But to be honest, cron jobs have a lot of cons:

• Cron does not offer tight scheduling constraints (neither sub-second nor near-real-time)
• Cron is often not distributed and is managed on one server, so if this one crashes, it does not work anymore.
• When you used cron for the example I gave above, it will poll your DB every minute to check if it has been more than 7 days than you executed the taskA (to be able to start the taskB). So it generates queries that could be avoided and again it will be more and more complicated when you'll have to scale.
• If you encountered any errors, you don't have the possibility to retry it

These are the major ones, I guess.

My idea is to provide a service that will solve this (not talking about a cron as a service. I like to call it a Wait as a Service.

If I take the example above

In your TaskA implementation, you'll add an HTTP call to my service, where you'll give as a body:

• a callback URL
• a timestamp to wait until
• a tag
• some more options.

What this service will do is that it will, wait until the timestamp you gave and when it's ready, it will ping your backend on the callback URL you've provided, with the options you gave.

So in terms of executions:

In your code, you dispatch the taskA using any background manager you want (Celery, Sideqik, Laravel Jobs, RabbitMQ, etc), the taskA is executed, a wait is registered on my service, 7 days later you receive a webhook on your backend and then you know that you can dispatch the taskB.

The service will provide an API to create / update / delete those wait programmatically and also you'll have a dashboard to see everything.

What do you think? Would you want to try this solution?

If I understand your use case of "Wait for 7 days " correctly, this sounds like something that would be done offline asynchronously. So not inline within your running program but scheduled by your program to be executed by another dedicated program at a desired time (such as sidekiq, Resque, Delayed Job, etc.)

Most servers have the concept of a cron job (or something similar), which are basically a way to run a command in the future or on a set schedule. From there you just point that command to execute whatever code you need to run, which will be different depending on how you deploy your code.