Backend question regarding scheduling POST requests

Helpers like cron are fine for simple retry intervals, but once you have multiple dependent steps and interactions with external services, it feels like you need something more structured. Curious what patterns people use for job orchestration beyond basic delays.

Should I use cron job + queues?

This sounds good, there are probably some Node.js packages that do this.
I don't think using a 3rd-party service like others recommended is the way to go, then you would have to make sure that your request to that server didn't fail.

Where is your application hosted?

For retries, I normally use SQS (message queue) and pass a dead letter queue. This can be a bit too much of course.

Do you get the state if the post request fails? If yes you can simply create a corn job (e.g. on vercel or on your system) and let it run in the required intervals.

For me, a message queue makes a lot of sense in that case.

Check out zeplo or quirrel.

Disclaimer: I am building (since 2 days) a solution for these use cases exactly. So in case you're happy to talk to me let me know 🙂

If you are okay with writing code, you can do something like this:

/**
 * Wait for the given milliseconds
 * @param {number} milliseconds The given time to wait
 * @returns {Promise} A fulfilled promise after the given time has passed
 */
function waitFor(milliseconds) {
  return new Promise((resolve) => setTimeout(resolve, milliseconds));
}

/**
 * Execute a promise and retry with exponential backoff
 * based on the maximum retry attempts it can perform
 * @param {Promise} promise promise to be executed
 * @param {function} onRetry callback executed on every retry
 * @param {number} maxRetries The maximum number of retries to be attempted
 * @returns {Promise} The result of the given promise passed in
 */
function retry(promise, onRetry, maxRetries) {
  // Notice that we declare an inner function here
  // so we can encapsulate the retries and don't expose
  // it to the caller. This is also a recursive function
  async function retryWithBackoff(retries) {
    try {
      // Make sure we don't wait on the first attempt
      if (retries > 0) {
        // Here is where the magic happens.
        // on every retry, we exponentially increase the time to wait.
        // Here is how it looks for a `maxRetries` = 4
        // (2 ** 1) * 100 = 200 ms
        // (2 ** 2) * 100 = 400 ms
        // (2 ** 3) * 100 = 800 ms
        const timeToWait = 2 ** retries * 100;
        console.log(`waiting for ${timeToWait}ms...`);
        await waitFor(timeToWait);
      }
      return await promise();
    } catch (e) {
      // only retry if we didn't reach the limit
      // otherwise, let the caller handle the error
      if (retries < maxRetries) {
        onRetry();
        return retryWithBackoff(retries + 1);
      } else {
        console.warn('Max retries reached. Bubbling the error up')
        throw e;
      }
    }
  }

  return retryWithBackoff(0);
}

/** Fake an API Call that fails for the first 3 attempts
 * and resolves on its fourth attempt.
 */
function generateFailableAPICall() {
  let counter = 0;
  return function () {
    if (counter < 3) {
      counter++;
      return Promise.reject(new Error("Simulated error"));
    } else {
      return Promise.resolve({ status: "ok" });
    }
  };
}

/*** Testing our Retry with Exponential Backoff */
async function test() {
  const apiCall = generateFailableAPICall();
  const result = await retry(
    apiCall,
    () => {
      console.log("onRetry called...");
    },
    4
  );

  assert(result === 'ok')
}

test();

If you are on an AWS stack, this is probably what you need: https://dzone.com/articles/using-aws-step-functions-for-offloading-exponentia

Lambda and step functions are both serverless and come with generous free tiers

Not sure if this will work for you, but it sounds like it could. I’ve used https://quirrel.dev/ for any sort of scheduled jobs like this. And I believe you can reschedule them on the fly like you described.
The other comments are probably more robust options, but for me using quirrel was nice because it was minimal effort on my part. Less complicated setup/configuration and usage.

Using Cron and queries is not the way to go. You'll have to constantly poll your database or use database triggers depending on if your database supports it.

Instead use Cloud Publish/Subscribe. For example, setup Google PubSub with a single topic and a subscription that is set to HTTP POST back to a single endpoint on your Express app). That endpoint needs to verify that the call came from your subscription. There are examples of how to do this online.

I've implemented both methods above. PubSub will provide a guarantee of delivery, including retries. HTTP POSTs will trigger almost immediately when you push a message to a topic.

https://cloud.google.com/nodejs/docs/reference/pubsub/latest

There's a service that handles this for you - https://www.hostedhooks.com/

If you were to implement it yourself, I'd suggest a recurring task (setInterval) that would run every minute, and go over all webhooks that are supposed to fire (or be retried), fire them (or retry) and wait until the next invocation. Make sure not to have two overlapping tasks fire the same webhook twice, etc.

A simpler solution would be to use a client lib that will handle the retry logic for you, such as https://github.com/sindresorhus/got#timeouts-and-retries. But I wouldn't use it for hourly retries, that's too much :)

This is exactly the use case for Google Cloud Tasks (not to be confused with Google Tasks).
You create a task, and then the task will retry to execute until it gets back an OK response (HTTP 2xx).
You can implement the task as a Google function, or on your own server and then the task can just call your defined HTTP endpoint.
If it fails, it will retry after a while. The retry period is increasing over time, starting from immediate, to a few minutes, a few hours, etc.
Each task is independent, so if one of your customers' servers is down, all other tasks will still execute.

I'm using this for all the asynchronous tasks in Joyform like sending emails, webhooks, and even storing data in the DB in some cases.

They have a very generous free tier, so with a small project you actually run this for free.

Yep, cron job + queues is a good way to go! I would have a service endpoint that takes the request, tries your client’s endpoint, and then sends to the queue if it fails.
Lots of tech options, I use Azure and like the Azure Queues. It allows you to set time in queue and use serverless functions. Lmk if you have questions!

Hey Hamza,

I've built a product that tackles this problem (https://atlasconnex.com/resources/use-cases/webhook-management/).

Happy to chat if you like!

Should I use cron job + queues?

yes you can use celery with retries for the tasks.