Ask IH: How would you set up async tasks processing for a greenfield project?

July 13, 2018

I realize that there are many types of async tasks, and I am mostly curious how would folks handle not so frequent and not so heavy ones. I.e. up to 2-3 times per sec or less, with latencies of up to a few minutes. What building blocks would you use? Can you recommend any off-the-shelve solution?

ilyastam

Say something nice to ilyastam…

Post Comment

1

I've had this problem many times when creating apps. There is always a lot of setup required when self-hosting a queue and you then have to worry about whether workers are overloaded, memory issues, etc.

I created https://taskbird.io to solve this exact problem (p.s. it's currently in beta).

Calum Moore

·
8 years ago
·
Reply
1. 1
  
  nice! cool project, congrats and good luck! I will keep it in mind for my future apps :)
  
  ilyastam
  
  ·
  8 years ago
  ·
  Reply
1

Sounds like a job for AWS Lambda to me. I've had success with Dkron clusters as well, but that's a lot of infrastructure to bother with.

Jeremy Phelps

·
8 years ago
·
Reply
1. 1
  
  Lambda or cloud function in GCP is a good way to go for most quick async tasks. However, the execution duration limitation makes them a bit risky for tasks that can potentially run longer (like an ETL job for example). So I guess I am looking for something that would be as easy to use as lambda, but without time limitations.
  
  ilyastam
  
  ·
  8 years ago
  ·
  Reply
1

What other tech are you using?

Google App Engine has great task queues for this, with scalability and exponential back-off for failed tasks. But not much use if you're not using GAE...

Graham Hughes

·
8 years ago
·
Reply
1. 1
  
  From what I recall, GAE's pull and push task queues still require you to manage workers that will do the actual work (plz correct me if I am wrong). I am wondering if there is a solution out there that encapsulates worker piece as well. At the moment I am stack agnostic. I just want to manage as little infrastructure as possible.
  
  ilyastam
  
  ·
  8 years ago
  ·
  Reply
  1. 1
    
    Not sure what you're thinking about in terms of management. A GAE app comes with a default queue - you dump tasks in it and they happen. It has configuration, but the defaults probably meet your immediate requirements.
    
    On the subject of time limits: GAE background tasks are limited to ten minutes, but after that they fail, they don't cease to exist. Failed tasks are re-scheduled to try again later so, as long as you persist your progress, you can pick up the work where you left off. Not sure about other products, but I'd be surprised if they don't have a similar arrangement.
    
    Graham Hughes
    
    ·
    8 years ago
    ·
    Reply
    1. 1
      
      Thank you clarification Graham. When you say "dump tasks in it and they happen" - what do you mean exactly? Shouldn't there be a worker that consumes tasks from the queue?
      
      ilyastam
      
      ·
      8 years ago
      ·
      Reply
      1. 1
        
        You get a default queue out the box. It comes with a worker. You just do something like:
        
        // define what your task looks like class SomeTask implements DeferredTask { // DeferredTasks are Serializable, so this will be persisted private String someData; public SomeTask(data) { someData = data; } @Override public void run() { // this does the work... // ... or throws an exception and gets scheduled for retry } } // get the default queue Queue q = QueueFactory.getDefaultQueue(); // create the task-instance DeferredTask t = new SomeTask("hello"); // dump task in queue TaskHandle h = q.add(TaskOptions.Builder.withPayload(t));
        
        Graham Hughes
        
        ·
        8 years ago
        ·
        Reply
        
        1
        
        I see, this makes sense. thanks for the detailed answer Graham!
        
        ilyastam
        
        ·
        8 years ago
        ·
        Reply