IDEA: Poll an API @ 1$ per month
Imagine you’re polling an API to look for the change in the response. Consider the following example:
{"product": "Wireless Fancy Keyboard", "price": 22}
Next moment the price changes to 5$
{"product": "Wireless Fancy Keyboard", "price": 5}
What if you need to alert your subscribers to the price drop? Or save the changes to the database?
What option would you choose?
- Serverless?
- Cloud Instance to monitor the changes and then perform the action?
I was working on one of my side hustles when I felt the requirement to ping the API to look for the changes. Based on its changed response at any given moment, I have to send the notifications to my subscribers.
Serverless:
Initially, I chose serverless. Why Serverless? Serverless enables you to roll out your code without managing a VM. And gives you a handful of quota, to begin with. Sometimes even a free tier for a lifetime. See Google App Engine.
Problems: I needed to keep polling the API 24x7 to look for the changes. Serverless does not maintain the state of the program. It times out after the timeout period, and the state is not maintained. There are workarounds but too much of a hassle for a simple polling function.
What we’re left with?
A Cloud VM
If you are looking for an online cloud VM and to do a setup. It would cost you a decent 5$ + taxes to look for the changes in the API. I’m considering building the SaaS, enabling developers to poll an API @ barely $ 1$ per month and look for changes in the header and body. After that, a developer could use Serverless webhooks and prevent context switching from serverless to managing VM during development.
What problems would this SaaS solve?
-
The developer doesn’t have to set up a VM to poll for changes and pay 5$ per month.
-
The developer does not have to switch contexts between server and serverless.
-
… Need answers from you?
I'd like to have some insights into the initial thought process. In what way would this app help your use case?
Hi there!
Your best bet from cost and architectural standpoints would be serverless (either function or container, your choice), for state preservation, I would go with a Redis instance.
Why? Redis gives you the benefit of persistent storage, but speed of memory caching. Setting this up as a LRU (Least Recently Used) cache, you would have data persistence over time and between runtime instances, but keeping that messy state of your application layer (app layer should never track state!)
In summary:
Now, something that also might be worth implementing is a traditional disk database that houses customers and subscription relationships, but this should be separate from the more ephemeral api data that we would house in Redis.
Hope that helps! (I'm a lead eng at one of the top Graphql companies during the day, this is a very well excepted pattern!!)
I think most people that have this problem will build out their own solution very quickly. Regardless I wouldn't recommend starting a business with such cheap pricing in mind - you'll never be able to make it worthwhile.
As far as pricing is concerned. I can optimize the pricing later. Although currently, I believe the pricing has to be cheaper than spinning up a VM on the cloud.
I've used async programming using Rust to reduce the memory footprints to solve the pricing issue.
Hi, I built https://memc.sh to solve the state preservation problem with lambdas. Essentially, it's memcached-as-a-service. It's designed to be cost effective and easy to use.
One reason I built it was because I needed a small but non-trivial project to upgrade my processes, so it might go down if my new stack has a mistake, but I have deployment guidelines to protect against that.
It currently runs as a single instance EC2 server, so it might go down for brief moment when I deploy an update. As I get more people using it, I would be more willing to upgrade the availability of the service.
I am looking for initial users and feedback, especially questions like what do people like about it, or what limitations are keeping someone from using it. If anyone has any ideas on how I can improve it, please let me know!
I like this, and using memc.sh / redis could potentially address my goal.
You can also spin up a Node.js VM on the Oracle Free Cloud tier, then you just have a
setInterval(fetch('https://api.com/price').then((res) => res.json()).then((price) => console.log(price), 30000). Being a Node.js instance, you can also save data or use database or do any complex queries.With AWS serverless, you can call a Lambda function once every xx time, similar to a cronjob, by using CloudWatch. I have functions that are automatically run once every 24 hours, once an hour, once a week, using this approach.
But what you are explaining is a problem long solved by webhooks, which are now used by most APIs. The description of your problem is that a price may change and while we call the REST api to see the change, the change may yet not be there so we just did a useless api call. Which is solved by webhook, easily. Am I missing something? What exactly are you bringing new, considering we can use webhooks? I'm interested, thanks.
The missing part is state preservation.
I'm only considering the scenarios where webhooks are not provided by the provider.
So let's only consider long polling a public API (usually not documented) like binance staking or a webpage.
While long polling to watch for changes, serverless functions time out and lose state after a defined duration by the provider.
To address this problem we need to bring persistent storage like a database into the picture to store the previous response to compare against. This brings an additional layer of complexity.
If I couldn't convey to you the value it brings. I would love to connect with you to explain.
I can't see the value you are proposing. If your target audience is developers, I can't imagine why a Dev would pay you to solve it. If this is a trivial day to day problem.
I could see you supporting the no-code niche where people are not developers. Still take a look at Bubble marketplace and see if there is someone already solving something like it.
You could try to build something that does long polling on a API and provide it as a webhook for places like Zapier or IFTT.
But the same holds true, this is a really simple problem for devs to pay for it. And still a simple problem for no-coders as well.
This is precisely what I mean. I can call it "Convert the polling endpoint to a webhook."
It is indeed simple. I believe devs have to spin up a server to achieve this simple task and whereas the rest of their app architecture depends on serverless. I want to simplify this. If proxies are involved, another layer of complexity adds up.
The longer goal is to add rotation of proxies and notify changes in the HTML page too.
Sure, I just saw this post again, read it again and I'm failing to see what you're trying to achieve. I'm sure it's something of value, I just can't see it yet, let's connect!