1
6 Comments

Where do you get data for your APIs?

I once read someone made an API for oil prices. Sounds cool, but how did this person get real-time oil price data. That's just an example, but in general, how do people get data to feed their APIs especially if it's something that updates more or less real-time like prices?

on July 17, 2022
  1. 1

    a) Type "oil price api" into Google. Replace "oil price" with whatever you're looking for,

    b) Some services actually provide a private API access. Email their support and ask if they provide any API access.

    c) Use API directories like RapidAPI.

  2. 1

    I found that it depends on the data you want. Some data can be retrieved for free, but usually there is a limit on the number of requests and/or amount of data you retrieve. Unfortunately, I don't think you can get a lot of data in realtime completely for free. Usually there are big companies behind the data that make sure it's clean and accurate. However, sometime you can find free APIs from public sources like governments, NGOs and the like. But often these APIs are not maintained well since there is no monetary incentive to do so. As with most things in IT, do a lot of digging and you might strike gold :)

  3. 1

    Google has a useful dataset search: https://datasetsearch.research.google.com

    But usually, the data is scraped from some platform or website

  4. 1

    From other APIs? Not sure about oil or other price/time series data specifically, but RapidAPI might be a good place for you to start, they've got a directory of APIs broken up into categories, that you could use to power your own API service.

  5. 1

    You have a ton of Open data portals, some of them with realtime data. Main platform they use is usually CKAN (ckan.org)

    If those assets are traded on the stock market you can get realtime data from trading platforms.

    If your question is about technical aspect of consuming data in a realtime, usually they use pulling mechanism.