hi guys,
happy to join the community, this is my first post :)
I was contacted by a friend who works in a startup (he is the CTO).
He is very skilled at building nice front-end tools but lack of knowledge regarding data modelling and good practices regarding data.
Basically, he asked me what is the best practice to apply when it comes to aggregating all the data they have into one database aimed to serve a dashboarding tool on which they will be able to monitor all the metrics they have.
They collect data from different tools/services:
I am tempted to suggest to load everything into BigQuery for example and link a dashboarding tool such as Tableau/Qlik/DataStudio. But my answer would be quite limited and I don't really know what are the steps to take to answer properly.
Thanks a lot for your time !
Copy/pasting from my website, because I'm tired/lazy/sick (not COVID19).
What is a data lake?
They’re very similar to a data warehouse but data lakes are fluid, flexible, and accommodate disparate data sources.
You may be familiar data warehouses as a place to store an organization’s data for use. We’ve all been in big warehouse stores, like Ikea or Costco, and have seen the rows of shelving for products. Think of a data warehouse in the same way: aisles (rows) with shelving (columns) full of products (data). One of the ways data warehouses have not kept up with changing demands is those shelves (columns) are very structured (inflexible) so they don’t handle new products well. Let’s say one of those warehouse stores wanted to start selling snowmobiles, but they don’t have a shelving unit specifically designed for snowmobiles. They couldn’t store/display the new products without new shelves and rearranging the entire store. Similarly, if you want to store new data in your data warehouse you have to change the column structure to fit.
You may have heard of noSQL or unstructured data; data lakes store an organization’s data in noSQL. Now when you want to store/display that snowmobile, you just dump it in the lake and the water level rises a bit. Post-processing, after upload, you might create a bucket of data (“all snowmobiles”) and you could use the data in that bucket.
A company or organization might create a data lake including website analytics, inventory, order history, human resources, and accounting data. Then they could create a bucket of data for “the last 7 days” to view website visits, production, and sales for the last week. Or, a bucket for “everything in Western Canada” could generate a report of marketing and sales activity in Western Canada. If the company adds a new product or a new sales region, the data storage is flexible and easily adapts to that new data.
What's the value?
A single source to draw all data for dashboards, business intelligence, data-driven apps, etc.
More organizations, every organization, have more data stored in more places. Every web/mobile/desktop app is a new data source. The only way to leverage the value of all this disparate data is to pull it into one place; a data lake. Once the data is accessible it becomes possible to leverage the value for almost anything you can imagine. Updating the data in the Lakebed data lake is quick, easy, and often automatic.
Separating "operations data" and reporting/analysis data improves business continuity and reduces the load on a single database. There are many stories of businesses asking staff to "logout of XYZ app so month-end reporting can run". Instead, the data required for reports can be exported from your business applications and dumped into a Lakebed installation.
My additional thoughts:
A lot of people think in terms of ETL (Extract, Transform, Load). I've heard people starting to talk about ELT, which I much prefer. By loading data into the data lake in it's pure, raw, unaltered form you do "analytics on analytics". I.E. How many errors exist in the raw data. What format was the raw data in originally? What work was needed to get all data into the same format.
What sort of data amount/levels are talking about?
If they're a tech company with a CTO do they maybe want something in-house like a local Mongo setup?
I'm looking for early "proof of concept" of Lakebed so could look at helping if needed.
I hope that helps,
P.S. The other day I built almost this exact dashboard in almost no time:
https://twitter.com/lakebed_io/status/1219639259562418177