Custom analytics sent from front to backend - how would you approach this?
I only have theories in my mind at this point and I'm curious what's your suggestion, opinion, different point of view about it.
The goal
I would like to build a simple custom analytics into my app, let's say for the sake of simplicity of this example, if user visit website, I want to send it to backend and I can only track - in this example - the user visit on frontend.
Idea
Would it be an overkill to send an xttp request every single time or it wouldn't if I collect data into redis and update the database when a certain batch collected or certain time elapsed?
The goal 2
What if I also want to track how many times the widget was hovered or became visible in users screen?
Idea
It's more tricky. Maybe I should batch together the info to an object or even to local storage and send an xttp request only when this batch has data and in a certain time cycle like 30 or 60 secs.
I believe that most of these tools make a request on every hit, just make sure that the request is asynchronous so that it doesn't block the rendering of the website and doesn't negatively impact performance..
If you look at the dev console when you visit a website, you will seen tons of http requests which don't impact so much a website if well coded.
You can check the source code of Plausible that is an open source analytics tool or test it on your own website to see what happen.
However, I think the most challenging part of this would be scaling on the backend, i.e. gathering and processing all of the data when users want to access their analytics panel to make requests and perform analyses.
By the way, it doesn't need to be real-time analytics (especially not for MVP), I could just build this feature as a daily summery and a cronjob aggregates the data once a day to a different collection which would be called when the used visits his/her dashboard - this way there won't be a performance issue.
You're right, I should just check the dev tools. I found an other tracker in a news site I sometimes follow and it sends calls an endpoint every few seconds when you browse their sites. So there isn't a mysterious way, I just thought it can't be that simple. I was also concerned about the scalability. I should check AWS and Google Cloud Platform which provides better deals for many requests / seconds. If my widget would be on a website that has high traffic and if this widget would be on many websites, soon there would be a lot of traffic to my backend too which would probably raise the price of the cloud usage too. So I should do some math and hypothesis beforehand.
Well, yes, data structure will be an other key part. I assume I should leave MySQL out of the picture too and use a serverless database like MongoDB which I didn't use yet - but it's all part of the fan of building the product out of my mind. :) Thanks for your comment!