First 1000 sites tested
by ChrisI was not prepared for this. After one single tweet from Guillermo Rauch I could run tests for over 1000 sites.
The good: He tweeted it at 1 AM in the night.
The bad: My DB was on the free plan. I woke up and saw that my DB had no storage left.
After a few adjustments and bug fixes, it now runs smoothly without any issues.
The best: My AWS forecast says ~ $0.80
-
1
Hi, Chris!
By exploring platform I accidentally found this projects and desided to share my feedback. The reason why it resonates with me is because I was also thinking about the similar idea more than year ago. More specifically, I wanted to meassure regression of performance/pwa/accessibility for my projects in the particular PR in the Github. I even created small PoC that was running docker with headless Chrome and Lighthouse and outputs the JSON with results. Initialy I used the Lighthouse bot that talked to hosted docker with headless Chrome but that is currently deprecated. Now it's possible to achive the same goal by using the Github action.
When I was thinking about it I was planing to focus only on API part more than a metrics dashboard that stores the historical data, as I've seen it more relevant to not pass the regressed functionality into master rather than continiously monitor performance. IMHO in case of properly implemented web app you going to monitor your hosting provider capacity or in some cases indirectly your API performance, but I could be wrong. I was thinking the opposite - to concentrate on provide some free amount of builds per month and paid for what is extra. But again, with the GitHub actions my goal was achieved, so I was abandoned my idea.
I don't know if you know but there are a strong player in this field https://treo.sh/ This service is pretty similar to what you are doing and his creator also create and main the Github action that allows to run the report on every PR.
But here comes an interesting and tricky part - in order to test performane regression for your web app in Github actions you have to provide an URL, meaning deploy the web app into some environement that is the same as production by performance characteristics. With other types of regression such as accessibility or pwa complience it's easy as is not such crutial and don't have to be performant optimized.
I have a few questions about your vision of the product:
- Are you going to store the generated reports as a historical data?
- Are you going to add some continious monitoring ability to run hourly/daily and therefore triggers some actions if treshold is not met (webhooks, emails, Slack)?
- What would be a competative advantage compare to https://treo.sh/?
I'm also interested in this problem so if you are lookgin for a co-founders or want to discuss any questions, or need help with beta testing of the features please feel free to contact me in any of the social media that is listed here https://pavlenko.nl
-
2
Hey Ihor,
indeed, treo.sh is a big player in this field. I built Lighthouse Metrics in the first place to get quick performance insights (globally) for your site. Also important for me: this should be free for everyone.
So the current "feature set" will be free for ever. I'm planning to offer more services and features. Like you said, historical data and continuous monitoring are on the list.
I'm still in the process to find a good set of features I want to offer in the next couple of months which are really useful and are different from the other competitors.
For now I will focus on the current MVP. I will add more details and insights from those generated reports.
If you have further questions, just drop me a DM on twitter I'm always happy to answer your questions!