How I came up with the idea of GraphMe
by pvcntI work in a big tech company, and being on-call is part of my duties. When in the middle of an incident, quite some time is spent in front of Grafana looking at graphs, analysing their trend and sharing them back in Slack ("look, this metric is improving!").
During an incident, some time may also be spent in the UI of our time series database (not your standard Prometheus) crafting more or less esoteric queries, and again analysing trends and sharing them in Slack.
There are two main aspects that are not optimised in this process:
- The loop of querying -> waiting -> taking a screenshot -> sharing in Slack is repetitive and results in wasted time. When we do it 10+ times per hour, this accounts rapidly to a non-negligible amount of time. It is also difficult to be accountable (e.g., which query exactly was executed to produce this graph).
- Crafting queries against our time series database is known to be error-prone. Available metrics and tags are scarcely documented Idiosyncrasies of how our time series database work are easy to forget for most engineers like me ("why do I need to specify two different aggregators?"). In the best of cases, the query I am looking for is somewhere inside a Google Doc.
After watching several talks about how GitHub engineers use Hubot to solve most of their tooling issues, I was inspired to built my own "graph me" command.
This lead me to build GraphMe, a Slack app that queries various time series databases, and displays graphs in Slack.
While GraphMe allows to execute random queries, it also allows to save some queries under an alias and reuse them later. This effectively creates a source of truth about metrics queries. No more time spent crafting the same query than last week! Saved queries can also be parameterised, which allows executing them in different contexts (e.g., a different environment or different host).
This is how I built a product from a pain point I was encountering during my daily job.
