How do you track exceptions?
Hey, I work in a company with ~150 engineers. We're using Ruby on Rails and track exceptions in Sentry. I like it but we're working on a monolith and there are hundreds of errors that people miss in the noise. What do you all use?
Also using Sentry. I'm just one developer at the moment so it's easy to track. Sentry has a lot of nice features that you can use to configure it and reduce noise.
You can create releases and connect with GitHub or other code hosting services and it would make a guess at which commit caused the issue and can assign the user to that error.
You can configure various notifications with different rules. For example, send direct message to tech lead for the error that appeared for the first time, or send the backend error directly to the backend team.
I also used different projects for different parts of the app: backend and frontend, but you can split this even further even if you have a monolith. For example, throughout payments module use one project id, for your service layer use another. Sentry has various hooks where you can do that.
Interesting idea on splitting monolith to multiple projects. I may actually figure out how to split it per feature/team and then every team has its own project to look after instead of us having a huge project with 1000+ exceptions.
I use Rollbar and also AppSignal. I believe it's not that different to Sentry. At my job I pay attention to practically every exception. If you have recurring exceptions that are expected you should find a way to filter them out (ideally not hitting Sentry at all).
If you are referring to JavaScript errors, then I hear you. There might be quite a noise. You can usually sort by number of occurrences to find the most problematic ones. Still simply going though them, closing them, labeling them, or preventing them to go to Sentry would be the answer.
Do you miss to capture the Exception in Sentry? or Do you miss to attend it?
Exceptions are captured but I noticed engineers tend to miss when their code causes exceptions. Let's say you introduce a bug on 0.1% of requests. When an engineer tests it locally and in production, they are probably going to miss it. If you have a million requests per hour, that means there's a 1000 exceptions per hour that are probably being missed because engineers don't randomly go to Sentry and search around errors (except for a couple of us that find it fun). We set the alerts in Sentry that message us on Slack when an exception is seen more than N times over 30 minutes and then we open GH issues manually for teams.
This seems like more of an accountability issue and not the technological issue.
Few suggestions for this.
Basically, creating an environment or a development culture to be accountable for the problems. You may also want to consider the priority or severity of the Exceptions.
Many a times, such issues has large opportunity cost.
I agree, it is an accountability issue. All the big issues get caught but the smaller ones with lower probability of happening are getting missed. We have a few engineers that check the Slack channel with Sentry alerts every now and then and then we message the responsible teams.
Cloudwatch scans logs and sends emails when an exception happens or whenever the term ERROR is found in a log.