Logging API calls

We use our cloud provider's logging solution to store and analyze the relevant data. It has always been way easier and cheaper than using a 3rd party solution or trying to set up something open source.

However that isn't all the data relevant to the API call, for instance:

• request body
• response body if error
• status code
• headers

Are all (sanitized) and logged which works great for the cloud logger, but there are also additional important things like:

• user
• frequency of user sessions/login
• resources being accessed
• authorizing action

This is all logged through our authorization solution which already tracks a lot of that.

If you've got a budget for that, look at Datadog. Their integrations with the code and other systems (e.g. Sentry) are smooth. Log search and Traces work like a charm
Otherwise, look what you can get from yourcloud provider: e.g. Cloudwatch from AWS is a bit cumbersome to use, but powerful. Also you can export the data (metrics mostly) from there to Prometheus + Graphana, or just integrate them directly.
And of course, there is Logstash/Elasticsearch/Kibana for logs, which you can also integrate with Cloudwatch or use directly

I take two approaches here:

1. Log out a JSON object that is then captured and sent through LogStash, parsed, and stored in Elasticsearch. This way, I can use Kibana to analyze and search the parsed JSON when I need to troubleshoot anything and can even set up alerts if I wanted.

2. Request metrics get sent to Prometheus for realtime performance analysis. This includes things like response times, status codes, etc. This could technically be done with Kibana as well, but I also have Prometheus scraping cluster and container level metrics (Im on Kubernetes) so I can correlate response times with, for example, CPU/memory usage and have a nice telemetry dashboard in Grafana.