Zac Propersi, Engineering Manager at Sentry, can tell when a page is not loading as fast as it should — just by looking at it. While working on our new Metric Alerts feature, Zac noticed that the alerts pages were rendering slowly. Being the super Sentry user that he is, he wrote a custom query in Discover to see just how slow the transactions were.

Large-scale cloud applications are usually built using interconnected services that can be rather hard to troubleshoot. When a service is scaled, simple logging doesn’t cut it anymore and a more in-depth view into system’s flow is required. That’s where distributed tracing comes into play; it allows developers and SREs to get a detailed view of a request as it travels through the system of services.

Tracing is a widely adopted solution to provide performance insights into distributed applications. It is a valuable resource for developers to view the service call graph and track service latency at a granular level. It’s also a handy tool for on-call engineers to drill down and debug a problematic service during an outage. There are a number of open source distributed tracing frameworks out in the wild, including Jaeger, Zipkin, and OpenTelemetry.

Humans are naturally visual creatures. Several of us are visual learners, meaning, we learn by seeing things in action. Tracing is seeing things in action. Troubleshooting where and why something is slow or flat out broken, with clear visual indication, is incredibly powerful.

With the advent of microservices, technologies like Docker, Kubernetes and services like Cloud Computing, have showcased the broader need for observability. Collecting valuable information about the communication endpoints and how they propagate through the discrete components of the application stack is the key to understanding when, why and what happens in case of failure.

I’d like to share some of the best practices we’ve learned on our journey to battle performance issues with the Jaeger tracing tool. Some may say we are experts in logging. We log for a living, and have our log analytics service (which we based on open source ELK Stack) to prove it. We’ve mastered logging to the level where debugging and troubleshooting our system is a no-brainer.