Searching Sentry Logs with Regex
We use Sentry's new Regex search for Logs to hunt for non-obvious bugs in our app. What do you do when traces show that Postgres jobs are backing up? Logs are trace-connected, so by looking at one of the affected traces, we can see all of the logs on that trace. In there, we can see Postgres has logged that it acquired a lock only after a long wait. Using regex, we can find all of these logs that describe acquiring a lock after a 10-second-plus wait. That narrows our search from thousands or hundreds of logs down to about a dozen.