Building a flexible, realtime data warehouse at Sentry with Beam + Dataflow (Syd Ryan)
Syd Ryan describes two hard problems they've solved at Sentry with streaming Beam pipelines. The first solution combines Postgres change data capture and SQL views to produce a table that appears to be updating in real time within BigQuery. The second solution is aggregating 1000s of events per second and backfilling historical data effectively with Beam's unified batch/streaming interfaces.