Even though NoSQL databases like Amazon’s own DynamoDB are very popular today, for many business use cases, there’s almost no way around using a traditional relational database. Amazon Relational Database Service (RDS), released back in October 2009, is one of Amazon’s first cloud services and can therefore be seen as a very mature service.
The term ‘big data’ is used so commonly in research and marketing terms, you would think it’s been here forever. But actually, big data evolved from database marketing (read first efforts of targeted direct mail) in the mid to late 1980’s. This was the first attempt of marketing agencies and large corporations to begin segmenting their customer databases to create different offers based on the demographics or psychographics of their customers.
At Catchpoint, my role can be summarized at a high level as two halves: designing and taking care of Engineering teams – and working with those teams to design and take care of the various distributed systems that run our platform. I recently attended Sapphire Ventures' Hypergrowth Engineering Summit (thank you David Carter and Sapphire for the invitation!) where the sessions focused on creating and scaling high functioning engineering.
In this guide, we’ll set up an OpenTelemetry Collector that collects logs and sends them to Grafana Loki running in Grafana Cloud. We will consider two examples for sending logs to Loki via OpenTelemetry Collector. The first one shows how to collect container logs with a Fluent Forward receiver. The second one shows how to collect system logs with a Filelog receiver.