Operations | Monitoring | ITSM | DevOps | Cloud

Ensure data quality in your S3 data lake using Python, AWS Lambda, SNS, and Great Expectations. Data lakes used to have a bad reputation when it comes to data quality. In contrast to data warehouses, data doesn’t need to adhere to any predefined schema before we can load it in. Without proper testing and governance, your data lake can easily turn into a data swamp.

Modern complex systems are easy to develop and deploy but extremely difficult to observe. Their IT Ops data gets very messy. If you have ever worked with modern Ops teams, you will know this. There are multiple issues with data, from collection to processing to storage to getting proper insights at the right time. I will try to group and simplify them as much as possible and suggest possible solutions to do it right.

Whether from a monitoring tool such as Datadog, a collaboration tool such as Slack, an automation tool such as Chef or a ticketing tool such as ServiceNow or JIRA, AIOps seamlessly integrates data from all of your IT sources. A robust AIOps solution with integrations can help your DevOps and SRE teams better know where to begin fix problems, resolving incidents before they affect services and reducing downtime.