Two Methods to Automate Data Observability at a Larger Scale: Agents and Collectors


This session is designed for data practitioners who wish to maintain control and confidence over their projects even after deployment in production.
We will explore two methods from the O'Reilly book ""Fundamentals of Data Observability"" that can be easily adopted to ensure the reliability of data pipelines throughout the whole process, from ingestion to analytics. The main outcome will be hints and tricks on automating the best practices and standardizing them across teams, people, and even organizations.

Session Outline:

In this talk, we will cover the following learning objectives:
- Identifying the characteristics that make a pipeline or any data application """"data observable""""
- Understanding how data practitioners can activate these characteristics with agents and collectors
- Recognizing the benefits for both data practitioners and final users.

Background Knowledge:

Python, Scala, R, Engineering best practices, data engineering/science/analytics experience


Andy Petrella is the CPO and founder of Kensu, a data observability solution that helps data teams trust what they deliver and create more value from data.

Andy is an entrepreneur with a background in data mining, data engineering, and data science. He is known as an early evangelist of Apache Spark and the Spark Notebook creator in the data community.

Since 2015, Andy has been an O'Reilly instructor and author, including the first O’Reilly book about Data Observability: “Fundamentals of Data Observability"

Open Data Science




Open Data Science
One Broadway
Cambridge, MA 02142

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Consent to display content from - Youtube
Consent to display content from - Vimeo
Google Maps
Consent to display content from - Google