Unlocking Climate-Related Data Through Open Source and Data Mesh Architecture

Abstract: 

The lack of reliable and readily accessible climate-related data has been a hindrance for financial sector stakeholders to properly assess financial stability and manage climate-related risks. This data gap substantially raises the barrier to channeling global capital flows towards climate change mitigation and resilience, by forcing businesses to engage in costly ad-hoc ingestion and curation efforts that cannot benefit from shared data or open protocols.

At the Open Source Climate (OS-Climate) initiative, we are building an open data science platform that supports complex data ingestion, processing and quality management requirements. We take advantage of the latest advances in open source data platform tools, machine learning, and the development of scenario-based predictive analytics by OS-Climate community members.

In this talk, we will present how we are implementing the OS-Climate data commons platform based on a data mesh architecture to make data accessible, available, discoverable, and interoperable across various development streams, while supporting strict compliance requirements around data access and regulatory disclosures. We will explain how the OS-Climate platform leverages open source tools including KubeFlow, Trino, Jupyter, Elyra pipelines and OpenShift to build maintainable and collaborative pipelines, and federate heterogeneous data sources into a controlled, common data resource for our community of climate data scientists and financial sector stakeholders.

Bio: 

Erik Erlandson is a Software Engineer at Red Hat’s AI Center of Excellence, where he explores emerging technologies for Machine Learning and Data Science workloads on Kubernetes, and assists customers with migrating their Data Science workloads onto the cloud. Erik is a committer on the Apache Spark project and contributor to the Ray distributed compute platform.

Open Data Science

 

 

 

Open Data Science
One Broadway
Cambridge, MA 02142
info@odsc.com

Privacy Settings
We use cookies to enhance your experience while using our website. If you are using our Services via a browser you can restrict, block or remove cookies through your web browser settings. We also use content and scripts from third parties that may use tracking technologies. You can selectively provide your consent below to allow such third party embeds. For complete information about the cookies we use, data we collect and how we process them, please check our Privacy Policy
Youtube
Consent to display content from Youtube
Vimeo
Consent to display content from Vimeo
Google Maps
Consent to display content from Google