Article Source

• How to Build a Metadata Driven Data Pipelines with Delta Live Tables

How to Build a Metadata Driven Data Pipelines with Delta Live Tables

Abstract

In this session, you will learn how you can use metaprogramming to automate the creation and management of Delta Live Tables pipelines at scale. The goal is to make it easy to use DLT for large-scale migrations, and other use cases that require ingesting and managing hundreds or thousands of tables, using generic code components and configuration-driven pipelines that can be dynamically reused across different projects or datasets.

Talk by: Mojgan Mazouchi and Ravi Gawai