Building Adaptive ETL Pipelines with Apache NiFi, LLMs, and Apache Iceberg - Kamesh Sampath

This talk addresses the common challenge organizations face with data integration, including numerous data sources, inconsistent schemas, and the manual effort required to prepare data for analytics. The speaker presents an innovative approach using open-source tools and artificial intelligence to streamline this process. Participants will learn to build intelligent data pipelines that transform data uploads into unified, analytics-ready tables by utilizing Apache NiFi for orchestration, LLM-powered schema inference for automated mapping and validation, and Apache Iceberg's schema evolution to accommodate changing data structures while ensuring backward compatibility. The session emphasizes practical implementation and real-world architecture patterns suitable for large-scale enterprise environments.