Hellmar Becker: Let's Do Data Lineage in Kafka, Flink and Druid by Tracking Aircraft!

🌟 Session Overview 🌟

Session Name: Let's Do Data Lineage in Kafka, Flink and Druid by Tracking Aircraft!
Speaker: Hellmar Becke
Session Description: Data lineage means you can track the data bits in your system and know at any time where they come from and how they have been processed. Enterprise systems need to be able to prove lineage for compliance reasons, but in general, lineage is also a significant aspect of data discoverability and governance.

In this talk, I am going to connect several Raspberry Pi devices that collect ADS-B (aircraft radar) data to a KFD (Kafka-Flink-Druid) stack for analytical processing. I will deliver the data through Kafka, cleanse and enrich it with Flink, and run analytical queries on the results using Druid.

I will demonstrate how to track data lineage through Kafka metadata and show how this information can be maintained throughout the processing pipeline. This relies on using Kafka headers, an underused feature of Kafka that also integrates readily and easily with Druid!

You will learn how to implement data lineage using the open-source KFD stack and readily available data sources. This way, you can try out enterprise-style data lineage processing at home and prepare yourself for a question that will inevitably arise in any enterprise data engineering project!

🚀 About DevDays Europe || DevOps Pro || CyberWise 2024 🚀

Unlock the latest advancements in software development, DevOps practices, and cybersecurity with the DevDays Europe || DevOps Pro || CyberWise 2024 Online Conference Talks! 🚀 Join us virtually to gain insights from top industry experts and leaders who share their knowledge on emerging technologies, best practices, and innovative solutions in the tech world. Whether you're a developer, IT professional, or security expert, these talks are designed to inspire and equip you with the tools you need to excel in your field. Don't miss out on this opportunity to expand your expertise from the comfort of your home! 🌐

📅 Yearly Conferences: Curious about the evolution of tech? Check out our archive of past DevDays Europe and DevOps Pro sessions. Watch the strategies and technologies evolve in our videos! 🚀
🔗 Find Other Years' Videos:
2023 DevDays Europe || DevOps Pro Onsite https://www.youtube.com/playlist?list=PLqYhGsQ9iSEpBBmsfPQ3J6izgfl6adGcz
2023 DevDays Europe || DevOps Pro Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSEqG4MhmIbZfDN-2Iq2knemb
2022 DevOps Pro Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSEqA1pIaVJgYre4fsqozch1u
2022 DevDays Europe Online https://www.youtube.com/playlist?list=PLqYhGsQ9iSErhN1NL8e629N0zMbSqla39

💡 Stay Connected & Updated 💡

Don’t miss out on any updates or upcoming event information from DevDays || DevOps Pro || CyberWise. Follow us on our social media channels and visit our website to stay in the loop!

🌐 Website: https://devdays.lt/, https://devopspro.lt/, https://cyberwisecon.eu/
👤 Facebook: https://www.facebook.com/DevDaysEurope/, https://www.facebook.com/DevOpsProEurope/, https://www.facebook.com/profile.php?id=61550635566454
🐦 Twitter: @DevDaysEurope, @DevOpsEU, @CyberWiseConf
🔗 LinkedIn: https://www.linkedin.com/company/75524033/admin/feed/posts/, https://www.linkedin.com/company/30633688/admin/feed/posts/, https://www.linkedin.com/feed/?trk=guest_homepage-basic_nav-header-signin
🎥 YouTube: http://www.youtube.com/@DATAMINERLT

💌 Your Feedback is Valuable 💌

Enjoyed this session? We’d love to hear your thoughts! Please leave a comment below to share your takeaways, questions, or feedback for the speakers and the team.

For more information and updates about the DevDays Europe || DevOps Pro || CyberWise conference, make sure to follow our social media channels and visit our website. See you at the next conference!