Hellmar Becker: Let's Do Data Lineage in Kafka, Flink and Druid by Tracking Aircraft!

This talk focuses on data lineage in the context of streaming data governance using Apache Kafka, Apache Flink, and Apache Druid. The speaker, Helmar Becca, a sales engineer at Imply, explains how to track aircraft data using a Raspberry Pi as a data source. He details how Kafka manages records, allowing the use of headers for metadata that captures lineage. The presentation highlights the use of Flink for streaming ETL processes and Druid for real-time analytics. The speaker emphasizes the importance of data quality, discoverability, and lineage in an enterprise setting, and demonstrates how these can be implemented in a practical way using open-source tools.

[Music] hello again we're back after a short break and I have Helma Becca with me today uh who has over 25 years of experience in uh data analytics and he's going to show us a really cool topic that uh you can actually experiment with at home if you have a raspberry pie or if you were looking for a reason to buy one um so over to you

Helmer yeah thank you l so uh yeah my topic today is let's do data lineage in kafa link and it by tracking aircraft and well we'll come to all the aspects of that in a moment but first a few words about myself my name is helmar I'm a sales engineer at imply and imply is the company behind apach Druid if you will um apach is an open

source project more about which later as well I'm from Germany I live near Munich with my family I write blogs more or less regularly mostly about Apache Druid and if you are interested in my blog the QR code here takes you directly there so um data governance is an enterprise Enterprise subject and U my uh goal today is to make it to bring it a little bit

down to earth and to make it to make you able to experience it at home with things that everybody has at home so we're talking about data governance and specifically what streaming data governance um and then we are going to put together some parts of what we are going to need so I'm going to talk a little bit about Kafka and Kafka headers I'm going to talk

about Flink and streaming ETL and Kafka headers in in stream in Flink I'm going to talk about Apachi Druid because what we want to do in the end is um streaming analytics and for that we are going to use Apache Druid Kafka Flink and Druid they are all um open source tools that you can easily use at home we also need data source or data sources and

uh for that this is where the Rasberry Pi comes into play so these are the components of our endtoend data processing chain in which we want to track data lineage and I'm going to put this all together and hopefully if the demo gods are with me today I'm going to show a live demo about all these these things so data governance um I shamelessly plagarized this from

a Blog that was um on the conference website I think is a Col collection of Standards processes roles and metrics that ensure that data is usable accessible and effective sounds great right um so it's about first of all it's about data quality second it's about data discoverability which means you need to have a data catalog you need to be able to find your data and the third

is um uh is a data lineage which is endtoend tracking of where your data comes from how it has been handled how it has been processed and um all this nowadays in a world where we're talking about streaming data event data not batch anymore we also need to put a streaming aspect to all these things so it's stream quality stream catalog and stream lineage there are great

blogs about the first two aspects stream lineage is also something that tools exist about but like I said I'm going to talk a little bit about lineage and how to track where your data is coming from so um let's do this with respect to really practical data how are you going to track lineage in data well basically what you do is you attach metadata to your data

and because we are doing streaming data we have to attach metadata to every single record um the standard streaming platform that most everybody uses or actually no let's let's put it another way um the standard for streaming apis nowadays is more or less Apachi cafka so if you're talking about data streaming uh you have either Kafka or something Kafka compatible such as red p or um you

have something that is working in a different way but usually um exposes a compatibility layer that might be the case for Apachi Pulsa or for Microsoft Azure event hubs where you have a Kafka layer as well um so Kafka is pretty much the standard in Kafka you have got uh messages that are records and the records have a key and value everybody knows about that it also

has a time stamp which also most people know about a lesson known feature is that um you can add any kind of headers to your um to to your data Kafka headers are key value pairs where the key is a string and the header is not even a string the header is basically just bite array so Kafka does not really put a lot of internal intelligence into

interpreting these things so uh you can read these uh header pairs you can add header pairs to the to each record and if you process data with the chain that we are going to use today then you can also pass them through these um through through these stages so in Kafka you have headers Keys key and value pairs in Flink you would introduce these to your table

schema you add these as metadata columns to your to schema uh it would be a map of bytes to bytes and if you really want to do some processing with on top of these data in Flink you have to decode your bytes so you have to use decode and encode functions you can do that now the nice thing is um that with Druid uh Druid is an

analytical database and I'll come to that in a moment uh Dr can read Kafka record headers natively so if you have all these pieces together then Kafka headers give you a wonderful way of attaching envelope information if you will so data lineage information where has your data gone along uh all across your processing chain so um what we are going to do here and I'm really showing

a very very simple example you can do a lot more um Flink uh you can process your data and I'm going to show this with the Flink SQL API and um in that you put a table on top of a Kafka stream in a way that you have like in this example here and this is actually the example that we are going to use so you um

connect to a Kafka stream specifying the Kafka connector and your format stuff and then you have got your Kafka headers map from byes to byes uh is a metadata file field where we are going to land all those data is in Apache Druid now Apache Druid is a streaming analytics database if you want to see what is happening in whatever system that you're looking at you are

putting analytics to the end uh the classical analytical thing and I have I've done a talk about that at the Big Data Europe last year is um how we used to do things is in batch yeah you have operational databases you have a bch process that then um transforms and loads those data to a um analytical database and then you have put your business ch's client on

top of it so that you can show your graphs and charts and maps and what have you um in uh the streaming world you have a similar challenge but you want to do it all live so you want to have uh subsec queries that give you your results immediately and you want to have results on top of data that is streaming into your database so basically um

you want to have uh an picture you want to have a flat snapshot of what is in your stream at any given time so that you can slice and dice your data you can anal your data Every Which Way and um that is what has been uh realize what has been implemented using Apache Druid apach Druid is a distributed database that is built for streaming analytics and

what you see here is a vastly simplified model of how Works uh it is a distributed architecture shared nothing microservice space if you will and you've got various processes here that have various functions we have got to the right we've got uh so-called Master processes that um deal with data Discovery service Discovery process coordination cluster State all that then you've got query servers that um run query

Brokers so they are client facing and um then you've got data servers that do the heavy loading you've also got um a an archive storage called Deep storage which would be mostly cloud storage or something cloud storage compatible or it could be hdfs something that so what happens when data streams into this system um data is already handled in a parallel way so to the left here

this Arrow this might be your Cupa stream it has n partitions you could have n so-called indexer processes that listens each to one pro one partition of data and collects uh Kafka messages analyzes these messages and transforms the messages into a columnar format which is called the DU segment format the DU segment format is um partitioned by time sorted by time colorized fully indexed and uh the

indexes that are automatically created are um compressed bit map compressed and sorted bitmap indexes so this is a really cool thing because um well first of all bitmap indexes mean means you can have the indexes local to each segment each segment can be packaged with it with its own segment secondly it is super fast for Point lookups thirdly because the index is sorted it's also super fast

for range lookups and fourth it is super fast for combined filters because combining two bitmap indexes is basically just a bitwise Boolean operation so very very optimized these index these segments are built up in memory and once the segment has reached its Target size it is sended over to the Deep storage and then brought back to the historical process um these are also distributed and uh segments

are replicated on the cluster so that there's always more than so the result of this data ingestion operation is that um we've got readymade fully colorized segments that are accessible by the historical processes in local storage here and we've got those segments that are still under construction by the indexes and they are held by the indexes in memory now if a query is run against this system

uh the query broker accepts this query and again like the data servers also the query servers can scale uh the query the query broker analyzes the query creates uh partial queries after looking at at the query it finds out with segments to interrogate it asks the master Services where to find those segments and then sends part partial queries to all those Services retrieves the results and then

assembles the final result applies aggregations applies final filtering ordering operations and so on returns the result so this is the secret how this is all very fast and very flexible and this is how you can query a stream and the interesting thing here is sometimes people would ask me uh what is the difference now between FL and through it because Flink can also do SQL on streams

right now what you want to do Flink is a great tool to transform one stream or a set of streams into another stream through it is the final Landing point and if you put Kafka Flink and Drew it together you can create a great streaming architecture that looks like this so you would have all your data sources that are here on the left and usually if if

you built an Enterprise architecture then you would have all kinds of datab bases and files and streams and what have you that you could collect into Kafka as a central nervous system you would use Flink as the um processing streaming ETL and then you would land your data and do it and that is what you want to do um for any kind of streaming analytics architecture that

you would build be it big be it small and if you do this in a for for a big company then some Enterprise architect will say okay this has to be um auditable and that's why we need data lineage and that is how I came to towards the subject of this talk now what kind of data are we collecting here well um the internet is full of

it it is easy uh to collect aircraft radar data with the Raspberry Pi uh commercial aircraft and in fact most aircraft that are flying around are constantly sending data about their whereabouts their identification and such um on a frequency 1090 megahertz and this data can be freely received by anybody who has a tuner that can listen to to this frequency and um long ago some smart people

have found out that a cheaply available um digital TV stick contain that contains a software defined radio chip can easily um receive these data and there's a free software uh dump 1090 that you can U download install in the Raspberry Pi and um then collect these data and look at at at your aircraft on a map and there are commercial portals as you know probably like you

know flight radar and what have you that um allow you to watch all the data that is collected by many stations um on on a global map and uh look at the aircraft and the identifications and all things now what I've done is uh I've talked to some of my friends and colleagues and I've said um how about we start as a proof of concept we' we

we tried to figure out how to collect data from multiple points and get it all sorted out and that's what I've done here so um I have taken a Raspberry Pi with a TV stick and I've installed the dump 1090 open source software and then I've created a little script that um the um open source software exposes its data as a kind of CSV like format on

Port 30,000 and3 and there I collect these data and I put them into Kafka cat in order or Kat in order to send these data to Kafka so I say please send these data to this topic this bootstrap server which is in this case here well um because I I just I just have this available it is a cloud Kafka service so and what I'm doing here

is I'm adding these headers and that is what you can do very very easily with kafaat so I'm adding a header for the client ID so each of my friends and colleagues that is running such an a spray P I've asked them to put in a unique identifier as the client ID I'm also putting in the client time zone and I'm putting in the um geographical position

of the receiver of the antenna and um so all this data is then collected into one single Kafka topic and this is how I get my data so we've got all the components now right um we've got at the beginning of the chain we've got uh these raspberry pies that are collecting aircraft transport transponder data and I've got some folks that are doing that in the UK

and others in um we are sending these data to Kafka to a Kafka service in the cloud and from there on uh I'm going to continue with a demo that I'm running in Docker on my own laptop and this demo is running a local CF car so that I can transform my data and then put them not back into the cloud but I'm looking at those data

in my local Kafka I've got Flink Flink there and I've got Druid there and I hope I will be able to show how to land those data and Druid and how to track the lineage of those data and uh reproduce it this lineage inside to it so let's go and start and see if the Works um so so so so so here is um my Flink SQL

uh this is well like I said it's it's in a do compost file um the query that I'm running looks pretty much like this but I'm pulling my data out of the cloud kfka stream so I'm not going to copy paste the exact query here because it has um secret credentials the trick here is um that in the first step and my data is actually at this

point pretty raw I said before it is a CSV like format but actually the way this format is defined is such that the number of fields is not constant now Flink has various ways of interpreting incoming data one would be CSV but um it will choke if the number of fields that it expects does not match the number of fields that it finds so I cannot go

this path what I'm doing instead is I'm using a raw value format which means that I have as the payload from the Kafka um message value I've got only one one field that I can view as a string this is the Val field here um then I need to have my um Kafka metadata fields and these are three in this case so I'm going to get get

the Kafka Tim stamp I'm going to get the Kafka headers which are going to which are the ones that are tracking my lineage and I'm also going to to get the Kafka key so I need to define the key Fields prefix here in order to distinguish those fields from the ones that are in the value so it's a bit tricky to get this right um but uh

in the end once you figure it out it's not that bad so I have done this I've prepared this already a little bit like the like a good TV cook so that I can now do something like this here um I do a select star from this table adsb raw so this is supposed to have uh only a value field and I hope it has some data

now it can take a moment there and you see here what is going on we've got the Kafka time stamp which is interpreted here so this is some date and time I've got the Kafka headers and like I said the Kafka headers are interpreted just as bites and this is what is reflected here in this view I've got the kka key which is the aircraft ID and

I've got the value which is my CSV like uh data record that is coming from the dump 1090 software so this is pretty good this is what I wanted now in order to be able to put these data into an analytical engine I need to uh impose some schema on this and um for the purposes of this talk I'm doing a simplified way of doing this uh

I create another table which is going to also be associated with a Kafka topic but this Kafka this is on my local Kafka which is this one here in my Docker image and I need to do a few things here as well I'm passing through the uh the lineage information here with the Kafka headers so uh I'm also passing through my Kafka key and my Kafka timestamp

and Define um all the fields that I have in my message like they are defined and um I need to give a topic name so this table is backed by a Kafka topic so I need to give a topic name and um various other options that are listed down here and this defines my table and then I it's time to put some data in and for that

I take my uh data from my table that I had previously from the unstructured table and now I'm doing some pretty pedestrian work where I collect each field by applying a split index function so I I split my uh um my uh P load string by the comma separator and pick then field number 0 1 2 3 and so on and so on and so on the

nice thing about that is that if a field doesn't exist the split index function will just give me a null value so this is safe so this is a safe uh way of populating the data inside my new table this table is also and did I mention it before um it has a value from of Json so it will have Json data now let's look what this

does oh it does something why is it now not working I think my yeah that happens sometimes yeah there it is so now this um is now inserting my data and I think I should be able to see this job now here exactly I see this job now running in the fling console as well and now I have a table that is in Json format which is

backed by Kafka topic and let's look at how this looks in practice so what I'm going to do select all data from adsb Json and now you can see it now it's it's we have the structure we've got the kfka metadata as before but we've also got the message type transmission type and all the things that are now that don't fit on the screen screen anymore um

and uh this is data interpreted by Flink now let's do um let's let's look at the same data in Kafka and here you can see how the data that were before in the format are now in adjacent format now it's time from here to go to uh and look look at at the how to analyze these data so I go to the Druid console so duid my

analytical database in order to connect to a stream and analyze the data in the Stream there is a easy to use wizard where I say I want to connect to kfka and I need to tell D where to find that Kafka hila typo c car 1992 topic Jason and here you can already see what is happening we've got each record here listed like this this is a

preview the payload is the Json that we saw before but it also picks up all the header fields and here this this is my yug he is on British standard times and this is this is where he's located and um now we'll just go through the wizard the first thing is uh specifying the payload format it has recognized this as Jason so this is all good so

far we've already got the Kafka um metadata fields in here as well so I think the um here is the client ID again next we pick up the time stamp because I told you before that it's is all time based so you would have a time stamp in your data model you can apply Transformations here but we're not going to do that and this is my data

schema so now I'm already good to go um I need to define the segment size here and I want to read from the stream and I give my table a name which is atsp Json and I hit submit and that's all that there is to it and why is this now not working ah the demo get gots are really not with today I think another process crashed

okay one moment yeah now we are good again and it's running actually so here we can see that the process has started despite the little hiup we could have some some data in here as well and we can query those data so let's do something like here's my data and you can see it has picked up all the F the all the fields we've got a um

the receiver latitude the client time zone receiver longitude and here the ID um do a select star it's all there and what I wanted to do now is this one here so now let's see if we can track the data lineage because that was remember that was the whole idea about what we wanted to do we have the data that is that is easy and I've done

this before but now I want this is not called adsp Json now I want to see where my data came from okay so here's the data collected by my receiver this is collected by yuk this is collected by Peter and this is what we wanted to have actually I can also do some little things here like this is also built in chart over time now let's break

this by oh this is the wrong table need to do this one here break this down by client ID and there you go here you can see you can see it and the just counting the number of records and just like this we've been able to track data back to its source without actually altering the data record so this was the whole idea we have added metadata

we've passed these metadata through Flink and of course this is only the beginning of the story and in a limited time talk this is um but uh in a more complicated setup you would then do things in Flink like you would apply your your um uh time windowing you would join other data sources you would put everything on a nice map and all these things but uh

that is another story for another time so for now I would like to conclude with a few final final remarks um data governance you will hear this a lot whenever you talk to people in an Enterprise context if you will I've worked in a bank some years ago and their lineage was big thing lineage is a crucial part of data governance and the way we can track

lineage is by adding metadata to each data record for Kafka which is the most popular streaming data Platform One way to do this is using Kafka record headers and if you build an analytical stack out of Kafka Flink and through it then you can track data lineage end to endend with Kafka headers and with the builtin uh capabilities of Flink and through it it is actually easy

to do so the I think the the the main learning from this is don't be scared of enterpris things it comes down to very very basic Concepts and these basic concepts you can try at home with uh public data sources with publicly available free software in order to teach yourself the basic concepts there's a few I think we will We later um uh share the slides there

are few here are a few sources that I used in order to comp compose this talk um you will find some of the code in my public guub prepo and um with that I say thank you and we've got some room for questions thank you for the lovely session helmar it was uh very exciting to see actual aircraft data being used um for this demo so I

can't help but ask um if this was if this was Apache Pulsar instead of cfco would you still be able to get the same results because we had a session earlier uh about a paticles are being potentially finally a good replacement to Kafka so would it be a good replacement in this scenario um Lena that's a great question um I I have kind of well first of

all it it would definitely work and the reason it works is because pzer has like like I mentioned before uh the there is a compatibility layer which is called Kafka on pulser or cop and um that enables you to uh use du it with Pulsar I believe there's also a pulsar integration with Flink although I I'm not so sure about that I I suppose it has been

done otherwise you could also use the Kafka compatibility layer so that is the answer with within this context um and outside of this context what's your personal preference because you know technology choices are like religious choices everyone is very polarized into using one type of Technology yeah yeah and uh I I I admit that um having worked with Confluence for two years in my career I'm a

bit of a confluent and Kafka fan boy but um I have to say Pulsa is a great product Pulsa brought in some components or like some Concepts that were missing from the original Kafka like you know storage in like separation of computer and storage um having multiple name spaces in one cluster all these things um interestingly the folks at confluent have copied some of these Concepts into

Kafka some of which have gone into open source Kafka and others are only available in their commercial offering but I mean I don't want to make make this a sales talk so um uh like you said Lina it is um a bit of of a religious choice and I think kfka and Pulsa are both great products and they are both fit for this purpose okay and the

solution you you showed today um what would be the challenges of of operating that so if we go outside of the demo and you you implement this uh infrastructure and this combination of tools to get the data lineage what would be the operational challenges well first of all I mean uh obviously you have to operate some clusters right um yeah yeah you you have to operate a

Flink cluster you have to operate a kfka cluster you have to operate a druid cluster uh and you have to put all these together um you would also typically not produce your uh Edge data into kfka which is what I've done here so um an endtoend architecture would often more be some like something collecting data from The Edge from sensors from iot devices what have you um

sending data through a protocol like mqtt having another Gateway that is between the edge world and your data center world and then um yeah and then then like I said you you've got basically you have to manage three distributed systems which is why a lot of people that I talk to nowadays prefer to to use managed cloud services but of course um that is something that goes

beyond trying out things at home because that is usually associated with some cost yeah yeah well again capex versus Opex yeah um whether you want to pay for the manage service or go at it alone and it depends on the size of the organization as well and do you think that um yeah yeah yeah I was just say it it depends on the the will of people

it's like uh you I I I mean i' I've I've talked to a lot of companies where you know sometimes you have those companies that say um I want to be able to scale my organization I want to I I don't want to rely on um on a handful of Specialists that have the knowledge where you have a bus problem like if one one of my Specialists

gets run over by a bus then the the knowledge is missing um and then I've got I've seen other companies that say I don't want to rely on a cloud vendor that may at some point change their terms of service so I built the the knowledge in-house it's it's a philosophic philosophical question isn't it that's uh exactly the kind of U discussion that I've been in recently

um specifically around the cloud because it was like but before we migrated to the cloud everything was so easy because we were autonomous and we knew how to operate it and now we don't know how to operate it and we have to rely on other people but at the same time to me it's like you can't be a specialist in in everything and um you have to

at some point focus on what you need to develop as as an organization and let go of some of the things and maybe have a plan B for uh vendors yeah I don't know absolutely but again it's uh it's personal preference as as much as anything and do you think that organizations are um focusing more on uh data lineage and uh data governments governance these days uh

the big ones absolutely yeah yeah but the smaller ones it's still not that big of an issue I think data governance is predominantly something you hear in larger organizations it is yeah it the the point where I got first confronted with this was when I worked in ING Bank some years ago and their Chief Architect was doing really a lot of work talk evangelization about data governance

metadata management and such things and he was really hammering it in um do data lineage and then I noticed that well like I said in smaller organizations um many people don't even know what what what I'm talking about when I talk about dat lineage so um I also wanted to make this concept a bit more digestible yeah yeah yeah yeah I I get that absolutely because yeah

we we have to figure out a way for smaller companies to understand what bigger customer companies are going through without you know just uh stop with a bureaucracy and unnecessary complicated and in the end it boils down to relatively simple Concepts and if you look at you know the presentation of your CH Chief Enterprise architect which is like a a big slide full of boxes and arrows

where you easily lose track what is the the central message then it might be intimidating for some yeah but I think it's going to become an even more important uh topic uh for iot companies that also have machine learning uh for their iot devices because otherwise it's going to become impossible to track like what information was used to train your model where did everything come from or

yes and that is why I think that these Concepts will become more widespread and that even smaller organizations will have to to apply some data governance Concepts in the future yeah I I think you're right as well well thank you very much uh for uh the very informative session it was very fun to uh look at a a live demo quite as interesting as yours um and

uh we're going to go into a small break and then the last keynote of the day thank you bye