From data stream to prediction: Deploying real-time ML pipelines on Kubernetes

In this talk, Carlos Arnal demonstrates how to build a real-time content classification pipeline using open-source technologies and Kubernetes. He explains the importance of immediate data classification in various domains such as finance for fraud detection, social media for content moderation, and cybersecurity for threat detection. The speaker utilizes a case study involving the classification of Reddit posts from the 'Ask Europe' subreddit, using two machine learning models: a fine-tuned transformer model and a logistic regression classifier. The demonstration proceeds through data ingestion, schema governance, dual inference, and aggregation of results into dashboards for monitoring the model's performance and accuracy. Throughout the presentation, he emphasizes the significance of schema validation to prevent data corruption and the potential for using two models to enhance observability and reliability in predictions.

[music] First of all, I and even before getting into the into the presentation, I want to clarify something and it's that some of you might may be a bit puzzled by the fact that the title on the presentation is a bit different. You are in the right presentation. I hope it's just that uh when I was try building the slides for this presentation, I realized that the

title of the talk was a bit long for a for a slide. So you are in the data stream to prediction how to build the machine learning uh data pipelines and deploying them into production. Just in case you are wondering what happened with the title and I am am I in the right presentation or in the right in the right talk. I hope you are. So with

that who am I and what I'm doing here today? I'm Carlos Arnal and I've been building at Red Hat. For those for those of you of you who don't know I IBM both Redat a few years ago. So I've been at both companies at this point for six year. I'm a principal software engineer at at IBM and I work on a big registry which is an open

fully open source project. We practice the the open principle by upstream first and then we of course build products. We need all need to make money right but then we we we publish our code and we build our our projects. Today I'm going to show you how to build a real-time content classification pipeline using only open source technology. Apicure registry is going to be one of those

tools but there are going to be obviously more more tools being used and everything running entirely on Kubernetes. To be more specific I'm going to dep deploy everything on mini cube but you can deploy all of this in GKE AWS in any any Kubernetes provider of your choice of or preference. there is no no strict need for a for any of them. We'll go from the row

text and I'll I'll get to that in a in a moment to live dashboards with data and uh some graphics and I'll deploy the whole thing uh live and I hope the that the demo gods are with us today because it's going to be a live demo on on a stage today. So we are going to get uh to get this started. What kind of problem are

we are we trying to solve? So this uh this problem is a problem that is basically everywhere today. It's a problem that in finance you need to do fraud detection for instance and on trans on transactions as they happen you can't wait for a uh for an batch job to detect the fraud in a transaction that need to happen immediately in social media. Obviously content moderation is

another example of this that needs to happen immediately and in milliseconds or microsconds. It cannot wait hours or days for that moderation. And obviously trend detection needs to happen in in seconds as well, not just in in hours. And obviously in cyber security as we have here, thread detection and classification has to happen immediately as well, not just in in hours. It has to be instantaneous. The

common pattern across these all use cases is the same. data arrives continuously, flows continuously and then you need to classify that data in real time. You need to treat that data and classify it not in not in a batch. And the challenge we are going to face is in the just running a model. It's building a reliable pipeline around around the model. So this is not going

to be focused we are going to be talking about models a little bit but this is not just around the models. models in this case are just a tech choice rather than the focus of the of the presentation. So the challenge as as I was saying you need data ingestion you need schema contract you need inference you have analytics monitoring each one of those things on its

own it's a a data distributed systems problem a big challenge on its uh on its own and I'm going to try to show you how to wire all them together using only open source technologies in this presentation the studio I'm going I'm going to use It's a a case study I I've been using in purpose for this presentation. I'm going to use a very concrete one and

it's classifying Reddit post from a concrete a very concrete subreddit called Ask Europe. For those of you who don't know, it's a regular place where you can ask questions. There's lots of discussions around a million a million different topics. And I'm going to use uh the flare categories. Think of flares in in subreddits as simply categories. treat them as categories for this uh for the sake of

this presentation. So you're going to have things like travel, politics, food, economics, miscellaneous, media, all those all those different uh different things. Okay. Why Reddit? Why this case study for this presentation? It's messy. It's as simple as that. It's the most complex thing and the most complex text are is human beings writing text is the closest thing to think of Twitter or X these days and the

text that humans are are writing on the are writing on there. The nice thing about this uh this subreddit is that although posts are wildly informal, they mix languages, they have slang and abbreviations and they have lot of noise, they are there are data for those who already that still has the uh the label the category on in the in the post. So we can use that

for training the models and I going to show that in a in a moment. [clears throat] And let me move to the architecture. So this is the full and the entire architecture we are going to discuss. You're going to see here a CSV data and there is an explanation for this at this presentation and up until last week was supposed to be streaming data from the RA

API but that has changed because they changed the API and now you need a subscription another on on a developer subscription on their on their end. So now I had to use the the data I already had for my uh from my presentation for training the models. I'm excuse me but I'm faking the data streaming side of things from the Reddit API and I have the CSB's

data. I am going to have a process that is going to be processing those CSVs. is going to be sending that data process to to Kafka to the let me look at the name to ready the stream stream topic and in parallel spark structured the streaming that you have that part in here is going to pick up those messages in micro batches and run the two machine

learning models that we are going to discuss in a moment in parallel one is a transformer model and the other one is a scikit learn classifier both predictions along with confidence scores going to a second CFKA topic here called Cafka predictions and then we are going to have a Quarkus application that is going to read those predictions from the Cafka topic and it's going to build the

dashboards and offer the and offer the statistics and the dashboards around the these things. The key here is that is that the two machine learning models are going to be running in in parallel. One of the models that we have is that this Robera is a transformer. We are all surrounded by transformers these days in m in machine learning. Right? So this is going to be a

transformer model and then we are going to have a classic old logistic regression model with a TF, ADF and LSA. We're going to go to compare both models and see which one give us better insight on a rich side. We we are going to discuss things like agreement rates, uncertainty zones, confusion matrices and all those all those things for those of you who are into a little

bit into data science all all of this obviously without needing any ground uh ground truth labels. So as you can see here one of the pro the project I said I'm involved with a girl registry is in the is in the middle here and you might be wondering what's the role of this project in know this entire in this entire presentation essentially we are going to have

two schemas for the producer side of things the the the cafka producer that you see here that where it's python it's going to use a particular cafka schema a particular schemas a JSON a JSON schema and it's going to use that schema to format the data that is going to be sending to the the stream topic and on the other hand the the quarus consumer is going

to ensure that the data is consuming also deers to the same schema. What's the point of that schema evolution? If you have a producer that has changed and now has a different field and it's breaking your uh your producer you're preventing your data from getting into the topic and you're preventing your uh your data from being corrupted and from corrupting your dashboards at the end. We are

going to see that as well in the later in the This entire pipeline and I'm going to show you that that later is deployed on mini cube on my on my laptop and you can deploy that anywhere anywhere you please. Getting a bit uh a bit more deep the let's let's talk a little bit about the the machine learning models. I have spoiled a little bit that

we're going to be going to use a a transformer model and a and a bit more classical model. So let me move to the to the next slide. The two machine learning the two models that we are going to use. On the left we have the transformer model. It's a fine tuned dist Roberta model. It's a distilled version of Roberta with about 82 million parameters. It has

this layer 12 attention in the attention heads and [snorts] island dimensions. Lots of numbers for these things, right? If you're not into into this thing, you don't need to to worry too much about this kind of information. It's information for those of you who are into into the machine learning uh side of things. The the important part is that I have fine- tuned this model with le

label data from the ask the subreddit ask Europe. Fine-tuning for those of you who don't know we can discuss at the end. Fighting is essentially you have a general purpose model and then you adapt it to your to your data. We can discuss that a bit later. Now it's a very trendy technique and and technology obviously. So why this uh model in particular not another model? It's

simply because it's good enough for this real time classification problem. The aguracy is not going to be the best in the world, but it's good enough for this uh for the simplicity of the presentation and of the demo. And for this kind of a streaming pipeline, latency is what matters most. And this model is really really small and can run as you're going to see in a

in a in a laptop. For example, in This uh this model understands what a sentence means, not just what words appear in appear appear just appear in it. A post that says I visit my grandmother in Kov and she made the perogi. The transformer understands that's about food and travel and it can with the context to decide whether it falls under one category or the or the

other and which flare fits it best and and on the other hand the back of world models will see Kov and Perogi as independent signals. So the transformer sees the entire story not just the two signals on on both sides. And on the right we have this I learn model. This is a much simpler pipeline. TF IDF vectorization to convert into numerical features then truncate those features

also known and as LSA latent semantic analysis for dimensionality reduction and finally logistic logistic regression as the classifier of the of the test of the text. It captures the term frequency patterns and it cares about which words appear and how often really that's mostly about about it when it comes to this uh to this model and its classification. It's fast, it's lightweight, inference takes microsconds in this

case instead of milliseconds in the case of the transformer model. So low numbers still but microsconds instead of milliseconds. So, and one of the important things about the um about this this kind of model is that it produces well-c calibrated probabilities which means that the confidence scores are more spread across the entire distribution. You're going to get confidences that are might be 0.9 and also confidences that

might be 0.3. You're going to get that on the other hand and you are used on transformers you are going to mostly get confidences around 0.90.9292 it's very confident about the prediction that is doing the transformer model very very confident so both models have been trained based on the same on the same data set about 30,000 me posts from the ask Europe subreddit and those already had

the flares h assigned to those uh to those posts So that's why I used this uh this as the as the case study. The the transformer ended up achieving around 65% accuracy which is not great obviously and the escalating model around 55 again not a spectacular for sure but we have 13 categories with noisy in very informal text and this for if you haven't wor with this

kind of problem it's a very hard problem to to deal with and to and to treat. So a random baseline will give you around 7.7% for for just for the for the sake of it. So some of those categories are inherently ambiguous as like the examples I I post or another one will be what about a post about Italian food if you are not Italian obviously and

you don't live in Italy is that about travel or food? What category is that is that post about right? There are some categories that are inherently ambiguously. So on the other on the end both models output two things. The predicted flare obviously the thing that we care the most and also for the statistics and for uh building our dashboards the confidence score of the model between zero

and one. Now another interesting part of this uh of this thing and this whole this whole idea why running two models why there is there is just one reason the agreement rate between the two models is going to act as a proxy for pred prediction reliability. What that means when both model models agree we can be obviously more confident that the prediction is correct. So we can

normally obviously act autonomously on that prediction. That's that's the end goal of the of this of this thing. On the other hand, when they disagree, that's a flag that the post is ambiguous or unusual as I was saying with the Italian food example. Correct? And we get all these insights without any ground level of the data without doing anything. You have your data set trained but then

you are running on untrade data on unlevel data and you get the same you get those all that information from that data just by by running these two models. This idea is a called a supervision signal is related to assemble methods in in training in machine learning model literature but here we are not assembling ensembling them to improve accuracy in the in the models. Okay, we are

using this just as an observability tool and we are going to get that into that in a in a moment. That's the key insight observability about the data and about the things not really about improving the accuracy of the models themselves in this in this case. And now let me move to the next slide Schema governance. Before we get to the to the live demo, that's the

next part after after this uh after this part here. Let's talk let's talk a little bit about the schema governance because in any data pipeline for those of you of you who've been involved with this kind kind of things. We have very complex technologies. We have models. We have lots of the stuff someone has added the field in a schema in some pipeline and the pipeline has

bro broke entirely just because of that. That's the most common or the most common error, the most common thing of waking someone up at 3:00 a.m. That's the at least in my experience, that's the the thing I faced I face the most. So, what can we do to improve the to improve data contracts between the between the pipeline pipeline components? This is where the project I work

in in the in the open community cames in. Before any data flows through your pipeline, we need to define contracts, schemas in general. Okay, in this case, we register JSON schemas. Those are the two schemas that we are going to use during the during the demo and during the and during the presentation. We register those JSON schemas in auro registry for each CFKA topic. And if you

work with m microservices, I guess at this point everyone worked with microservices at some point in the on their career, right? I mean we've we all been through this at at some point. So you've been with microservices, you know, you know you for sure know this pattern. It's the same idea as an open API spec for a rest API. But for event driven uh systems, the schema

is the contract between the producer and the consumer. And if someone breaks that contract, you want to know that before the data flows and not after. So we have these two schemas. The the first one is the ready stream value is the schema that is going to be used for the output of the of the producer. It has an ID field and the and the content essentially

the the ID of the post and the content of the of the of the post by the by the user. And the second one defines the inference output of the of the two models. We have the uh the ID of this um of this inference instance so to speak. And then we have the transformation the the transformer flare the the flare in inference by the by the

transformer the confidence by the transformer the flare by the scalar model and the flare by the the confidence excuse me by the escal escal So what does this do? This prevents schema drift. If someone change the producer the producer here here to add the new field remove a field or change a data type the downstream consumer won't silently break the consumer won't consume that data it won't

be able to consume that data will continue to consume your data the data that ad to the to the schema that it's supposed to consume but it won't consume that that data you have a contract now between your producer and your consumer so the schema validation will fail and in our pipeline that's what that's what happens something uh changes that's it's going to it's going to to

break. Other things that you have to prevent even the addition of a malicious field or something from happening in your in your schema. A big registry also supports what we call rules. You have backward and forward and full compatibility rules for your uh for your schemas. Meaning that if you there because um the JSON schema is supported, protobuff is supported, a supported multiple schema types are supported.

All those comes with rules when it comes to add or remove fields or what you can do or you can what you cannot do with with them. So we have implemented rules the the rules in the in the server for that. So if you try to remove a field from a from an abro schema for instance, it's going that change is going to be rejected basically if

you have that rule enabled, right? you're not going to even get to the producer in if you have those those rules enabled in your in your application. And as I was saying, we support multiple schema types. I don't want to get now into the details. We support multiple multiple of them. Now, let me go to the to the Apigur registry UI. I'm going to get out of

the presentation here a little bit. This is how the how the UI looks like. We support what's known as groups. It's just to grab your to group your your information. And here we have the the schemas I was I was describing. Those are the schemas that we have in the in the presentation. The CFKA predictions and the and the Reddit uh and the Reddit stream. When you

go to this uh to this one here, you're going to see just in a moment you're going to see here that this has a version and the version one obviously because I just registered one uh one version for this schema. you're going to see the content of this uh and here you are going to see the JSON schema that I was uh that I was describing. Obviously

in the presentation it's way simplified but you it's exactly the same thing. You have the transformer flare, the confidence, the escalator, the flare for the logistic regression model and also the confidence. Same for the uh same for the producers I do things. If I go here, you're going to see exactly the same thing. The type, the content and the and the ID and here the type of

the of the field and everything and the the required fields. This ad to JSON schema and follow the JSON schema validation validation rules. I just wanted to show a little bit the the user interface of the of the server. So that's it. Now let's get into the most interesting part of the presentation, the live demo. I have everything pre-eployed except for a couple of things obviously and

I and in a moment you're going to understand understand why everything can be deployed in mini cube and in a laptop but it takes time. So we have a limited time in this for this presentation. But first here what you see here in this presentation and you may be wondering what's happening here. It's a live terminal embedded in the in the browser and inside the presentation. So

it's a it's working. This is connected to my to my laptop. This terminal is connected to my laptop. And what you can see here is what I already have running. So what you see here I have the Reddit realtime uh name space and the spark operator name space. As I was discussing we have a spark here up and running. What do we have in the ready real

time name space is the Apiguro registry server. This one here, this guy here. Then we have the Aiguro registry user interface. They are separate separate bots and separate applications. Then we have the Quarkus consumer, the Quarkus application, the one that is getting the the inference results from uh from the CFKA topic and the publishing the dashboards and the statistics. And then let me start with this one

actually. This is the streamc cluster cluster operator. This is streamc. A streamc essentially for those of you who don't know it's an operator that allows you to deploy cafka on kubernetes. It makes it way way easier as simple as just defining a yl and deploying in your in your kubernetes cluster. So and what you see what you see here is the actual broker for kafka. In this

in this case, this is Kafka 4. So no zookeeper, no things that are not really needed for this uh for this case. This is it for this side of things. And on the right side of things, you obviously see obviously see the the operator and the controller. You see the driver and here you see the two executors executors that I have I have for for Spark. Okay,

that's uh mostly about what we have uh what we have uh deployed here in the in the presentation. The only missing piece as I was as I was saying is the is the producer. So let's let's get to the to the next slide. And let's let's deploy the the producer. I have created the producer and now In the second terminal, let me tell the logs of the

producer. This has started and it has started producing as I was saying at the beginning. Now it's a unfortunately it's a CSV with the with the data. It was supposed to be taking data from the from the API. It is what it is. So what you see here is messages from the from this part producer. It's messy because I was debugging some things last week for for

obvious reasons. But you can see here that it's sending information um to the ready stream uh to the ready stream topic and it says the partition that is using and also the offset that is using for uh for sending th those those messages and this is going to be continuously sending data over the presentation. So there there is no point in on sticking on sticking here. And

now let me get back because this move this uh part here if I do this here this is going to connect to the quarus application because if a spark has processed and I hope so because I'm seeing that in on the on the other side if a spark has processed all the information it's going to load the statistics that the quarkus application is building based on all

the data that has been flowing through a spark to to quar from and cfka and quarkus I hope it's big enough to be uh to be seen you don't need to get into the details of uh all the all the different things but you're going to see a few things here. Here you see the basically a JSON that they that they used as a as a summary.

And uh we can see here per flare counts in the in this part here. You can see at the beginning of the of this uh of JSON object, you can see the the number of uh of posts that have been classified under a specific uh specific flare. You can see also the average confidence for that uh for that category. You can see the the average confidence for

the transformer for the logistic regression model, the agreement rate between the two different models for that for that category and also the um the confidence gap between the two between the two models. Okay. And this is very interesting but this is a JSON. We are going to get to dashboards in a moment where we can see we can see the things. Now we have everything you have

seen the data flowing from one side to the other side. The entire pipeline is working end to end CSV I'm sorry. So the producers cleaning uh that is click that is cleaning the information is validating the data in the using the schemas as I have described in a big registry is publishing the data picked up by by a spark running both machine learning models inference on both

models and publishing the predictions to the second cafka topic that I was discussing earlier and then that gets consum consumed by the quarkus application and it gets aggregated as you can see this into real time statistics. Those are stat statistics. You run that as you can see the the producer is still running. So if you run that command again the statistics are going to change obviously. Okay.

another important part not just for this presentation but for any presentation in How does this work? It's an important part about any presentation or any demo The data is still flowing in the in the background as I was saying and every post that you see in those logs in the producer logs is going through the full pipeline right now. It's going from the CSV. I'm faking the

real time thing in the in the CSV is processing uh processing the the post but it's flowing through the entire pipeline. Let me explain what's happening at every one of those uh stages. If you want to see the code, you're going to get that opportunity at the end because it's published on on GitHub. So if you're wondering about that, don't don't worry. You're going to get the

code and uh you're going to get you're going to get everything even the channel for the deployment into to Kubernetes. So this part uh this part here what's happening what's happening here what's the first step collect and clean the producer is reading the pre-colcted ready pro post as I was saying from the CSB files and each one of them 18 each one of the post is going

through a clean uh cleaning pipeline where is stripping out things that don't matter for this kind of problem of classification problem like URL removing a stop words remove puction you know what that is when you are dealing with with text and when you are reading that kind of information, right? You are concatenating the title, body, comments, any content that gets into that uh post, you are processing

that uh that that excuse me, you are processing that information. So you are able to do inference into that uh using that uh that content. Okay. So that gives the model a clean and unique input, not multiple scattered inputs. We are validating and streaming all the information and before any message goes to Kafka is validated against the JSON schema that is stored in Apicure registry using the

Epicur registry serial serializer and only valid messages make it to the ready stream topic at this point if I did everything correctly and I hope so all the messages are valid right now the all the messages that are flowing are are val are supposed to be to be valid. stage three as I as we were discussing dual inference spark structure the streaming is picking up the messages

from the cafka from Kafka topic in microbatches and for each post is running the two models in parallel in parallel inside pi spark and using ud udf the transformer is tokenizing the text is running it runs a forward pass through the the entire neural network and applies a this max to get the the distinct probabilities for for the categories is then the on the other hand the

logistic regression model is running the TF technique is applying LSA for dimensional uh dimensionality reduction and it's classifying the uh the post using logistic regression and both outputs the predicted the predicted category or flare and obviously the confidence score because we need that to the to the for the statistics is going into the CFKA predictions topic And last but not least because it's I think it's one

of the most important part of this presentation. We need to consume the information and analyze the information. This is uh I think this this last part is where human beings are going to be way more important in the in the future than they've been in the even more than they've been in the in the past. The Quarkus application is consuming the uh the information from the Kafka

predictions uh topic using a framework from the from the Quarkus team. It's validating again each message against the prediction schema from applic registry. It's computing as you as you saw in the in the terminal. is computing real-time statistics from the from that and you're going to see statistics and things like agreement rate and we are going to calculate the confusion metrics for the information distributions and certations

and all of this and it's going you're going to get more information about that is happening in memory and that's a ket about the presentation and the corus application is serving all of that through rest endpoints in this in this presentation and in this application some of the >> some of the design decisions I had to to make for this uh for this presentation are uh I

think are are worth discussing. The the first one and the most important one is I as I was saying is doing inference inside the spark. Normally in in a production environment you in 2026 you wouldn't do h inference inside the inside the spark to be to be fully honest but I'm running the models using udfs rather than calling calling an external model server like tensorflow serving or

tree or any any other tool basically because for this presentation it's avoiding me having to deploy yet another thing in into the into the presentation it's also avoiding uh for a system like is it's also avoiding obviously network latency for every prediction and it keeps the pipeline simple. There is no separate service to deploy. The trade-offs is that the obviously the inference is coupled to the streaming

framework to to spark. Excuse. And the biggest tradeoff in this case is obvious if you need to update a model because you retrained your model, you need to rebuild the image that you're that you're running in a spark and you need to redeploy that uh that image for uh for obvious reasons. But I think for this uh for this pipeline simplicity wins and I think the pipeline

is simple enough and you can uh you can use it. Another important thing is that the inmemory analytics for a presentation like this are good enough and I'm serving them through a rest endpoint. So you can still uh glue something in in there. But obviously in a in a production environment you will persist that to a time series database like influence DB, time scale DB um any

uh any kind of technology of your of your choice of or or preference. Okay. I wouldn't recommend having your uh your statistics and your information in memory in in production. Just uh just saying. So first third I I had to register the uh we register the schemas in APU before any data flows. That means that on on one hand every can evolve independently the producer the consumer

all can everything can evolve independently but at the same time there you have network latency and it's one of the reasons what why I didn't wanted to add even more network latency. Okay. So you have the network latency on this uh on this side but honestly in practice having uh and being being able to evolve your different uh components independently I think it's really really huge because

the the team that is developing this producer is is very simple and also the consumer is quite simple but in a in a in a in an enterprise scale the the producer and the consumer wouldn't be that that simple and those teams can work independently. ly with even the coordinating between themselves about what one is deploying in in the producer side of things and in the consumer

side of things. So that's an important an important thing. And the fourth and last in this in this part the lacy model loading both uh models are being loaded in uh [clears throat] on first on first used on first use. You are not seeing that because it's happening really really fast but they are being loaded in on first used. It's critical and it's a technical a bit

of a technical detail and a bit of a techn an implementation detail but if you need to serialize a network into a pyel that does not work very very well in in a spark just so it's a bit of an implementation detail but I think it's also an important an and now another important part that we have here are the the dashboards essentially we have four dashboards

The first one is going to be the dashboard. It's basically a counter. This the first d the this first one is a counter. I have the culture and you are seeing that the the dashboard is evolving uh live on on a stage. You're going to see that the the number of uh of post for every for every category. You're also going to see the average confidence for

every category. This is averaging the confidence for both models, not just one of the not just one of them. Here you can see a model comparison between the the inconfidence between the transformer and the and the scalar model. And here you can see what I was discussing about confidence when it comes to the transformer model and to the to the other to the other one. The transformer

is always very confident about the things that it's that it's predicting. It's always very very very confident. always. And then in this part here, we also have the agreement rate between the two the two different the two different models. And this is where where this thing starts getting interesting because for the categories where you have a an agreement of almost one between the two models, fine, great.

I'm I'm doing great in this category. For those like personal or history where your agreement is not that great, those are the categories where you really need to get uh attention in your in your data and in your and when you're returning your model. And the very interesting and in important uh and important part about this last two uh dashboards is that you're getting this. I was

going to say for free. Obviously not for free because you're running two models. But you're getting these insights because you are running the two models in production with the transformer model. Fine. It's confident. Great. I have a I have my pred my prediction. If you have the other the other model, I have some predictions, but I don't I don't know if I can trust those predictions. Really?

But with this two dashboards and with this information, you can start working with that and start u doing more more things. So I think that those those two are the are the most uh the most important part. And now getting back into the into the presentation I'm going to get into the confusion matrix uh dashboard. This one again is a very important one in the in the

diagonal. You have a you have the agreement between the between the two models. Fine. For those post that landed here, it's a it's kind of the same thing. I'm I'm happy with that. Then of the diagonal uh cells then you have systematic differences. For instance you might see that the transformer says that one postage falls under the travel category and on the in the scalar model might

say that this culture or something like that. It tells you just not how often they disagree but it tells you how how they disagree. How do both models disagree? And that's also critical for understanding how they are failing when they are trying to predict something. If you are building an active learning system o of this, those of diagonal cells are the ones that are telling you exactly

which category pairs you need to to pay your attention and where you need to to work on your on your training on your training data and on your labeling I'm going to skip this one because it's going we are running a bit out of time. This one I'm going to do it very very quickly. It's basically telling you the confidence distributions for both models. As I was

saying during the whole presentation the transformer is very confident. So you have most of the post from the transformer fall here. And obviously the escalate model is distributed more evenly through the uh through the chain. And then I'm going to get into this part here because I think if you leave if you leave this presentation with something I want to leave I want you to leave this

presentation with with this insight about this presentation. I think this is the most important takeaway from the from these dashboards. The this chart divides the uh the confidence agreement within uh between the two models into three separate zones. Green great both models are are agreeing and uh we are we are happy. orange both don't know what they are really trying and this disagreements cases here in the

in red are exactly the post you need to send for manual review or use as I was saying for active learning to improve your uh your models and to improve your data. Those are the post you really need and you really want to get attention. And in case you were wondering in my schemas why I was sending the ids of the post in the whole flow that's

exactly why because you need those ids to get those to back to those posts and then get that data and see what's going on with that with that data. If you get to this to this point great they are disagreeing and I have this this thing here. Great. And now what? I there's nothing I can do with this with this data here. Right. And now getting back

to the to the just a little bit of I've been doing this a little bit also during the during the presentation. There are a few things to be taken into consideration for production. You need to apply classical uh classical uh design patterns like uh dead letter patterns for your uh for your CFKA for your CFKA topics obviously back off jitter sequel breaker patterns obviously you need to

apply apply all those patterns when it comes to observability you need to apply things like instead of having an HTML like me in my presentation please have something like Prometheus Graphana for your for your metrics Jagger for tracing your um for tracing your request and your and your kaka messages. Please don't be like me and apply your patterns properly in This is the last part of the

of the presentation and you're going to get the code and this is very important because this is just a pattern. The pipeline you you've seen here is just a pattern. You can apply the same pattern really to any row data that you have IoT, Twitter or or expost any row data that you're dealing with and you're trading you can apply exactly the same pattern and exactly the

same pattern to your uh and we are running a bit out of time. The conclusions are the ones I was I was discussing here a little bit and I want to give you the opportunity to ask any questions in case you have in case you have them please I think we have a microphone we have a couple minutes just and I want to I don't want to

take the uh please >> thank you for the detailed presentation I'm working on something similar running workflows or the kubernetes just a tld tlddr of what I understood >> you are taking the stream kind of stream like data but now you have CV so you are kind of doing data preparation and data cleaning preparing the data set then you are passing that data set into producer >>

Mhm. which in turn is passing that into Apache stream that's a like message Q for inference on which the dual inference is happening first that data is textual so I'll come back to that and then the uh inference data is taken by the consumer and uh it is like partitioned into different categories like you mentioned so Kubernetes on the like if we see it is for the

scaling of both producer and consumer as the data uh from the stream or the CV set grows is my understanding correct Yes, for sure. >> Okay. So if we have say for the real world examples actual streaming data suppose like vision vision LLM something like you are monitoring a conveyor belt or you are monitoring something food food grade production level >> how do you think those dual

inference models we have selected you mentioned it's a choice but do you think they will be able to properly classify the data then if the sizes are increased >> at uh at some point there are multiple ways of scaling your your infrastructure for sure Kubernetes just is just one of them. But nowadays uh frameworks like uh I think I I need to remember uh one of them

is the name Ray is one of the frameworks multiple frameworks for for scaling the infrastructure also for inference and on Kubernetes are evolving and appearing continuously. Ray for instance is is one of them. It has a ray serve for serving models in in >> You mean ray by any scale? >> Yes. >> Yes for sure. Yes. >> Thank you so much. >> Absolutely. Definitely more questions. >>

Sure. >> Any more questions? >> No. Thank you very much. >> [music]