Building Adaptive ETL Pipelines with Apache NiFi, LLMs, and Apache Iceberg - Kamesh Sampath

This talk focuses on modern data engineering practices, particularly the integration of AI with ETL processes using Snowflake and Apache NiFi. The speaker illustrates how traditional manual ETL processes are being transformed through the use of an 'intent engine', which leverages large language models (LLMs) for schema mapping and semantic analysis. A detailed demo showcases handling varying data schemas from a music event management company, emphasizing the importance of a unified schema for analytics. The speaker also discusses Snowflake's Cortex platform and its capabilities for AI model management, data processing, and security measures, ensuring data governance and integrity. By simplifying the infrastructure required for AI-driven data pipelines, organizations can enhance their operational efficiency and adaptability.

Do you see the Bitly link right there? I shared a QR code where you can get all the resources of the demo that I'm going to show today. Um that was an intro given to me uh about and this is myself. I have my social handles on the QR. Um if you want to get me connected on LinkedIn, GitHub, Twitter, so whatever you are, anything around data,

AI, Snowflake, Red Hat was my old one. Anything you want to hear about it, please do ping me there. Happy to answer those questions around all these technologies that's involved. Um and if you want to uh grab the demo sources and other stuff that you're going to talk today, this demo is completely reproducible. I have all the links there. Um this Bitly link takes you a living

document. Um I keep made it in this document with all my new blogs, new videos, and other stuff around this area. Uh and then you can just also my socials are there so that you can connect me there as well. All right? Um so before we go uh deeper into what we're going to talk today, so how many of you are familiar with this kind of a

architecture on data and ETL? Like Um so let's go from the legacy side of things. I want to talk say so anytime you get like okay, I have let's say the fictitious case we are going to talk about for a demo is a music event management company. So that's what we're going to talk. This company has plays all over the globe. One of the things they were

been doing right now is that they were on the whole left side, I mean to your right side of the screen, my left, uh I think, doing all manual ETLs. Okay, what I mean by that is like let me quickly get to my editor and tell you what exactly I mean here. So is the font enough? Bigger enough? Back side? I want to blow it up. Okay?

So uh so what See, these are the three uh data feeds, the example that we're going to see today. So let's assume that the very first thing that I got from a one of my event management companies about the event data, the events and other stuff going to happen, artists and other stuff is this one. What do you see on the left side of my screen, right?

It has schema title like show ID, performer, location, date, time, ticket cost, and capacity, right? This is the first feed basically gets into your data pipeline. The data is inserted into a schema and everything is up and running. Imagine now what's happening is that this particular person is giving me a new company at joined a new application has been inducted and they send me something like this,

okay? Event ID, artist name, stage, start time, ticket price, genre, and sponsor, okay? And also like something like this. So, which means that there is a change in schema every time the data flows in. There are chances that, okay? So, but usually what we have been doing earlier, so with this in mind, coming back to this slide, what we have been doing earlier is that we have

been doing manual ETLs. Whenever there's a change in schema, I have to sit down, go back to my logic, change my logic, fix all the schema inserts and everything, find everything, fix the AM, DDL, and everything, and finally start the data flowing, which is like months of work, right? Weeks of work to be very precise. Now, what we are doing right now doing is like what I

call as intent, right? The intent engine is nothing but your LLM, right now. We're going to get LLM inside action here. For example, let's I get a data source one, which is the CSV file you saw. It could be any format, keeping it CSV for a simple example. And then that flows into the system and then everything gets inserted, which could be your baseline schema. Perfect. Next,

what happens? There's another one which comes, which is called as DS Evolved, which means that I have another new field added to it or a field name change, everything. It also gets through this thing again because there is an header resolution which happens where I can do semantic mapping, okay? I'll explain that in a second. What do you mean by that? My goal is that any given

schema for a particular use case, or whatever thing in this case like we are doing an event management, everything should land in one single unified schema so that the data can be used for further analytics. That's the goal of mine. All So, coming back to the stuff what I'm talking here, we go here now. If I can compare these things, I show ID, performer, location, date and

time. Don't worry about the data. Just saw the header here. And let's take this header for a very simple example. Let's not worry about an additional column now. If you see these two things, show ID is semantically equivalent to event ID. Do you agree? Right? Similarly, performer is name and then location could be main stage, okay? And daytime could be start time. Ticket cost and VIP cost

are a little different, but for this simple example, I kept them to be similar. Anything that is dollars or rupees or whatever you call them as everything is going to be same. So, which means that this CSV data load and this CSV data load can fast through the same pipeline without any problem, right? But what you have been doing so far, B pre before AI is that

we have been writing programs which can adapt these two things thinking these two are same things different, right? But they are semantically same. All right? The other interesting part what also happens is that this pipeline with AI and few other tools I'm going to share in a second is that I can also have these things adapt for extra columns which we technically in the data world we

call them a schema evolution, right? With Apache Iceberg, which means that it can do a schema evolution as well, right? So, all these three three things are in place for us. We are just going to send the same file. I'll show you a demo how the whole thing works. You'll see the data all these three files gets landed into one unified schema where you are running using

things like Apache Nifi and other stuff. All right? That's exactly what I define here. I think no, I need to go here. Yeah. We start from here, use AI and LLMs, map everything here, schema discovery, header resolution, and everything happens here. And then my infrastructure has intent like because in AI world, I always stop this world as intent, right? It's not code we are writing. We are

writing semantics. Right? You all developers are now becoming semantists than syntactists, which you usually like Python code or Java code or whatever code you write. Now we are not going to do this. We're going to intent define. So my intent here is that any format of this event management thing is going to land, this particular is going to land in this unified table in this format of

data. That's my intent. I'm defining my intent. If you remember sometime ago we used to do like, okay, write lots of architecture, then I write LLD, I write I write low-level designs, I'll have all the documents, and finally I write the code. End of the day, the code is nothing but the translation of all your documents. What is all your documents is what I mean by intent.

The intent of user build a system, right? Which is kind of handle this whole data pipeline. That makes sense? Okay. But to do this, what I have to do is like I need to have a very strong data foundation. So any AI to be very precise, if I don't have a strong data foundation, then AI is not going to work for you. The AI is going to

send no, right? It can't find it. It'll hallucinate. It'll do all kind of other things that you known for, So to bring data in a typical thing, so we are going to see something called as Snowflake OpenFlow. But don't worry, this is the enterprise version of Apache NiFi. You heard about Apache NiFi? So this is enterprise version of Apache NiFi that we bought, we call it Snowflake

OpenFlow. Both of them are same. We just added additional connectors or other stuff that we do here, right? What basically Apache NiFi does Apache NiFi allows you to bring data from heterogeneous sources into your analytics platform. In this case we're going to see it's going to get into the Snowflake for this all the demo that you're going to see. And you see here, this various heterogeneous I

mean, you can get it from SAS platforms, you can get it from streaming platforms, you can get unstructured data. You're going to get it from database, and also other kind of thing, right? You have SFTP, Amazon S3, etc., etc. All And people will be having a question. I assume that everybody is coming from a Java world. Okay, what does Apache Camel does? How many of you know

about Apache Camel? Okay, one, two, okay. So, I think if if not like that, I'll tell you simply Apache Camel is an app basically which does exact same thing, but Apache NiFi is an infrastructure that gives you this. That's a big difference. I think I wrote an article sometime around on Apache NiFi. If you go to my the blog link which I said, that has a document

which talks about what's the difference between Apache NiFi and Apache Camel. Cuz people usually get the confusion. Okay, I know Camel. Camel does the same thing. Camel does the same integration of data, but what's the difference? Right? This is what happens. So, with this, what I basically get is like I get a solid data foundation. Now, my data from all the sources flow inside. In our case,

the demonstration is going to be data coming from multiple different people are sending me the data, companies and organizations about my event. All right? Next, what happens is that I want AI to process it. Basically, so Snowflake Cortex is a platform that allows you to do AI management, which means that we already have those models. So, tell me one thing is that when I start to use

AI models to be very precise, the very first thing I get into is like what should be my infrastructure that I basically need to have for AI to run? The GPUs, the CPUs, and other stuff, the loading of models, refreshing of models, you name it. There's a lot of tasks. So, what we have done at Snowflake is that we have made things super easy for you. You

don't need to worry about what is there behind the scenes. You just choose, okay, I want Claude 4.6. I want Claude or Gemini. I want these models. So, what we made sure is that we have loaded tons of models already preloaded for you, right? Which means that all you have to say within a SQL function or a Python function or even a REST APIs is just give

what's the function name is. I mean, the model name you want to use. There's a list of models, large models, medium models, short models. Everything is available for that. What also we made sure is that we made powerful AI apps. Not everybody is going to be a coder. Not everybody is a technical guy. So, I want people to also to use AI with just clicks of buttons,

wizards which will allow you to do this stuff. So, that's what I meant by powerful AI apps, but I also have people who want to talk to AI using APIs. I want to connect my Python application, React application, or any kind of an application talk to AI, get the inference done, and do stuff. That's powerful API. The second piece I want to talk to data. So, what

do you mean by talking to data, right? Let's say I assume that I'm a non-technical business guy sitting at the top. I want to understand what's my next quarter's revenue or something. So, I don't want to run complex SQL queries. People can give you dashboards to all this stuff, but I want to do this stuff myself, right? I can definitely talk to AI, and what we call

as Cortex Analyst, which basically talks to data using AI. And the text to SQL thing which we have done there with Snowflake, it's a kind of like it's 92% accuracy converting your natural language into SQL queries. Which means that none of your business guys never need to write a SQL again. Just use Cortex Analyst, and then it can talk to natural language, talk to your data, and

it'll give you all the insights that you require. All right? Obviously, talked about the model, but the very important stuff I want to touch before we get to the demos that biggest question comes to our minds when you use AI is trust. Can I trust AI? Okay. And we have made sure that you can definitely trust if it's on a trusted platform like Snowflake. The reason why

I say is that we don't use the models to train the models again, your data, right? And we don't send your data. We say like we have OpenAI, Anthropic, Meta. We're keeping adding Gemini and all these models are coming. So, we don't send your data to these providers to kind of train their models. Which means that the data is secured within the boundary that you define. That's

the last piece you're talking about the governance piece here, our back guardrails, evaluations, monitoring, and AI gateway. All which means that imagine I have a table which has PII And then I run an AI model natural language thing on the data. Now, what happens is that technically if the model is weaker, but it's not trusted, it will go touch your P2 data and get the P2 data

out. Which is a bummer for you, right? So, what you had done with Snowflake is that the policies are applied, right? Which means that they are also carried to all your AI functions and AI queries as well. Which means that if the data is masked, if the data is P2 protected, if the data has our backs, if data has guardrails, none of these models will give you

answers for them. It will give you an answer like very abstract, like I cannot read this data because I don't have access to it. It's masked or something like that. Which is the perfect thing that you want to do. Okay? That's the kind of thing you needed. The biggest blocker for anybody to adopt AI is two things. One you saw about the data foundation, we fixed that

using open flow. There's other ways to bring in data as well. The second way is that you want to have some platform as which is easy, connected, and trusted. That's what we say by here. Again, trusted is what I'm stressing right now on. You can get into Snowflake, use the data by yourself, and see how powerful it is. Like I have a trial link on the doc,

which is a 30-day no credit card required trial, which you can try all the features that I'm going to talk today. Okay? Awesome. So, this is where everything stands around. We have a AI data cloud because people have a very uh what you call a myth or something. Okay, Snowflake is just a warehouse. No, we are not a warehouse. So, we are a complete AI data cloud

platform. We do data engineering, analytics, AI, collaboration apps, everything put together as one platform. In Snowflake world, the warehouse means is a virtual warehouse. That's is basically highly tuned compute that can run your SQL queries. It's end of the day is nothing but the computes that run on the major cloud providers. Okay? Great. So, uh I talked enough. What I'm going to do now is I'm just

going to show you a quick demo. And obviously, when I was talking about Snowflake, when you sign up for the trial trial account, is is it I want to blow it up or it's okay? The fonts? The fonts are fine? Okay, let me know if if you need to uh I will zoom it. I can zoom that. So, whenever you sign for Snowflake, you basically get this

page which is called as Snowsight when you read articles or documents or blogs, tutorials about Snowflake when it says, "Okay, what is Snowsight?" It's nothing but the web console that you land in. Okay, this is what it is. Then try to do a trial sign up. I think you can go I'll also show you the URL at signup. Uh signup.snowflake.com. When you go here, you sign up

for all these things. You get a bunch of the other stuff here. And then you fill it up. You get your email, 30 days, $400 credits, no credit card required. Sign it up here and get there. I have a link right there on the doc. You can use this link. Don't worry if you need to copy. I'll talk about one more interesting stuff at the end of

the session if I have time. That's Cortex CLI where I can go to do all the stuff that I'm going to show you with just bunch of AI commands, like a coding tool. Okay? Great. So, let's go. And then if I go uh and there are a lot of columns. And one thing which is interesting for today is open flow. That's what I've opened here. So, where

we have open flow which is basically Apache NiFi, managed Apache NiFi by Snowflake. In fact, I mean you can do all the same things I'm talking about with Apache NiFi also. Right? If you're running by yourself. Okay? But you're taking care of that. We're managing this platform by ourselves. So, this is what happens. I'm on a runtime and and then I go to a runtime, this is

what your flow looks like. It's very similar to your flowchart. Okay? Imagine this case, right? Um I'm going to go go back to this whole stuff and tell you again bring back the I have three CSV files, data feeds for me right now. The first data feed is going to be baseline. The task that I usually do first is that, "Okay, should this be a create schema

or an alter schema?" That's the first decision I do. All right? That's exactly what I'm going to do here as part of my analysis and design. Okay? These are all processors. I'll talk about processor. I don't want to go deeper because that itself is going to be its own session. So, processor are nothing but which will get connected to the external uh systems basically like your databases

or Google Drive or whatever you call it. In this case, you're going to use Google Drive and then kind of talk to this. For example, this one talks to a Google Drive and SharePoint from where my data feeds are going to come in. Okay, you can add anything you want. Okay, the processor are basically list of processor here. I can drag drop And then you get a

set of processors here. So, we have Snowflake which is proprietary Snowflake connectors and processors which are built. This gets available only on a Snowflake open flow But if you're using open source version, uh just pick Apache Nifi. Then you get all these components that are available in open source as well. Apache Nifi also has a Docker container. You can run it locally on a machine as well

and add all these containers. So, basically, these are the ones which just talks about So, the two things that we're going to be interested I'm not going to go deeper into this is capture Google Drive changes which means that anything any CSV I drop into Google Drive is going to be picked up by this processor and the thing is going to be processed. Technically meaning, you're going

to drop the CSV files into uh Google Drive and pick it up. The other one, just to show a heterogeneous connector, I'm also going to drop the same file inside SharePoint. Right? So, that like it also comes from SharePoint. It also comes from Google Drive, One Box. There is a lot of connectors I listed, right? That's what basically does it. The first thing does this analysis and

design. The next one it takes okay, I need to create a schema. Basically, it runs DDLs. They basically create a alter the And now once a schema is created, what I do is like I okay, this data is ready for me. So, I move this data into a different thing like in this case to an S3 bucket ready for ingestion. Okay, that's usually what your data pipeline

is all about. Okay? And then there is also a place that also runs some schema inference. That's exactly where I'm using Snowflake Cortex to infer schema. I'll send some sample rows, sample thing, make this thing say okay whether the schema is semantically same or semantically different. This is evolved. I bring in all those details inside. And eventually, if people have heard about Apache Avro, there's a unified

canonical Apache Avro schema that gets created that will be baseline, I'll also just trigger this one. This is my tool that I usually use to kind of visualize this whole registry all about, Uh so, let me go here while this starts So, let's start with this one. So, what we'll do is like um I'm going to take this Harmony Groove lineup. And then I'm just going to

throw this one into the uh drives. I already have this Beat Valley lineup up up here. So, I'm just going to run the script. I'm not going to upload it in the interest of time. Just going to touch the file. Uh once this touches the file, I'm going to start this whole stuff, which means the processor are going to start. And you see the file flowing inside

here. And then once it flows, it basically does all this analysis. This is a very first baseline thing. You see that's what this says here. There's nothing no sequel to respond because my database doesn't have any of these data. I cleaned up for the demo. The first one is running here. And then it went into the all the pipeline. It does not move into ingest yet because

there is a gate which I'm putting because all the schema changes I don't want it to be done automatically. Somebody has to go review it. You can connect to a workflow. And from there, people can say yes, no, and then it can start from there, okay? That's what happens here. So, and then like I used to I have this Slack integration here. Basically gives you details about,

okay, I have uploaded this file. Given analysis and everything. And it also generated an Iceberg file. We'll see that file in a second, okay? And then move the data into an S3 bucket where the same file is moved for ingestion to happen. But ingestion is not started because I have not enabled it yet, okay? I'm going to go into the visualizer. So, let me refresh the data.

I should have one thing. The ingestion doesn't happen because it's not ready. Somebody has to go manually click, okay, ready to do this stuff and all that stuff, okay? We'll do And we see the schema details, uh it will not have much analysis done because this is a very first schema that's landing inside my old application, which means that this is the baseline schema for me, right?

As I said earlier, what I'm doing is that I'm kind of converting this whole schema into Avro as the end of the day, so that Avro schema is the schema that I'm going to use across because one of the biggest features of Avro is aliases. Each Avro field can have aliases, which means that all semantically mapping field I can drop into aliases, so any different shape or

form of your CSV or data feed can actually land in here, That's what I'm doing here. So, uh right now there's nothing here, so what I'm going to do is as instructed in this it asked me to run the SQL file, so let's go to my CLI. And then I'm going to start this uh whole stuff here. Task It's just a bunch of uh scripts that I'm

doing here. It just creates a schema. It's going to download the file. Even the SQL file is generated by Cortex. It's not handwritten. It's going to be generated. I'll show you the SQL file in a second. So, it's downloaded this even create SQL I'll put put it here, okay? This is a create a SQL file, create an Iceberg table. It infers even your what you call your

what is the type of the data. This is an event data as I told you. It understands this event schema. What type of event? It's a music event because we have some music-related stuff there. So, A is able to infer all these data from the schema and created generated this Iceberg table schema definition, which you can run against your database and anything I open table compatible cloud

format. In this case, I'm using uh Snowflake here, all right? Once I do this, uh things have got your open flow I think should start to pick up this one right now. You see here like in a few seconds you should start to see the data. The data is moving right now. Let's refresh this for a second. There you see, the data flows in again. And then,

sooner or later should have the data inserted into our table, right? I think this this has basically go here. Once ingest the data, I have a table like that put Iceberg, which means that it stores the data into Iceberg specific open table format. Okay, it takes the same data the CSV understand the schema, use the APIs to basically insert the data, and this is giving you the

information that the data has been inserted here, okay? So, if you go back here on to my um let me go run this one. Let's run this thing. I'm going to refresh the data first. Okay, and then we'll do a count on the It takes a few seconds to refresh, and then if you do a count, the count basically is going to return zero now. I mean,

15 here. Let me do the description of the data uh So, which means that the data is there, right? So, all the crystal waves and whatever it is, let's go back here. That's what we are kind of doing it let me go here. This should be the one, right? I'm not I'm not going to go. It's Beat Valley lineup, sorry. this is the one which I put.

Sorry, it's the wrong one. So, we have the crystal waves one. This one 15 record has been inserted here, okay? Perfect. Assume that if I send a semantically same schema, okay? That's the first thing I'm going to do before I you going to evolve the schema. When you see here, you when you going to push this file, these two are exactly same. Semantically same. Though they have

different headers, right? She instead of show ID, I'm having event ID, okay? Let's push this one here. Go back Let me go back Uh and this is the file I'm having here, sound wave events. I'm going to edit all. This one again I'm going to touch uh so that like it picks up so that I don't need to upload. Save it here, and then click save. And

I'll go back to the flow. Back here. And you should see the the whole thing flowing back here. Now, there is no interruption. Nobody needs to interrupt right now because the schemas are technically same. What now happens, it automatically flows through these things. Now, there is no alter, no schema. It basically goes through that ingestion pipeline flow. And then it's just like a flowchart. Imagine that it

starts from here. Now, it's going to go here. You see this one coming up here, right? Which goes here and then sooner or later the data ingestion should start. And then you see here it also given you the information that everything is mapped. I will go even a better one that's more for my understanding. So, now it says, "Okay, uh it's loading the analysis for you right

now." And says, "Okay, there is no mapping between." The event ID maps directly. This is the actual AI analysis which is being done there. Okay? It says takes these two things and says that, "Okay, the event ID is mapped to event ID, artist name to artist name, VIP price and this." It says that, "Okay, this is a rational that I given here." Okay? In this case, what

I have done is like, "Okay, you might be asking a question, okay, what if I send a customer uh file or something else, some different thing, right?" In this case, what happens there is a threshold that I have in the pipeline, which basically says that I should have at least 60% of the fields should map semantically. Which means that if you're bringing it all together a different

data, customer data instead of a music event data, then it's totally two different schemas, which means that it doesn't map. For me, for this to map, I should have 60% of semantic mapping. You can change this thing, but this is what I said here for this right. You see this threshold that I'm saying here? It's 50% threshold that is required. Now, my baseline schema has not changed,

but sweetly what happens is my data has landed in inside my pipeline. Okay, let's go back here to my table. Right about here, I'm going to go and do a refresh. I think let's see a count. I have 15 rows. I should have 30 rows now. Uh I'm going to do a manual refresh. Usually the refresh takes some And then now when I do the select star

from here, you should see all the data gotten. Uh let's do crystal waves and then if you remember like let's say um neon wolves, right? That's what we had You see here? This data's gotten here, but sweetly into those respective schema that we already had. It did not create anything. It did not change anything. It did not fail. It sweetly adapted to the schema that we already

defined and that option is taken care by AI which kinds of inferring the schema. How do I basically do this is very simple. Take this headers. Give the five rows to the AI query and ask it to analyze what is this. And I already have a baseline schema, Avro schema. I ask it to compare these two things. Once it maps, then it goes inside. Otherwise, it shows

an evolution or something. It generates a sequel for you to create the whole stuff. Okay? Yeah. >> People in the data loss function like decimal, I had to you will see a data is exactly same as value. >> See, as of now in this example, I don't have a transformation done. But ideally speaking, what happens when I generate the Avro schema? So when I actually run through

the thing, it also generates reduces the data types as well. So which means that the data type has to be converted. All I have to do is go back to the prompt and say, "Okay, convert this data type." That's it. Otherwise, yeah. >> It's a pure one. That's good like this is not nested one. Pure one row data >> Yes. No, I've not I just taken the

CSV example, but it could run with any other nested example as well. So the only thing is like it it's normally I say that music uh event use case here. But it could be adapted to any use case because I'll show you a couple of other files that is involved here. All you have to go touch is prompts. It's a vanilla English prompts that you need to

go touch to adapt your use cases. Like let's say I'm going from customer order thing from here, right? I can even use this use the same stuff. Convert the whole thing to this. But all you have to do is like you need to have something to draw the pipeline like Apache Nifi. And you should have solid platform that can give you the AI features. That's what we're

doing. now. are leveraging that very very heavily behind the scenes, all right? The next one what you'll do just to complete the demo, I'm going to go and touch this one. This is where something interesting is going to happen. So, let me go here and I'll edit this file. And then I'll Let me remove this one and say save. Good. The data is saved. Then go back

here. Two messages here. I think it's a lot old ones. Go to the flow again. The flow should start right now. And now the evolution should happen, right? There's one more file which is full flowing in. So, ideally it doesn't flow so fast, but since for the demo, I made sure that the sinking time is very faster, like less than a minute. But ideally like by default

it's 5 minutes. You can change that on the configuration and properties of all these things. Now, what should happen is that since I'm getting the new one here, this which has two additional columns here, right? The rest of one are all semantically mapping. I have two additional columns in this new feed, which means that my table has to evolve. The schema has to evolve, okay? Which means

that additional columns needs to be added. That's exactly what I'm going to do right now. Let me go here. If you see this one, I got this message. This has been processed, but it says that I cannot ingest your data because I need to run this evolution script. Okay, this is not ready to be ingested, all right? Let's see what the analysis has been given. Okay, let

me refresh the schema So, if I go back here and refresh this one, it says that there are direct mappings. Event ID, event ID, artist name, stage, and everything. This is a perfectly direct one-to-one mapping. There are two fields. Schema evolution is required because genre and sponsor, this basically are two new fields, which needs to be added onto your schema, which means that schema has to evolve,

And now what I basically do is like go back and run. This already have a sequel generated. I'll just clear this one. I'll go up and then do evolve schema. You will see that the generated sequel is just an alter table to an iceberg table, which has two columns, which means that all my other columns are going to be intact. Now, there's an evolve schema. It's going

to run the evolve schema, basically running the sequel behind the scenes using the database here. Alter iceberg table, two columns are getting added, genre and sponsor. And nothing is changed, and everything is intact, okay? Now, what happens is that when I go back here, the schema would have been changed, and now the open flows thing should basically start running this whole stuff again, processing its schema and

every Let's wait for a few seconds for this to be done. And then, like I'll show you the one of the One of the questions that's running in your mind is, okay, do I need to handcraft all this stuff by hand? Okay. How old things simple we make for people? We come back again to get this is hard, right? We Okay, obviously there's a bit of learning

curve with Apache Apache NiFi. Instead of you writing manual ETL jobs, you're drawing something which is going to do this job, which is one step ahead from manual writing. Now, you feel, okay, I have to do manual writing. What is this processor? What is this processor group? How do we do add these properties? How do we do all these things, right? Again, with AI in place, you'll

always feel, okay, I have I need I need things to be done much easier and much faster. So, let's wait for this to happen. It I have less, and then I'm going to show you a very simple way how we are made things super easy with Snowflake Cortex, again with Cortex So, uh in this case, uh okay, this is done, I guess. Let's finish this one and

get to that. The back processor. The schema actually better than what I would try. Back state video. We'll be able to do that. See, every every schema right now, it's kind of like have a registry and I have a version that is maintained here on the schema registry here. I can always go back and roll back to the old version. The average schema gets updated. So, the

pipeline backs in like it takes the take the schema and inserts the data again. So, this is not a I should say this is not very super sophisticated like a Git or something to have so much thing, but it's just a very basic concept of what you can do, okay? All right, let's go here and see if my data is there. So, let's go here. I'm going

to refresh the data again. And the same table, same schema, everything is same. Uh and then let me I should have more rows. Genre and sponsor has been added. And then there are some rows which is not there because they don't have genre and sponsor. There are some rows where genre and sponsor are there, they get inserted. The same same table, same schema, one unified table so

that your app doesn't need to change. In the in Iceberg world, you you can always point to a snapshot uh with this respective tables for your application so that you can query the extra fields or don't query the extra fields and all these are allowed within Apache Iceberg spec. All right? Look. Yeah. >> Okay, there's there is a grab the schema cost like that. >> Yes, infer

the schema first, the first shot. So, let me explain this again. That's a good question, actually. So, if you go here, the first space, it basically understands the schema, the schema that comes in. It takes the schema, sample rows is given, the headers is given, and try to deduce what should be the table name, schema name, and everything, right? And then it goes and checks your database,

in this case Snowflake, to see if there is this this table and schema is already there. If it's not, basically generate the SQL for you to give. That's the first baseline thing. Next time when a new data comes in, it basically checks the takes the old Avro schema, compares with the new sample data and schema, again run it through AI. It compares these two things and infer

whether this correct or not, okay? Let me show you an example. I think this is a little big query, but since you're asked, I think I'd love to show you the query that actually runs These are the functions that we did. I'm going to close these ones. Uh let me go here. Sequel. This is the first one, the evolution check. This is what happens. So, uh, it

takes a table name, table name space, average schema as I told you, and then it runs through AI complete model, which is the function, the sequel function that Snowflake Cortex gives you. Lot of AI data cloud platforms gives you this. But, this is very sophisticated as I told you earlier, to choose a model is as simple as this. You want to Claude 4 on it, Claude 4

Opus, Claude 4 or Gemini or Lama or Mistral or whatever model, just pass it as a parameter to the sequel function or a Python function for that matter. And then I define a prompt. This is where everything gets inside. I give the average schema, I give the headers, I give the sample rows, and I give some analysis rules here, semantic mapping, schema evolution, match priorities, all these

things which are like technically defining whatever thing. Like I remember I telling you intent. My intent is to infer whether the schema is required or not. Okay? What schema needs to be generated, right? That's what we need to give here, kind of going this, and required response format is more for my application to process. All the functions returns you JSON, so I'm just telling, okay, on the

JSON, I need these fields, like whether it is matched, schema evolution is required, what should be the schema analysis. Like you see that, right? It was mapping, it was evolving and all these things. And then I'm giving some analysis guidelines, what it basically needs to do. Finished. And then one of the important thing that we need to take care of is this temperature here. Because whenever you

run anything with AI, the randomness or thing is very high. So, which means that the randomness is parameter somewhere between 0 and 1. Okay? By default, it's 0.7, if I'm not wrong. Which means that every time I run the query or the same prompt, it's going to give me different responses. Obviously, even now it will have, but it's since I've given it as 0.1, the thing is

negligible, right? Because I want consistent response every time I get back, so I try to reduce still we get some amount of inconsistency if you see that the ticket price thing and admission cost these two are two different thing because VIP price is different from a normal admission cost. But still it is in taking okay it's dollars so that like it takes in dollars, right? But we

have that kind of a difference which is still there because the models are not perfect. But that's the little thing I kind of reduce the temperature here and lastly I'm giving the response format what schema I need to have because this is a schema I'm using it back in my data pipeline when I extract the data out. That's something which I require so that's again in this

one. So this is the first question, okay? And finally I just do a base 64 encode and all these stuff, The second one what happens is that the ingestion mapping. This is basically where the ingestion happens, right? Where the data ingestion happens. So the basically why I need here two AI calls is that one first to infer your schema, my schema's got created. Now the CSV comes

in and then let's say the first baseline you don't have any issues this is directly inserts inside. For the second one I need to compare with the baseline schema to see how to map the incoming headers, right? That's how the actually adoption happens, right? That's what I'm doing here. Same question here. Again using the same model. I can choose any model that Snowflake supports. I'm also giving

some examples here. This is very important when you write AI prompts because we need to go tell AI what exactly you want. Just run do a mind mapping here, right? I'm saying these are the some examples of event ID and all that thing etc. etc. what you want to use. Semantic mapping rules that I'm defining how should it semantically map everything, okay? And also I'm adaptation instructions

I'm giving like whenever there is extra columns or something like that put as aliases and all these instructions being given. I'm giving some input sample data for it to analyze and understand how the data structure looks like and finally it gives me like the theme again the model temperature and other stuff the adapted schema, right? So which means that this schema is what your applications will basically

use to extract data from incoming data files. Right? So, these two are the important things that runs across to basically do this job. And apart from that, as I said if you go into this repository, again, I said the repository is there on on the The system message and the user message. So, these two are the two different prompts. I think this is a little larger prompt.

I think you can just take I think I define all the basic rules including data quality rules, code rules and everything. This are around 600 lines of prompt that basically says how to generate the file, what should be there, what syntaxes, semantics you need to follow, the naming conventions to follow, how to verify what is generated. I go into that level of details. Now, you can take

this same one, give it to any other AI tool, ask it to format, simplify, it will do again. Right? It's a kind of continuous process. So, system message and user message, these are the very two important things that I need to give to AI calls again to make it to generate those files. Example, the sequel files that I generated like here, for this I have to give

all those details. So, we should also send in data like whether it's an evolution or a creation or whatever use cases. All So, these are the very important thing. In case if you want to adapt to not to music events, let's say for an order processing system, go back and touch this prompts, things will change for you. You don't need to touch any code. There is no

code involved in this. Okay? That's why like this one is little the only complex one maybe the little learning code to understand what is a processor, how to configure a processor, how to put it in process tools and everything. Okay? Even we are making that even more simplified for you. Okay? So, since I have like 20 more minutes, is let me open another runtime here. For sake

of simplicity, so what I'm going to do is like I'm just going to deploy one processor here. So, as I said earlier, we have a lot of connectors. And one of the connectors is an unstructured connector. So, where I can pull data from Google Drive or any other thing. I'm just taking Google Drive as an example here. And take the unstructured data, your docx and everything, and

then you can index them as a search data, right? That could be searched by AI. What we call as Cortex search, okay? To do that, I'm not going to handcraft or put raw drag and drop like what I told you here like this, okay? I'm not going to do anything like that. So, I'm going to I opened up another canvas. This is a canvas, if you read

canvas in Apache NiFi, it's nothing where you draw your diagrams, okay? So, what I'm going to go is I'm going to go down uh to my Cortex code. I think Cortex is a CLI. Uh Cortex code is like a cloud code, Codex, and other stuff. But the only difference is that Cortex code is very, very tuned for enterprise not for the generic coding things. Meaning to say

like when you connect with Snowflake, it understands the whole Snowflake's ecosystem and all its features. It's going to help you do the job very easily. Which means that it's not It should not be compared for a normal coding agent like your cloud code or a Codex or something, which is very generic. This is very specific for data and enterprise data very specifically, So, what I'm going to

do here is like uh I need to do a quick thing. Connect to my VPN so that I don't have the IP garbles here. Give me a second. So, it's going to go there and then connect to my uh VPN. Now, what I'm going to do is like I'm going to say uh basically do one little thing right now. So, it'll be a little slow, but I

have a lot of talking time and it's I'm just going to say uh if you go back to this uh runtime here, so when I do analysis and design, I had this capture Google Drive changes, right? And then do all these flow. I'm just going to basically say do this flow and create this particular connector and do all the configuration that's required. That's what I'm going to

say to that, okay? Let's go here. And I'm going to go back uh to my CLI. And then say cortex session. I think it's I think I remember goods 2026 if I remember correctly. Let me see if that's a session. Okay, sorry. It's a session name. Go here. It needs no authentication. A second. Just going to authenticate. There's a lot of authentication that goes inside All right.

I'm good here. And should be having here. So, all the things are going. Let's go and clear the screen. And then I'm going to say uh set up connector demo. I'm just using demo here just to be very specific. That's all I need to do. Now, it has something called a skills. I think probably I read about an AI what is called a skills. Okay? I'm going

to talk to that about in a second while I run this one. I'll also talk about those things as well. Let give me a second. Let me authenticate as well. And then sign in here. Come back here. it starts processing. It takes my skill here. What skill is basically, okay? I have to keep jumping between this because it keeps stopping at places because I have to say

yes, no before it proceeds. I've not given the YOLO moment So, if I go back to the the presentation here, so I go back here and talk about cortex code in a second. I'm talking about not I'll come back to this in a second. This is what I think I'll leave it here so that I can jump between this, okay? I give all yes here. So, this

is what I'm talking about skills, okay? As a developer, right? So far, what we have been doing is that we've been a lot of production experience, a lot of deployment experience, and other stuff that gets inside us, okay? But eventually what we have been doing all these years, we have been just considered all developers have been considered as just operators. Right? Somebody takes in open up IntelliJ,

VS Code, you name the IDEs and write start writing code, deploy the code, finish. But with AI, that job is taken out. If you remember like when I first started my Java code way back in 2000, like I wrote it on a notepad. Right? There was no IDEs, nothing. Okay? Before Eclipse, IntelliJ, everything came out. Right? So the evolution happened, right? The IntelliSense came in, dot things

in the docs, and then generation code snippets, lot of other stuff. Now, we are in the place where we say something, generation automatically happens. Okay? So what we are doing right now is that we're changing the mindsets from being a syntaxist to a semantics. What I mean by that? We're changing yourself from being an operator to a mentor. I will mentor my AI tools to say what

it basically needs to do. Right? Exactly that's what I'm doing here. For example, in this case I'm telling okay, go back and deploy this Google Drive connector. Like you don't need to do everything hand-crafted because I know what needs to be deployed. I made that as a skill. That's what I call as codifying of wisdom. Skills are nothing but vanilla plain markdown files. Nothing more than that.

Okay? And all all of these tools, Codex I mean or Cortex I mean or another tool like everything understands what the skills files are and runs through the instructions. Which means that you are in power of how do you basically do this stuff? Okay? I write this transition from operator to mentor, codifying your production scars. I know this is wrong, this is the way is right. And

you have I codify every other stuff inside my thing as skills. Give the skills to this particular AI tools which can run through this. Right? If you remember I was talking that was the first thing like I'm kind of building my own intent to do this. Okay? So let's see if that is in the prompts. Uh this should go here. Back. Why we do this is because

I want to flip this ratio. What I call as field velocity problem. Every time I run something within Snow, I mean anywhere building an application, the customer asks you something to do, you spend 90% of the time doing the infrastructure. And only 10% creating the value for the customer or anybody, right? With AI and infrastructure as intent what I define as I'm going to flip the whole

ratio because my AI, like as you see here, is going to do all the job that it's going to do here. Let me give one little instruction to this. This Snowflake connection local what? Okay. So, that like it takes the connection, doesn't do. So, it auto- automatically recovers as well. So, coming back, sorry about that. I'm just kind of coming back and saying I'm flipping the ratio,

want to create more value to the customer, more value to the people rather than spending time on infrastructure and all other mundane tasks that we usually do. So, that's what we're going to do with AI, making AI coding your skills and bringing this one and then telling it to what to do, So, that's exactly what you're doing here. Let's go through the steps. As I said earlier,

I've not done this stuff like I can basically do YOLO moment, say to do yes with everything, but that's the most dangerous thing that you can do with the AI because still it has hallucination problems. It can go back and do all the stuff that is not required, okay? It's kind of going to do all of the stuff that is required. Sooner or later, I'll show you

what is there. So, if you go to this particular open flow of canvas, I don't have anything else, but sooner or later you should see a particular connector added, configured, connecting to my data and getting the data and everything, all Uh, I think I have to go back and uh, say something about skills. Um, when I was kind of was kind of skills, this is what the

skill looks like. It's a skill markdown file. And then I say, okay, I want to deploy the open flow connector demo. This is some front matter that I'm giving to it so that like it can understand. You remember these are the triggers basically like when that particular skill get triggered. So that like it gets invoked. And what your basically AI tool will do is is run through

the skill MD and does each step by step. Okay, that's what it's doing basically here. It does each of these steps one by one and then kind of does the stuff, right? So why I call this as infrastructure of intent, right? So because if you imagine for a second if you're writing a Terraform script or an Ansible script or any other script, this will technically be your

DSLs. And how many times we have stuck with those DSLs? We don't understand what it basically means. Let imagine a higher CIO CTO kind of a guy wants to come and see what basically is being done. He doesn't have an idea to go through all the C DSLs, but now with these markdown AI tools, we're basically making them a simple markdown instructions where you are as I

said earlier, you're codifying the wisdom and crunching, putting everything inside this particular stuff. So that's what you're doing here. So let's follow this one. This This basically a schema get created. Now it deploys a connector flow, does all of the stuff. Give it a second. You should see the connector in a second coming up And that's it. I think probably I have 10 minutes or 15 minutes.

Okay, I think um I can take some questions while this is done. Uh it says it deploying a connector right now. It should I think I should have a There you go. I'm not writing this stuff. I'm not pulling anything from there. Now I've codified my wisdom, which means that it's deploying the connector. It knows which needs to connect, which API to connect, what needs to talk.

We just give the credentials earlier on and it goes back and does all of the stuff, right? Now you'll be wondering I can use Apache Nifi with ease. I don't need to worry about how do I get the processor, how do I get other stuff. So we are packing all the skills for all the connectors, which means that all you have to remember is that what are

the configuration details that I need to give? For example, in this case, I have to give a Google Cloud essay JSON file. I need to give a SharePoint credentials. I want to give other credentials. All these things is what you need to remember and give it there. And then rest of all is taken care like it goes step by step, does all this stuff based on this

thing and then kind of get this thing done. Okay? Any questions that I can take? Yeah. >> Is it ready? It connects back you showed me here. By there was a prompt I think the sequel cluster. >> Right. >> Like you said if you did that change you can go back and change the prompt but I thought the prompt was off this from no play. I did

it again. >> No, there are two things here. Uh in this file I think I remember what you're talking about. Yeah, this is see the first one is the initial one to start the whole flow, In between the processing, so one of the things what you have bought within Snowflake is that we have this AI functions which basically that's normal SQL function but it uses AI inferences

behind the scenes, right? And even this also takes a prompt. That's where I define instead of So what it basically does it let's say I have a million rows. I want to do some scan on the million rows. Instead of you going doing all the complex queries, you can ask AI to do the scanning of the million rows by giving it a prompt. That's all we are

doing here. >> What is coming like this this you do you This is that you know what I think is a >> See this one is what I created. Yes, yes. This is what I created. This prompt was tuned based on my need and then you can also have it generated should not be a problem. And all these have these feelings here. These are parameters that I'm

parting within the CD. That's all. But I need to write them. >> Please complete a function. It's >> It's it's Snowflake's. Snowflake has a ton of AI functions which already there. If you go to the docs.snowflake.com you'll see all the AI functions like your sentiment analysis, your AI thing, complete summary and all these things translate. So all these functions already there, which can be passed each of

them takes this model name as parameter. So, you can just try it around this which model you want. And you also have like while this happens, I can also quickly show you also have something called as an AI/ML studio. When you try for the same. So, that you can also do all the other stuff that you want to do with AI/ML. All the models get loaded for

you. For example, the very my my own famous is this Cortex playground. Even before I choose the model, I want to understand which model I want to use for my use case. So, this allows you to compare the models. All the models that we are supported here. Plus, I say compare. I can give cool prompts and compare side by side. What this model is giving, what this

model is giving. For example, let's uh let's say I take uh Lama, okay? And then say uh tell me about gets, okay? Let's see what it has, So, now you do a side-by-side comparison here. So, I think this is what the power the platforms brings in because now as an application developer, I want to understand which model best fits my use case, which is giving me the

right response, okay? This is giving completely something here. Let's see what it gives. Lama 38B I think it should be giving something else, right? Which means that it kind of gives closer to what it basically you want, right? Until then you can do all these things. These are all comes out of the box. You don't need to do anything extra. Now it says something else. Lama 3B

gives like this. This is more clean and neat or anything. So, this kind of a comparison also you can do within the models that's already loaded. So, there's nothing comes from your thing except for signing up for a Snowflake account, a subscription. And then all these things, all the models, the infrastructure behind the models, the GPUs, and everything is already done for you. You don't need to

worry about hey hey what I GPU I need to have, how many CPUs I need to have, etc. etc. All right? So, that's the power you get. I think there's a lot of other stuff you can do. There's something called a Snowflake intelligence where it's an agentic framework already built for you. You can connect all tools together to do your agentic stuff. So, that's again comes out

of the box. So, if you go to this bitly link bit.ly that's there on my thing. I'll share it again in a second. So, this has all these things. There's a trial. There's a Cortex Score trial. Exactly like if you want to do the trial that I'm doing right now. You get again a month free of Cortex Score to try all these commands that I'm saying right

now to you. There's a Snowflake trial which you can use. Again, I've given all the demo repositories. The structured demo is what we saw today. And there is other additional information I've added into this. I keep updating this. There are a bunch of videos, blogs, and other stuff that I basically do here. I think you can do You can go check this one out. This bitly link

right there is there on this QR code. If you've not grabbed it, just do a thing out of that as well. >> [music]