Anurag Kale: Building Data Lake Platform Fully Serverless

In this talk, the speaker discusses building a fully serverless data lake platform, sharing insights from their experience at an electric car company. They outline the challenges faced in integrating diverse data sources, including both home-built and off-the-shelf applications, while emphasizing the importance of a solution tailored to specific organizational constraints. The speaker explains the necessity of a centralized data management system to standardize data ingestion and governance while minimizing process waste. They go on to detail how they implemented AWS services like Kinesis and S3 for data storage and processing, establishing a solution architecture aimed at maintaining data integrity and accessibility. Key takeaways include the significance of user experience in data lakes and the careful consideration of cost management strategies during implementation.

[Music] ladies and Gentlemen please welcome our next speaker in arac KY presenting the topic building data lay platform fully serverless I'm going to try to be a little more entertaining than usually what I try to do and try not to bore you to that all right uh quick questions before we start does uh how many of you have here heard the word data Leakes before okay how

many of you built data Leakes before awesome uh have you done this on AWS or something else okay couple of them U what does serverless mean to you I that's open-ended question so maybe just shout out what does you no takers exactly exactly that's that's a good uh let's say summary of how uh how you should perceive serverless uh so this is what we are going to

do today let's let's assume that you are a engineer perhaps a senior engineer who or you have been an engineer working at a company and you have been now given a new task at hand and it's basically it's an open-ended question and you're trying to solve that particular uh question for company so before the chat GPT world what you would do do you would go to Google

and try to search certain things right so uh you you you were given a certain problem statement and you go on Google and you say try to search data Lakes you will find something like this uh popping up every here and there so this is taken from awss website it's a great great architecture but you will see that there is plethora of services that are moved around

and I'm the kind of at least I'm the kind of person who would want to understand the paradigms of data L before I go on into the details of the services so I continue my web search and something like this kind of pops up where they are talking about various blocks and pieces of let's say uh pieces of systems that are uh talking about it in the

data Leakes uh you continue searching and you might stumble upon something like this which basically is talking about patterns of how data leaks should be done right now we we we need to understand that software architecture is very nuanced top uh everyone which I mean is me you you we are dealing with separate different constraints so some things that are important to you might not be important

to me so when you look at an architecture pattern like this that I have shown you on the screen uh it is it is not fine-tune to your use cases it's not a solution architecture similarly when you look at reference architecture um it is basically a set of services that have been built together that can give out a generic capability both of these pieces are really again

they are really important for us to understand uh the services how how these various AWS components work and all of those but what we really need as a as an engineer is we need to find a solution to our problem statement right so you need to cut the noise out of all of these diagrams that you saw and just pick what is the best for you pick

what is the best according to your constraints and that's what that's my intention for today so I I really want to talk about the problem statement uh and what what is it that we did in order to reach to that particular goal before we go there um a little bit about who I am my name is anurag and I currently live and work in gothenberg in Sweden

I work for an electric car company uh unfortunately I'm not allowed to tell the name but you'll figure it out later um and I'm also something called as an AWS data hero which is like Microsoft MVPs and of that kind of sorts uh and um if you are interested to talk about data leaks or data in General on AWS feel free to connect with me on LinkedIn

and we can take the conversations later so this is the agenda for today that we are going to try to do we will talk about the problem statement first what is the problem area what are the constraints that I'm dealing with I will also talk a little bit about the framework that I used in my mind in order to choose or reach to the conclusion that the

data lake is the right Paradigm for this kind of use SC case I'll spend certain uh a little bit of a time to walk through the solution that we are using and why why we made certain choices uh during that particular phase and I will try again I will not nothing promis but I'll try to give certain takeaways or certain things that I learned during the process

that might help you in your next Journey when if you're going to try to build a data L for yourself or your company or your assignments all right let's talk a little bit about the problem that I had at at my hand so I was working at a company uh where the landscape was very diverse when I say diverse so the internal landscape which is basically what

we also call as a trusted u space because I own the entire application we had a bunch of products that were built in-house and they were all built using seress Technologies so uh all all the components all the apis all the orchestration everything was serverless and everything was event driven applications so event was the M was was the first class citizen that we were using for exchanging

information between the systems but at the same time we already also had a bunch of commercial of the Shelf products that you buy off the of the market example sap example workday example multiple of these Automotive uh manufacturing Mees solutions that you just buy off the shelf and you try to make it work uh we also had a team data analytics who wanted to collect the data

from all of these let's say things and information security because we were a company that was listed on nasak so so here's what happened so then what we did is uh there was there is also multiple partners that we are working with so these are manufacturing Partners these are delivery Partners these are uh people like um stripe who are let's say processing the payments and stuff so

they we had plenty of those partner systems SAS manag platforms and even Factory systems where the cars cars were being produced that we that is something that we didn't control so naturally we wanted to get the data which is in untrusted space and we want to bring it into the trusted space so that we can make use of this data so like in any startup is phase

happens everything went berserk and people started building Integrations towards these external systems and soon we had a spaghetti mess of Integrations and typically in an Enterprise setup again I say startup and Enterprise it's a weird setup but it is it is a weird setup uh so in this setup wherever there is an integration they will go and pick up a Enterprise service bus of the market and

try to build these Integrations now it might work okay but in this kind of setup I'm now not even representing all of the applications that are involved and it still looks a mess and I had 60 different products that were built in house we had 10 different carts product that we your building and each of this Integrations might have had like 5 to 10 pipelines so you

could you could imagine so this was like we were going through a nightmare we lost the raw data every time somebody made an ESP connection to the external world because they took the data what they wanted right so they I only wanted five attributes out of the external partner and I'm going to ignore the rest of the five or 15 uh let's say message components so we

lost the raw data in the process this was a govern nightmare of course because nobody knew what kind of money are we paying for the external systems because everybody had the freedom to build their own Integrations at the same time the boundaries of these Integrations were unclear so if for example if I took a take a payment SAS uh so the deal uh the data about payments

is required by the Erp system it is also required by your order system it is also required by your uh let's say invoicing system so all of this built the three individual pipelines towards it and it's basically who is exactly owner of that particular pricing object we were not able to place that and this information is critical if you are trying to establish ontologies if you are

trying to establish U the in general data landscape in your organization so it was very very problematic and this kind of created a big process waste in the process in in this entire thing so what we really wanted to do we wanted to change this we wanted to take control of the situations and this is where the problem statement kind of started so these were some of

the desired outcomes from this process or from for this project that we had in mind we wanted to standardize the data inje because we wanted to have a fixed governance way in order to predict how much cost are we going to pay for the external people to in order to get the data in we wanted to have better governance of course no who who doesn't want that

um we wanted to minimize the process waste so that uh you don't wait for the vendor to return back to you in 5 days when you're asking them for for a new data object and we wanted to build something which would help multiple personas these personas being developers business analysts uh data scientists so that they can utilize this data and not lose the value of this data

but there were certain certain technical constraints there were certain budgetary constraint that we had so due to that we made three critical choices that we are going to build this particular platform as a data Lake as a self-service based platform which basically means minimal involvement from the humans to get the data in or out of this particular platform that was my first constraint second is because we

it was a very small team we had to build this Frugal so we wanted to make sure that we don't pay upfront very much so we wanted to have some sort of a Opex model over capex model that's why we chose serverless and it had to be built as Loosely coupled system because although this seems like a project that came from top top down this was a

project that was built bottom up which basically basically means Engineers saw the problem Engineers thought okay this is a problem and we need to fix it so I needed to build a system that would evolve over the time so it has to be Loosely coupled uh in uh Loosely coupled in in theory so if I if I if I forget about the technical constraints if I forget

about um budget limits and anything right so if there was a magic wand that I could just push it and build some sort of a solution uh what kind of blackbox solution would we think about right so this is what we thought so we had an external space internal space how about we build a centralized data inje platform and this platform that that's the black box that

we're talking about and it needed to have certain attributes for it to work right it needed to have integration options that could integrate with this external partners and push support to the maximum uh possible permutation and combinations so that the we can integrate with the external systems now once the integration happens we needed to store this data so that again we didn't want to lose the raw

data right so we wanted the possibility to store this data somewhere once you do the storage you want to make sure that the governance is intact and and it needed to have a governance Pipeline and we also needed to have distribution options so that the rest of the systems downstairs which are in our systems which require different mechanism themselves we should be able to provide those options

out of this black box then basically we went uh I was in young architect I went on we started to question how can we build this particular blackbox what kind of terminologies can we use what are the different options and if you're working in enterprise Enterprise generally tends to default to a data warehouse everybody's like hey we need a data warehouse for this but there is there

is there are some inherent issues with data warehouse that I'm I'm going to try to address and why it is not a good Sol solution for this particular problem so what kind of architecture patterns can we solve this again at the beginning I said I tried to be the first principles guy and U I'm like okay before even getting into the details of data Lake data warehouse

we should you should really think about this problem from first principles and I started digging around and ask some questions how is it that people decide which is the right Paradigm right so this when I digged around this is the summary of what I found there are essenti usually two different approaches to get data from uh value from data the first approach is a top- down approach

now in this top down approach you have the data you have some sort of a theory that I want to get uh some and you have an hypothesis about this which is basically I want to derive I want to make like let's say an example if I use data I will be able to streamline my processes that's an hypothesis that you make right now to make that

hypothesis you do some sort of an observation on the data that is available and then you get a confirmation that's a top- down approach right um and that's top down approach then there is also another approach which is bottom up wherein you don't have a theory at this moment but you know that there is a lot of data available you know so you can observe the data

once you observe you start to see some patterns inside of this data right so once you see the patterns then you say I okay now I see something and there is some kind of hypothesis in this and then I'll make a theory out of it right so these these are two different the top- down approach is what data warehouses uses because when you have a when you're

trying to build a data warehouse you're in control of the source you're in control of the target you know the structure of the data and this is and now if you think about it how you build your ETL pipelines you will see that it's a top down based approach the same time if you're doing a bottom up kind of approach you will see that when you implement

typically Implement a data L you are implementing a bottom up kind of approach where let me take the data as as much as possible and then I'll decide how to do it so basically these were the two theories that we had on at hand you use topown approach when you know the question that you want to ask to your data you have the data and you you

know the questions that you want to ask to the data sure it's a little meta reflexing right now but if you think about it that's how um bi systems works if you do data warehouse you have to do a lot of upfront work you have to figure out Source schemas you have to figure Target schemas you have to do data modeling right so this is this approach

is what I call as schema on right approach so before writing the data anywhere you already need to have the schema but on bottom up approach what you what you do is you you basically don't have a question right now but you know that you need this data for something later you will be doing little upfront work and you will do schema on rate so you'll put

the data and put the on put the requirement on the consumption systems to figure out what the schema is overall if you think about my constraints what what kind of things that I was trying to achieve and what data Lakes gives us in this Theory it was natural for us to choose a data Lake as the Paradigm for of choice to build that black box now being

that first principle guy I'm like before even I go there let's try to understand what the hell is data Lake why what are the building blocks inside of the data lake so this is what in summary what data lake is essentially about at the heart of data Lake it's data lake is a paradigm that was born from cloud and the Heart of the data lake is an

object storage and object storage is basically your S3 or Azure blob storage or whatever equivalent of gcp is and that is where you that is what you use as your place of storing all the data in in data veras approach you use a fine-tune relational database similar Sly replace the find un relational database with an object store you have a data link that's very simply simplistically now

to get to use this data storage as sorry Object Store as data storage you will start getting data from databases iot devices applications disk flag files whatever you can imagine so data laks actually because they are just flat files they support multitudes of ingesting inje algorithms so you can do ETL you can do change data capture you can do uploading streaming up yada y so it supports

multiple paradigms works really well for our use case because the systems that are going to send the data from outside of my control I don't know what kind of uh ecosystem that they're going to use so I can if I use data Lake I'll be able to fine-tune it for my use cases now once let's say the data streams in and it's in object storage because it

is flat files if you have a millions of files you will really don't know where your data is right so you need better ways of discovering that data so a data lake at heart should also have something called as an as a catalog which basically tells you what kind of data I have and if there are frequently used access patterns you should have indexes on top of

it so that you can speed them up now from the object storage to build this catalog and this index you need some sort of computing right so for that compute layer in a Cloud ecosystem you will use function as a service which is AWS Lambda or equivalent or SQL as a service like some services like Athena uh that will allow that will take the object store derive

the metadata out of it and build that catalog for you or keep that indexes fresh right now so far I'm able to put the data in into the data R I'm able to make it useful then your rest of the ecosystem can start taking data from it so you could build report s you could build dashboards you can or maybe another database or another data warehouse can

take the data from this ecosystem and it supports multitude of protocols you can do batch operations you can do ETL you can do SQL as a service or you can just plain read the files using file operations in any language of your choice so so far data ingested data curated uh data provided by to the rest effectively but if I don't manage this if I don't keep

an eye I don't keep a lid on what's Happening inside this ecosystem I'll have a governance nightmare so I also need to have what I call as a governance layer on the in the data Lake on this data Lake governance layer what it'll do is it'll have identity and access management so that we can provide users certain uh limits on what kind of data they can access

I also need to have policy I because without policies identity and access management doesn't work and you really want to track the lineage of your data as well so that you know where the data came in where it went out and you can have the entire life cycle of the data mapped so overall looking at what I wanted to achieve and how data LS are working in

first principles this seems to be a great choice but now this is very generic in nature right this is this is this is the this is the reference architecture this is the uh stuff that I was talking about so what we are going to do spend next few minutes is take this exact building blocks and try to find tunit for our use case using AWS components so

let's try to implement this using AWS services so what I so then what we did is we broke down the last the the entire system that we wanted to build into four logical blocks the first block was something called as a landing Zone again this is all in my head right this is this is not physical this is like for us to understand that's a mental model

so we we need to have a landing Zone which basically is a layer of software that allows Integrations through external World which you can fine-tune depending on what kind of services your external alert provides now once the data lands into your ecosystem we want to build a layer once the data is stored managed curated you need to have a distribution layer which again that distribution layer needs

to be find fine tune for your landscape in in your in your controlled area and I and finally I need to have a control plane so what we are going to do now is we will go deep into each of these blocks and try to build them together so this is what we did so uh in landing Zone we decided that we will use a streaming service

because the systems that are going to send me the data they don't put it on a Cadence they send it when it is available there are so many multiple systems so a stream seems to be a logical choice where the data can come in and converge so as as soon as the data comes in I send this data to a stream processor again we'll go into the

details this stream processor makes two copies of this data the first copy goes into a data Lake which is going to be kept for historical purposes and the second copy is put in a temporary data store because I IM immediately want to send this data to an event store so the people who in who are event d even get the messages in near real time for their

consumption right so this was like majority of my components that were built in house were serverless event driven application so I'm serving the biggest customer for my internal landscape so they as soon as the message lands in puts on a puts it on a stream we make a copy temporary copy permanent copy and the message is received by the uh seress or commercial of or sorry or

internal inbuilt event driven architecture now for some of the now when you look at streams streams are inherently have some limitations on the amount of message that you can send at the same time so we needed to try to build a system that could also allow for messages that are bigger in size like 20 MB file 50 mb file 100 MB file that still need to be

processed so we had to build something called as a file uploading mechanism that would let you allow allow you to put a file and then we also had these historical data needs that perhaps will be required by some of these applications later so we were able to build this logical uh let's say uh logical way of building this uh application so with the near realtime delivery I'm

able to support all of the applications that are event driven with the historical data that is available you can use as Python and you can use queries that will allow applications that are 20 years older uh which cannot do event driven architectures they can still you bring the data out through API actions or through ETL and get it to the ecosystem now let's try to go into

the component design of each of these so for the streaming now we are now diving deep into the AWS uh ecosystem for the streaming ecosystem we had three uh we essentially had three choices to make from AWS that were fully serverless and matched our criteria they were Kinesis data streams there were Kinesis fire host and it there was Kinesis analytics which now has been changed to as

managed Flink but out of all of these three options only Kinesis fire host is fully serverless which basically means I need to know I need not do any configuration I need not do any provisioning it will just work out of the box if you're using so Kinesis fireh host became my first that became the first service that we chose because it allowed streaming for the data in

now this is where we hit our first problem or first challenge so some of these systems although they were able to send us messages they were Legacy systems and for authorization of Kinesis whenever you use AWS Services you have to use something called as a simple token service STS or AWS users so the Legacy systems could not use STS because STS is like a dynamic token vending

system so what what we had to do is we had to figure out a way to work around this issue so what we did is we just put a small API Gateway facade in front of it and used AWS users from IM am and I am users can have long running tokens so it it gives you an equivalent of username and password that can then with with

your policy you can keep it alive for one month 6 months one year or never it's not ideal but if it if you have Legacy systems in your in your ecosy or in your a vendor ecosystem you have to have it so this is the this is how we designed the first component ENT of the zone now if you look at Kinesis fire host the Kinesis fire

host also has a little bit of a limitation that it can only process a message that is only 256 KB in size so if you have a message that is more than 250 KB in size and if you look at text which is greater than 256 KB it's huge data it cannot process that particular data because the rest of the ecosystem in AWS like SQL s event

Bridge or all other API integration Services there their limit is 256 KBS so we needed to find a way in which if the uh partner systems are sending us data which is bigger than 256 KB we need to figure out a way on how to make that possible so to to do that we built this oh sorry yes here we are so we had to build um

a file upload mechanism now to build this file upload mechanism what we did is we view uh we bought together a bunch of services that worked in a flow so we had an API Gateway where the external partner would tell in the first thing that I am this service and I am intending to upload some data that is bigger than 256 KB on your system so we

take that service and we record that in a Dynamo database which basically says okay this guy or this system at this time is trying to upload and we capture some metadata from them which is basically who you are and what kind of data are you serving at the same time we trigger a Lambda function that goes into S3 and returns back with a S3 signed URL now

if you are not aware in S3 there is something called as a signed URL which basically creates a link to the folder inside of S3 which is valid for certain amount of time imagine of this like a Google Drive link that you shared your friend so that you can upload a file on that particular link so something similar but programmatically created so then at the step five

we sent back this S3 URL to the uh caller or the client and they took this s uh took this particular S3 URL with the tokens and service and authentication and all we gave and sent that message directly to S3 so you can upload through assigned URL you can upload the message directly to S3 once the message lands in the S3 fold ER you can use awss

S3's native functionality of event notification and it sends a notification saying that a new file is available at this particular link uh in S3 but if I only give that notification my Downstream applications will not know which kind of file it is or who is sending this file so in this process what we do imagine sorry in the previous step in this the DB we did capture

some of the metadata that we had so we took the event notification that S3 created we took the message uh the metadata that Dynamo DB had and created a notification custom notification that we could send Downstream so this was my component design of the landing zone right now once the landing Zone was complete we needed to build the uh processing and storage zone so to for stream

processing uh when when I do stream processing uh I use Lambda Lambda triggered this message Lambda made a copy of this message in a Dynamo DV which served as a temporary storage with a time to live for uh only two days and we also made a copy in an S3 bucket which was the data Lake now if you remember from the previous setup when the file upload

functionality completed S3 uh S3 had the file we got this message and we use we triggered another Lambda which put the message on event bus basically saying a new file from this particular system is available for your consumption and essentially then you can also use e uh you can put an ETL on top of uh the S3 bucket for your bulk data needs now looking at the

distribution mechanism so for event distribution we use something called as event Bridge because in event Bridge what you can do is you can configure rules that are dynamic in nature which basically can look at the contents of the message and say if the message meets certain criterias it needs to be forwarded to this particular endpoint you can configure that so we use that and for ETL kind

of use cases we use something called as AWS glue which is manage spark runtime now coming to the uh control plane uh so in order to make sure that everybody the data is readable and we have a catalog we used a service called as uh AWS glue crawler the crawler basically goes into your S3 bucket and looks at all of the files and derives the metadata out

of it basically name of your columns what kind of data type do you have it crawls it and creates a schema for you once the schema is available you can that schema is put into uh glue data catalog and this glue data catalog is used internally by Athena that allows you to do SQL queries in place in S3 on on your data so with this we we

were enable we were able to enable inpl squaring for the analysts and for uh for managing role base access controls we use something called as AWS formation that allows you to register this S3 bucket as a lake and you can do role base Access Control by using um IM am which is built in into uh Lake formation and then for some API oriented designs we used a

simple API setup that allowed us to do that now this is where we hit the final challenge that we had which was that if you are setting if you are taking the data from uh Kinesis streams through and Lambda putting directly into S3 S3 creates a combination of files which is broken down by year month date and hour so if you collect the data from various systems

in a single place glue crawler would is is gets confused because it gets message formats that are different for different systems so what we needed to do is we needed to figure out a way in which we can dynamically break down the message into the specific subfolders inside of S3 so that it is collected together for us if if you do this so an example the all

the messages from stripe should go into S3 SL stripe so that uh the message will remain mostly consistent again it's okay to have one or two columns that are missing but it remains mostly consistent if you have another endpoint which is uh which is coming from uh let's say another payment provider you will have S3 SL that payment provider's name and the messages will be collected inside

of that but for this to happen we needed to put the enforcement on the source systems to uh to provide us data in certain format so for that what we did is we created this metadata data kind of format so whenever you send a event to us or a message to us you append this metadata tag to any data that you are kind of sending now using

this data I have then we use this source and resource targets to create subfolders inside of S3 so all of the messages that are coming from one particular Source are collected together in a single subfolder inside of S3 and then glue crawls it over and treat it as a separate data source so this was the outcome of all our exercise uh this is how it looks uh

together uh in the in the entire process so um sure this this is this is how it looks today but it's not the end I said that I wanted to make this as a self-service platform I want to ensure that there is minimal human involvement involved when you want to use this particular service some of this stuff we have already did but I just want to let

you know so what we wanted to do is we wanted to automate onboarding for the new producers so what we did is we worked with our service now team for the Legacy systems they can make a service now ticket which is auto approved as soon as it is auto approved it goes through an S3 uh it goes through an AWS account creates a im user for them

and Returns the uh credentials of that IM users for systems that can do uh STS we created a common GitHub repository and they can make a pull request the the the moment they make a pull request on certain criterias they will get back a STDs role and that they can assume and build it we also wanted to onboard the consumer systems so we created an API action

that that was um a facade on top of event Bridge rules so basically you could send us rules that we would register in back of us uh we also want to add a service called as glue data quality which basically lets you uh Define quality checks on the data that is coming from the source so you can put something like if the data is coming from stripe

then I we need to ensure that the value of the uh transaction is not less than 0 I don't want negative transactions so something like that you can Implement that also we want to U uh provide mechanism for full transparency so for that we are what we are doing is we are building all of these uh dlq all of these um um dlq for sqs SNS that

are that are used in internally we are exposing them through a data dog dashboard so anybody in the uh in the company can look at the health of the messages and if they have some messages that are struck stuck inside of that uh let's say uh messaging cues they also have API actions to let them retry those failures for certain amount of time if they do not

then we will will have to intervene so what I've tried to do is I've tried to talk you through the entire process of how a problem statement was figured out how we went for certain let's say Paradigm why did we make choices for certain of the services and how this stuff came together and how this is this entire process that we talked about this is nothing but

solution architecture so if tomorrow somebody tells you what a solution architect does this is what they does right so this is their core job a solution architect should be able to ask your ask the business people what are the constraints what are the the use cases what are the things that you really want to do what are the things can be okay not having in the system

what are non-negotiable and once you do that you can use those to make these decisions that we did in this process I could have used Kafka as an inje point because Kafka was like universally available but it did not fit into my constraints of building it frugally because Kafka would require me to basically uh have some servers running all the time right so it did not fit

into that ecosystem but if it if this solution was for some other company where Kafka was already there it would have been no choice Nob brainer so I've tried to talk you through the ecos through the process of how uh we how a solution architecture could be thought about some key takeaways from this process was then people basically asked us is why when should I build a

data lake that is centralized like one for your company or I should have multiple of these data legs so uh for me it it if you need to centralize in our case we had to centralize because we wanted this to be the single source of Truth for the entire organization so in that particular use cases you should centralize this but if you are utilizing data L just

as a landing Zone like a temporary place where the file comes in and you want to read the schema and decide what we want to do then you can decentralize it so then multiple of your application systems can have their own smallish data Lake and they can be okay with it but in this AR use case centralization was required a key thing with data Lakes is we

need to pay really good attention to ux and DX when I say DX developer experience user experience because data laks are notorious for becoming they they are really hot when they are launched in a company people will use them for a couple of months and then they will forget about it so it becomes just a place where something comes in nothing goes out so in order to

make sure that it does not happen to you make sure that you are having you have the personas in mind that you're trying to build and build the interfaces that are required by those personas so in my case there was analysts that I was catering to and analysts are good with sqs sorry SQL so we utilizing Athena in this particular process meant that I am serving my

analysts okay at the same time when I said that I want to uh you help my developers I created API actions that they can use so they don't have to rely on a person to fix their errors for them so my developers are happy at the same time uh poor you again I have already mentioned this uh if if the poor uux then data will turn into

Data swamps which is basically data just goes in nothing goes out uh there is also a little bit of limitations when it comes to Open Table formats so if you are building data LX and if you search on data LX right now you will you will hear that there is Apache hoodi there is Delta lakes and there is Iceberg again two of them are Apache projects and

Delta Lakes is something that data bricks kind of presented that lets you do acid uh let's say transactions on top of data leges so if you intend to use some of those services in your uh in your data Lake then uh and you want to manage that data Lake through Lake formation then iceberg is the only protocol that is mostly supported it also has a lot of

limitations but that's the best format that you can use if you're in AWS ecosystem and you're building your uh data LS that said thank you very much for Patiently listening to me I hope I didn't bore you to death and uh if you would like to have I mean if you have any questions please ask them right away if not please connect with me on LinkedIn uh

if you want to ask me some questions later thank [Applause] you thank you anorak for your presentation and there are two questions on slido AWS can become notorious with costs while building this data Lake did you do any tecture changes to reduce pricing yes yes we did so in in the process if if you see that there was this component called as glue crawler now this GL

glue crawler is required uh in order to make sure that aena can query on your s uh on your S3 Lake now how frequently you run that crawler decides how much cost you're going to pay so we looked at the patterns on how analysts were using uh that particular query we saw that okay most of the analysts were coming in onto the lake and querying it every

morning right so I didn't need to have that uh crawling um done every 15 minutes instead we deferred it to 2 hours so again lessen the amount of the stuff another aspect of it was when we were looking at the data Lake in S3 S3 is also sure it's it's not that costly but it it gets costly so we put on life uh life cycle mechanisms in

which data that is in S3 Which is less than which is greater than 6 months it it went into Cold Storage which is basically Glacier so you have this uh you have this Auto life cycle mechanisms that S3 provides when after certain criteria you can move your data from hot S3 to let's say cold S3 so that in that way also you can uh we we were

able to manage uh some of the storage cost uh Dynamo DB we put TTL on the temporary uh database that we were using in between all of the databases that were temporary in nature the TTL for them were from few hours to 24 hours so if the data is if the data is registered well in the data Lake I don't need it in the temporary place so

that the data goes away in 24 hours so you're not paying for the storage cost I think that those are the top of the things that come top of my head that uh we kind of did uh for it for um cost optimization yes okay thank you there's another question so what approximate size was the S3 Object Store in the end of the day did you run

it into a scaling challenges yeah so within 6 months uh we we were I cannot say the exact numbers but we were in hundreds of GBS uh of data that was plain text that was stored and if I consider other data other other let's say things like images and all of those we were close to one PB one TB kind of data again it's not a big

scale but it's going to grow because it's it's only 6 months uh 6 months old um right now so the scaling issues that we we particularly had was with um with with the uh Lake and Athena like Athena is also you pay for Athena for the number of queries you make and you also pay for the amount of data that Athena scans in your query so we

had to put some best case let's say best uh we had to put we had to teach our analysts that you need to limit the scans by using using the day month day and hour partitions so if you do that you'll be able to reduce some of that particular cost that was a big bottleneck because if you didn't put that we Clause with this this thing it

would scan the entire 100 TBS of data to give you the result which was which was like really costly at the same time it was slow and this is exactly why we started looking into previously when I talked about iceberg hoodie and uh Delta Lakes this was the reason why we started to go into that particular route because if you store uh so we were using csvs

and Par as the storage format for the data laks right now but if you use Apache Iceberg it has a natural compression algorithm built in where when we tested it out it it took the 50 gigs of data and compress it to 5 gigs so we are looking at those Avenues in order to uh reduce and try to use U the hoodie capabilities or ice BG's capabilities

to reduce those overheads I I wouldn't I wouldn't claim that we are successful because we are hitting the limitations of AWS Lake formation and Lake formation is a little behind in the curve when it comes to supporting these three um open source table formats okay thank you are there any other questions in the audience so yes uh hello so question is did you done some research in

the market and uh maybe some other platforms have this solution they do uh I mean it it U data breaks is a great example so if if you again U if this was a small te this is kind of came up as a POC that turned into a production kind of use case uh one big Pro one big area that we thought was Data bricks um there

are a couple of other vendors who are building something similar uh like a managed data Lake uh but then again we didn't want the data to go out of the our control at this moment we didn't want the uh account to be in somebody else's control uh so data Delta Data bricks you can still if you have data bricks you can put data bricks in your AWS

ecosystem through Marketplace so it does not leave your ecosystem so data data bricks was definitely something that we considered we also considered uh basically scrapping all of this and uh bringing in um bringing uh bringing in a combined Paradigm which is lak house in which certain of the data before processing is in data Lakes but the moment it is processed or there is certain business criteria that

is met it goes into Data Warehouse uh but all of this we never considered going outside of our control so data brakes was the only choice but there are other companies like on source and a few others that provide data Lake um commercial software if you like to call them I don't know what the right word is but yeah you can buy something from them and it

gives you some more optimizations on L yeah sorry I've overseen one question on slider what's different between data L and Big Data yeah so big data is like the the name that was coined when the data scale got really really big and data lake is just one of those architectural implementations that you see data warehouse is an arch arital implementation data lake is an imp data uh

let's say big data implementation so in general uh it's a bracket term data Lake Big Data where it was invented around 2000s when the scale of the data became IM immense and they just didn't have any other smart way of saying it they just called it big data okay any other questions thank you very much for your insights and yeah this is your applause