Mikhail Larchanka: Seamless Automation: Transform Your Data Processing with Azure Data Factory

This talk covers the challenges of traditional cron jobs for data processing and introduces Azure Data Factory as a robust alternative. The speaker, Mel Lenka, describes the limitations of cron jobs, such as manual configuration, limited functionality, and scalability issues. He explains how Azure Data Factory streamlines the creation of data flows using a user-friendly interface, allowing developers to easily set up data automation without extensive coding. The session highlights the scalability, integration capabilities, and advanced transformations offered by Azure Data Factory, making it an attractive solution for modern data management. Although there are cost considerations compared to simpler cron job setups, the benefits of ease of use and system management can justify the expense for larger organizations.

[Music] ladies and Gentlemen please welcome our next speaker Mel Lenka presenting the topic seamless automation transform your data processing with Azure data Factory hello everyone welcome to my presentation and this is the description of it I don't think you need to read it yeah it's just for people who missed everything on the website let me troduce myself um my name is m the person the left one

in the white hoodie here and I work in a company called scitec in Netherlands uh it's a let's say out staff company and my current client is bank I have yeah I'm 30 eight years old I'm Belarusian uh I have experience around 20 years so I built my first website in 201 and I live with a wife kid cat um yeah I made this presentation a bit

early so hamster is already reap sad but and a couple of fishes still have them uh I have also social networks you can imagine it's on the website on the slide I have here YouTube Twitter and telegram but it's in Russian so if you don't speak then just don't go our plan for today is uh we have introduction I just finished it we have challenge you will

see what is challenge we have a solution for this challenge key features of the solution benefits then we will talk about the price of this solution and the conclusion um I feel really bad friends of mine said don't tell you have a well a Handover I have a Handover challenge uh where's my water yeah thanks lot usually when there are two types of people who can speak

about software development regular people they are not related to the development they users and when they talk about software development they usually talk about like so uh social networks different applications they know they don't know how it's built inside but they know these are the things I'm using and this is software which was developed when second uh second group of people who talk about development if they

talk uh they are literally developers so we are and we usually talk about development from the perspective of performance or like creation of something beautiful maybe design U UI you know all these conferences so people just come there and like yeah let's talk how good and how quick we can build an application and then we also discuss Frameworks we discuss libraries on top of this and other

stuff which is related development but there is one more thingy in the software development which people barely talk about it's uh it's data flow or data transformation or data something so when we see development we see interfaces we have we know there is some kind of backend doing something when we do a request it sends information but also we have a lot of stuff which happens and

we don't know we don't see it's happening we don't call it uh this stuff we don't like interact with this uh usually it's called Data flows as I said and it's something which automatically happens on the back end oh back y still here and and uh we don't discuss data flows and the conferences usually we don't discuss them between each other they just build somewhere there and

they just work and the most known probably example of the data flows is the cron jobs whoever wrote CR Crown jobs yeah simple ones yeah you just write a script you set up the schedule for the script and it just runs in there and there are a lot of problems with them but usually it looks like this you have your Source somewhere you have your data then

you have your script which once day couple of uh times per day just runs grabs data from The Source transforms it in a way you need and pushes to some kind of Target and Source it can be be whatever you want it can be database rest API it can be Amazon storage Asia date L FTP server whatever uh Crown job as I said it just a script

which does all these data transformation and manipulations and the target can also be whatever you want it can be another database it can be some API FTP server SSH connection Etc and then when you work with this you fac with the issues so to build a chrome job it's not a big deal you just write your script you have Source you have targeted works but then uh

you if you write something complex and you have issues with it you face with the problems so you have limited functionality of course you can write whatever script you want and it will be working but um how to say it it's really hard to combine if you have many sources and many targets for your crown job for one job it's really hard to combine them and you

are kind of slightly Boom come on then when you work with the Chrome jobs you should do everything manually you need to manually write it you need to manually configure it and then it just uh especially if you are let's say not really skilled in the Linux or Unique Systems it can be also a problem Oh damn hang over monitoring and login uh cuz crun job is

just a script which runs for example uh so if you want to log it you need to implement some special mechanism if you want to monitor it you just probably cannot do it ability and high availability this is also a pain if you for example rent some kind of server and you run Crown job on This Server it means you are limited with the performance of this

server and if you need more performance or more resources you cannot just get them from the server rent and then it if you just rent a server and it has an issues your provider has an issue or you just don't fit in or like it has some downtime uh so if provider doesn't work your application also doesn't work your crown job doesn't work because because uh computer

which is is a is a host of your script is just stopped and also limited error handling yeah the same reason as we had with monitoring and login just hard to do with uh with this sitation when you have just a script which runs in some specific period of time uh and we try to find the solution there's everything in software development we're trying to find a

solution for the problems we create ourselves so we find a thing called Cron job and we decided yeah let's just do Crown jobs and then we found it's not a bad thingy and we started to search for the solutions and here you can see the list of probably most popular today Solutions except one which I will talk about in the future but we have uh things created

by different companies we have AWS glue data pipelines cloud data Fusion Oracle data flows so we have some things which can replace Chrome jobs because they're ideal and I'm going to talk a bit about asure data Factory it's another solution built by AIA by Microsoft of course which allows you to create uh scheduled tasks or even non-scheduled tasks uh which will do a data flows which will

just transform the data from the sources with the specific requirements and push it to the Target and usually when we talk about data factories or some other analogs we are thinking about something like devops thing or data platform engineers and cuz I'm not a data engineer platform engineer or devops I will tell you about this from the developers perspective and I think we should start from Pros

what's good in the data factories or analog of the data factories why we want to use them why I will show how they works it work uh they're really like easy to use and the thing here is you don't write your code in data factories as you did for uh Chrome jobs you can use interfaces of course you can still use manual writing you can write a

code or configurations but you have interface and with an interface you have just a drag and drop thing which is much easier so if you are not really experienced developer you just turn on computer go to aure to ADF aure com and just drag and drop elements and provide URLs so it's much easier than just opening a file and writing the whole big script in scalability uh

cuz it's a aure so it means if you need small amount of resources it will use small amount of resources but if your demand grows it means that Asia will automatically uh extend your resources and you will have always enough and if we will compare with the chrone jobs I mentioned before you have hosting with 512 megab of RAM and you need one gigabyte you will have

52 it will not just scale integration uh when we're talking about Asia we usually assume it's a huge huge platform which has a lot of stuff built in and it means that when you build a data Factory you can integrate it with everything Asia has inside uh with the cron jobs if you want to integrate it with something which is outside of your scope of your you

need to integrate everything manually so you if you want to connect to source and it has some firewall you need manually go to this firewall that's bad much alcohol uh uh so in Integrations uh and it's pretty simple then monitoring and management aure Microsoft provides you all the tools you have login systems you can attach additional login mechanisms top you can manage uh your application you can

run restart it and see how it works again you can just extend it and send to different uh like within the Microsoft you can run it on the different servers in different countries which is pretty cool and because uh Asia normal uh is a big huge company it also provides you ability to run it closer to your consumers and if your consumer is in States for example

it will be run close to the to the uh to them in States if it's in Europe it will be run in in am Amsterdam for example or frankurt uh Advanced Transformations uh the thing with the data flows is that you can have multiple sources and you need to combine sources based on some uh things uh and you will have kind of extended data for example you

have whatever information about some kind of let's say about some tool and you have additional information about this tool in different in some other sources and you want to grab this all information combine it all together and show to the user in the end so with uh with Asia or analogs from AWS or Google Cloud it's usually much easier because they have already this built-in Transformations done

by other developers you just reuse them and I think uh it's it's better to see the real example instead of like listening to me so I will not show you live coding it's always during the presentation it always breaks so I just made a lot of screenshots and we will go through them so that's how our Crown jobs are looking again we have the source we have

some kind of script which is your your job and then you have your target where you push the the result result of the transformation when we talk about data Alex uh you have just data Factory instead of CR job so the the the overall idea is the same steel Source something here Target created two endpoints one is a source endpoint one is a target endpoint hosted it

and and for the source for example for the example I've created basic house and for the Target we uh oh it's Source information and for the Target we have another end point with O2 in for uh authentication and we need to run this uh this example I will show once a day 1 this is the interface of data Factory I don't uh I don't have my own

account in there so I was using my work accounts that's why I blured it slightly and you will see I blur some other information here and this is uh an interface of the data Factory when just when you just enter it it's completely empty and on the left you have navigation where you can select which exactly you wanted to create and on the right side where the

blank space is you just build your data Factory in the in asure data Factory you always start with source so if you will go for example to the data flows uh on the left in in the navigation you can create the whole flow of the data from source to Target and you always start from the source of course because you need some information to work with uh

as I said we have endpoint which requires basic authentication to to grab the data and as soon as you create a source in asure data Factory you can select what is the type of the source and you have whatever you want in there sip snowflakes uh Parky files and cuz we are using in Endo we need rest so we can just type rest in this uh list

select the the only option we have and then we have configuration for it for it and this is pretty simple configuration so you provide description if you need Etc then you provide URL uh do I have some pointer thanks so we provide here URL of the endpoint we want to connect to then we select authentication time type in our case it's a basic [Applause] one then we

just provide information about this basic authentication for example username and password and we feel it here we click create and now we have connection to our source uh there are multiple options how we can connect uh but as I said we use uh base authentication and as soon as we've created a source connection we can already see how it works so we can click this button here

data preview click refresh and you see in our case my example is about currency exchange changes so we figure we receive a lot of currencies from the API and then we need to filter it out and find like we need to do some transformation in our example we will just filter it out find the ones we need it will be Euro in do US dollar and push

it to the Target so as soon as we connected to the source and click data preview we can see everything we are receiving from it and you you can see I have a lot of currencies in it so my next step is transform it just to select two ones which which I need uh so I click you can see after the sord there's a plus button so

I can build the whole flow here I just click plus and I can select whatever I want to do with the data I have for example I need to filter it out because I only need two currencies so I select filter and then that's where I need some kind of code to write but in uh in comparison with Chron jobs I'm not writing the whole code on

a python or whatever PHP I'm just writing here what I want to filter out I'm putting like body. cod based on the structure of my source uh cuz it's ajon I know exactly how the uh the the past looks so I just put here body. code equals Canada or USDA or SSS Frank uh it can be Euro so I'm just kind of selected here the currencies I

need from the list I received and again I can click data preview click refresh and I see how the data transformed so I had source with a lot of currencies but then I wrot this simple filtering where is this one and now when I click preview I can see that all my currencies were filtered out and I only have the the ones I need bad data transformation

uh in in Chron jobs or aure data vory it's not only about like filtering out the data it's also for example can be that we need to change something with this in this data rename columns or maybe rename content even so for this we also have we click plus so we can build as long as uh data flow as we want but we click plus and for

example select select and in this case select here allows us for example to rename name uh columns we have so I just put here then just change the name for example of the columns I had body code renamed it to C body price to Euro renamed it to P click preview and I can see now my Json my like final result Chang it so I have cusd

whatever p and this is price of currency uh so if file return back come on that's what we had when we just source a lot of currencies with a lot of columns in it and the name of the columns is code name and price to euro after my transformation I have three currencies for example with two columns only which is cod and price but Cod is named

C and price is named and cuz I'm not building a complex one I'm building a simple one next the next step I want to do is just to push the transform data to my target so I click plus again and select sync with the data Factory there is always a rule you always need the sync in the end so you cannot create a data factory with a

data flow without the target so as soon as I selected syn I can uh set it up also so I can select type of and cuz it's also end point for me I'm selecting rest and this a different one cuz we have o two for the Target so I just set it up here and you can see it's a bit different so I selected all two now

I have token end point client ID client secret and a scope I need to provide to to receive token as soon soon as I enter this data I can again click data preview refresh and it will push all the data so start from the source from the beginning go through the whole flow and push the data in the end and also I can see the result in

the preview now I have a data flow but the problem with the data flow data flow is not uh is not the complete Factory the main uh thing in uh asure data Factory to like create the complete application here it's a pipeline so now we need to create a pipeline which will reduse our flow also pretty simple I I I think people who worked with it they

already familiar uh how it works so in activities we can select data flow cuz we've created one it will provide us ability to select the data flow we created and of course later we can we can even extend it we can combine data flows we can combine pipelines so I've added our data flow but I can add another one here and I can can connect them to

each other and also I can connect it based on the success if it was successful like a green scene here or if it failed if it failed I also can rerun it or do something else so I have all these options in the my in my Pipeline and you can see I just have one data flow I can uh see the name I can uh integrate with

some R runtime and now my data flow will work the only thing I need to do I can or manually run it debug or I want to do it once per day so I need also to set up how often I want to run a data flow yeah I can see if it successfully worked but then I go to the left menu there is item triggers that's

where I can just select how often I want to my data flow to um data Factory to work and you can see it's for the Chrome job Chrome jobs usually manually set it up you have these stars and you put like star one star is a minute second star is like hour and here I can literally in the interface selected like every 24 hours just run my

pipeline start from this date and I can even provide like uh end date if I want to stop and like to work just for a few some period of time and stop all is done man uh via graphical interface which is pretty cool now we have a data flow it completely works and I showed you a simple one on the screenshots but you can really build something

like this so we have multiple sources they do something all together and then pushes the data you can build something like this when you have one uh Azure data Factory but you have a multiple tasks happening there in parallel you can see this huge one is one data flow this one another data flow but they work together and as soon as uh it's triggered they both start

to work uh and that's time for the conclusion and you can see here in the conclusion I especially highlighted con because uh there's one bad thing about data factories let's take an example real example we have in our project so we have function data function it runs once per day it has three simple mutations One Source three mutations it filters out currencies it's also about currencies it

filters out currencies it selects uh it renames a couple of columns so source is 100 kilobytes and the target uh receives we push to Target 5 kilobytes pretty simple one and the problem is the price you can see here 8 7 82 per month for the crown jobs for the crown job which runs once a day receives 100 kilobytes and pushes five somewhere pretty simple application and

price 87 per month and if and if we will just write a simple PHP code doing the same stuff put it in a chrome job and run once per day it will be $2 dollar per month maybe three for the very cheap hosting so this is the biggest problem and the biggest issue with data Factory but from other point of view you have a really simple interface

to build it you have Microsoft you have scalability e so this is the biggest problem we pay like 40 times more we can pay uh if we use just a simple hosting but as I said you have a really scalable complex solution which is pricey but again it's much easier to build much easier to control much easier to Monitor and log and check logs uh and maybe

price is not so bad because if you are a big company like a bank I work for you don't really care about the price questions