Konstantinos Passadis: Microsoft Fabric & Content Safety: Analytics on Metadata

This talk covers an end-to-end workshop on integrating Microsoft Fabric with custom metadata for image uploads. The speaker explains how a React web application communicates with Azure services, including blob storage and content moderation through the Content Safety API. By using tools like Terraform for infrastructure management, the session highlights the creation of a storage account to save images and a logic app for metadata management. The analysis of uploaded metadata is performed through Microsoft Fabric, enabling further insights into user-generated content. The talk illustrates the practical application of these technologies in environments focused on content safety, particularly concerning minors.

[Music] welcome everyone to this uh the next talk uh with uh Costas um and uh let's see I'm sorry I was I actually have your stuff open up and you will be talking about uh Microsoft Fabric and content safety analytics on metadata so that this is really cool and uh so uh Costas let let me introduce Costas pasadas uh is a Solutions architect from epam systems um

he's also a cloud event specialist and specializing in hybrid and Cloud Technologies and uh he has impressive credentials from all major Cloud providers and with a very calm and easygoing personality he's constantly seeking to acquire new knowledge and share his own experiences and expertise and so with that we welcome uh Costas and uh the stage is all yours thank you thank you very much for your introduction

I'm so happy and excited to be here welcome everybody uh my name is constantinos Costas for short badis coming from Athens Greece uh today we're going to speak about um an endtoend Workshop which I have already built the most interesting and time consuming parts for us to save time what we're going to see is how we can import custom metadata into pictures that we upload through a

web UI uh into a storage account this uh custom metadata on each picture is going to be analyzed with the help of Microsoft fabric so it's an end to solution starting from a web interface build with react container images and let's have a quick look on the architecture this is the architecture so we have our uh react web app it is a container coming from a container

registry we are saving the images and we are triggering a function in CP that is communicate with content safety the content safety API approves or blocks the content depending on the content itself so should be violence it is blocked should be something U that is not of the categories that content safety uh checks it will be approved so this metadata along with some user metadata that you're

going to see we are saving as a CSV file with the help of logic apps and then Microsoft fabric helps us analyze what's going on with this data it's a solution that can be utilized for example in learning PL platforms or in um cases where minors engaged so I'm going to pass this very quickly it is a presentation of what I have already done and some parts

that we will do together we have our environment Visual Studio code running with asli and core function tools we can start a react up with the commod that you seeing lpx create react up my web up I'm using terraform that helps me build quickly my infrastructure it's already built as I told you earlier Terra for minut it of course we have to initialize our providers these two

commands are optional but I'm using them all the time fmt allows me to have a nice um identation into my files and validate very very useful in case you have some errors or something is wrong it will inform us of course the terraform plan will do the same and it will plan our infrastructure it won't deploy it terraform applied apply will deploy our infrastructure which uh we

have to add a storage account name and the S toen into the do end files for the react application so it knows where to store the pictures that the users will upload and then Azure container registry two commands just to log in and then we can see our registry name we have to use Docker I have already done this build and then we will push our app

to do registry this is the part that we're going to do together we're going to deploy directly to web up from the docker registry very very useful and very nice this is the interface that we are going to get so we're going to upload photos select some categories that will be stored as custom metadata into our pictures along with approved or blocked um flag from the content

safety endpoint I'm not going to uh spend time here this is my contact details you can contact me I will maybe I will have time to show you this also at the end let me reshare my screen from the start give me a moment please on that because I'm using a virtual machine I'm going to transfer now to my work station where you're go you you'll be

able to see the whole thing yeah I think you can see my vs code so let's take it from the beginning what I have done going to show you now what we have already here we have our uh Azor subscription of course here we have a resource Group I'm going to filter it out to the interesting subscription so we have here our Resource Group let me visualize

that for you we have a logic app plan the storage account one that is accompanying the function apps and the other one that it will be used for our content we have the safety um the content safety end point application research and logal analytics I don't think that anyone does not use that we have to use application insides because we get great insights on our solution the

logic apps is using a managed identity to connect to the storage account and of course we need an application service plan everything is declared through our teram home file I'm going to zoom in for you so you can see a little bit better I'm going to zoom in once more so yeah let me show you quickly our ter form files so the main is the interesting file

okay it's just one file and is very straightforward I'm creating around stream so I can have a random names I'm using a resource Group in West Europe log analytics and application insights I'm getting the keys and the application ID I'm building an con registry of the standard plan we're creating a storage account the course rules are very important because you're going to have trouble when the react

app is trying to write into your storage account so I have already included the course rules to allow everything I could filter it to allow only from the URL of the react app but okay we're creating two containers the uploads where we are going to store the user photos and the content where we're going to to store the CSV file with the custom metadata the content safety

resource an an app service plan with a up and the function app uh storage account of course the function app itself it is a net SARP function that it is triggered once we have uh the images uh each image that is uploaded it is a trigger so I have already built my Docker image so I'm going to push it now to our Azure container registry I need

something that I yeah I have to go to the directory that is holding our Docker file yeah I think it's correct something I added that it's not needed it's the dot at the end because of course it's not needed so push we have to be into the directory of the app I was there but the error was through because of the dot that I forgot to remove

now so it is pushing the react application image into our Docker container into asual container registry it won't take uh much time the interesting part is that we are able to go directly to our container registry here it is and in the repositories we're going to find our image I think it's already built see what Azure offers we can go here to the three dotted menu and

we can deploy directly to a web app this is very very helpful for us to create our mod apps our apps I'm sorry I'm going to name it mod app uh so we don't have to do anything just click there deploy to Azure web app leave the resource Group the same it's okay I can select my plan create a new of course because the other one is

for the function up let's name it B up 02 and and I'm going to add it into the same region that everything else is deployed you can see that it is uh selecting by default the standard plan that's okay it's not uh very expensive but we can always scale down if we want to save on costs everything looks fine the tagging is version one it is a

best practice not you use always the latest tag because you can use version one version two and then you can play around with the with different images because maybe you need to to make an update or you need to make a change uh you don't have to always change the same image you can use tagging version one 2 3 4 10 20 so you can have various

um uh images variations so to say it correctly of your images and play around now we can we could always use container apps but I wanted you to show to see the the this menu that is uh offered by Azure container registry where we can deploy directly now an important detail that it may look uh useful to you is the logic apps thing the logic app it's

very very was very difficult for me to create and I'm going to explain you why when you have metadata on your storage account it's not a problem they are there but when when you want to do something with your custom metadata then things get very very complicated what we have to do is we have to build a specific HTTP rest uh API call that I spent almost

two days to find out how I'm going to build it finally I was able to build it and then we can grab with a for each function for each image into the storage container grab the custom make it a variable make it another variable for the moderation so we have the categories that you saw in the web VI you're going to see it again we're going to

make a demo there create an array string another array for the categories and another array for the moderation so so this is for each image uploaded by the user then composing you can see here we're um this uh these variables and we are creating a new CSV file the new CSV file here is a new CSV table which updates a blob uh we have the ability to

create this logic where we updating but the first time we cannot update we can create so what's going on here this will run only if the other fail fa so the first time where the blob is not there this one will fail so this will run only if this fail very useful uh logic from logic apps and that's why they are called logic apps let me show

you again what is this thing this part will run only if the preview one has failed so the first time where the blob is not there the CSV file this part will fail so it will create the for us and each subsequent time it will update it once someone goes and create deletes the CSV it will rerun because this one will fail you this is the logic

of the logic apps now the schedule can be whatever we want we can do it recing sliding window we will run it by hand so uh I think our web app is ready I don't want to save anything here our web app is ready and the next thing that we're going to do is the function up the function up is the say sharp trigger let me show

you here our code the code was taken initially from Azure uh documentation the thing is that content safety it is a very very uh fresh uh resource and we had changes very frequently so I have to readjust the code each time that a new uh change came to the content safety resource now for example it was rejected uh and or approved and now it is changed to

rejected and accepted small details that you have to be aware of if you want to have your um App working it's a very straightforward trigger you see that what it does it is connecting to the account it uh watches for changes into the upload directory uh container to say it better and once it is triggered it gets the response and it is storing it as custom metadata

into each blob this is what it does I hope we don't have a change now to this end point let me see uh and build our function up we have to go to the Azure menu you will find it here into your workspace projects and you have to I'm sorry you have to right click have to deploy to web up let me find the menu I want

to deploy this into our web app but it doesn't give me the the menu that I want just give me a here yeah this is our project deploy to web up no I'm sorry I have to deploy to a function up this is why deploy to function up it's going to uh bring a list up of our subscriptions be careful you have to sign in from the

command pallet and also don't forget to a login from your CLI it discovered my function app resource and now it will deploy this for our convenience all environment variables are already there via the um uh via the terraform configuration file let me show you in the web app if you go to environment variables you will see that we have already set the con safety API and point

the key automatically from the terraform configuration because as you may know terraform allows you to do that once you create a new resource you are able to um each variable from each new resource you don't have to write it you just refer to it and it is getting into your resource whatever you want to do so for example the point was inserted with this function with this

line with this syntax content safety end point so go get it from our con safety resource okay so we have our uh deployment ready I think we don't we are not missing something here let me open my react up I think I think everything is there and we should be able to see the web interface again this solution can be used into various environments and various uh

um you know can be added as an addon into various uh projects uh especially when it has to do with minors or with uh uh the need to flag content that could be inappropriate and then you can for perform analytics there and see what's going on with our uh app uploads for example okay I choose content safety it could be changed to anything else so let's upload

our photos I'm going to uh find some photos that I have already stored here some photos so this one is a save photo and I'm going to select you know some categories people and art these are also stored as custom metadata into the blog so re is uploaded let's go to the storage account and see what happened storage accounts this is our storage account the other one

is used for the function up to store logs and uh Telemetry in general in the uploads we should be able to see Rita here there it is and if I click on the file and go to the metadata you can see that I have the category these people and art the categories that we selected via the web interface and we have already our moderation status it is

approved by Conant safety that means that it that it is working our trigger I was a little bit scared with the all the changes that are happening with AI and all the services in a have great great great people working into great uh projects but you know changes are inevitable especially when something it's new so let's see here a quick look on metrics and see that uh

that we got you know a request total calls let's see for example total calls be updated let's add a new metric uh data in maybe successful no add data in it's not yet updated or it's too small the data let's upload some more photos nice let's add this one it's a little bit of a violence there okay so if anyone of you feels uh uh not well

seeing this kind of picture I will request you to not uh look at it it's not that violent and you didn't show a lot of things so let's change the categories to people and outside and society and submit it is uploaded let's go out now let's make a refresh here yeah we have already data total calls are 10 and we have 7.7 megabytes of data sent over

to the con safety endpoint let's go back to happened with this Photograph that showed a couple of people you know fighting each other it's it is considered content let's see here what's going on you can see here that the category is what we selected as well we have the moderation status as blocked and we also have what is the uh threshold and the category of the issue

here of the content okay there are four categories in Cent safety it's VI uh violence seex uh self harm and uh another one I don't remember it four categories with different thresholds you can be very strict or regarding the content I am very strict and the severity level is two but it is blocked because we are very strict we are not allowing any any uh window of

inappropriate content so we have already two photos let's upload two more so we can have a little bit of data into our fabric this is a funny one it's always know I think that safety recognizes that picture and uh yeah it doesn't uh it doesn't flag it as inappropriate but let's see what's what's going to happen with this one again it always is surprised me with this

photo I don't know what's going to happen yeah it is approved yeah disapproved anyway probably and safety has a bit of a humor there let's add one it is self harm and people outside yeah let's leave like that two categories so this is the web interface right this is what uh supposedly users were going to use to upload photos so you have to have a mechanism in

place to check the content flag it and you can do do a lot of uh things with the metadata stuff now we have this one let's see what happened with metadata yeah it is self harm and violence so it is blocked with uh out any questions there we have enough photos to start using our logic app and analyze what happened what are users uploading where do we

have a uh most uh information about our categories what do they upload is the content blocked or approved what's going on with our uh you know solution there so let's go over to our logic up again we could have a trigger to run it every day uh once a day every night once a month once a week you know logic ups are very flexible regarding their schedules

I'm going to tell you a little bit here what I have done to make it uh faster because because it is quite a complex logic I told you earlier especially the the rest call in order to get this metadata it is quite difficult to create it you have to make a a call and you have to synx it like like like like like this one so this

line only took me two days to find out uh apart from the yeah apart from that and I think that I have to add here something because I forgot to do it yeah let me do that for you I need the token here so I'm going to take it from the environment variables that I have for the react it is the front end it is the Dot

N and it is this one this one is the token and I think I have to be very careful regarding the synx yeah it's this one and it's starting from the SV yeah starting from the SV imagine imagine how many hours it took me to understand this and why should I add this with uh without you know the question mark or with the question mark and all

that stuff so this is the token I think I have it correct yes I think it is good yes everything is good so this is the rest API call I don't think there is anywhere else that I must add it I have changed the storage account name and uh I have created a new API connection from logic apps to the stage account so what I did I

just changed here the subscription name the resource Group name where it was needed I I created the same connection with the one that I created the first time so what I did I exported the first time this logic app and I just pasted the code here so I didn't have to create it step by step via the logic app designer this is quite useful and I'm glad

that the logic gaps has the ability because this was the harder part of everything you are saying right now so let's try to run our first uh workflow here I think everything is good and we're going to be successful let's hope so what uh it's going to do it's going to connect to the storage account using the manage identity it has the permissions of storage blob data

contributor because once it get the blobes from the storage account that you saw earlier with the pictures it will run this logic so for each you know get the metadata add them into variables the category and the moderation create the arrays create the logic to compose this uh data for us to be able to have it as a CSV variable and create later the CSV it is

not easy to do that again because saw if you saw it we have multiline data into our storage accounts and uh apart from separate metadata we have also comma separated values inside each row so that was also another challenge to separate this data and yeah I think it was good let's see what we have into the content container should be a CSV file and it should have

data yeah so let's have a look here and then we off to fabric I think we are on time okay this is our data right two columns but we have multi data into each column so fabric is our friend here that uh is allowing us to this no it this app. fabric. microsoft.com you can get it I'm not sure but you are going to need the trial

license because it uh U um enables you to do more stuff and run the data science thing and all that nice stuff I think this is the user I should and I need a password to here please please let me add my password it won't allow me to add my password and I'm not happy about it I probably has to do something with uh the live streaming

multifactor yes now for anyone that have already worked with synapse or similar data you won't be find it hard to connect uh your uh pipeline to a source U uh destination uh sync we have our workspace here we have workspace and we need a a data uh Warehouse let me see if I'm correct it yeah this is the one yes we need a new Lake housee here

okay you have to create a new lake house because you will be you will not be able to create the um theing model I'm going to sign out and connect with another account but uh I don't think we will be able to see that part let me try I'm not sure that we will be able but I will be able to show you what we're going to

do what we're going to achieve I think this is the user and here fabric data warehouse yes this is so let's go over directly to our copy pipeline we have to build the pipeline right what we need to do is to connect the fabric workspace with our storage account so it will be able to grab the CSV I have already here our pipeline so what we can

do is we can change the source and create a new connection forour storage account storage glob and I need just the account key so from the storage account get me the account key get me the account key and put it here update yeah I have to change I think the storage account name I think and afterwards once it gets this CSV data it is running a python

script a python notebook to say it better that uh is let me show you what the that does it's the python analysis this is a very very easy thing to do you you just use python and a notebook directly into the pipeline so it gets the files that you have imported from the pipeline and uh it makes the split over the categories uh multivalue uh rows so

you are not you don't have to worry about the comma separated values into your CSV so that's all let me see if the copy line if the pipeline yeah we have to create a new connection here uh the storage account BL yeah BL the account name or URL is I'm going to get it from the endpoint here the blob service is this authentication account key and I

can get this key now copy into my Pipeline connect and it is ready I can now run pipeline it will create two files uh uh it will create the file into CSV and then the python not will create a secondary so once we are able to see the secondary file which will the uh let me show you it will be let's go over to the lake house

and see it's better we will have these files refresh I think is this one yeah first file and this is the second so once we have we're creating yes we are creating a semantic model and then we are able to perform analysis directly as you can see we have the split here and what we are going to get I'm going I'm not going to um get more

time you here so I really hope you like it and I think I try to make it as as understandable as possible uh usually when you put the for you are getting uh some nice stuff so after the data is has come to our fabric lak house and with the transformation we are able to create visualization through the fabric uh resource because this is the future of

data analytics guys you know it uh a managed resource that offers everything and the new uh things are coming uh almost every week so thank you very much guys uh I think uh I can close now if you have any questions I'm here for you yeah we have about three minutes so if anybody has question although I think I did see a question and I posted on

the you can find on the Q&A so here Costas uh um maxims Talus asked this question what component is responsible to set picture status and what AI model moderate the photos the the AI model uh was the content safety endpoint and the resource it's named content safety so you can uh also use it for you to understand how it works through the content Studio but I created

it directly from terraform and I was able to communicate with it via the API endpoints so it's the content safety resource and the first part of the question could you please rephrase it for me let's see okay I can see it I'm sorry what component responsible to set picture status yeah the first component is the user input that he is selecting the categories so it's a way

to for users to add some extra data into the lob and uh the secondary is the content safety so we have metadata of the user and the content safety blocked or approved and why it is blocked in case it was blocked I I thought that I think that I I answered that okay I guess maims if you have more question you can we still have like a

minute you know and 10 seconds left if anybody has question on maims you want to uh you know augment your question uh yeah thank you guys for being here yeah I'm not seeing anybody an honor for me great here are my details you can contact me for anything that you want to uh you can find my solution in GitHub don't uh hesitate to contact me for anything

you can find more solutions on AI and a integration have a great day guys and uh thank you for everything bye-bye for