About this talk
This talk focuses on the Edge framework, an open-source platform for embedded AI. The speaker, Arena, alongside her colleague Axel, describes how Edge enables AI capabilities directly at the source of data production, providing benefits like real-time processing, privacy protection, offline operation, and reduced energy consumption. The framework addresses challenges such as fragmented solutions and interoperability issues that exist in the edge AI domain. Detailed features of Edge include high interoperability with various deep learning frameworks, support for CNNs and SNNs, and innovative optimization techniques such as post-training quantization. Additionally, the framework allows for easy manipulation of computational graphs and code generation for diverse hardware platforms, showcasing its adaptability for various applications including industrial IoT and healthcare.
Full transcript
[music] Hello everyone. So my name is Arena. I am a researcher at CA list of France. And today with my colleague Axel we will present uh edge which is this uh independent and open source platform dedicated to embedded AI. So as I mentioned our focus is in edge AI. So what that means integrate AI capabilities where data is produced. So it comes with uh some key benefits.
Um and you can mention uh real time processing uh means ultra low uh inference time uh privacy protection because data and model is stay on device uh offline operation we usually have no internet dependency and also cost reduction with uh energy consumption in terms of energy consumption and infrastructure in general. And there are many applications that uh we can have with uh with edge AI. So just
to mention a few we have industrial IoT mobility uh um special uh special applications uh aeronautics healthcare and you can name that. The problem that we find in this domain is that uh it's how to call a structure uh limitations in terms of uh the to chains that exist uh that comes with uh edge AI. So you have a lot of deep learning frameworks. Uh some are
open source and some are are um closed and uh they come with runtime dependencies uh some limited interoperability and uh yeah as I mentioned some closed sources execution. Um we have also a lot of uh uh optimization compilers that have this really low level approach but come with a uh usually with uh extremely uh complex pipelines and they are kind of difficult to to be appropriate of
and um we also have a lot of hardware specific SDKs uh which which also are usually uh black box. So all of that leads to uh fragmented solutions uh some uh expertise scarcity uh innovation bottleneck because they autoerate a lot together and uh yes it can be really time and cost uh consuming so that's uh the con context that is behind uh the development of age is
that deep learning framework dedicated to embedded AI embedded AI and that is uh it's a end to end framework work um that that means it includes uh optimization and deployment. Uh so it's essentially built around a modular library uh with a light core and uh a system of plugins that uh you adapt accord to the users uh needs and it has a real life field dependencies. So
and also of course it's uh the the entire code base is open source and is hosted by the Eclipse Foundation. uh we've adapted the license to uh uh proprietary software and the uh um contribution. So I will go through the the the most important uh uh steps of this framework. So starting with the design as I mentioned you have this uh a high interoperability with a different
uh frameworks and the formats and the uh conventions of of the machine learning uh tool chains and support most of u the most popular CNN's uh rens and SNN's architecture. So it's essentially like uh what is uh the state-of-the-art and we have uh this uh really efficient uh intermediator representation where it's uh it um uh allows us to easily uh manipulate our computation graph. So the machine
learning model that is behind it and um and also have a lot of tools for in-depth analysis of our of our algorithm. So as I mentioned for interoperability we have this uh um a strong interperability with onex which is a format for the for neuronet network has changed that that what the acronym means and then in edge we have this an extensive coverage of the operators that
are supported by onix we have also an integration with onx runtime and uh we also support so uh the import what we call is the is that the uh the the fact to bring uh oper uh on operators inside our platform and also the export that is that means uh u outputting the operators that are present in edge uh to the onx uh standard so it's a
so that are um functionalities that are really um simple to use I mean I I have an example here of some with some uh line of codes you can do that uh really simply. We also have this uh compatibility with u uh Google's model explorer that was adapted to uh interpret our uh internal representation edge internal representation but we are also developing our own um uh user
interface to visualize models. So yeah, you can just um you can simply explore your your computation graphs with uh in web interface really easily and um understand the different uh formats that we have. So, so this is an example you can just uh uh we can uh follow your graph search inside your graph with um using um reject expressions. So, and then uh uh pick up some
specific nodes like you see here we are uh adding a color to nodes that are for certain types and we can afterwards uh I don't know manipulate them in a particular way. So just an example and yeah so some of the key features of edges uh internal representation. So we have this what we call a transparent graph representation the which means that the compute graph is not
inferred from the user's code the user has to explicitly write the graph. Of course uh if you are uh talking about interoperability of an or the model already exists you don't have to do that of course but if you are creating your graph and then you want to be sure that uh uh I mean uh the our framework will not uh suppose anything it just has to
be uh explicit written and uh yeah just we have what we call this mood paradigm abstraction model where we can represent um different uh data structure Sure is not not only uh tensors which are the most common but we can adapt according to our needs. And then uh you have this um what you call a aeros graph model where you have multiple uh granularities that can be
uh handled uh um in the same graph. Uh and we have this functionality that we call the meta operators. So you can create bigger operators uh that contain smaller ones and that can be pretty much anything that you want. You can create an operator that doesn't exist that you create as long as you define it uh according to the to the to the to the rules of
of the framework. So yes, so just on a really simple example where we can see that uh just creating uh what you call expand a meta operator is just um exposing what is inside that and that can be done with a few lines of code and then you can also create your own operators. Here's the line uh dow here where [snorts] just to say that uh my
operator is composed of these operators and that can be done in a single line and then yeah I was saying it's really adapt so adaptable so you can transform your graph to match the granularity that is required by your implementation afterwards for your needs and then so yeah you have what you call uh some recipes that we um that we adapt your graph according to uh to
the to the to your specifications we we also have what we call generic operators that uh it is really useful for example I mentioned uh the interability with onix but um let's suppose as I mentioned we don't have support for all the uh operators that exist but if you If in your model you find an operator that is not defined inside the framework, you can simply treat
it as a generic operator in the first place. So maybe just to test if you can uh work with your model inside your framework, our framework and then you can afterwards define in with uh the Python API uh simply said what this operator that we first identified as a generic one is made of. So it doesn't break your workflow and and it's really easy to do. And
also as I mentioned um the graph inside edge it's searchable. So uh we can uh you can just uh look like in with um uh expressions uh you can look for uh some patterns in your graph in a very simple way. And yeah, so you have this um you have this what you call it's a clean uh transparent uh uh representation of the graph and uh you
can just like here adapt the graph uh adapt the memory layout, apply uh quantization uh um techniques uh cast your operators for different types. You can do uh pretty much what you want with uh with your with the computational graph that is represented uh inside edge. And you also have the uh the scalway model uh which you you essentially uh get uh the real early and late
uh logical uh uh steps of your graph. So in here you can see uh um a representation of uh the the logical logical time logical steps of uh when your when each operator is u ready to be executed. So it's just a a static scheduling uh that is that can be obtained from uh pretty much and a composition graph that is loaded inside So as I mentioned
we have u a lot of analysis tools. So we can uh obtain uh uh before before deployment uh in the moment where you are uh studying a model you can obtain uh number of uh uh statistics about uh the number of uh um inter operations floating pointing operations and yeah there are a lot of uh of um features in this sense. So what I mentioned was the
part uh what um is part of the design uh umbrella let's say and then in the in the second part you have uh uh some u tools for optimizing your uh machine learning model your computational graph. So as I mentioned some recipes to uh simplify your graph. We have a techniques such as uh quantization uh post training and quization our um um our training and also um
techniques for tensor uh compression that are implemented in our in our codebase. So yeah was mentioned about the onx uh it's a inspired by the onx cleaner something that exists. So essentially is a is [clears throat] a collection of um of routines that uh can be applied to a computational graph to simplify it and then um we have different levels of uh of optimizations that can be
applied. So what we call the training safe accuracy safe or for performance safe. So it it will define the level of um transformation that will be applied to your graph. So when you are in a training safe uh model is just you simplify uh some uh operators but uh uh keep information that is required to train again your model in case you need to to do this
operation and yeah so that can be done automatically and um I mean the user can just say simplify my graph or um which uh each routine can be uh called indep independently. It really depends in the level of uh control that the user the user uh wants to have in their code. So yeah, some example with quantization quantiz uh quantizing sorry your network with um post training
quantization or or before uh training quantization. So we have this those are techniques that are done um as most of everything that I mentioned with some uh some lines of code that are really simple and uh they reflect they are always reflected in our code. So as I mentioned here when you do some steps you can see the new operators that are that appear and everything can
be traceable back. Yeah. So per deployment we have a big uh um uh farm of of boards. So it's uh it can be uh it's it's pretty much uh portable to and um and a microcontroller that supports uh C++ uh code. We also have some some work on um a trustworth AI. So that that um validates our approach from the beginning about uh having this uh uh
cut that is open source that is transparent and that is um able essentially and then you also have some features to uh validate and benchmark our framework. So the the the output of the framework of course with a different uh frameworks of of with sorry different frameworks of the state-of-the-art. So yeah just concerning um code generation that's called the export part of of um of our tool.
So essentially pretty much is based on uh templates. So it's have a general templates and uh the cut that is generated it's open and transparent. It's uh it's not a binary developed and you can you can um generate uh different codes for different um different um uh hardware and uh yeah so we can also uh integrate uh low-level uh um primitives in our code if uh we
know uh that we are going to uh use it afterwards in a specialized hardware So it can be easily adaptable and then yeah so we have this uh each we are based on templates as I said and each kernel what you call the the the the implementation of the operator it comes with a definition that we call the specification that defines the data type data format dimension
and everything is adaptable. So yes, I was saying here about the automatic graphical um adaptation when you say uh that we are going to export this model to uh for example in here in the example is the to export it just generically uh C++ code. Um it will it will automatically adapt your graph. I mean you can tell it to uh automatically adapt your graph to uh
better match uh this uh this structure and then yeah of course if you say uh for example that you know you know you have a an hardware that has u some um uh specialization to do a matrix multiplications you can turn the framework uh please adapt my convolutions to matrix multiplications so you know it can be uh uh it will be uh useful later. So that's something
that you can do uh as well. And yeah, so you have some uh other adaptations for uh I mean tiling uh match specifications, convolutions. So pretty much in the in the idea to adapt the code to your hardware and then you also have uh this uh his static memory allocation. essentially everything is is is uh static when it comes to code generation and with uh what you
call the memory manager. So it's essentially this uh a view of how your how the intermediate tens tensors and of the tensors that you're using in your computation model how they behave I mean how they are consumate. So here is it um is again in the idea of the logical time. So when that you will be needed uh during uh inference time. Uh yeah we can also
uh do some adaptation uh regarding this. I have an example just after. So yes this is essentially uh the export system that we have in N. So we have um as I mentioned this uh generic uh C++ code and then uh we can adapt it to different um different uh hardware. So there are a lot of those uh here those um uh collaborations that were done by
our partners. So that have joined um and that are contributed in our uh in our um uh open source uh uh codebase and then yeah so it's a it's a it's a framework that is is done to be u um complemented. So just an example here of graph ting tiling and adaptation for a specific use case. So here is for um passive acostic um monitoring and uh
the the the idea is that they had this um this model that was kind of that was uh too big to fit in their architecture that had a limited uh memory size. And then with edge it was possible that was something that didn't exist uh um in a first place and then the user was able to uh just uh implement with edge the need to uh cut
cut their graph in two parts and then you can see here that the memory uh the peak memory was reduced just by modifying the graph in a really uh customized a way. So, so I think that can be really difficult to to be done in in frameworks that are uh just uh as and then yeah so about the about the benchmark model we have this um this
integration with different frameworks for on the state-of-the-art where you can uh compare uh the the uh inference time of of a entire model or offer different operations operators individually and then you can compare for an index runtime pytor G TVM and uh yeah which is already done so yeah so it's something that uh can can be used to to test uh um your uh our approach with
a different specialized u uh framework of the state-of-the-art so yeah just to complete this part some use case that were enabled by edge so in Here we have the different partners that have worked in in with that. So there there are a lot of use case. So um the effect detation and classification uh as I mentioned the passive acoustic monitoring. We have some work on hardware design
that is done in our our lab in France. Um with the energy distribution uh company you have this uh some work on uh heat pump monitoring and then so most recent ones with visual uh based landing with uh aeronautics um stakeholders and then yeah some um sometimes uh defense work and um yeah even electronics with um smart uh sensors. So some perspectives and conclusions before we go
to the uh uh tutorial part. So yeah so I have some upcoming features uh essentially this part of uh we are really working on the on the export system. So the goal is to be able to really uh seamlessly uh generate a code hardware. So which has uh which you need uh different libraries and I mean different even uh um ways to to to deploy your code.
We are working on quantization of transformers uh based models. Um you we are also working on the integration of compilations to chains to to make our our framework more um I mean to go to this uh really low level uh um optimization and uh yeah some uh essentially just more support for different uh types of neuronet networks and different types of of hards. So yeah, so our
project is it is a really large uh it's part of really really large ecosystem with a number of collaborations. So we have academics, industrials um mostly in France but also a lot in Europe in general. So yeah don't hesitate to if you if you want to know more as I mentioned our work is uh the entire codebase is hosted in in Eclipse. uh we have um a
documentation page that is uh always uh being uh being improved but yeah so it's also um hosted in Eclipse uh we have this um matrix chat that you can join and uh if you want to to work um be more involved in the platform and you can where you can directly interact with uh the developers and uh yeah so just uh my reminder of the different partners
that we have and uh yeah I believe you can go to the tutorial part and then afterwards if you have any questions you can take that. Yeah. So you just throw uh >> Yeah. >> Yeah. Thank you. Uh can you hear me? Well, cool. Uh, yeah. So, as Arena said, I will just present you um um well, get on the the walk of the whole framework uh
but on a more practical way. So, I'll show you an example on how we actually use uh H. And just first of all, I just wanted to show you um the actual uh the actual documentation. So, >> [snorts] >> Uh yeah. So I just wanted to show you the actual uh documentation that we try to maintain. Uh so we have some uh for instance some tutorials uh
that explain each core part of the framework. uh and the the idea is for the user to uh quite easily get into the platform and so there are some tools that are more focused on each part of the the framework and the one I'm going to show you uh now is uh this one which uh as I said gets on the whole workflow of a simple a
simple example >> yeah we have a a binder here which allows to uh actually run this example uh which is a notebook uh this example from the navigator uh but actually it takes a few minutes to to launch. So I just will show you from uh my local VS code here. So uh yeah so so on this example as I said we will get uh from the
importing a nonx model uh to exporting it in a standalone project C++ project then you can then compile and run and going through um manipulating the graph uh playing some and so the the model that we are using in this example is a lunette model uh it's a really simple model which uh classifies handwritten digits. So basically you you write uh a two and it will just
classifies it as a two. Uh so here um oh yeah something that I've not said yet uh so in order to use the framework you can just get into the gitlab and clone the the codebase compile it and use it. uh but you have we have also uh pip uh packages so you can just install it using a pip which sometimes it quite uh uh handy and
uh yeah so here we just have to okay so here we import some uh some libraries uh as Arena said we uh so the the framework is quite modular meaning that we have uh module for each uh core part uh of the the code. Uh so basically here we have the core module which handles the core parts and then we have the hback CPU which uh um
which has the CPU implementations. Uh we also have a hback uh CUDA in order to handle GPU implementations. uh the age uh handles the interoperability within with anx eventually the age export CPP to uh export a model uh in a CPP code. Um and this modular uh modular uh approach handled to um helps to add uh quite easily your own modules. If you have um I don't
know maybe an optimization technique that you want to add to the framework, it's quite easy to create your own module and to plug it uh on on the existing framework. so here we uh import the main module so the edge core uh and then we can import the edge back and CPU to get the implementations uh and eventually the edge of Unix to be able to import
our model from uh from our Linux. Um so basically here we just download the model and then we can just load the model using this simple uh this simple function here. Uh and so as Arena showed we can visualize the model. So this is kind of the idea of a transparent uh uh transparent framework is to be able to actually see at any point uh what your
uh model looks like. And so as I said this is a quite simple uh a quite simple one composed with uh only three fully connected layers. Um and so once you have imported your model you can apply some transformations to the graph. So there are a lot a lot of a lot of transformation that are quite basic that we apply most of the time. uh for instance
here we may want to remove the flatten layer uh because the flatten uh in at age it's handled directly by the fully connected layer. So basically here we will apply um this recipe to remove this layer and this is done again quite easily uh with one line of of code. There's a lot of uh log logging here and that's just because I didn't uh act I didn't
deactivate the debug mode but uh usually it's it's less uh verbose uh and then we can visualize and we can see that the uh flatten layer hopefully has disappeared. So this is kind of the idea to be able to quite easily manipulate the model. Um so we have also uh static analysis um uh some function that allows to see actually which layer takes the more uh compute
power and so this is actually what what we can see here. and then once we have simplified our model we may want to run an inference. uh and so here we simply create a tensor. So this is a seven here and I think in this first part yeah we load the the the results we uh which uh come from PyTorch. So we will compare us with and
here we compile the model meaning that we will uh prepare the model for the inference and this actually does three uh main steps. The first one is setting the implementation. So we we will say to the model okay uh you will use the CPU implementation that we just imported in order to perform the inference. Uh as I said it could be the CUDA implementation as well. uh
then we set the data type uh on the model. So here it's float 32 and eventually we uh forward dimensions on the model. So we give the input dimension which is here is a 28 28 by 28 and it will just forward these dimensions through the whole model. Um uh eventually we can check the graph uh validity meaning that uh we can check with uh the given
implementation if uh this is supported for all the operators of our model and so this uh returns a valid so it's okay. uh then uh then we can run the forward and this uh gets first bar by by creating what we call scheduler which is uh kind of an ordered version of the graph. So it will just schedule the uh and eventually we run the forward uh
um and get our output here. Uh and so we have a seven as a prediction which is a correct result. Um and also once we have awarded we can uh visualize actually which layer took the longer to process. So this could be useful. and then we start thinking about exporting our model and often we as Arena presented we want to to apply some optimization techniques. Uh the
maybe one of the most used one would be the quantization. So changing the the the precision uh of the of the model. So getting from fraud 32 to uh integer 8 for instance. Um so here we simply clone the model and um we we're going to uh not sure yeah okay so we're going to apply the PTQ uh so the poster quantization meaning that we quantize the
model after the training and so to do so we need to uh get um uh calibration sets meaning some a few examples of images for the process to go through uh and then we just have to tag all the nodes with the right precision that we want. So for instance here we want the activation to be uh in integer 8 the weights to be integer 8 and
the biases to be uh integer 32 for the accumulation and eventually we can run the quantiz quantized network function which will actually quantize the model. Uh so it takes a few seconds here I guess. and so eventually it will add node to the graph. So contiser node uh which will uh when we will be running the inference uh this will apply uh the actual contisation it will
apply what we call a scaling factor and hopefully we will be able to visualize what it has done uh at the end of the process. So again this is something which is quite important for us uh to be able once uh actually every time that we are applying an optimization to the model to be able to actually see what has what has happened and um to maybe
interact with it. So it takes a few second here. Uh maybe I will just wait for it. and so once uh the model is optimized uh we can uh we can actually export it meaning that we can from uh the model uh we can generate standalone code that then then we can compile and run into target for instance. Um and basically this is quite simple here. we
actually just have to run the export function from the H+ module uh giving the the model the scheduleuler and an input. And so it really won't. Okay, maybe I can just show you. >> uh so yeah so basically what we what it will generate is uh something like this and you can actually see it on the side here. Uh and as I said this is a standalone
model with um with a forward function which will actually run every uh layer of uh your model one after another. And this is as said this is kind of fully customizable. So if you have uh your own function for running the fully connected layer you can just add to the export and this will your function will be called instead. And so that is one uh crucial point
of the export is to be able to fit any targets with any implementation uh that you may have. yeah, so this really won't go through. >> Uh anyway, so that's the end of the tutorial presentation. Uh if you have any question uh we can answer it. [applause] Okay. So yeah, if you if you want to try our our framework later, don't hesitate. Uh it's uh like I
said, everything is available uh online. Everything is uh open source. So uh yeah, so the p the power for the project uh builds on the community. So really don't hesitate. Hello. U maybe start with an icebreaker question. Tom Tom Flashman [snorts] from Accenture. Um it's super interesting project and uh uh in the slide you showed your community. Um it was pretty obvious that you had many defense
companies as well looking into that and I guess they don't do bird spotting but potentially uh use uh I think it's known uh use microphones to spot incoming drones. uh distributed across uh countries that are attacked and stuff. Can you elaborate a little bit on or maybe otherwise is it allowed for you to elaborate on some of the use cases that are super important for the defense
market or can't you speak on this? Uh yeah, I think we can't maybe really go too much into details here, but something that we can say is that uh this is something quite important for the our uh industrial partners uh that they can actually know what's happening and that they can actually know what's uh what uh system that they are using. So um we have uh quite
long partnerships thanks to uh the fact that we uh can guarantee that uh we can see the optimization that we're applying and because uh often when we are using um um framework that actually optimize uh you can't really see what's happening and it's often an issue for these kind of companies uh which work on really sensible uh areas. so yeah that's kind of an interest for them.
Also the fact that it's a uh European project uh it sometimes really matters to not use uh because sometimes they just are not allowed to use foreign uh forign uh systems uh to train AI etc. So I I think these both element are uh quite important for them. >> Thank you. >> [music]
