Bringing ExecuTorch To the Next Frontiers of Edge AI - Mergen Nachin, Meta

This talk focuses on the future of artificial general intelligence (AGI) and the role of Executor in advancing this field. The speaker, Morgan Nachen from the PyTorch Edge team at Meta, highlights the importance of on-device AI, emphasizing its benefits such as enhanced privacy, low latency, and reduced costs. The session discusses how to effectively deploy PyTorch models across various device types, including AR/VR headsets and mobile devices, without complicated conversion processes. Executor, which reached general availability in October, enables seamless model deployment and execution on diverse hardware platforms. The presentation also explores voice modalities for AI agents, detailing real-time transcription capabilities and the orchestration of complex voice interaction processes using various backend supports. The speaker concludes with a call for adoption and contributions to Executor, emphasizing its role in the open-source community.

Hello everyone. My name is Morgan Nachen and today I am really excited to talk about the next frontiers of AGI and how Executor is enabling it. So, and I work at the PyTorch Edge team PyTorch team at Meta Meta super intelligence team. So, roughly the agenda is as follows, you know, we're going to talk really briefly about where we are today and I want to touch on

two different topics. One is the voice voice transcription voice voice modality for AI agents as well as desktop laptop right in the consumer GPUs you want to run local AI. And lastly we're going to talk about the announcements specifically about the open source community. So, All right, so let's start to understand about you know, why we are doing on device AI in the first place, you know,

there are a few different reasons such as being able to run inference on your locally can have enhanced privacy. You can have really low latency or almost real-time user experience cost aspect. You can save on some of the API cost to cloud services and then be able to operate in a low and also low connectivity environments. Moreover, if you if you also look at it the models

have been becoming smaller and also more smarter. So, this is a great opportunity to capture this. So, the problem statement is let's say you train your model in PyTorch. You have your great model here and how do you deploy it into different form factors such as you know, glasses or AR VR headsets or mobile devices or wearables right or desktop right. So, a lot of times people

go through either a expensive conversion process or use like a very proprietary SDKs on on these um And if you are a startup, it actually adds a lot of this kind of development cost because you have to maintain all these different and different different combinations for different targets. And this becomes a hard problem. And we um announced executor's general availability 1.0 last October at the PyTorch conference.

And if you for a brief introduction executor, the way it works is that you take a PyTorch model and you take an torch.export and gives you semantically equivalent graph representation that can further be compiled into different backends or in a different hardware and platforms. And then you do all this kind of compilations ahead of time. At the end and they during the runtime, you basically take this

model and you be able to run do inference in a very you know cheap way, efficient way, right? And then the the bare runtime is can operate in a very small microcontroller even like a bare metal around 50 kilobytes standalone. But you know, you can also run it in a very beefy you know gaming PCs for instance. And you know, we we also support the various different

platforms, Linux OS, macOS, Android, iOS. Uh I also want to talk briefly about you know what we have been doing in the microcontroller space specifically since the general availability, we've been improving our microcontroller support specifically for arms cortex and you know microcontrollers. We are in a better status now. Uh leveraging sensors and then accelerators. And you know, this is the early benchmark you know, on MobileNetV2. We

are beating TFLite micro in in various different metrics, latency, load time, and and RAM usage. We want to like more expand this to other quant schemes, more model more and then get more much more efficient in limited constraint devices. In fact, you know, we all have a few different talks throughout this PyTorch conference in embedded earlier today RJ and Matthias had a talk on microcontrollers. But you

know, there will be other companies who are also talking about their respective backends. who's using executors right now? And executors actually started with at Meta. So we've been using within like glasses, Ray-Ban Meta glasses, Ray-Ban display VR headsets, and all the apps, Meta apps or Facebook, Instagram, Messenger, and WhatsApp. And outside you know, we've been being adopted by startups like in liquids, LM Studio, and software mansions,

and private mind apps. yeah, so that is basically oops, where we are now. I want to change gear and talk specifically about voice, right? So If you think about you know, we've been operating in this kind of chat LLM chatbot era where you're typing. But you know, a lot of agents, you know, you we are now we want to be able to talk, speak, and then hear

back, right? You don't have to type it all the time. And the key voice is a key modality to support this and the in terms of the models, you know, these are proliferating and quickly, right? So all these labs and companies are generating um building open source open weight models. Earlier today, Patrick from Mistral talked about the Vox Trail Vox Trail model. Um but, you know, they're

all open source and open weight, but they really kind of at this point lack a more the unified deployment story, a unified native deployment story. So, Um so, let's think about what a voice, you know, this is a an an a an example of a voice agent building blocks would look like. If you think about it, like you are talking, there's maybe a wake word model, there's

a voice detection model, and, you know, to figure out who's talking if there are multiple speakers, differentiate speaker one against the speaker two, have the real-time aspect while you are talking, and just be able to transcribe in real time. and, you know, understanding the the emotions and tonality and context from the and transcribe or translate, right? Think and plan, and at the end be able to talk

back and actually respond to in a voice. So, uh So, um compared to like a just a whisper, which is a very offline transcribe and done, you know, this is you can it is much more richer and more nuanced and more complex pipelines usually exist here. specifically on on So, we enabled Vox Trail real-time the the transcription ASR, and I'll give you just a case study of

what we did. You know, if you're looking into the Vox Trail, these are an audio encoder, it's a very standard quote-unquote standard audio encoder, text decoder, embeddings, and what we do is that we basically export in different components individually and have a thin C++ layer kind of orchestrate some of the complex logic uh specifically some of the moving moving window uh uh logic so that you can

get some context from from from the past uh and moreover uh we can actually support the multiple backends, right? So, CUDA and CPU uh metal and and Qualcomm, you know, these are some of the the backends that we support. So, the idea is that you have the same C++ code uh same export uh and you just link it to different backends and actually get all the uh

accelerations and uh right away. So, I haven't uh I I want to actually demo real quick uh uh so, this is LM Studio. You know, they just gave me a better build uh like a few hours ago, so let's see if it will work uh I'm on stage right now and talking in real time talking about voice transcription and as you can see this is able to

transcribe in real time running fully locally 100% on my laptop and it's an M1 MacBook uh laptop. So, all right. It's good. It worked. Yeah. Well, the the the other one and this is just running the uh this is not using ExecuTorch uh this is using I think MLX backend, but the transcription itself is using ExecuTorch as an inference engine. So. All right. Well, good. The the

demo That's uh All right. So, uh it is using uh Vaxer real time. Uh it's using 4-bit quantization. Uh and in the you know, the I just showed you the macOS, you know, but it can work on Windows, CUDA, and Linux CUDA Um uh uh as well, so. Yeah, so that was Vaxcel specifically, you know, but we are not only focusing on that, right? So, we these

are some of the other models across the board. Um Nvidia has like a Parakeet and Sort Former, which is quite different. If you look into traditional LLMs, it's quite different, and they're very much voice focused, right? Whisper and it's a classical one. Yes, Celero, which is a very small 2 MB for voice activation detection. And in all the way to the embedded systems, we have a Conformer

ASR built by the ARM team that can run on the like a small Ethos-U microcontroller embedded system. Uh okay, so here we also have Gemma, Gemma 4. It's in the works. Hopefully in the next few days we'll be able to support Gemma 4, so. Yeah, so I talked a little bit about voice specifically. So, I think the idea was to we want to showcase different backends and

you can do it easily. Uh I change another gear to the think look into inference on desktop laptops, right? So, um I just showed LM Studio, but you know, there are so many other LLM models, CPP, Ollama, and these are, you know, great frameworks you can run LLMs. basically we're asking like can, you know, we traditionally Executor has been mainly on mobile and in the glasses and

embedded systems, and then we're trying to see like can we use the same principles to to do inference on on your laptops as well. Um you know, you know, some of the requirements is that we don't want to have any Python runtime during during inference stage and no libtorch dependency, right? So, libtorch is a multi-gig library. You do you kind of want to not have that and

then embed into your application the AI capability there. Um, so in order to showcase this I want to anchor and specifically on Qwen 3.5. This model 35 billion act total primary 3 billion active primary model. It's a mixture of experts. and you know, for people who are not familiar with it, um, this the attention layer has like a hybrid mechanism. One is the standard attention with like

a full KV cache. Uh, that's like only 25% of it and in the 75% is a linear attention. And it has a recurrent state, right? So, that the KV cache doesn't grow. So, it is a fixed size recurrent state. And the mixtures the the only eight experts are activated at during inference. so it's a relatively sparse And this is an interesting model because on one this is

quite capable model um, that can run on on consumer GPUs. Um, and memory doesn't grow because of the recurrent state linear attention mechanism the memory requirement doesn't grow linearly in terms of the context length. and you can see some of the similar techniques are happening in other models such as the GLM or or the Nemotron models, right? So, uh, I want to actually go over some of

the examples of how we enable this. So, if you look into the model definition, um, you know, this is just PyTorch, you know, uh, written written exactly and then we'll have in a custom custom kernels. This has the chunk gate delta is a flash linear attention kernel and you have fused MOE kernels like you know just like what you would see in a normal PyTorch program and

then you can actually export it right you can directly call torch dot and it will it just works. on top of that we are doing some sort of a few different optimizations. The first one is quantization so you can use torch AO to actually quantize your model to four bits and use like a different type of PTQ algorithms to do that and we recently also start supporting

turbo quant which is if you look at the code you literally have the KV different KV cache implementation that compresses when you're storing the KV cache entry and during the SDPA attention inference you decompress it kind of on the fly and do this kind of tiled decompression SDPA implementation. Um, there are other like fusion techniques you can directly do it on your PyTorch program like you know

for KQV linears you can you know batch them together and fuse them together and split it afterwards. Um, the the kernels itself the Triton kernels we are basically just adopting from FLA flash linear attention the exact like basically they import it directly and then just using it as is and you can use the regular Triton auto tuning auto tuning mechanisms to tune on your on your host

to host machine if you want to maximize the performance. So you know why am I talking about this? So, here is the here's the magic part, right? So, you know, I want to take away that executor this is PyTorch here. you are in the model developer or a machine learning engineer, you are doing all this, you know, loop of like if in the phase of optimizing and

research, and you are doing everything in Pythonic way, right? PyTorch Pythonic way. And you can use that exact same thing for your Python loop, right? Um and maybe you can use directly eager or torch compile or even like AOT precompile. But the thing is you can use the exact same Triton kernels, exact same, and be able to run it in a non-Python environment in native C++ environment

without libtorch, and then it directly actually just in embed in your C++ application. And the idea is that you still get a lot of the flexibility and experimentation, but you know, you get the easy of deployment story. And you also get the Triton kernels there for quote unquote for free, but also they a lot of the inductor PyTorch just compiler technology inductors fusion like happening kind of

underneath the hood for you. um yeah, I mean, this is numbers we are able to get like 77 tokens per second have like, you know, in four quantization as well as turbo quant. And but you know, this is basically showcasing, right? But we want to close the close the performance gap with like a state-of-the-art very soon. Um and you know, have like cross-platform aspect of and as

well as CPU, right? Um uh CPU offloading, you know, this is something we didn't have time, but you know, this will hopefully get be done in the next few weeks. So, be able to not only use all the VRAM, but if the VRAM doesn't fit, you know, maybe you can share with the DRAM VRAM aspects of So, yeah, here's, you know, a recap of, you know, what

we what we talked and we talked about voice, uh, how with voice we showcase like, you know, for voice modality a lot of different complexities. You can uh, describe it easily in PyTorch and have the thin layer of C++ code and orchestrate there. and have multiple backends. and and in desktop laptop you can actually still get a lot of this kind of power of experimentation and uh,

profiling and research and then still get the native flow easily. Uh, so what's coming? Uh, obviously we make sure we improve some of our voice and the desktop laptop inference and especially Um, robotics is is definitely an uh, interesting space. Uh, you know, it's growing. Uh, we want to make sure uh, we um, we support it. You know, I don't have um, you know, it's we we

don't have plan to actually announce anything at this talk, but, you know, we have a proof of uh, concept stage at this point to figure out what to do. Uh, and, you know, TinyML microcontrollers there are like in fact five talks, six talks, five six talks at this PyTorch conference on the TinyML part. So, um, please check it out. uh, that's, you know, where we are, um,

today. I This is mainly a little bit of the technical bits so I want to share Um, I want to talk a little bit about, you know, how we started, um, executors. It started, um, like few years ago, three, four years ago to really support Meta's internal customers, right? The you know, family of apps, Instagram, WhatsApp, Facebook, as well as the Oculus and Ray-Ban the smart glasses.

And really we see that it has actually grown like not only meta, but you know, I think that if you look into it, we have multiple back ends, you know, this is contributed by the companies, you you know, for 12 companies 14 companies, um and in within the ecosystem really we close collaboration with the transformers team, um react native, you know, we have uh a startup who's

building a SDK specific for react native. Uh their model um model builders like ultra lyrics supporting executors, so um yeah, I think this was a good great time to announce that um we are joining the PyTorch Foundation and becoming PyTorch core, right? So, this will be um just like any other uh you know, you know, PyTorch PyTorch that we want to be you know, be able to

uh be part of the open source movement and be able to leverage the community, be able to co-design both the hardware and software uh together and move the the industry forward. So, what this concretely means, right? I want to be uh clear. Um the maintainership it'll be maintained by PyTorch core maintainers um list of existing core maintainers list. We also have an executors module maintainers list, so

this will not be this will be individuals based on not company, but on merit based. we want to make sure we can have increased more contributions from different hardware partners, kernel authors, model authors, right? So, uh um that makes it makes it quite if you are part of the PyTorch Foundation it just makes it quite easy to do that. for specifically for our partners we want to

have more shared infrastructure to be able to run CICD in a shared manner. Um, and lastly kind of able to co-design both the software and the hardware specifically in this new era of local AI. So, I think being able to open source it the way we would do as Mark said in its in its keynote the way the only way to do that is to be open

source. So, yeah, uh So, going back to so the call to action right now I want people to start adopting executors and let us know and give us feedback, you know, if there is any models and please contribute back if there are new models that you want to make sure we have it. If if there are definitely contribute um and integrate executors in your middle SDKs, right?

So, as a end user sometimes executors can be a lot of different knobs and low-level configurations. End users may not really want that, right? So, maybe a library a higher level library can actually abstract it out Um, you know, transformers would be a good good example but many other like in the in the embedded space uh there there are so fragmented SDKs so you know, that would

be a good integration point. Um, all right. Uh, so I want to just end on the note that you know, the executor just really the power comes from open source collaboration, you know, we have seen again and again that this is working. Open models, open backends and everything and open ecosystem and uh and the the the PyTorch announcement the foundation that we are joining the foundation and

being part of PyTorch core is really the capstone that makes it even stronger. All right. Thank you so much everyone and have a good rest of the talk.