Lightning Talk: How DeepInverse Is Solving Imaging in Science and H... Andrew Wang & Minh Hai Nguyen

This talk covers the significant application of AI in imaging, particularly through the lens of computer vision and PyTorch. The speaker argues that imaging represents the final frontier of computer vision, emphasizing the importance of translating real-world data into usable images. They discuss various challenges in medical imaging, such as the high costs and long processing times associated with MRI and cancer imaging, advocating for the development of cheaper and faster imaging technologies that leverage AI for high-quality reconstructions. Additionally, the speaker highlights Deep Inverse, an open-source library that integrates advanced imaging techniques with AI, and invites collaboration from developers, scientists, and the open-source community to tackle ongoing challenges in the field. The session aims to inspire engagement and innovation within the imaging and AI communities.

Hi everyone. I'm Andrew. So, it's quite an intimate audience here. So, I think if you we should have some time at the end to answer questions. So, if you kind of reflect and have any questions, then please ask them at the end. So, I want to talk about an application of AI and PyTorch which you might not have heard of, but which I think is probably one

of the most important applications of AI today, which is quite a bold statement to make. This is the applications part of the conference, so this is my claim. So, I want you to go away at the end um um with an idea of what imaging is and what what is it is as an application of AI and PyTorch and why it's important. Um and I'll also make

the connection to PyTorch later on in the conference. And at the end I have three slides about how you can get involved as well. So, there should be something for everyone here. So, first of all, I want to start off with who has heard about imaging before? When I say heard of, does it does it conjure up anything in your head or is it just a just

a null vector or something? Um please feel free to put hands up. Okay. Okay, so maybe half or something. Okay, it's good. It's good challenge. kind of my thesis today, which is quite a bold statement, I think, is that imaging is the final frontier of computer vision. So, kind of in AI we've seen I mean many talks here about natural language processing, LLMs, etc. And kind of

computer vision for me is maybe the more interesting side, the application of AI with many scientific applications. So, in kind of computer vision that's presented today, most of the time, you know, you have images somehow, you magically have images, pixels, and then you have tasks. And you know, many people are building amazing startups, you know, one of the most important people in AI is just started their

own startup in Paris um building for example world models, perception, reasoning, blah blah blah, all these kind of things. And they're all saying we're building models going from pixels to decisions or to model modeling some kind of um you know, it's some kind of uh um yeah, actions on these on these images. Um and then somehow there's a big gap here when we go from the world

or the universe and and then some biological and health applications um just things that exist into images. And what is in this big gap here? So, this is imaging. So, imaging is the process of going from the world or the body or whatever whatever is of interest to data. So, this is for example data captured by your smartphone when you take an image or when you go

to hospital and it's you go into an MRI machine or you're a microbiologist and you're using a a microscope. And the image reconstruction is this big stage here of how do I get a nice image from these data here? So, my belief is this is the big bottleneck today in computer vision. It's not anything that's going downstream. To get these images in the first place is the

most important problem and that's still unsolved. So, why is it a bottleneck? Um I'm going to talk about some applications I find really interesting um that maybe talked less less about on these stages which I um which is a bit of a shame. So, for example, medical imaging. Right now, to obtain these really beautiful pictures of a brain, it's very very expensive and takes a very long

time. Um so, if you go into a hospital, MRI machine takes up multiple rooms. You have to often build a hospital around it. It's very expensive to run and to to maintain. So, what if we can build for example cheaper MRI machines that take worse quality images and then use AI to reconstruct these high-quality Uh similarly in on the cancer imaging Hello. Hello. Oh. in another medical

imaging domain, so this is in nuclear medicine, we have these um pictures which kind of like trying to figure out where the cancer is in your body. This is extremely slow. When you go to a hospital, you can see they're only processing tens of patients a day, and there's a huge waiting list uh to to process these. What if we can make this these devices go faster,

um they produce lower quality images, and we use AI to reconstruct high-quality images. Um another application in healthcare is in keyhole surgery. So, you know, we have beautiful cameras on our phone. These cameras are never going to fit inside the tiny tiny hole that they put um to image your brain. So, to do this, we need very small cameras that can image very high resolution. Uh the

only way to do this is to take really crap photos and then reconstruct them. Okay, a bit bit more blue-sky up. Hello. Okay. A bit more uh blue-sky uh black-sky thinking uh is imaging black holes. That's a really cool application where we have never seen black holes in our life before. Like, how can we ever build something to image to look at these without ever training a

model to be able to do it? Okay. So, this is um yeah, highlighted what image reconstruction is about, and it's kind of where a lot of the lots of advances in AI today are being implemented. So, foundation models, uh lots of math stuff, uh diffusion models, probably one of the most probably the most interesting application of diffusion models um right now. Um and this is what we're

doing at Deep Inverse. So, it's an open-source library. We're part of the PyTorch ecosystem now. Um kind of very French place, so very happy to be here. Um uh we we won a prize uh with the French national prize for a documentation, and we're kind of trying to do everything that there is around imaging. Um so, there's a lot of kind of code side of things, but

there's also a lot of math side of things. How do we model, for example, uh these type of uh real kind of uh physical objects? And then we have uh for example, AI models, um blah blah This is kind of maybe less interesting to you. This is more for uh imaging audience. What's maybe more interesting to you is um how I you involved. So, I've got three

slides here. First of all, for the developers or engineers amongst you, there are big engineering problems still to be solved in imaging. We have huge data, for example, like microscope volumes can be like 10,000 by 10,000 by 10,000 pixels, absolutely huge. How can we have models that can process this data efficiently and also be used for people in the real by microbiologists or by radiologists, etc. we

are Yeah, we're friendly community where, as I said, we're global but we're primarily based primarily based in France. uh yeah, so kind of um it's nice to be speaking to this community and you we're obviously it's all on GitHub. Um second No. Yeah, here's some more pictures, so um Yeah, engineering problems are not on the not just on the mathematical side, but also on integrating state-of-the-art deep

learning. Uh so, for example, we recently integrated Hugging Face, you know, that there lots of kind of state-of-the-art artificial models are on there. How do we start using them for real scientific applications? Um and then so the image in front of our community hackathon last year. Um and we have global usage. Um secondly, for scientists and mathematicians, so Paris Um in Paris, you know, there's there's lots

of amazing science coming out of the universities. Um there's still lots of open problems at the intersection of machine learning and physics and and science. So, for example, you know, physicists have built these amazing machines to image cancer, but then these are really expensive and then we need we need to be able to model these um in PyTorch code, differentiate through these um so that we can

start using them in practice. uh we've just received some French public money to hire an engineer as well, so um you can Google Deep Inverse hiring as well. so final thing I want to mention, so that's so I've talked about engineers, for scientists, third about is for the open source ecosystem. So amongst you there will be many who are kind of like floating around the ecosystem. So

apart from the open source side we're also just recently co-founded a startup building this stuff and bringing it to the market. So for example bringing this foundation models to medical imaging applications. We're Paris based yeah, if you're if you're at all interested in that let's have a chat. You can email us, you can Google blur labs. That's I think my personal but yeah, please I'll be really

I really love to chat to any of any of you Thank you very much. If you have any questions please let me know now or we can talk That's us. Recently published in ICLR and some other marketing crap. Any questions now? Please. Yeah, yeah. So it's kind of like we're kind of in parallel with the entire computer vision community so everything that gets proposed in computer vision

community gets gets applied. So like this foundation models here are primarily convolution based, you know, we're coming from academia these are still the best models today to reconstruct these things very efficiently and to be deployed. Diffusion models are really cool but you know they they're very slow, they're very big. Trained from scratch as well, yeah. That's a really good point because you know most pre-trained foundation models

here is for applications like I don't know, generating faces or like other computer vision applications but you know we're The interesting applications are for example like MRI of a brain, you know, there's you need to train these models from scratch. Cool. Thank you very much.