Lightning Talk: Deep Learning in the Wild: Embedded PyTorch for... Taraqur Rahman & Owen O'Donnell

This talk presents a project focused on deep learning applications in remote environments, specifically for tracking endangered bird species like the chestnut-mandibled toucan and the three-wattled bellbird in La Amistad National Park, Costa Rica. The speaker discusses the challenges of deploying machine learning algorithms in areas without power grids or cell connectivity. They detail the use of IoT devices called 'ducks' that communicate via the LoRa radio protocol to relay bird detection information to conservationists. The talk also covers the initial phase using the BirdNet model with Raspberry Pi for audio analysis, followed by a second phase that optimizes power consumption and machine learning efficiency through a more lightweight custom model and digital signal processing techniques. The overall goal is to improve battery life and reduce false positives while ensuring real-time bird detection.

Hello everyone. My name is Tara Kora and this is my colleague Owen. Uh we've been working on We're from Owl Integrations and we've been working on something different than LLMs. We're working on deep learning and deploying it in very remote places like rainforests in Central America with the Phoenix Zoo, Proca, and ASU. Uh next slide. So before we begin, I would like to play a quick game.

Um it's called Guess That Sound and I'll be your host. So I'll play a sound and you guys can guess if it's a bird or not. So the first sound is Do you guys hear a bird? Yes. So specifically, it's the chestnut-mandibled toucan. And the second sound? >> [laughter] >> It's No, it's actually me with imposter syn- syndrome. And the third one? How about that one? It's

actually a bird. It's called a three-wattled bellbird and it's one of the loudest birds in the world. so all jokes aside, the two birds that we'll be looking at today is the chestnut-mandibled toucan and the three-wattled bellbird. The bellbird is a keystone species, endangered, and critical to the balance of the bio ecosystem. And this is a bird's-eye view of their habitat, La Amistad National Park, and it's

located southern Costa Rica, marked by the yellow X's on the map. And scientists needs a way to track them to prevent extinction extinction. And as you can see, they need a scalable way to do that and they need to have a acoustic machine learning algorithm deployed in the field. And this La Amistad is a very remote area. There's no power grids. We can't There's no cell towers

and there's We could probably connect to satellites, but that's very expensive and power-hungry. So next slide. So basically, it becomes like a three-body problem. The first one is the machine learning algorithm. Uh there's We have to deploy in the edge, so there's no cloud, there's no server, nothing. Second one is battery optimization. Like I mentioned, we need these devices to last weeks, months. We can't just plug

into a tree and hopefully gets power from that tree sap or something, which is not possible yet. And third is connectivity. If the device gets a detection of a bellbird, how how do we relay the those detections into the hands of biologists so they know exactly where it is? We can't build cell towers because that is literally the opposite of conservation. Next. first, let's talk about the

connectivity piece. So for a To close the connectivity gap, we're using Owl IoT. Uh Owl IoT is we deploy devices that we call ducks. So coincidentally, we're bird-themed company solving bird So we deploy IoT devices called ducks that communicates each other through a long range and uses low power using LoRa radio protocol. So when a bird detects a When a duck detects when the duck detects a

bird, it'll send that information through LoRa. It'll hop from one node to another node to another node until it reaches a gateway node and pushes up to a satellite. The difference here is we only need We don't need every device to be connected to a satellite. We just need one gateway node. And so yeah, phase one. Let's talk machine learning. So now phase one was in October

2024 to December 2024. Next. uh phase we used a model developed by Cornell Lab called BirdNet. It's an amazing model. It takes a It takes the audio file, converts it to a spectrogram, and then runs convolutions on it to get detection. And we use PyTorch to transfer learn and for our use case and quantize it. And to have BirdNet running into in the microcontrollers, we had to

use a Raspberry Pi and a microphone, and we use a LilyGo T-Beam as the Owl IoT Next slide. And then this is the deployment. In January 2025, we have Kinley from Phoenix Zoo. She's finishing making the final touches on the ducks. We have Garth Paine from Arizona State University. He's testing out the radio signals in the rainforest. We have a tree climber who's actually deploying on a

tree. And then the last picture we have a local bird visiting them as they're deploying. And this is the deployment for phase one. Each bird icon is a device deployed and we got messages from each of the devices relaying in the network into the Zero the office. That's the gateway node where pushes up pushes up to a cloud. So we had some detections and we were successful

in getting ML model on the edge and being and getting detections, relaying the detections. However, uh we were getting some false positives and the battery lasted like 3 days, so we definitely improve on that. So I'll pass it on to Owen to talk about phase two. All right, thank you, Tara Kora. So back in January, the Phoenix Zoo reached out to us to do a phase two

and later this year in July, we'll be deploying that back in Costa Rica. All right, so the two main areas we wanted to focus on phase two were power consumption as well as improving the machine learning model. And so the Raspberry Pi consumed a lot of power because it's a larger board compared to an embedded system. And we needed the Raspberry Pi to run the BirdNet model.

So that got us thinking, maybe we can replace BirdNet with a more custom lightweight model. So before we talk about that, I would just want to talk about digital signal processing or DSP for short. Uh it's basically manipulating using math audio signals like bird calls, which you have a spectrogram here, and we'll pull out uh the strongest harmonic bands. So we We've We're going to filter out

basically our calls and we'll extract features like the frequency of So DSP fits into our pipeline by acting as like a filter before the machine learning stage. And the benefit of this is reducing the amount of potential false positives like background noise from going to our machine learning model as well as reducing the time the machine learning is running because DSP is more power-efficient than machine learning.

Uh and so we use feature selection to dwindle down our amount of features to about 20. And from there we decided since they're mostly all integers or floating points, we kind of use a random forest or a light a lightweight neural network since these are optimized for the edge or not optimized for the edge, but they can be run on edge devices. So our tech stack is

looking pretty similar to phase one except we replaced the Raspberry Pi with a Teensy 4.1 board, which is optimized for a DSP. All right, now let's take a quick look at the model performance. So uh we can see that the inference time all under 1 second, which is great cuz we want real time. We don't want to miss any any birds by accident. And then the F1

scores are all relatively similar. But the big difference comes in the size and the estimated lifetime of our the boards where we have a dramatic decrease in both the All right, so just to recap what we've improved so far and what our next steps are, we've introduced DSP, which will help both the model and power uh be optimized for this deployment. And we've also um we've replaced

the Raspberry Pi with a lighter and with a lighter weight And then to summarize what we've been working with, we start off with BirdNet. It's an amazing model. It was able to detect every bird, uh but it didn't fit the constraints that La Amistad uh put on us. Uh originally, we thought what most people would think, inject more data, train harder, grab a Nvidia GTX. Uh but

in for phase two, we went back to the drawing board. Uh we talked with acoustic experts like University, and we made some tweaks, and we ended up with a 100 kilobyte model that was a better fit than BirdNet for our three-body problem. So we want to leave you with this. Next time you're experiencing something similar problem like this, before grabbing GPU or adding more parameters, we recommend

going back to the drawing board, make some tweaks, and uh see if you can make any new innovations. And Sara Hooker puts it best on her in her recent paper, we can now stray from the beaten path of boring predictable gains from throwing compute at the problem. And that's the path that we're on. So yeah, uh thank you. Uh we we skipped few details, so feel free

to connect with us and we can answer your questions and use these QR codes. Thank