Lightning Talk: Bridging the Gap: Engineering Compliant... Muhammad Saqib Hussain & Mohaddisa Maryam

This talk explores the intersection of artificial intelligence and clinical practice, highlighting the challenges of deploying black box AI models in healthcare. The speaker, a fourth-year medical student, discusses his work on creating an AI application that augments clinical expertise rather than replacing it. He focuses on the implications of the EU AI Act, which emphasizes the need for transparency in AI systems, and introduces the concept of 'glass box' AI as a solution. Utilizing PyTorch Captum's integrated gradients, the speaker demonstrates how to interpret model predictions and explain them to clinicians. He showcases how the Neuro Bolt model processes EEG data but points out issues related to its interpretability and accuracy. The session underscores the critical importance of involving clinicians in the development of AI models to improve usability and trust.

Hello everyone. My name is Mohammed and there are a lot more of you than I expected. But yeah, that's perfect. So, I'm going to talk about like bridging the gap between the engineering and the application part of AI. So, a little bit about me. My name is Mohammed and right now I'm a fourth year student of medicine. And yeah, you wouldn't expect someone like me to be

speaking here, but I think there's a real need of people like using the product that you guys are building and like knowing what they really need. So, yeah. Right now, my thesis is just building a system or like some kind of AI application that can augment rather than replace clinical expertise. Yeah, I never took a course in AI or ML. I'm entirely self-taught. So, yeah. I'm really

honored to be here. What I will talk about today is the clinical dilemma, like why the EU AI Act makes this black box AI impossible now. Like you have a model that is pre-trained, you just use it. That's not possible anymore. The architecture of an example model that like I'll be presenting, which is Neuro Bolt. The glass box engine, like going from the black box architecture to

a glass box architecture. A clinical proof, well, like why is that even important? And next steps, what can be done about it? So, a quick question. You guys all are building AI applications for your users, right? What level do you think your users currently stand? Like at a level one where no one knows what a GPU is, two where they know, but like they don't really understand

the stuff, three they know AI, but don't really understand it, four they can write some applications, or five they can write complex applications. So, hands for level one. Two. Three. Level four. So, you guys think your users are level four? Okay. Because like my experience is like doctors stand all the way to the left. They're the ones like using all of these AI models doing clinical research

in cancer and all of that stuff, but like they don't understand what AI is. You deploy a model to a hospital, one hour later you get complaints, it's not working like like they expected. Why is that? Because they don't really understand what it is even doing. Uh I stand at the gray one right there and I know you guys are the red ones here. So, Uh the

black blocks black box problem in healthcare is like really important because the EU AI Act makes all of these like black boxes impossible to deploy right now. You need to be able to explain what the AI model is doing and it has to be like interpretable by a human being. And like all of these articles are from the EU AI Act. so, there's a glass box solution

and it's already built into the ecosystem with the PyTorch Captum. Uh it is clinically explainable. Uh PyTorch Captum's integrated gradients can give you some insights about what the model is doing behind the scenes and you can use that to explain to your users or like in my case to the doctors what the AI model is doing. yeah, so the output would be mapping these deep learning uh

math functions back to the human anatomy that uh the doctors understand or like your users are lawyers, they understand the law part of that. So, they can understand what these models are doing. Uh so, yeah, very briefly, uh just as as an example, I'm using Neurobolt. It's a very complicated model. from the left, you have these uh EEG electrodes. They give you some readings. You can tokenize

them, uh pass them through the AI model. The architecture is there, and it gives you some values of like fMRI region of interests. Now, the details are not important. The important part is that it gives you some predicted values. You can plot them and go to doctors or clinicians and say, "Okay, my model is performing at like 60% accuracy, which is pretty good, by the way, uh

for this." Uh it misses sometimes, but it's still okay. The problem arises when you use integrated gradients, try to explain to the uh clinicians what the model is It is taking a look or like it's focusing on the electrodes like at the back of the head, but also at the front. I know this might sound very complex to you guys, but like this is real problem because

the model is not looking at the correct electrodes. The 60% accuracy means nothing here. It's taking a look at the wrong How can we fix that? expected was like at the occipital electrodes at the back of the brain. It should just be read at the back, not at the front, right? Uh the unexpected is that it is also focusing on the middle and the front part. My

first hypothesis was that like the models uh successfully learned uh all of the visual pathways. It's like AGI right The second hypothesis is that it's cheating by using the front electrodes and using that to just guess if there is visual activity. I think it's the latter part. It's just cheating. So, this ambiguity is exactly why clinicians need to be involved or like your you end users need

to be involved in the production of these models because like just accuracy means nothing. So, what was missing? When this model was trained, it was trained by researchers, academics, not clinicians themselves. So, that was a huge red flag. That had transparency, the clinical clinician visibility into the training data about how this model was trained, if the training data had biases. No one really knows. Action level clinical

UI, doctors don't understand these numbers. I myself don't. So, I think there's a real need for UI. And like translating this back end math into visually showing what these models Just as an example, simple Streamlit dashboard would be great for this use case. Yeah, that's all from me. You can take a look at the GitHub repository if you would like to like see what kind of implementations

integrated gradients can be used for. And yeah, this is the GitHub for the our Neurobolt architecture that I was showcasing. That's all for me. Thank you. I think if we have a few minutes, so we can take some questions if anyone has them. Yeah. Uh the integrated gradients? Uh so, it's this one. Yeah, so it's this one here. Uh we just got the predictions and this is

what the paper ended at. Like this this These were the results. But like when I went further with the integrated gradients and like mapped all of these points to a visual dashboard that I could understand. Uh yeah, these results are wrong. Like it shouldn't be looking or like paying attention to the electrodes on the right like at the top at the left. So, you can just take

these integrated gradients, take them to your users, and like ask them if these predictions or these answers are correct for them.