Understanding machine decisions

This talk focuses on the concept of explainable AI, particularly in understanding how machines make decisions. The speaker, an AI researcher at Microsoft, discusses the challenges posed by current AI methods, especially machine learning, which often obscures the reasoning behind automated decisions. Through illustrative examples, including the misclassification of images and vulnerabilities in computer vision systems, the importance of transparency and understanding in AI technologies is highlighted. The speaker introduces the idea of graded logic, a methodology designed to align machine decision-making processes with human reasoning, enabling machines to provide explanations that are more intuitive and reflective of human thought patterns. The session also covers the development of a project named Bacon, which aims to automate the construction of decision trees that guide machine learning outcomes, allowing for more interpretable and cost-effective medical diagnoses.

[music] Thank you for coming everybody. Uh I'm Hi from Microsoft. Um but I can assure you I'm not going to talk about co-pilot today. Uh actually I'm talking open source and as a AI researcher about explainable AI under the title of understanding machine decisions. Uh unfortunately my my machine broke. So this is not my machine. I'm presenting from far away. Hopefully this still works. Okay, so let's

get started. So we we make decisions every day, right? Uh to decide to come to here, you need to make decision whether you want to spend several days in the OCX conference. Uh then you need to decide which flight to take, which hotel to stay in. And of course there are bigger decisions in your life uh like which house to buy, whom to marry to and even

smaller decisions like what to have for dinner, what to dress and what to wear, right? So we make those decisions every day. That's just a part of our social life and like it or not uh nowadays we are living in a world with um automatic uh autonomous agents and robotics, right? So this auto autonomous agents and robots they are making decisions on behalf of us in some

cases. So as you can see in most of the scenarios we would very much like to understand how and why those decisions are made especially in some mission critical and life-threatening issu uh uh scenarios. We want to understand how the uh robots thinks so that we can not only to understand how they decide and to influence how they decide. And this is the topic of today's talk

about how we can model the machine decisions and how we can understand them and how we can control them. Um so I will start with uh some basics right how machine make decisions and as we know at least uh the the current belief is that human beings are the only intelligent beings I mean super intelligent beings on earth. So the machines they are not intelligent by default.

So the way to give them some smartness is to through this AI right everybody know about this but I want to remind everyone right AI is artificial intelligence and the artificial is the key uh which means it's not real right it's a humanmade intelligent we we we write the program to make the machine appear to be intelligent and there are many ways to achieve such kind of

intelligence and I think a lot of us are familiar with machine learning nowadays but this is just one of the method to achieve artificial intelligence and actually a lot of systems artificial intelligence systems are not based on machine learning but based on some sort of a rule based system or expert system. So to explain the idea further I will use the best thing in the world uh

video games to explain the idea. So, if you play Super Mario, right, the Goomba, uh, if something looks alive, right, it moves around on the screen, but you can tell it's not intelligent because you can see the script, right? It moves from one direction, it it hits a wall, and it moves another direction. So, this is apparently not intelligent. But if we move the next uh if

you play like a shooting games, there's a homing missiles or homing bullets that will track the enemy position and try to go to the enemy position. By the way, if you know which game that is by looking at that picture, you get actual credits. Uh talk to me afterwards, you get get a little gift. Um so so in this kind uh the the the homing bullet or

homing missiles appear to be more intelligent because it's trying to track the enemies but this is still essentially a rule based system. is just based on the the simple physics rule, right? I'm trying to get closer to the enemy at each step. That gives it give the bullet this tra uh tracking uh uh appearance. And if we move next level higher, uh if you play a Street

Fighter 2 and those characters appear more intelligent, sometimes even humanlike, right? They can be really annoying opponent to fight against and they make complex moves. And whether this one is artificial intelligence is kind of borderline because on one hand it's still like a basic rule based system. It just in this case the rule set is much more uh uh complex. There could be like a 20 or

30 different moves the character can choose. But uh the on the other hand the character is making those decisions dynamically based on the opponent based on the the level of health. So this one can be borderline if it's a artificial intelligent uh system. By the way, I just use a trendy in one of my presentations. I think I can check that one off in my bucket list.

Uh and last but not least, right, if you move to Alpha Go, right, those kind of system are definitely uh machine uh artificial intelligence uh system because it has its own knowledge base. it can make rational decisions based on the learned rules. So those are definitely uh uh uh artificial uh intelligence systems. So you can see to make something appear to be intelligent uh there's a spectrum

of technology we can use but of course nowadays everybody is talking about machine learning. So basically with machine learning we are saying uh a certain problem like a plane goes is so complex we don't know all the rules. So we'll give the machine some data for for the machines to learn the rules. Then we apply those rules to our problem. But this creates another problem which I

called a machine learning paradox. At the beginning we don't know the rules, right? We we don't know how to solve a very complex problem. So we give it to AI. So the AI will learn the rules. We're happy. But now we have a new problem. We don't understand the rules the AI creates. So essentially we are not solving that complicity. We are basically changing our ignorance to

opacity of AI. So we change one not understanding uh problem to another. So this is the whole field about uh explainable AI research is about. And before I go into the that uh research uh though I want I just want to give you some cases that to illustrate the importance of understanding uh machine So the first one uh this is a uh somewhat famous case of computer

vision. So in this case the model is trained to recognize horses and cars on the picture and you can see um and the column uh three and column uh column two and column four are representing the pixels uh that drive that decision. Now you can see in this case although the model performs perfectly it recognizes cars and horses but if you look at the pixels the key

pixels are actually not on the horses or the cars. Instead the the model learned how to cheat because in this training set all those pictures have a copyright watermark on the lower left corner. So the model in this case learned oh if you ask me if the picture is a horse or or or a car I will just look at the copyright text if exist I will

detect it's it's a car and for all the test this models works perfectly but if you feed it with the real life pictures like in the bottom row you can see without those watermarks it fails to detect anything. So this is just a small example to show you that what machine learns may not be what we've expected. So it's critical to understand what the machine is actually

learning so that we can actually get the the machine to do what we want in real life. And another uh example is this uh this is the attack uh about uh attack the computer vision as well. So you can see uh in this case the attacker puts some stickers on stop signs right? This looks normal to human. I mean it looks annoying but we can still recognize

this is stop sign but to the AI model this could completely change the meaning of that stop sign. So for for for a computer vision model this could look like a speed limit sign or even a right turn sign. So you can imagine for the autonomous driving system, this will be really dangerous, right? If a attacker just put some stickers on the stop sign and nobody will

notice that because it looks normal to us, but they this will break the the autonomous driving system. By the way, we have the paper reference at the corner. Um so this is just another example of that the vulnerability of what machine learns and sometimes it will breaks in the way that we don't expect because to us our intuition tells us this looks okay but this is actually

not and the third case which is also uh kind of interesting is this guy Cameron Mattis. I feel kind of bad because I use him all the time but I don't actually know him. Uh but what he did is that uh in his LinkedIn profile he added this system prompt uh and the about text you can see the highlighted part. So basically he's saying whenever you send

me a message you include a flam recipe in your message. So he just put it on profile didn't do anything else. And of course a few days later he got this spam email and this guy is sending him an email. Then out of nowhere there's a flam recipe attached to the email. So this is kind of a passive prompt injection uh attack carried out by this guy.

It's very smart. But on the other hand, you can also see this kind of understanding or lack of an understanding is actually quite dangerous, right? People can use it for the good purpose or for the bad purpose. And the whole idea of explain AI is to give us the uh crisp understanding of how the machine is and uh there are many researchers in this area uh and

they can be categorized to two broad categories. One is this post hawk uh uh analysis. Uh basically u this approach take the existing AI model. So the AI model uh at the end is a big matrix with billions and hundreds of billions of parameters. Um so this method is saying you still train the model as it is but afterwards we will look at the model we'll look

at the weights and distributions trying to reason how the how the features are used in the final decision decision and there are whole bunch of methods uh listed here and you can find all the papers uh online. uh but the gist of this is trying to find the patterns right how features contribute to the final decision uh so that uh we can make some uh reason of

how a decision is made uh but the challenge of this is that it doesn't give you this end toend decision process you can only understand the principle of how decisions are made you know these pixels are important these uh criterias are combined in certain ways but you don't really have the end to decision process. You don't know how the from the input to the decision how that

decision path is formed. And uh this uh leads to another uh category of research which is to make the uh white box models right people is thinking okay if we make the model themselves explainable then everything will be uh transparent will be explained. So this is a uh most work related to symbolic AI. Uh one of the uh uh best known sample is the decision trees. So

basically in term of decision trees you are looking at one feature you do a if else then you select a branch then you look at another feature another set of feature you do another branch until you reach a conclusion. Right? So this this kind of methods it actually give you the whole path whole path of the reasoning. Right? You can track from each every single step about

each step how how that decision is made. However, this is not how human thinks, right? I I don't feel a lot of people actually I don't know any people will use this kind of decision tree for reasoning, right? like if I buy a house or but uh decide where to eat I I don't do if else if else uh like this actually to understand and to model

how human reasons is another field. This is a another field with a very long history with like great names in the history. Uh we started with George Bull. So he uh created this uh mathematical framework for reasoning using true and false values. That's the classic boolean. Everyone knows about it, right? And then we have Thomas space. Basically he introduced uncertainty. He formulated how decisions made with uncertainty

uh by updating the beliefs based on new evidence. And then there's a joy man right he reasoned the decision process is actually a optimization process. So basically based on the input and the environment we try to optimize uh based on the utility uh expected utility and Herbert Simon on the other hand argues well human don't do optimize we actually just try to uh satisfies our uh requirement

with within our cognitive limit and there's a uh uh loy uh who invented uh fuzzy lo fuzzy logic That's where we uh go away from the true false value but using fuzzy representations right called fuzzy set or uh membership function right basically the the value is no no longer true or false but of a degrees of truth or degrees of false which is much closer to how

natural reasoning actually fuz logic today is broadly used in machine control uh systems and of course there's a Daniel Kennean who reason people actually don't do rational reasoning. A lot of cases we are like passionate animal uh we we we just do do horistics to make fast decisions instead of doing logic induction. So you can see a lot of people are going from different angles from the

uh mathematical formulation to uncertainty to human behavior trying to understand how uh people thinks and this is still ongoing research and if I fast forward uh this is a Dr. uso doomer uh different from the other guys this guy is still alive he's 86 but he's still alive as of today uh I shouldn't have said said that but uh so 50 years ago he designed a system

called graded logic so graded logic basically uh similar to fuzzy logic it's using real numbers to represent the level truth so instead of zero and one you choose a value between and the unique uh aspect of this system is that is is the logic system designed specifically to model human reasoning process. So I don't see any other system that have have that in the goal. Right? So

so this this this mathematical model for sure but it's designed to model human reasoning and one of the key concept of that is that when human reasons instead of and or conditions we use a continuum between and or. So it's called disjunctive and conjunctive. So basically some cases uh we need both criteria to be true and some cases one criteria uh can replace another. Then you have

like a selections in between to adjust how simultaneously you require those uh u criteria to be true which is called andess in the grad logic system. And based on the greater logic system he created this method called RSP. So RSP is uh like a aggregation tree constructed using uh greater logic primitives. It's used by professionals to make serious decisions right like uh to buy a nuclear submarine

right that usually takes a few hundred or even to to a few thousand uh input and you create a very complex aggregation tree. Then at the end you decide whether you'll buy the submarine or not. But in this case I'm just going to show you a really really simple tree with just the two nodes just to give you some intuition how RSP aggregation works. So in this

case the the scenario is that I'm choosing a new job. Um I want to pick a new job and I have two criteria. One is the the money right? I want the job to be well paid. The other is attractiveness. I want the job to be interesting for me and assign a slightly more weight to the money part because money is always important. Then I aggregate the

two criteria using this uh operator called HC. So this is one of the graded logic aggregators. Imagine for the classic boolean you have and or not those kind of operators but for for greater logic you have like a much many more aggregators to express how those uh conditions should be combined and this whole tree can be expressed by a very simple uh notation uh like was shown

on on the bottom. You can see this is a very simple structure but the power of this is that it actually carries a lot of semantic meanings in this simple structure. Uh I don't think I will go through all of this uh uh one by one but essentially what this tree is saying uh for a job to be preferable to me uh it needs to be have

both high salary and high uh attractiveness. But if the salary is high enough, it can compensate the other side. So if the job is less attractive, but the money is really really good, I can compensate for that and vice versa. And of course, I won't accept any job that is too bad that the B salary is just unacceptable. So all that is actually captured in this one

single expression. And this system has been used in the past 50 years in many many projects to to be proven that it it's it's a very reasonable uh modeling of human decisions. Of course I I or either uh Dr. Yo Dumarich would claim that this is the method but we believe this is one of the promising uh method we can use. Uh I actually have the opportunity

to study under uh Dr. uh doomer uh 20 plus years ago when I was doing my master's degree and at the moment we were talking about some ideas to use AI to build this RSP aggregation tree but then we didn't do much we dropped the idea for 20 years but now we we picked it up again uh so that leads to this project uh which called bacon

I mean the the name is um it has a like a very long history be behind the name but but we can talk afterwards So basically the the the the goal of this project is to give machine this reasoning layer that is aligned to how human reasons. So because we believe the greatest logic can model how human reasons. So we want to force the machine to say

uh you can still do all your trainings behind the uh be in the lower layers but when you make a decision we force the machine to follow this um uh graded logic semantics to use this aggregation tree to make the decision. So in this way whatever the decisions the machine uh makes it will be aligned with how uh human reasons because we believe the greater logic aligns

with human reasoning. So this but we don't see this as like a replacement of human experts instead we consider this as a addition to human expert because now we are constructing that RSP aggregation trees out of data. So in the past 50 years the way RSP was used is that the human expert will analyze the problem decompose the problem and create that uh aggregation tree manually. But

what we offer here is that to automatically build that tree just based on data. So this data can reveal additional patterns um for the expert to make better decisions. Uh for instance uh in one of the papers we published not long ago uh we used this method to do a breast cancer diagnosis and the data set contains about 30 different features and we use a bacon to

analyze the the data and we decided only uh four or five of the features are important for the final diagnosis and then we used only five features to generate a very high accurate uh prediction and this has great implications patients in term of cost, right? Because in medical uh in healthcare, each feature has an associated cost, right? For for blood pressure or for any test, you have

to perform a corresponding lab work. But if we can reduce the lab work, then we can make the health care more approachable by a broader uh audience. Uh actually we did another study uh using the uh UC Davis data set. We looked at uh 10 uh important diseases and in general we can reduce the diagnosis cost by 75%. Just because we only need to look at fewer

features and by the way all the all the code what I talk about is all available uh on on GitHub. Uh unfortunately I have to skip the demos because my machine is broken. Uh but uh but the GitHub uh repo is there. Uh it's a little dimmed. Um but hope you can see it. Uh this became public uh 48 hours ago. So it's a fresh out of

the box. Um just uh I hope uh you can uh stop by check it out. Um maybe make uses of it or make contributions right because I skipped all the all the demos. So I actually zap through all the contents I have. So next I I'll just open up to to questions. anything that you want to discuss or even if you want to ask about the copilot

any questions comments or I'll point people to ask like a school teacher okay yes please thank you for present. Thank you for the presentation. Uh the subset of features that you picked eventually for the diagnosis of these diseases, what was the process how you picked those? Was it like did you determine um automatically by applying machine learning or and why did the model not figure that out

by itself that it only needs to look at >> a subset of those features? >> Yeah, that that's a great question. Actually we we wrote like a 18page paper to explain that. But in the gist is that uh we actually use a a binary tree in in this case. Basically imagine if you use a banner tree basically you are forcing the most important feature to surface towards

the top because the deeper you goes the less weight you have. So by this training process we actually force the features to surface at the top. Then we start to trim the trees to see if we trim the tree if there's no accuracy changes we can just discard that. So actually we didn't uh anticipate this would work at the beginning but it worked amazingly well not only

for breast cancer like I said for for 10 different diseases. We we can in most of cases we can use like a 30% of the features and we can still get the same result. And of course we used uh because we we don't really have access to like doctor expertise we asked chat GPT to see if the features we come up is reasonable. Of course GBT is

always very positive right they say oh yeah that makes sense. Um but but we did do do some uh actual analysis for the Gstone case. We actually talked to some physicians to see what the actual test they do and then we measured what are the tests can be reduced. That's how we come up with this uh 75% reduce uh uh figure. Yeah, that's a great question and

I think the the the paper was rejected many times but it's on archive. [laughter] So if you look at the bacon uh paper you can you can see it on archive. Yeah. Thank you. Ah actually I don't know that if this time is accurate. Does that mean I only used 19 minutes? Is that the case? How many minutes are left? I have 15 minutes left. I really

want to show the demos. I don't think it will come back. Yeah, because it's not even connected to Yes. Yes. Thank you. It's becoming awkward. [laughter] >> Uh maybe just a quick question. You might go into this detail in the paper, but in a lot of these AI explanability methods, um you can introduce like a proxy to say like, okay, the model needs to still fulfill the

original objective and then fulfill the reasoning objective or that I've put in. Mhm. >> Um maybe in your paper did you look into okay is this actually then what the model is doing or it's just fulfilling this extra thing that you've put in place but actually the underlying reasoning is still doing something that is not aligned with how your method wants. >> Yeah. So uh thank you.

So basically yeah explains the uh uh uh explainable AI a common approach is to use uh what is it called surrogate model. So basically uh you you still have the black model as it is but you use a surrogate model try to emulate the behavior of that model. The reason is that if they behave somewhat similar then you can use the surrogate model which is explainable such

for instance if you use a symbolic model then by explaining that you hopefully you reflect how that model works but um I don't know I mean it depends it could be coincident right I mean I think this method only works when you have like a significant uh large data set so that you don't u drag into this bias of overfitting problem. Yeah. >> Yeah. Yeah. Yeah, I

mean I suppose the question was maybe >> sorry maybe I'm not >> No, no, maybe it was a bit of like a general more philos philosophical thing of like how can you be sure that the ML model that you're trying to explain is actually following the logic that you want rather than fulfilling these objectives independently say oh okay I need to my the training wants to do

this I'll do this but actually the underlying logic still follows the old thing because that still fulfills the >> uh next token prediction task very well you Yeah. Yeah. Great points. Actually, this is a one of the problem of the so-called VIP coding, right? Because in the web coding is very goal oriented. If you give the goal, you will try to achieve that goal, but it sometimes

it it violates your architecture design that doesn't have the practice etc. So this is why this transparency on this reasoning layer is is important. The reason we believe this will work is that as a human expert, you can review the entire tree and you can mimic that reasoning process to see if it's it fits in with your intuition. And one of the key uh value proposition of

graded logic is that it's supposed to be intuitive. So if it's not intuitive, then something is wrong, right? So so so with that explicit uh traceability that you can actually uh trying to fully explain the the reasoning process and also what we also support is called called policy injection. Let's say uh when you uh let's say buy a car, right? You know the brand is the most

important then you can actually put a policy to say the the the the brand of the car would be a mandatory condition. It will be at the root of the aggregation tree. So you can inject those kinds of policies. In that case you are um combining the data insight and also your expertise to to the final tree and this final tree you can examine step by step

and you can change it you can tweak it. So hopefully uh that will give you like a more trustworthy decision process. We're not saying it must be corrected but it's more intuitive. Yeah. Yeah. >> Thanks. >> Thank you. >> Yes. Somehow it showed on that screen, right? It doesn't show on this screen. Yeah. >> H That's >> Yeah, that's >> Oh, yeah. [laughter] Okay. That's >> bacon.

>> That's a great point. Since we still have time, I mean, I think the next one is coffee break, right? So, we just have coffee break uh a few minutes early. So, bacon name. Uh this goes back to my high school. So I was reading a programming uh book, the basic programming book. And in that book, there's a bacon program. I don't know why it was called

bacon, but the idea of that bacon program is that you feed it with some data, then it will discover some physics formula for you like the Newton's laws and such. And I was so fascinated about that approach. And then later on when I started to do this uh bacon essentially as another more complex neuros symbolic network. So I wanted to use that name and I went back

to Google and everyone trying to find that reference but I couldn't find it anymore but I decided to keep the name uh anyway. Yes. So thank you for queuing it up. I was wishing for some opportunity to talk about that. Okay. I I think uh this is it. Thank you very much. Sorry that I went too fast because I didn't have the demo machine. Um thank you.

Thank you. [applause] >> [music]