About this talk
In this session, Matthew Khouzam discusses the integration of AI with trace analysis through the Machine Communication Protocol (MCP) and the Trace Server Protocol Machine Learning Library (TMLL). He explains the significance of tracing in capturing system events at nanosecond granularity and the challenges associated with analyzing large volumes of trace data. By implementing MCP, users can interact with complex tools using natural language, making it easier to draw insights from intricate data without the steep learning curve typically associated with traditional tracing tools. Khouzam also highlights key features of the Eclipse Trace Compass and TMLL, including support for the Common Trace Format version 2, and demonstrates how to wrap existing command-line interfaces as MCP servers, thus enhancing their AI accessibility. The importance of using deterministic methods in trace analysis is emphasized to reduce the risk of errors when utilizing AI.
Full transcript
[music] >> Hi. Um my name is Matthew Khouzam. I'm going to be presenting AI power trace analysis adding MCPs to TMLL. And this is a talk aimed for those that are AI adjacent and that potentially have tools that they want to bring into the AI sphere. But it's also a bit of a war story. So, I hope you enjoy. So, let's start off. Uh here's the agenda
of what I'm going to be talking about. I'm going to set the context on tracing. It's a very interesting domain and I'm not just saying it because I've been working in it for a long time. Um then after that, I'm going to be explaining MCP. And we'll walk through an implementation step by step, show the debugging tools, and uh I'll show off the results at the end
across multiple AI clients. So, before getting too deep into it, a quick background. My name is Matthew. I've been working in open source for over 16 years. Uh mostly in tracing and observability. And I work closely with universities. I am a former board member and uh for the Eclipse Foundation and I also have the privilege of being paid by Ericsson to make the world a better place
through open source and for that I am eternally grateful. next up, before getting into MCP and how to connect it uh how to connect your application to uh AI, I want to discuss tracing and Trace Compass and why do we want to add AI to it? Because we don't want to just do it because it's cool. You need a business case. Tracing captures what actually happened in
your system. It's nanosecond granularity in a machine le- uh readable way. Unlike logging, it's designed for minimal impact to the system and minimal overhead. There is a downside. It produces enormous volumes of data and they're hard to analyze manually. Eclipse Trace Compass um could be seen as the Swiss Army knife of trace analysis. It can handle kernel traces, user space traces, network traces, stitch them all together,
put them across a whole a whole system, a whole cloud, and trace the cloud. it's very powerful, but because of that the learning curve is incredibly steep sometimes. Um you need to know which views to open, you need to know how to interpret the data. One of the most most brilliant men I know yesterday stated, "I just implemented the tool. I don't know how to view the
traces in them." why add MCP? I'm going to give my personal motivation. I have spent 16 years building these tracing tools and using them, and I watch users struggle. And they're struggling where I see that it's very easy to get to the next step. The MCP lets users put natural language interface on top of your tool. And so instead of teaching someone the tool itself, you have
the tool adapt to the person. I think this is a much more natural and human way to compute. So, before getting a little further into this, um Trace Compass has a 11.3 release that is coming out and uh we're on the road for 12. And uh the new big feature is parcel CTF2 support. Now, CTF2 is the Common Trace Format. It's um the standard format for LTTng
and OS level traces as well as user space level traces. It's binary, compact, efficient, and uh the new one is um the same, but with JSON metadata and handling of many new features such as binary blobs and compression in the trace. It's very exciting, especially if you get down to the bit level. And on top of this, we have a library called TMLL. Now, the TMLL, or
um Trace Server Protocol Machine Learning Library, wraps machine learning techniques into a clean Python API, and it talks to the trace server, uh which uh which is a library that is the back end for Trace Compass, and makes the data AI accessible. Until now, users need to write Python code to use it, or Jupyter notebooks, and this is a gap that MCP is filling. So, what is
MCP? It's an open protocol. It was developed by Anthropic. Um it standardizes LLM interactions with external tools. They advertise it as the USB for AI. Uh one connector for every device, and what I would say is that it's very much the USB-C of AI, where it's mostly one connector for every device, but you have to have a few um points of caution. Uh fun note, it was
released in November 2024, and I believe Eclipse Theia supported it in December, or the first week of December 2024. So, um the Eclipse ecosystem knows MCP well, and there's a lot of extra expertise around here. in a nutshell, MCP is um something where your AI receives tool schemas, and it decides autonomously which tools to call. And the protocol handles the plumbing, the AI handles the reasoning. So,
with this, uh we can keep it very simple for the user. Why would it matter for you? And I'm going to give you the practical pitch you can give to your managers if you want to try this out. why should you care about it? The AI picks the right tools and parameters. Uh the actual computation is in your deterministic code. I'm an engineer. I don't like non-deterministic.
I really like it when I give the same input and I get the same output. So, this will reduce the amount of hallucinations and more on that later. without the MCP you'd have to paste trace data into a prompt and ask the LLM to analyze it. Now, traces are big. I don't know if you've ever tried pasting 1 MB into ChatGPT. Well, traces are 1 GB to
1 TB. So, you would be running out of tokens very fast. You don't need to rewrite everything also and that is very fun. You just wrap I'm going to be showing a workflow where you wrap an existing CLI as an MCP server and this will allow you not only your tool work with a CLI but an MCP. A win-win. So, before we build anything though, let's justify
the architecture with some numbers. Why do we need MCP and TML at all? Again, why can't you paste the trace into your favorite AI? Let's call it Gemini this time. I did some calculation. And this is very important. Um I tried with Claude and it decided to read the trace using Babeltrace. it started reading and reading and reading. I stopped it after 10 minutes of reading. Um
and I saw that was about 2% through Reading one trace might cost you $1,000 in credits. We don't like that. We like not paying With MCP and TML at all your the amount of tokens that it took were 2,000 because it basically said, "Open the trace, read the trace, post the results." And the AI was just the natural language interpreter between and the deterministic software. This is
I think is a very valuable future for tracing and for many other projects within Eclipse. We have a lot of brilliant business logic that we should be considering wrapping in this. Um so the sub-process wrapper adds about a second per call. it's completely dominated by the time of the LLM inference, at least. And even worse when I try it locally. so in my opinion in terms of
performance, the overhead is not bad in terms of time. as we've mentioned earlier, I'm going to go back to the previous slide, it's not much in terms of memory because your tiny compared to before. And another thing that I really like is that it's running the data is running deterministically on your computer or on a computer that you own. The trace data is high sensitivity. I can
reconstitute a lot of things from there. You can figure out a lot of what people are doing and you can even figure out their basic patterns and behaviors. Uh this is not something that is you should take lightly. It could be leaked. You can have the calculations done locally and have the LLM just interpret the abstracted aggregated results. this also, um I would like to mention the
free compute. It's not free. Nothing is free. It's running in my example on my laptop. And it's just that we You paid for the So it's a it's already in fixed investment. We don't have to pay extra So, let's start. When I create a MCP first, and this is not the only way to do it, but it's a way I like. We create a CLI for the
tool. What's the advantage of this over just doing it over a dedicated bespoke API? Well, you get a CLI. People like CLIs. It works well with your CI also. So, here is a normal Python CLI. It's It's arc parse sub commands, print output. There's nothing special. It's wrapping the TMLL analysis. Now, I don't know if you can see well. I will bring the mouse here. TMLL client
is run, then you get your experiment. You find your outputs. You do an anomaly detection. You find your anomalies. And then you print them. Easy peasy. Now, you got to write this in an MCP. Now, I made the MCP fancier since last And the patch you'll see it on GitHub. So, it's actually now about 700 lines. I'm sorry for that mistake. But, most of the work is
adding the at sign MCP tool. And this is using fast MCP's library. And then you do your a wrapper for detect anomalies. And the very important part, and I love this. If you want your MCP to behave well, comment your code. Explain the justification. Your comments become API. Because the MCP or the LLM gets the comments from fast And therefore, when you're doing this, you are making
the the project a lot less prone to mistakes. I'm not calling it hallucinations, but these are just So, here's the basic stack. You have your on the very bottom, any agent that you want, any AI agent. Uh be it Hero, Gemini, Thea, Goose. And then you have your MCP server running. so the um the AI agent will call the MCP server which calls the CLI and then
the CLI calls the library. Now, the library is resident in memory. Therefore, it's going to keep a state. It's not purely restful. And this is very good. You don't have to juggle contexts. So, why am I suggesting this kind of architecture? I really want you to separate your concerns. You'll get a good CLI out of it. And MCP is relatively new. You're wrapping a CLI with this
new tech. I suspect MCP is here to stay for a long time. It's already got a brilliant ecosystem. But I would like to make sure that if you want to wrap something else in it, you can and that you're going to be on a hardened code path. And um that is for me the most important part. One code path. You don't want to have many code paths
to the same data because then testing, debugging, and all that jazz becomes a nightmare. So, we exposed 12 tools in MCP. They're very simple. Each maps to a CLI sub command. It's Once again, we just at MCP tool them. And another feature of MCP, and this is please remember this you can now progressively discover your tooling. Progressive discovery means that when you and when you announce your
capabilities, you announce a subset of your capabilities. For example, with tracing, the first thing we announce is, can you check if the trace server is up? That's all we announce. And because of that, until you use it, your context loses like three tokens. It's tiny. If anyone says that MCP is slow and eats your context, it's because they had a bad tool, not because the protocol was
bad. once you use it, then once you use your tool, then you can you'll progressively display other tools that are available, such as um create an experiment or open an experiment. Because of the sequence, it actually forces the LLM into a specific UX, and that is very powerful because at this point, you no longer are going to be um prone to hallucinations because the AI has no
choice. The LLM has no choice but to pick the flow that you're giving. This is one of those cases where security, and UX are perfectly aligned. So, next up, MCP apps. This is one very hot feature. Um you might have seen it if you opened um like a chat GPT, Gemini, Claude and asked for airplane tickets and you get these ads in your um response. You get
rich context returns. And it's basically you could put an iframe in your um AI uh return. You can put a PNG, and the AI will embed it directly in the So, now, for me a very important part, also. You're developing it. And now, I'm I'm a pretty decent developer, but I never get it on the first shot. How do you debug this? there are two tools that
I really strongly recommend. Um Eclipse Theia and the MCP Inspector. Let's get into Theia first. So, Theia offers this feature called the agent history. The agent history allows you to see the raw JSON that's coming in and out of your MCP. This allows you to figure out what worked, what didn't with real-world I'm going to go back and forth here. The MCP Inspector gives you the ability
to give fixed input and get fixed output. is a standalone web tool that allows you to connect to an MCP server, browse its tools, call them interactively. You don't use AI, you don't use tokens, testing that everything works. This is what you use once you think it to make sure that it works with real-world data and real-world prompts because when you're writing into your prompt of your
AI, typically you're not controlling the entire prompt. You're controlling the last user chunk of your prompt. So, this will allow you to see your entire history and see why it works on your machine but not in deployment. This is a game-changer. So, with these two tools, you actually can deploy things very fast. I was frankly shocked with how fast I was able to get things up and
running. So, let's look at the some basic I made the one MCP server and you saw how. the server was working across multiple AI clients. I tried it across Goose, Hero, and of course Theia. Um and I wrote it once and it worked everywhere. That's another feature that you want to have. So, with Kiro IDE, Kiro calls the TMLLMCP, and the user asks in natural language what
he wants. In this case, uh we don't see it in the screenshot, but it was plot the anomalies in CPU time for a given trace. And you can see on the bottom right here that you have the anomalies. I will use the mouse. The red dots are the anomalies. You basically limited your search scope 10 seconds 100 microseconds. By doing this, you can pinpoint your problem a
lot faster. And I'm going to take a step back. When you're doing tracing, you have to first know if there's a problem. And then once you know if there's a problem, you have to find where the problem is, and then you identify it. You cannot identify a problem before knowing where it is. You cannot know where a problem is before knowing if there is a problem. This
is a very logical sequence, but people tend to try to do everything at once. So, here we were able to find it. And this one also is very special for me. Here we have the Thea example, and I used the free tier of Google Gemini Flash 2.5. Why? Because it has billing that's transparent, and because it is the least performant free tier AI that I could find,
and it still gave the great on Thea, and you'll notice that the same query gave a different picture. And this is something that I would say is um needed in the industry that there's a kind of alignment between all the players on this on what is being outputted. Um I like this one a bit more because you have a scroll bar, and you have a decent size
picture. With this one, I actually had to take the magnifying glass to see what's going you know, there's This is my taste. So, the very important part, and this I need to remind you, is reading a 1 GB trace file. It took, if you look up there, five requests. Oh. Five requests. kilo tokens to find the anomalies in this giant This is savings, and it's also savings
using unused compute on your laptop, keeping your data safe. And this is the kind of mindset that you should have, that MCP will allow your current legacy apps, and I don't mean legacy in a bad way. I mean that they are properly tested. You can use the You can use MCP to allow these applications to be AI driven and accessing the power of the prompt. Okay. particularly
fast. The main message um is that making your AI tools or making your tools AI accessible via MCP is very easy. Um it's surprisingly easy, to be honest. I have the patch up, little surprise for later. I do want to give some words of caution, like bit of my war stories on it. Um there are a lot of poorly written MCPs out there, and they are giving
the technology a not so great name. so, don't let that that color your view. I think you should really try out the tech and understand it before making your own judgment. Um there are and there are other technologies out there like skills and we can discuss that. Um but you know, a bad REST API design doesn't mean that HTTP is broken. And a sub-optimal MCP design doesn't
mean MCP is broken. Uh so let's go through a few minor um suggestions. If you are making an MCP this kind of pattern your library to a CLI, your CLI to an your MCP to the agent very easy, very maintainable, very debuggable. if you make your MCP server test it over several agents. Now if you are 100% hardcore open source resolve. Thank you. Uh secondly Eclipse Theia
and Blocks Goose are two excellent examples of AI agents that you can use to test both ends on the extremes. progressive discovery is a game-changer This not only improves the performance but the security and the UX. when you do your MCP, you can test it locally, you can test on a server. This is all not AI specific. This is basic computer architecture. AI didn't mean that architecture
is deprecated. You need to architect your data flows. The bottleneck it's pretty much never MCP. The LLM is going to be slow. You have to accept it and understand a large pa- a payload and that the results are very high quality typically in terms of the UX and the look and feel. you'll take that hit on performance, you'll have a 5-second latency but you'll get in a
good result or a happy result. Um yeah. And main takeaway also, MCP is the protocol, not a product. this is the summary of what I wanted to say. there's one little bit of good news, and I added this at the very end. This is the PR for the MCP. And as of last night, the maintainer of TMLL, who is not me, his name is Kavisha Heddi, and
he is a beautiful, brilliant, and funny PhD candidate approved it. It was merged. This is in the wild and ready to be consumed right now. And I hope that we can have a discussion now on what your needs are and how we can help you with making your MCP designs. Are there any questions? It's not on. It's not Now it's on. Anyway, uh given if you had
a CLI that has a proper help like uh description, can you actually automate the generation of the the MCP? Yes, you can. Okay, the question. If you have a CLI that has the proper help, will you be able to to automate uh MCP? Uh the CLI that I did has a proper help. It did not automate the MCP, but it made it a lot more enriched. You
still need to explicitly state what you want. And when you're doing progressive discovery, you need to explain the order of discovery. This does bring up another point that I forgot. When you're making your MCU When you're making your CLI, you need to make sure all your errors go to standard error, not standard out. Because the MCP is expecting a JSON and getting error messages in your JSON
is the best way to make it break with a really bad error messages. Thank you for reminding me of that part. Thomas, another question. I have lots of questions. Good. I have lots of voices and things to say. You said it's expensive to actually analyze traces with uh directly right with AI. Yeah. But isn't AI quite good at at finding patterns? And is there research on whether
that would be something good even if it costs a lot? Um So, the question um would AI inference on trace data be something good? Now, there is a lot of research being done in that and I am participating in a lot of research about that. So, we can do this for a few hours, but the gist of it is that AI typically will have the good ones
will have a context of a million tokens. a trace is billions of tokens. So, it's a thousand times larger. Which means that you typically will read the full trace, compress, read, compress, read, compress, read, compress. Then at one point it's compressing you have to compress again and do that a few thousand times. Which means that the context kind of grays out. Now, one statistical advantage is that
typically you trace and the crash is at the very end of the And that is the most weighty item your context. However, at that point your context is so full that it your AI will tend to hallucinate given current technology. this is something where we have deterministic means to get good results and these deterministic means can give hundreds or even thousands of results and you can use
the AI to summarize those results. that would be where I think the quick wins would be. Is that is that uh answering the question? Uh yeah, more or less. Uh thanks for the great talk. Um Thank you. I really can confirm that MCP is awesome. Um I'm a maintainer of of things needs for traceability data. Mhm. Um I've made a um experience that MCP is great um
because you as a maintainer or developer can define what's the description of it it um of the features. And then the agent can decide, "Okay, how to use it." The problem is that if you as a maintainer does not know exactly how the feature gets used inside a project or something like this, then sometimes it gets misused by an agent. That's my experience, at least. Oh, absolutely.
And then I have um at least the possibility to tell via agent description, "Okay, use me the command line tool or interface because then I can give a more specific description when to use it." Can you confirm this or do you have some some workarounds? Okay, so um the gist of it is that you list all the tools available and then the agent uses it in a
creative which might not be considered right. Or might be considered the opposite Is that correct? Okay. So, progressive discovery is a really good counter to that. Where you only show the first step of your chain at first, your first tool. And then, once that tool is called, the next tool is available. By doing this, you limit the error surface for the LLM and the agent. I've had
very good success with that. Um another way that I would recommend looking into it is there are other texts out there such as skills, I can't recommend skills for my problem space because are non-deterministic compared to MCP. They give very good results, but sometimes they don't. Whereas MCP will always give the same result. It might not be good, but it's always the same. you have to look
at your problem. Um, I hope that if there's one thing that's retained about this is that engineering is not dead by AI. Software engineering is not dead by AI. You're going to use these tools and keep We'll keep on trucking for a few years at But, um yeah, I can confirm everything you said. I can There are some tech tips and tricks that can minimize it, but
I've never seen it completely eliminated. the other problem is that "It works on my machine." will become "It works on my model." Uh, is this This kind of confirms what you've experienced, right? >> Yes. Okay. So, people, it means if you have MCP questions, you can go to him, too. Are there any other questions or comments? If there's nothing else, the coffee is exceptional in OCX compared
to EclipseCon. Thank you to whoever is sponsoring it. >> [music]
