About this talk
This talk covers the Mosaic project, an initiative aimed at the management, orchestration, and supervision of AI agent communities. The speaker, Massimo Tisi, discusses the challenges of using AI agents to generate domain models from natural language specifications. He introduces a multi-agent approach where a primary agent delegates tasks to specialized sub-agents for model generation, syntactic validation, and semantic validation. The session highlights the benefits of this architecture, such as increased reliability and cost efficiency, while acknowledging its potential drawbacks, including communication overhead and error amplification. Tisi also describes the project's open-source nature and the collaborative framework involving various partners across Europe, as well as its implications for software engineering and AI reliability.
Full transcript
[music] >> Hello everybody. So actually I would make the question how many people were here on Tuesday and saw my presentation. Okay. So I'm sorry but people were here because I'm going to show um several of the same content to introduce the project. Uh so if I don't want to waste your time, if you want to come back in 15 minutes >> [laughter] >> Ah, okay. Okay.
Okay. Okay. Okay. Okay. So um so I'm Massimo Tisi. I'm professor from IMT Atlantique and I'm coordinator of Mosaic that is a European project of management orchestration and supervision of AI agent communities. So it's about AI agents. Um so in this slide you can see a little example that I will show throughout the presentation where myself in the picture I use an AI agent uh and I
give a very simple task. The task is to uh generate a domain model. The domain model is a set of classes in practice with attributes etc. in a particular textual format. The format is from another Eclipse project that is called Eclipse Emphatic. Uh then I give a uh little specification of the domain in textual natural language. And uh of course I want this model to be um
syntactically and semantically correct. And of course uh the task is very simple so any off-the-shelf agent that I get today probably will answer correctly. Uh you see the correct answer in the picture. But of course you can imagine and if you tried these agents in your work but when you start writing a specification is much bigger, much more complex like few pages specification, you will get all
sort of errors, syntactical errors, semantical errors etc. LLMs are not that good. Uh that's why people uh write agents that are a bit more complex than that and especially they have some specific logic to handle some cases. Uh there are several approaches. I'm going to talk only about one of these approaches that is called the multi-agent approach. So, in this approach um the blue agent uh that
I used before is not directly inferring the answer, but is inferring a set of uh sub prompts to uh delegate to some sub agents. For instance, it can delegate to the first sub agent, the green one, uh a prompt about uh generating the model. To the second sub agent, it can delegate a prompt about syntactically validating the model. And this is a This is This makes sense
because maybe in the second sub agent you can equip it with you can embed some tools to check the syntax uh like the emphatic uh parser and validator. So, it can be trained to uh to use well these tools. And then you can have a third uh sub agent that uh is in charge to validate the semantics of the model against the specification. And of course, here
you want to use an LLM that is different from the first one to raise a bit the objectivity in uh in judging the work of the first agent. Um of course, then the the blue agent needs some logic to uh gather all the results of the sub agents and to maybe iterate how to get um some satisfiable answer for the user. Okay, this is a very popular
approach today. Uh a lot of people are doing this. Even when you launch Copilot, etc., you uh there is some multi-agent thing going uh in the ground. Why? Why it it it can make sense. Uh one reason is because it allows you to specialize the sub agents. So, you can make agents that are optimized for some sub tasks because they have specific personas, specific tools, and uh
they have an LLM that has been benchmarked well on that task. Another reason is variability. Uh variability means that you can replicate the same task a lot of times on different agents. Agent that has some difference in LLM persona or tools. Uh we hope that this will increase your success rate or if you do some kind of uh majority voting maybe getting a more reliable answer. Another
reason is cost. You can route uh simpler task to cheaper uh agents, But of course it it comes with some problems. Uh it's not trivial that this is uh this is a positive approach in the sense that it does a lot a lot of overhead. The more agents I have, the more of the communication, the more the LLM calls, and the slower the response time. And also
it's not clear what happens with errors. They could cancel themselves in the in the network, but they could also amplify themselves. So there is a a lot of research. Research is quite active here. Um and I would say that the the the research field of was kind of positive up to 2024. There was very discussed paper which was called more agents is all you need. Uh that
was showing that we with a very very simple strategy that is a replicating the same task and the majority voting of the most popular answer, you were getting some advantages, let's say some performance gains in a lot of well-known benchmarks. But okay, today instead of the the view on multi agent is much more let's say skeptical, but in in a sense more critical. Um there's a nice
paper from Google that shows that I mean if you don't use any particular structure, you let the little group of agents, I think six agents if I remember correctly, um work independently with each other, uh you generally don't have any advantage with respect to using a single agent when you use frontier model, let's say the top performing model in this graph. This will be the top Google
model. You can get some advantages when you when you less reliable model, let's say, but with the frontier model usually it's not that's not the case. So, anyway, the also the setup the experimental setup is very limited, as I said, a small group of agent in a very dynamic way. So, the agents computes the collaboration at run time. So, a lot of research is still still has
to do be done. Mosaic is really about that. So, Mosaic means management orchestration supervision of agent communities and the purpose of the project is to enable and study the cooperation of a possibly large number of LLM agents. And the idea is can we use this collaboration among agents to increase the reliability of LLMs in software engineering. It's an Horizon Europe project. We have 12 partners in Europe.
IMT Atlantique is coordinating. We proposed this project in March 2024. The idea comes from 2023. I just say it because I'm again I'm proud that we could foresee a problem that in 2026 is very actual. And okay, we are at the the beginning of the second year of the project. one year passed something, we have something to show. So, the first thing that we can show is
a client. So, it at least gives you an idea of how Mosaic looks like. The client you can see a screenshot in the right hand side and it really looks like so it's it's a VS code extension. So, it really looks like a copilot, somehow. Um so, it's an AI assistant. Um the for for implementing this uh UI, we forked an open source project that's called Continue
uh that has Apache license. Uh so, you see here the same example I showed. You see the the answer. And the only thing that you will see differently than what you see in copilot is some kind of tracking of what happens of the backend. So, this is exactly what I showed in the example. I have a sub-agent that computes the model, a syntactic supervisor, and a semantic
supervisor. And you can track what is their feedback, what is their evaluation, etc. So, this client we made it available during the during OSX, and it has already 5,000 downloads. So, uh we are kind of happy of the popularity. Um and you can download it in open the SX right now. Of course, the client is only the left hand side here. Uh so, it sends the prompt.
But at that point starts the complex part. Uh the client communicates only with one agent that is called the reference agent that is this red agent here. And in Mosaic, we have a hierarchy of agents. Actually, these are all the kinds of agents that you can have. So, the reference agent typically delegates the whole task to one agent that is called the collaboration agent. The collaboration agent
is an agent that has the task to devise the collaboration. I will talk a bit more what it does later. so, it computes a workflow that involves a lot of other agents that you can see here. Some of these other agents, sorry, some of these These agents are still collaboration agents, so are still green. Uh because there could be some task that require some further uh decomposition.
Okay? So, it's not sure that all the task are actionable by a single agent. Um so, that's why the decomposition is a hierarchical. But at some point, some of the task are given to the blue agents that are the the ones that do the the real work. The we call them solution agents are the one uh either an output or an artifact. They want to perform changes
on your file system or to launch some NCP tools on the cloud, etc. So, this is the blue agents. Uh since we try to increase reliability on the multi-agent system, the work of the blue agent is uh typically passed to some yellow agents that are called the supervision agents. Supervision agents have the job of the only job that is evaluating uh the task of the solution agent
the work done by some solution Um that's why supervision agents typically can be replicated a lot. For instance, to for having several different kinds of evaluations of a single uh solution. And that's why since we have several uh evaluators, uh at some point the um the framework needs to take decisions and the decisions uh need to uh reach a consensus among the evaluators. Uh the consensus can
involve a debate. So, it's not uh is not trivial to to be reached. And we have indeed another kind of agent that is called consensus agent uh has this task. So, this is what happens at run time. Then, we have some global two global uh components that are called the orchestrator and decision engine to support both uh the collaboration agent and uh consensus Finally, we have a
repository. Uh the stores the code of uh these agents [clears throat] or points to the code of these agents what you use external repositories and then performs in background a benchmarking process. So, we compute metrics on these agents and these metrics can be used by the collaboration agent to select what agent to invoke. So, what are we delivering? Uh we are delivering four components that are the
four colors that you see here. Um in orange, starting from here, we are delivering an orchestrator with the collaboration agent. The agent that devices the collaboration. in violet, in the second part, we are delivering a decision engine with the consensus agent. We are delivering a protocol to make all these agents communicate among each other and we are delivering a repository uh the benchmarking infrastructure. Um okay, so
what are the principles? So, the point of this slide is that there are a lot of other agentic platforms. So, this is just one of the agentic platforms that you will see in 2020-2026 appearing. Um so, what are the the the design decisions that make Mosaic different from from the others? Uh first of all, it's a platform that is polyglot and very and really agnostic from the
way you are developing your agents. So, we the whole project is about the collaboration among agents and not the what happens inside the single Um that means that the agents are isolated, each one runs in its own Docker container and they only communicate by protocols, by network protocols. So, you can use the language that you want, the framework you want to develop each single agent. And this
allows us also not to compete a lot of bigger actors in the market. Then the second important point is that it's a platform based on reputation. So, as you saw, we are computing these metrics by benchmarking. We are also tracking the performance of these agents in online during the tasks. And this has the purpose to build a reputation for the agent and to make it more suitable
to be invoked the next time some similar task is needed. This really to us we are really this metaphor of the community. So, reputation is a bit like in a community where agents somehow know each other and they know who to call to to perform a certain task. Another factor that you will not see in other frameworks is explicit ness. So, when we go a bit more
technical, we have explicit representation on one side of the multi-agent collaboration. So, we have artifacts that represent the And also that represent the what the consensus agents works on that are governance rules. But also within the agent, we suppose that the agent has an explicit representation of the mental model. Okay? We we use as an abstraction for the mental model an old an old idea that is
called belief desire intentions. As I said, the the implementation of agent is opaque. But in the protocol as sometimes we we ask for this structure. So, the agent can compute them on the fly or it can, and this is more advantageous so often, them maintained in some data structures during the execution. So, all of these all of these makes things explicit and we can then the framework
can query this information at runtime to optimize the collaboration. And that's what makes our collaboration working. Uh fourth point, we use uh languages with formal semantics uh throughout the platform. So, all things that I told about uh we tend to uh represent them in languages with formal semantics and not in simple Python code as uh or or or JSON JSON files as you see uh And finally,
um Mosaic is open uh in all senses of the word, I would say. first of all, we have uh MIT license for all the code in the platform. All the data sets, all data, and all the experimentations are published in Zenodo freely, so you can download the data sets and do your own things. Uh all publications are in open access, and the technical reports too of the
project are all public. Uh Eclipse is a partner of the project, and the project is also open to contributor collaborations, especially we really need in the project um to to develop agents. So, we are developing agents in the consortium, but we also ask for help from from the community to start developing agents and making the platform useful. so, I took some time, so maybe I will skip
the technical side, but if we have time, I will go back to them. Yeah. The consortium, we have the four um universities that uh deal with four technical uh sites of the project. >> [snorts] >> Um we have an integration that is Intrasoft, a company for integration. Uh Cododo is a company in that is uh quite well known in VS Code for tools that use LLM for
software engineering tasks, for instance, pull requests uh um reviews. Um Eclipse, as I already said, FC says this with communication and dissemination, and we have the four use with four in four different domains. And the use cases are these one. We have a immersion, that is a company that Okay, in all cases, they try to use Mosaic to help them build some software, of course. What changes
is the software. So, for immersion, the software is mixed reality content for VR and XR device. For Collins is a prediction component for estimating remaining useful life of of a component in the in the airplane. For Unparallel is maintenance and testing of a catalog of IoT solutions. And for National Technical Greece is a recommendation system for investments based on user profiles. So, the best way you have
today to give a look to Mosaic is to download the demonstrator. You have a URL there, and in the URL you will find also one liner that uses Docker Compose to download the whole bunch of things that you see in this picture and to run them. What do you will download? You will download the client, the reference agent. You will download a few agents that include also
the agents of my little talk. My little example. They will download the repository with its database. They download also a an observability platform and Fuse to monitor what happens with your agents with the data storage for it. Maybe don't do it today because I think that because of sex we were overcharged, so now things are not downloading. Actually, we have too many too too many requests, but
maybe tomorrow will be better. Thank you. >> [applause] >> This session question uh I need the I assume you are familiar with the stuff of 20 years ago like >> FIPA and the JADE platform, right? >> Absolutely. Sorry, I I really need to go >> me? Do you know who I am? Uh JADE is my PhD thesis. Ah yeah, perfect. >> So, I wanted to understand. So,
did you fix what France Telecom did wrong with the semantics of the BDI in FIPA? So, what what what what semantics did you use? Sorry, so what we are doing about FIPA and actually we are really looking at FIPA, yeah. What we are doing, so I I was Sorry, um So, at some point in the proposal we said we are we are developing a new protocol Okay.
uh for making the agents communicate. Um actually then the let's say industry is aligning a bit on an existing protocol that is called A2A by Google. Okay. So, what we are doing is extending A2A with some things that we think it's are useful. On one side the things about observability and the other side some performatives. The performatives are taken from from FIPA. uh Yeah, so so so
far we are staying with FIPA but we are looking also at KQML and uh because there's some things that we like there too. and the idea is that in A2A you can have you already have some primitives for the life cycle of a task Okay. but you don't have any primitives for the uh for the beliefs of the agent, the the intentions of the agent, so the
commitment of the agent to some actions, etc. So, this is what we are using FIFA for, or at least the primitives. Okay, but but you are really trying to do a formal semantics that let's say judges admissibility of interactions based on BDI. Uh no. Okay, so what what what what do you use BDI for? Because you said that you require that an agent kind of declares or
I guess believes I'm sending this because I believe that, right? Yeah. So, we Maybe I have another Yeah. So, I have slides for everything. Um so, for instance, we try to use it for to find a compromise in disagreements. So, so as I as I said, there is a a notion of consensus at some point. So, at some point we we query the beliefs to try to
find inconsistencies between the beliefs of two agents that give us a different result, and we try to see if we can come up with something that that can be a compromise between them, or at least to communicate We communicate with them to see, "Okay, look, you did this because the the other agent has a different belief on this point." Or we use intentions to block conflicting actions
typically. I mean, this is very common because all agents try to write on your face system. >> it like arbitrary So, sorry if I interrupt you. I don't because otherwise I'm going to steal all the time. So, I just want I need to speak with you more. Yeah. Okay. Yeah. So, Well, I didn't know. I I I wouldn't have I wouldn't take any coffee, Rosario, if I
knew that this was a Ah, no, I'm so sorry. Yes. Yeah, but uh But no, I'm sorry because I skipped all the technical part because the the workshop here is about collaborating on projects, but I think it's very useful, yeah. You ready, Sebastian? Yeah. Okay. >> [music]
