MOSAICO: Management, Orchestration and Supervision of AI agent Communities

This talk presents the Mosa project, which focuses on the management, orchestration, and supervision of AI agent communities. The speaker, Masimotiz, introduces the concept of multi-agent systems where a primary agent delegates tasks to specialized sub-agents, each responsible for specific operations such as model generation or syntactic validation. He discusses the advantages of this approach, including specialization and redundancy, while also addressing the challenges associated with increased overhead and error propagation. The Mosa platform features a structured architecture based on protocols like A2A, enabling agents to collaborate through defined belief-desire-intention models. The speaker outlines the project's current phase and its components, including a user interface akin to VS Code, and emphasizes the open-source nature of the Mosa platform, encouraging contributions and collaborations.

[music] Uh hello everybody. I'm Masimotiz. I'm a professor from IMT Atlantic and I'm here to present the mosa project that I coordinate that is about management orchestration and supervision of AI agent community. So it's a project about AI agents. Just to know in this room, how many people can you raise your hand if you're using AI agents for your work or a lot of you? Half of

you. Great. I I I do the same. Um so to illustrate the project, I start with a simple example. Uh this is me in the picture and I'm giving a a prompt to an AI agent. It's a very simple example just for pedagogical purposes. In this example, I am asking the agent to generate a domain model, a set of classes, okay, and attributes in a specific textual

format that is from another Eclipse project that is called emphatic. Then I give a short description of a domain and I of course I want this uh this model to be syntactically and semantically correct. Okay. Now, if I ask this to any of the shelf agent, this is a very simple task. I will probably get a the good solution that is the one that you see here.

But you can imagine if you if you try to do something like that you can you can agree uh as soon as the specification grows pages of specification you will find errors you will find errors syntactic errors semantical errors etc. So what people do they build agents that are a bit smarter for specific tasks and there are a lot of different approaches. Here I'm talking about one

specific approach that is the multi-agent approach. Uh in the multi-agent approach we uh don't have the blue agent inferring or computing the answer by itself but the blue agent let's say computes um the composition of this task and then delegates to other sub agents. Uh for instance, in this little example, he could delegate to the green agent uh the task of generating the model. Then he could

delegate to another more yellowish agent the task of syntactically validated the model. And this is nice because the yellow agent maybe is an expert in using some tools like the emphatic parser, the emphatic validator. So it could use tools to synthetically validate the model. uh and then it could delegate to another agent, the till agent the task of semantically validating the model. And here maybe you want

to use a different LLM than what the first agent was doing to to have some objective uh validation and things like that. So in general uh this is uh kind of common today people are doing this uh actually even when you uh when you call copilot for some task there is some multi- aent thing going on and of course then the blue agent needs the way to

put things together uh even iterate to arrive to a model that is satisfactory for the user. Why? So you do multi- aent one because you want to specialize. So you want to get agents that are experts in a subtask in a specific task. So they have specific persona uh tools LLMs that are suitable for uh or optimal for a certain task. But then another good reason is

some variability. Sometimes, especially for critical uh critical uh environments, you want to replicate tasks on several different agents uh either because then you increase the chances of uh task success or because you want to do some voting to to get a single answer out of them. Sometimes people do it also for cost reason. Uh typically you know you you if you there are there are some setups

where an agent delegates to a cheap agent when the task is simple and to a expensive one when the task is complicated. But of course the method comes with a lot of problems even if now it's it's becoming very fashionable. One is that there is a big overhead. So the more agents you have the more communication the more LLM calls and the slower response time. And also

because errors propagate sometimes unpredictably and so you may have errors that don't cancel themselves in the network but that actually um amplify. So research is active in the era. Um let's say the feeling of researchers here uh was very positive on multi- aent up to 2024 I would say where there was this fashion like more agent is all you need. This is a kind of well-known and

uh discussed paper let's say that was showing that even by getting a set of agents and a very simple strategy like replicating the task and majority vote uh for getting the best solution you are increasing the performance of LLMs in a lot of as you can see in the picture in a lot of well-known benchmarks but actually in 2025 there is a very nice paper by Google

uh that shows that things are much more complex and for inance that uh if your agent is smart enough like you see in these graphs here uh at a certain point multi- aent actually uh hurts your performance you would better get the the top frontier model from Google or from open AI and you would get better result in a task that you were doing in multi- aent

way but of course also here the research is quite kind kind of limited uh the setup is very uh un so very dynamic let's say so the agents compute the collaboration dynamically every time. Uh and it's up to six agents. So I mean the studies has its limits but still it it shows us that this is a very active area of research and and nothing is very

clear. So this is what the project is about. Uh Mosaic is about enabling and studying the cooperation of a possibly large number of LLM based agents. uh and asking ourselves can we increase reliability in software of LLM in software engineering by uh this large number of agents it's an Azen Europe project 12 partners in Europe IM Atlantic is coordinating uh we submitted the proposal in March the

project started in 2025 for three years and actually the idea comes from 2023 when we started writing this proposal and so I'm pretty proud that we kind of predicted something that gets very very fashionable in 2026 and you can find more information of this in the website. So we are at we finished the first year we starting the second year of the project. Uh so we have

uh something to show uh uh in this first phase we are dealing with some components that enable at least the platform etc. And it's in the second phase that we will deal with more the scalability part. So the first thing we have is an interface. So um a user interface for mosaic. Uh you can see a little screenshot in the right hand side and you can see

that it looks a lot like copilot. It's a uh it's a VS code extension. It's an it's an open source extension. It's a fork of continue. Continue is an open source project released in Apache license. And um you can see here I'm using the same example that I showed you before. I I give the task I get the right answer. The only thing that you can see

differently is that you can track each sub agent here. It's exactly the architecture that I showed. So I have a generator agent that has completed. I have a synatic supervisor and the semantic supervisor that each one uh do their job and report on their uh on their activity. uh actually the client talks with a single agent that we call the reference agent that it's has the role

of interfacing with the user and then the client has this um purpose of exposing resources from the ID up to the up to mosa but this we we are still working uh at finalizing it. You can already download the client from opens. Okay. So uh you you're here in the left hand side you send your prompt to the client. The red um agent is the reference agent.

But now things start getting complicated. So the reference agent first of all it delegates to an agent that we call the collaboration agent. The collaboration agent has the uh is a is an expert in the composing tasks. So the collaboration agent computes a uh a plan a workflow. Um actually in dynamic setups it computes a workflow but we also support collaboration agent that have hardcoded a certain

workflow because we know that a certain kind of task are better done with a specific workflow. So you can have different kinds of collaboration So the workflow computed by this agent involves a lot of other agents that are all these agents that you see here. uh this agent can be other collaboration agents because sometimes one of the tasks of a workflow needs to be further split to

arrive to actionable uh tasks. Uh so actually in Mosaic the um the composition architecture uh is hierarchical but then the the main agents that perform tasks are so-called solution agents that compute tasks that performs action on the file system that u launch uh API calls on MCP are the blue ones and then in most of the workflows we want to have some evaluation that the task performed

by solution agent is correct or report respect some parameters and some KPIs. So we have the alloy agents that are called supervision agents and the task of supervision agent is exclusively um to evaluate the job of a solution agent. Nice thing about supervision agent is that we can replicate them. So we can have sometimes a lot of uh different agents that evaluate the same uh work with

respect to different criteria etc. Uh and then we need at some point to arrive to a consensus about all these evaluators and that's the role of the last kind of agent that we have that is this one uh that is the consensus agent. The consensus agent enacts a policy for uh decision based on the evaluation getting from the uh supervision agents. And these decision they look a

lot like uh we have a lot in in mosaic the metaphor of the community. They look a lot like governance rules in a community. So it's a community that needs to take a decision and there are deciders with different roles with different powers uh that can influence the decision. Uh the solution agent especially is uh enacts a lot of times a debate between them to arrive to

a consensus. Okay. So this more or less gives an overview of the main platform and runtime. There are two global components, the orchestrator and the decision engine that uh give global uh structures that improve especially uh performance and then we have a repository that runs a background process where agents are stored. So the code of the agent is stored on a repository that we are producing in

the project and we have a benchmarking loop uh that allows us to uh run some benchmarks on some KPIs to give values that are then used to select the agents uh by humans or by other agents that better perform in a given task. So this is uh the platform and so what does okay mosa is a aentic platform okay there are a lot of other agent platforms

around there at least in 2026 so what makes mosa different than other agent platforms so one thing uh we are heavily protocol based that means that we are agnostic to uh languages you can write your agent in any language you want and we are agnostic to implementation technology You can use a lot of libraries to write your engines. What you need to do is implement a protocol,

an interface to make your engine uh collaborate with with um with mosaic. And that means that also agents are isolated and every agent has his own docker docker container and the platform um starts and stop docker contains on demand. Um so this is to make the platform open. As I said, the platform is based very much on a reputation uh concept. So we track the uh performance

of the agents either in background with benchmarks or online during um uh the workflow workflow execution to uh derive some metrics some KPIs and a lot of full work package of the project is given to define these KPIs and measure them. Then and this is very important for us uh the platform is very explicit. We make explicit two things at two levels. One, we we make explicit

the mental model that's what an old concept from multi- aent system theory. The mental model of of agents as beliefs, desire and intentions and then the platform reasons on beliefs, desire, intentions of the agents to optimize collaboration u over the network. And we also make very explicit the collaboration patterns, the collaboration workflow. that is another artifact. When I say explicit, it means that there is an artifact

that represent it and uh governance rules. Finally, uh we care about formality. So all throughout the platform, we use formal languages. Most of the industry uses Python, let's say, or or JSON, let's say, but we use we try to use formal languages during the platform with the formal semantics. uh and this allows for precise communication but also uh precise implementation of the platform uh in in other

technologies. So now for the rest of the presentation I'm only um talking about some components that we have that you can already look at right now and give and illustrate a bit uh this principle that that I described. So the first one is uh the protocol. As I said the platform is heavily protocol based. We uh proposed a protocol uh the originally the proposal it was expected

to be a um totally new protocol but actually uh things changed. Now A2A uh that is the protocol from Google looked very nice to us was fulfilling a lot of our requirements. So we're actually extending A2A is already as a mechanism for extension. Uh so how are we extending A2A? On one side we are adding features for observability. So we need to add the concept of hierarchical

um hierarchical organization of agents. So here you see uh an observability platform. Uh we are compatible with all open telemetry systems like Lfuse, Lmit etc. And you can see that you can uh check traces hierarchically. uh for instance you had the reference agent contacted the collaboration agent but then had the three agents I talked about in my little example. Um so you have you have some metadata

that you can check about the uh the performance of this collaboration for instance time to first etc. And these are the kind of metadata that then the framework uses to take decision about uh optimizing the communication and also we are adding performatives. Performatives are also an old idea from multi- aent systems. uh actually we are uh using a standard standard that's called FIPA ACL and uh so

we are adding to A2A also some information about how much the agent commits to some actions so how much it commits how much it commits to objectives um if at some point it needs to drop them from some reasons etc. So 8way if you know about it is more focused on tasks. Uh we are actually with performance is we adding something about the mental model of the

agents that perform the task. So this is an example of what we mean about our work on protocols. Um as I said the belief desire intention is important. So uh if you know uh what the parend is about. Uh so we want each agent the agents are here to have a uh precise view because at some point they will be requested for what are their beliefs. So

what they do know about the world uh typically is a is a set of sentences let's say but they could be in natural language or we also support a formal language of this called agent speak. Um but imagine a set of sentences that uh represent what the agent believe uh about the world. Uh what are the knowledge that he extracted about the world? Desires that is the

objectives that he has and the subobjectives that it also has to arrive to this global objective and intentions. Intentions is what is the commitment to some actions that he wants to perform right now as soon as possible. Okay. So, Mosaic has a library that gives you so you you could implement your own agent and uh and just implement the protocol and be able to answer to our

request for beliefs desire intention. This is up to you or otherwise you use our library in the implementation of your agent and our library gives you uh some nice utilities for BDI management. Um and again so we use BDI to orchestrate and moderate the communication by uh among agents. For instance, we can use it to find compromises. When you see that two agent disagree, we can use

it to share knowledge. So, one agent learn something that could be useful to the other, some new belief, we can share it automatically with the others. Uh we can block two agents that they want to do some conflicting actions because they have some intention that uh conflict with each other. And finally, I have an example. I think I'm on time. I have an example about um formality.

So uh when we represent all these things, we try to use formal language. For instance, to represent the collaboration among uh mosaic agents, we are relying on a language that comes from a very different domain that is CML that comes from system engineering but we really like it. Uh so we defined a subset of CML that applies to agent collaborations and the subset uh we like it

because it has CML has a native textual notation that is uh that is here. It's um it has an imperative style. I will show the workflow right after. It has a formal semantics that we really care about. It's already known to Frontier LLMs and there is a lot of tooling. For instance, Eclipse has the season project that has an editor for it. So, we we we selected

we also published a big data set big 75,000 uh CML models uh workflow models like the one I'm going to show that we are using to fine-tuning an LLM for being a better planner. So, a better collaboration agent in our framework. So in the picture you show you see the part where we define that there are three types of agent and we see we can add some

constraints. For instance here the only constraint I added is that these agents that need to include some skills that are emphatic generation and the syntax validation and semantic validation for the three agents of my example and these skills are automatically used to search in the repository for agents that satisfy these constraints. But these constraints can also include a lot of KPIs like you know performance in these

skills etc. And then you define some instances of these uh types of agents. uh for instance in this ca in this case I'm getting for the first one uh in our demonstrator we have a python implementation that uses small agents and they use uh some tool for producing emphatic code uh by ginger in the quen 3 lm uh we have a synthetic validator that does not use

any lm at all actually just uses the infat parser and gets nice um errors if it finds that the um uh that model has some syntax IC errors and we have a semantic validator that uses launch chain and is written in Java. So as you see different uh technologies and different frameworks then once I defined these models I have these these agents I have uh I can

specify a workflow. So here you find a little workflow that I dro for this example. This is generated by plan toml actually automatically but this is the native textual format where uh I just have as you can see a very imperative notation in the sense that starts from here and then I perform this action and then I perform this other action. So even if I'm drawing a

graph it looks more like a procedure and the LLM there is a nice paper also recently that showed that LLMs are kind of bad producing graphs but they are much better at producing imperative code. So uh that's more or less what this looks like. Okay. Uh so this is what I wanted to show. Demo consortium. We are 14. We have four um four universities uh with different

competencies. We have net company that does integration. We have codo that is a big actor in VS code LLM based tools uh that deals with fine tetuning and prompt engineering. We have Eclipse as a partner F6S for communication and we have four use cases and for use cases just a couple of words we have we try to apply this to very uh developments. So with immersion we

develop energy efficient mixed reality content. With call in cyers of space we try a very challenging thing that is applying to some development that requires some certification too. In this case the prediction component for the remaining useful life of uh an aerospace plane. Uh we on parallel we deal with a catalog of IoT solutions and then national buffer greens. We do some investor recommener system recommendation system

uh with some data science. Um all of the in all of these we evaluate how the set of agent helps normal developers in reaching in in developing these uh these systems. Finally uh Mosao is open is open in all senses. I would say we have MIT license for all the code and all the code is on the clips GitLab. All the data sets and experimental data is

on Zenodo. So you will find all data we have. Uh all our applications are on access. Our technical reports from the projects are public. Uh Eclipse is a partner and also we are open to contributors collaboration. So the consortium is not closed. Finally uh and I close with this. we have a demonstrator that is the probably easiest way to get the platform and have it running and

try it out. So you can go to this tiny URL but if you go also on the clips GitLab you will find it. Um, so the demonstrator, uh, you can also try it at the boot in the [snorts] main room. Uh, I think more or less at the center of the room, right in front of the coffee. Um, and in the demonstrator, you will find the client,

you will find a little set of agents, and you will find all the agents that I uh discussed in my little uh you will find the repository with the vector DB and you will find some observability uh with lang fuse so an installation of lfuse with the storage in it um that is all thank you [applause] [cheering] so I don't know if you have any question [clears

throat] yes we have a question there. >> Um, so Mir German Aerospace Center. Um, thank you very much for the uh for the talk. I have a question concerning your uh so your initial remarks about what may be the drawbacks on uh on having multiple agents. So do you already have some uh let's say data points now? So when when you cross the threshold let's put it

this way when when does it apply this this mechanism when when is it best better to use just a single agent um so we don't have anything that I would feel confident in sharing uh [laughter] okay but the the point is that we we find the best let's say uh advantages we find is when it's for our coded uh workflows. So uh when you have a clear

workflow, you are cod workflow, you have your sub agents etc. It's easy to get gains. Uh but in terms of dynamic workflows so far um our results are kind of um unstable I would say. So I think next year we will have better answer on that too. [snorts] >> Yes, we have another question. First of all, it's a very interesting project. It's very exciting to to see

those things. Uh so uh what would be the open-source uh let's say building block that you want to propose here? Is it a novel system or Yeah. So it's a system uh this is it. I had some backup slides. >> Yeah. Yeah. Um yeah, it's actually there are four main components. One is uh the protocol, uh one is the repository, one is the orchestrator with the collaboration

agent. So the orchestrator and the a and the collaboration agent go together since the collaboration agent devices the orchestrator and the orchestrator enacts it and one is uh the consensus agent and the decision engine that go together in another different component. So this is what we are delivering here. >> So just this is very important. We have an implicit architecture with a cate taxonomy of agents. The

supervision agents is so important because it's validating the solution agent because otherwise you don't trust the solution agent. Okay. So this is why I like the architecture and my question is uh will you promote this at standardization level knowing that uh there's no reference architecture on AI agents right now. So there is an opportunity. It's hard but I can I can help >> that. That's a very

good uh point. So we didn't dare to talk about standardization in the proposal. Uh but yeah, I understand that is a very good point. So um the point is that it looks to us that in in in industry uh lot of agentic a lot of agentic platforms are out there. Um in a lot of times they give you some tools and they say okay now you deal

with it. you you build your agent and they will just work or at least it's up to you to make them work. Um yeah so we want to try some to do something much more managed that what you see in the company. So supervision agents are one step uh collaboration and consensation are the other where everything should should work because u there is some intelligence in make

it work together. Okay. So, we have more work to do. That's good. Uh, and uh, I have some a couple of them, but [clears throat] I think I need to choose some of them. you always are free to don't answer. I remember from my study believe desire intention and that's the agent that you are building about the beliefs do you have standard beliefs that you inject to

them because if I remember beliefs is where you have problem in agents. >> Yeah I mean um actually the development of our agents passes through there. So in when you develop an agent you need to put an agent in mosao you typically put things in these terms from the beginning. So you see what are the initial beliefs that are more or less the the persona that you

write down and what is the objective and what uh should be even some little plans. So when the belief is this you you want to do this etc. So we really use it to design our agent uh because it's convenient to do so. As I said it's not mandatory. You can write to your agent as you as you prefer. But we do um we do do we

do that and in this case yes we do a lot we have a lot of initial beliefs that are the system property. >> Can we say supervised beliefs are supervised? >> Yeah. Yeah. This we can say it. Uh so the beliefs are visible. So this mental model is um inspectable. So we inspect it for two reason. one the frameworks inspects it and two you can also inspect

it yourself for explanability of a decision so you can look at why you got some decision and what are the beliefs that are there um but yeah it's inspected also automatically by the framework and the idea is really to to supervise that the there is not a wrong belief that then gets propagated and convinces all the other people that uh >> yes that's that was I was

remembering from my study yes so that's >> there will be a language to describe the belief. >> Yeah. So we have two for for now we have uh a formal language that is called agent speak also comes yeah also comes from uh the old times we actually just uh integrated it somehow with LLMs. So we will find for instance a translator from agent speak to natural language

but you als we also support beliefs in natural language and in this case for do every reasoning on those you use lms the excessent that we are talking about natural languages but we have another >> Thank you. My question is a little bit related to the governance and once again this specificity um in in the governance case when you have the uh what you call collaboration agent

giving the orders or the actions to the solution agent it can give the the the action to different ones at the same time. So in that case the governance how is it linked and how is that the receiving one is going to get the exact objective they want to without changing their beliefs. per solution agent. >> So in our terminology, let's say uh we have two kind

of workflows. One for flow is the one devised by him and yeah in the the collaboration agent sends maybe the same task to n agents. Okay. Uh and this is very procedural kind of workflow. And then here the consensus agent instead is the one responsible of the governance. So at some point some decision needs to be taken. For instance, a decision is should this file be put

on the file system of the user, let's say. And so to get this decision, he uses a set of rules that are the governance rules. And so for instance, it needs a validation from these agents etc. or it needs that the majority of them thinks that this is a good file. Um and these rules actually can as I said can also invoke a debate. So actually it

can also trigger again these evaluators and even trigger again some solution agents to produce new solutions if none of the solutions are are correct. I I'm not sure if I I answered exactly your >> Yes. Yes. I have more questions now but yes I did. [laughter] Thank you. Uh just one thing who defines the governance rules like is already from the beginning or they are taped. >>

Yeah. So the governance rules so far are all are coded. So we write them manually. Okay. Uh for instead the the collaboration part we have two options. Either we write them manually or it's automatically derived by the LLM. Uh just that the project is limited and we don't have time to study how to derive these governance rules automatically. And I was uh about to say that um

that's outside the project but just a discussion between me and you because one main point that is discussed in some small research is about the language not the programming language but the language like the English that is using in the agent because the English is a is not really a great language for describing things. So it can be mistaken, you know, it can be ambiguous. do you

think that's outside? Do you think that we need in a certain point to have dedicated agents for dedicated languages? >> Um that's an interesting question. Uh it depends. I think that there are some agents that are really that could be tied to languages. There is no reason why they should be the language problem should be a problem of the LLM. So there are LMs that are very

good in some languages, LMS that are just focused on English etc. So the problem should stick with the LLM but actually a lot of times people are programming agents putting English prompts and so there are some issues related to the language in the agent development. Right? Uh well honestly this is it's a mosaical problem in the sense that we try our agent to be language independent. Let's

say the our agents are actually just the collaboration agent and the consensus agent. Um but up to the each solution agent and and the and each supervision agent it's up to the user to write them in the right way. Let's say >> yes of course. Okay. Yeah that that is >> languages languages. Yeah, that that was uh that is an interesting very small get to research but

it's a very interesting one. >> I will say that it's time to wrap up for this morning. Um we have lunch as I mentioned to you we have mandatory breaks. So we have lunch at one five minutes for now and we are coming back here at two. Okay. So we will see each other in 1 hour. Lunch will be served in the exposition area and it will

be um catering buffet. Okay. So it's also a way to know each other or you know know other person. Okay. So enjoy the lunch and see you later.