About this talk
This talk discusses cross-platform collaborative coding, specifically focusing on the Open Collaboration Tools (OCT) framework developed by TypeFox. The speaker introduces various existing tools such as Google Docs, Confluence, Figma, and Visual Studio Live Share, highlighting their impact on collaborative work. The OCT framework aims to enable synchronized editing across multiple platforms, including IDEs like Eclipse Theia and Visual Studio Code, using a TypeScript library for seamless integration. The speaker also emphasizes the extensibility of OCT, which supports different editors and enhances team collaboration, even among non-developers. A demo illustrates how users can connect and collaborate in real-time, while the integration of AI agents into collaborative sessions is also explored. This feature is designed to enhance the collaboration experience by allowing agents to participate in sessions as virtual teammates.
Full transcript
Welcome to our session about cross-platform collaborative coding. I'm Miro. This is my colleague Jan. We are from TypeFox. And uh TypeFox is a company from northern Germany. Founded in 2016, which means we are now celebrating our 10th anniversary, which is a really Thanks, Matthew. Which is a really nice reason for us to celebrate. Um yeah, we do innovation in open source tools and frameworks for IDEs and
DSLs. We also have a booth here just in front of the entrance to the main stage, so by and chat with us. So, we heard this morning at the introduction presentation that collaboration is no longer optional, it is foundational. Well, of course, that was meant in a very much broader sense, but and here we talk about collaboration in the sense of synchronized editing, so real-time collaborative editing.
And uh looking at a few reasons why I think our world is already collaborative, uh the first thing that comes to my mind at least is Google Docs. Um Google Docs in the in the way we know it today has been around since 2010 or so, and I think it's it it was not the first tool of that kind where you can really write and see the
changes from others in real time in a document, but I think it's has been maybe the most influential one. And that is yeah, for documents, spreadsheets, and presentations, and so on. Then later others came, um for example, for structured documents, uh there's uh Confluence, where Atlassian added such live editing features. And also Notion, uh for which also allows that for structured documents. And there are similar tools
also also for the graphical um collaboration like Figma added added Yeah, they called it multiplayer editing in 2016 already and Miro for virtual whiteboards and brainstorming also has this multiplayer mode. And finally for coding there is Visual Studio Live Share. Maybe the most well-known extension for Visual Studio Code where you can developers can connect and edit the same code in the same workspace in a synchronized way.
Now today we're going to talk about an open source solution open collaboration tools at Eclipse. And so this stands out in comparison to the previous products that I showed because it's open source and the main thing we want to present today it's a cross-platform. this opens OCT open collaboration tool open collaboration tools or OCT framework is meant to be a product ready building block so you can
really use it in your own products or in your internal tools in your organization to build tools where people can collaborate either your your colleagues so the people at your organization or also your customers. And there is text editing built in so that's the simplest use case comparable to Visual Studio Live Share but it's fully extensible to integrate other editors as well. So the underlying technology is
not limited to text. So this is our symbol for today to symbolize cross-platform so we have a tool that spans many different things and we are going to go through them one by one. The first one is the IDEs. We already heard in the previous talk about Eclipse Theia and also VS Code very important tool in that domain or in that area. Uh these two are connected
to ACT and there are two ways how you can connect Thea in particular. these two ways are built on a TypeScript library that that is shared by those so that they share a lot of the common functionality and it that library can also be used to build an integration for your own TypeScript based tool if you want. So the first way to connect Thea is simply a
Thea extension uh which is a compile time package that you load into your Thea based application. And the second one is the using a plugin and so that uses the VS Code um extension API or you sometimes you call it plugin because it can be loaded at runtime. And so you just search for open collaboration tools extension on the VS Code marketplace or open VSX and can
just install it. And by the way, we're not going in much into the overall architecture in this talk. There have been there's has been a previous um presentation um about how it works. Just to mention it, there is a server instance that handles all the communication between participants and for if you just use the standard deployment, that's a kind of a testing server. Typically, if you want
to uh use this yourself in your organization in your product, then you would create your own instance, your own deployment. It's very easy to do and then by that you can really secure it in the way you want. So next up is uh the two Java based IDEs that we have on our list here, Eclipse platform and everything built on top of it like the standard Eclipse
IDE and also the IntelliJ platform which is also a family of IDEs by JetBrains. Uh both of these handle the integration with OCT in the same way by using a background process. Call it the OCT service process, which is again built in typescript, which has the advantage for us that we can reuse the same base functionality but now we we communicate with the host and handle certain
things so that we don't have to re-implement them again in Java. So it's much much easier to keep the same functionality in the same language and platform of course. And so the service process runs in the background. It communicates with the host IDE and handles document synchronization. So changes made by the user on the current IDE on the local IDE versus changes made by others you're connected
with. And also all the messaging of the open collaboration tools protocol that goes to the server. Important to mention here, this is still work in progress. So current state is the IntelliJ integration is done. The Eclipse integration not yet. So next is the OCT playground. The playground is a web application that we provide as an example. It's on the OCT website. It's open collab.tools. So it's an
open source website of the open collaboration tools project. And it's meant to be a lightweight environment to explore OCT. So really for testing, you can use it very easily to try it out and see how it works, how you can integrate it with different tools. but it also demonstrates cross platform which we'll see in a recorded demo in a minute. And in that sense, it it's an
example of how mixed teams can work together. Mixed team is for example, uh you have a team working together where uh there are developers and also other stakeholders who happen to not be developers, so they don't have VR or VS Code or Eclipse installed, but you still want to share to have shared editing on important documents that for example are in a Git repository. And so that
can be achieved with this as we will see. And it can be a blueprint for your own apps where you can uh you can see how it's implemented and do something similar in your case. And when you build your own apps with collaboration inside, it can be from the same from from that app to the same to other users on the same app, so that would be
kind of in the same uh web application, uh multiple users connecting to each other, or also to other applications, uh covering the cross-platform aspect. So here is the uh demo setup. Uh it's a split screen with two participants. The one on the left-hand side is the host, so there's always one participant who defines what is the workspace content that we're actually sharing here. And uh so that
has opened a workspace in VS Code in this case and will share that with the other. And the other one connects to that through the playground. So the host user on the left starts a session with LCT. It it gets a room token from the server, copies it to the clipboard, sends it to the other user via chat, we don't see it here, so there has to
be some additional channel for this or by email. The other copies the room token into the uh website. It has to do some login. This is really depending on your own app, it can be different. And then the host has to allow the user. So, you see who is trying to join, and now they are connected and they can chat for it So, there is also an
integrated chat that works as soon as everyone is inside, and so they can start sending messages. But typically, you would also additionally connect via some meeting app so that you can talk in real time. So, now we can see some example of coding. So, the the playground user on the right is typing. You see the cursor position and the synchronized text on the left. So, the left
one uh user is currently watching. And um Yeah, and on the left, we also see some error. So, the left hand um user takes over and fixes the error and starts typing. And now uh we can see on the right how the cursor position and changes are applied, which are triggered by the other user. And so, this is really VS Code with a simple web app working
together. Um so, web apps is one thing. Electron is also an important platform, which is used a lot nowadays to build uh custom uh desktop applications built with web technologies, and both can uh can use the same technology, the same underlying frameworks. So, what how it works just quickly, we use the Monaco editor, which is the component seen here on the website. And Monaco is the code
editor that powers VS Code, and so it's really a standard widget used a lot nowadays. And And using OCT, you add real-time collaboration to that existing editor. Um maybe you already have Monaco editor in your app. So, then you can add the library provided by the OCT project to it to connect to an OCT session. and with that you get the same editing experience and also the
same live sharing experience across all applications that use this. Good. Uh just quickly and on the underlying thing want to mention Yjs is a library that is used a lot by many tools to handle the actual synchronization. It uh conflict-free replicated data types CRDTs uh which is also a standard method to to solve the problem of synchronizing over over a network which has a delay in in
messages. And yeah with that we handle synchronization of changes as we have seen but also awareness. And and awareness is making others aware of where you are currently and that can be a cursor position or a text selection. Uh if you're doing this on a graphical editing then it could be a mouse pointer that is moving around something like that. this is extensible and now I'll hand
it over to Jan for the interesting part about agents. >> Yes, thank you very much, Miro. Um so nowadays we see AI agents integrated everywhere, right? So it's a modern and very fashionable way to enhance every piece of software with some um agentic assistance. And in some cases it's also useful we wanted to try if it would be useful also um using an agent or integrate an
agent into a uh open collaboration session. we created uh a first prototype, experimented a bit, uh came to this. Um the idea behind it is that an agent should be added and should act as if it would be uh at as if it would be an normal participant in uh a OCT session. Um at the very least, it should be um which the uh participant uh could
talk to, um and which responses the participants could read and discuss. so for that also um the agent should be able to read the workspace um in the OCT session, and uh should make changes to the files in this workspace. Um from a technical point of view, there's no big difference um to the other client implementations. So, um we are using here the OCT protocol, which is
client agnostic. Um and so the agent um is is not a big difference to the VS Code extension or to uh to Theia. so you could say now, yeah, I I I could run an agent on my machine, and uh I could let it make some changes, and then I could sync these changes uh to the OCT session, so everyone could see it. But, um the advantage
to really integrate this agent into the session is some kind of transparency. So, every participant could see the changes, could make requests uh to the agent, and could decide what the agent should do. And what kind of changes we accept, right? To be fair, our implementation of the OCT agent is not really an agent in the traditional sense, if you can say traditional. Um it's more something
like a bridge or a connector to the to a third-party coding agent. And so, we need to communicate to this. And for this, we're leveraging uh the agent-client protocol, short ACP. When we started, we also experimented a bit with MCP, but uh very soon we recognized that this is conceptually completely different, and um we needed some real bidirectional communication, and the ACP offers that. ACP is an
open protocol uh from the company Z. Um it standardizes the bidirectional communication between editors or any other clients which implement the messaging. a agent. And it does it that with JSON-RPC over standard IO. So, there's already a a huge list of uh third-party agents like Cloud Code, Open Code, uh Goose, Gemini CLI, Cursor, you name it. The list is huge. You can read it up uh look
it up on on the ACP webpage. They all understand ACP. Yeah, as I said, our OCT agent is more something like a bridge. which handles all the incoming and outgoing messages to the OP OTC OCP session, sorry. spawns the actual third-party coding agent. It forwards the requests to the third-party coding agent. It handles the responses from the coding agent. It provides some tools for writing and for
reading files. Um I don't know if I forgot something, but yeah, it's easy enough, right? Yeah, okay. You see my gray hair, so maybe not sometimes. Um An agent without any context is just a stupid piece of software. And so we need to think about where do we run this agent? Um so that that we can give them a lot of context to work. um the first
case would be we let the agent run on the OCT host machine. So that means um this is the machine where the actual workspace is living. Um and hence uh the coding agent has direct file system access and hence context is not a problem here. Right? So, this is the simplest, most robust case and hence we decided to use this as a starting point for our experiments.
And we sided against case two. Case two would be um, that the agent runs on any machine which is connected to the OCT server. In this case now on a machine from another peer. Um, OCT supports uh, requesting file content. So, you get the virtual file tree on your machine. Every peer has it and uh, at any time request any content um, from any file. Um, that's
great, but you don't have all the contents at once. So, which would be a bit too less uh, for a whole context for the coding agent. we leave this here as a future challenge. Um, we have to think about how we could uh, efficiently um, provide all the contents uh, to the coding agent when it not runs on uh, on the host machine. The next question was
a bit easier to answer. So, it is the question how do we talk to the coding agent? It's small um, yeah, how do we provide the prompt? So, in the beginning we started to um, added the prompts inside a comment uh, the the actual text file. Um, and it was more uh you can see you can see it here. We have this pattern. Oh, where is it
now? Okay, it's gone. we have this pattern uh the the agent name preceded uh with a with a at sign, and this is followed by the actual prompt. and we very soon said, "Okay, we can't do that uh in the comment." So, um we enhanced the open collaboration protocol by some uh chat messaging. Um, which is actually already um merged into uh OCT, so everyone can use
it now. Everyone one can um implement um chat window in his client, as you can see here. This is the chat window uh from the playground, which is not merged already. yeah, as I said, it's it's part of the collaboration protocol. Um, it's a main interaction channel to the agent, but not only to the agent. It's peer agnostic, again. Um, you can also just use it uh
to chat with the other peers. Um, yeah, and the agent does the the um direct responses in natural language in this chat, and of course explains what it did and Yeah, as I said, humans and agent share the same channel. Um, and that leads us to the next question about output. So, how do we show the agent output? And our first idea uh was to be very,
very fancy. So, yeah, let's let the agent just type as if it would be a normal human participant, right? So, we show the live cursor and and we let them type into the code and so on, make direct changes, and then we very soon recognize that this is challenging as hell. Uh did I mention my gray hair? Yeah, that's also coming from this. Um because it's not
really possible to use an AI agent and encoding agent to calculate the exact position and so on. So, it it just delivers a whole new document. So, we decided against this and we go for option B, which is the classic diff view. And it makes also from a UX standpoint much more sense because no one wants an agent to make direct um changes to the files, right?
Because I mean, everyone everyone knows that agents are not perfect everything must be reviewed by a human participant. So, when we get back the the code changes from the agent, um we just show uh the changes to every participant in this diff view. And that gives all the peers um the chance to discuss the changes and eventually accept it or reject it also in many cases. this
is also an enhancement in the open one doesn't have to use it uh directly with an agent. So, we could think about enhance our UI later um that uh human participants could just propose changes to other ones to discuss it first before um we um um the uh source files directly. Um that I I could think about um learning solutions or something like Yeah, and again this
part of uh the open collaboration protocol as everything there it's it's peer agnostic. It can be used by every client. now we have a look um as before uh to a recorded session so that we can see how it looks to use the agent. So, at first we have to start the OCD agent. We can do it with this built-in command. It's built in the VS Code
extension. We get provided a link to this login mask that we have seen before. Um we have to give the agent a name and allow to join So, and then it appears as every other um participant everyone can now talked talk to the agent. Uh so, as here uh we can we can uh do a we we we we can request a prompt directly in the playground.
And let the machine do its work. And eventually the diff you on on every machine. So, this is in VS Code. The diff you respectively cursor. and this is a diff you on the And yeah, of course it was a relatively complicated change as you can see here, but it was good. So, I think yeah. The agent did a very good job. now it's um synchronized. So,
for the future, I think it is a very very very beginning of a love story between open collaboration tools and coding agents. Um but like with every love affair, there are some challenges and open questions and so on uh that needs to be answered, fixed, and um If you like uh to contribute uh you can talk to us. We are out there. maybe Miro, you want to
have uh you want to make some >> Maybe to the next slide that we can mention again uh the uh Birds of a Feather session that will be this evening. It's um yeah, Eclipse net next gen tools AI web and cloud. So, it's um also from multiple working groups that are at Eclipse. Uh one of them is the Cloud DevTools working group which we are also part
of where the open collaboration tools are also part of. so that will be an open session. Otherwise, uh yeah, that's it from what we wanted to show and we are of course open for questions and discussion. if you you were first, okay. >> He had his hand up first. >> Okay. >> Um I have a couple of questions. First, you you showed um VS Code and Eclipse
as different connectors, but do they connect together or would they connect together? >> Yes, uh you could then yes, this cross-platform means everything uh connected to the octopus can talk to everything else connected to the octopus, yeah. Um so in the demo, we only showed these two playground with VS Code, but everything else works as well. Um of course, the um well, I mean, open showing file
contents then will be I think self-explanatory on every Eclipse has a slightly different concept of a workspace compared to compared to VS Code, right? And so that that's not done yet, as I mentioned. So there uh we have to see how the workspace of VS Code would be visualized within Eclipse. But yeah, that's really a tool-specific way. And the playground itself, you don't see the workspace at
all because that's also meant to be a client that is simplified. I mean, you could of course have a web app that shows the file list, but here we chose to only show the active file, the one that the host is currently opened. I have a second question, if that's >> Yes. >> Um have you done anything on having shared debug sessions? >> Uh not yet. Um
I think that's something Does the Visual Studio Live Share you have that? Yes. So so that's something that that could be on the road map for OCT as well. Um but I haven't worked on that yet. >> Hey, um I have many questions. So, cut me off whenever you get bored with me. Uh first off, I have a new idea. It's amazing. No one's heard of it.
I want to add support to um for OCT especially that I'm looking at this and saying this is the USB of collaboration. You just plug and play everywhere. How complicated is it for me to add my own connector? >> Okay, so what technology is it based on? >> Rust. >> Rust, wow. Uh then the service process could work for you. Uh the one that we also use
for the Java based ones. So, you would take that as a basically a an executable app Node.js app. You start it. You have to connect it for example via standard IO and have to talk the particular JSON protocol to it. Um it's not the same as the protocol that is talked to with the server because it's a different role. Um as I mentioned, the the service process
already covers some of the needed functionality like the synchronization and so on. So, um yeah, we would have to look at implementing this respective messages it would be done. >> Sounds great. Uh so, that would not give me too much white hair, right? >> No. >> Perfect. Um the next question though, and this I was particularly triggered in, the uh agents use the same protocol uh users.
is there a way to identify the traffic from agents versus users? And I'm talking about a security and compliance perspective for large corporations. >> Uh I Well, we don't have any any traffic log on the server side currently. Um I mean, for the test instance, we see it as a good because it means uh people can use it and we don't have to specify in our terms
of usage or privacy policy anything about that because we don't store any information from the users which is good in that case but of course I understand in some cases it might be desirable to to have some tracking of what is possible. That would have to be added to the current server. It's not done yet and I would think we could implement a special flag for the
agent users to identify them as not non-human. >> Yeah, of course. >> That could be an option an easy option then so that in your logs then you you would easily find them and make a difference yeah. >> In this very early stage everything is possible so >> The last question I would have is how would you configure an agent? Is this something that you could do
from the client side for example? I'm I'm going to make an agent that does vocal poetry so I'm going to use Claude Haiku of course. >> You can send with ACP you can um do some configuration. So you can say what tools to be used what not and so on. This is possible yeah. >> Thank you very much. >> All right. >> Over here. >> So I
know it's quite early days but my question is are there any plans for SSO or LDAP integration to the server so that let's say if you have on-premise hosting and you want to say okay we open a playground but we don't have to worry about usernames or anything we just identify on whatever the user login is on the computer stuff like this. Single sign-on integration so basically
whatever the active directory or whatever is used within a company for instance. >> Yes so you can you can connect this with your own single-sign-on, yes? Uh there is already uh support um for for multiple authentication providers. And and basically when you do the server deployment, then you you can choose using server configuration authentication options you want to offer. Uh the public deployment for testing offers multiple
ones just to make it easier, so you can use uh I think uh Eclipse, Google, uh GitHub, and also a kind of simple un- unverified where you just enter a name. Of course, in a serious deployment, you would uh not do that and just use your SSO. >> And and then uh you know every you know that every user accessing the service has gone through your own
login flow. >> Okay, good. Thanks. >> And and the same also uh applies to agentic users because they have to go through the same login flow, so which means you would typically then create a service account for them and use that service account to lock them in. >> Exactly, yeah. >> Anyone else? >> Okay, no more questions. Thank you for attending. Let's enjoy the conference.
