NODES AI 2026 - Semiont: A Graph Based, AI Native Wiki and Annotator

This talk introduces a project developed by the speaker, who works with IBM's AI alliance team, focusing on an open source AI native wiki designed for collaborative knowledge management. The speaker outlines the project’s capabilities, highlighting its function as a knowledge base allowing users to create and annotate documents while integrating AI workflows. A live demonstration showcases how users can search for resources, add new entries, and utilize AI to generate documents based on context. The discussion emphasizes the importance of creating usable knowledge graphs, addressing challenges, and promoting document-grounded knowledge. Additionally, the speaker explores the significance of making AI and human interactions equivalent within the architecture of the knowledge base. The project utilizes tools such as Neo4j for graph management and supports multiple languages, aiming to provide a flexible, scalable solution for various industries.

[music] All right. Thank you. Um, all right. I've got plenty of slides here for the 30-minut talk. I'm going to try to stick with the slides for the most part and pop over to um quasi live demo. Um, probably early on. Uh, but we'll stick to the slides for most of the content. Um, so I am uh with with IBM. I work on the AI alliance team

at IBM which is a consortium of 200 organizations that have been working together for the last couple of years on a number of fronts. Uh I organized a knowledge uh work group a little over a year ago and uh this is a project that I've been speaking about to that group um through the last half of last year quite a bit and then I've been giving more

uh more talks like this in recent months. So we'll dive in. There's a QR code for you that'll take you to the the GitHub repository. This is an open source project. It's released under an Apache 2 license. Um, it's been going through a lot of changes. It is uh I would consider it not quite in beta release. Um, there's still probably a couple of um not major,

but probably a couple of breaking changes coming up. Uh, but with that said, it's it's a lot stabler than than it was a couple of months ago, and I've been using it quite a bit. Uh, so this is a general outline of the talk. and I think at this point I'm just going to pop over to browser and just show you what we're talking about before I

get too deep into the how or the why. Uh, let's make this concrete. So hopefully you can uh see this. So um, one one way to think about this project is it's an AI native wiki. Uh now that's that's been um uh you know kind of a hot topic the last couple of weeks. There was uh some noteworthy uh noteworthy discussion of that. Uh so I'm going

to define what a wiki is uh to me and I think to the broader community. But uh hopefully you know this kind of format uh you would recognize as something that looks like a wiki. So you if you come in um from the the top page, you'll see uh the the opportunity to search for resources and then of course you can add them uh collaborate with others

on the content, edit the structure and so forth. Uh so I'm just going to pop in here. Uh I just put this together before the talk. Uh I've annotated this tiny little document with uh with a couple of uh reference annotations. So uh we can uh we can generate a document for Neo forj. And a few seconds later um we should There we go. All right. All

right. So, we've got a new resource we've just added to the wiki. Um, so that at a really high level is the pattern. We can search for resources. We can add them. I kind of skipped over that part. I did that before the talk. We can annotate them. And I showed a a reference or a linking annotation, but there are a few other uh motivations uh for

annotation. Uh and then um we can resolve that unresolved reference by in a few ways. Uh what you just saw was was the utilizing the context which was pretty minimal at that point, but the context of that initial uh hello document was used to uh help generate this document. Okay. So I'm going to pop back over to the deck and keep going. And the name semi there

is just a it's just a neologism uh mashup of semiotics and biant. And uh if you want to construe the aunt as as uh implying something about ontologies, that's a welcome connotation as well. Okay. So going back to the why, um these were some of the big questions that I and this knowledge work group were looking to answer together as we met uh every couple of weeks.

And then you each member of that group has their own project uh many of which are evolving. um I would say in similar directions but but not identical. Uh sort of interesting to see what um the different ideas that people are bringing to the table. As I was thinking a couple of months ago about the the biggest questions that that we're trying to answer here, it's you

where do the where do the graphs that that underly a knowledge graph come from? Um I had done a lot of work uh about a year ago as a part of the alliance on on graph rag and hit the same stumbling block that I think a lot of folks do which is the the techniques for creating those graphs in the first place were not not well understood

or or felt risky. Uh so we wanted to invest some time in answering that question for ourselves. Um and then also think about how they relate to the to the user experience through the UI. Um that they those graphs may not just be backend data structures. they may actually be intimately um woven into the user interface. And then uh you know how can we get all of

the graphs? No organization is going to um have at hand or be able to produce all of the graphs that might be useful. So how is it can you know a larger community organize itself to uh to publish those? Uh and then another another um inspiration or or constraint is this belief in document grounded knowledge. Um I uh I had a professor years ago who used to

talk about in in times of great technical uh technological disruption the uh the anthropological uh uh universals uh the things that have been around for thousands of years are the things that are least likely to be disrupted. So I think one of those in our case would be documents. So I have a little picture here from a couple decades ago of the uh the code of Hamarabi

from uh from almost 4,000 years ago. Um there have been documents for for longer than that and documents are probably here to stay. uh and in in those um AI systems that are operating in environments that demand high trust and verifiability, staying close to them either by being able to cite them with high confidence or being able to create new uh authoritative and meaningful documents uh is

really important. And then the other the other tenant the other conceptual tenant is that and this is part of why I arrived at this word wiki in the first place was um making humans and AI agents architectural equals. We'll we'll talk more about that but in some of my experience in building AI systems in the in the past I have a career in in legal tech and

careers in music before that and affiliate marketing and and and some other things. Um, this it always seemed like the AI was kind of often bolted on in previous generations. Either that or the humans in the loop were bolted on. And so this was an opportunity to step back and say, well, what if they were uh from an architectural perspective equivalent if they had that the same

uh API surface area with which to interact with the system. Um, and then just a little bit uh you know working through some concrete examples. So what do we mean by a knowledge base? It's it's not a database. Um, wiki gets it gets at a lot of the intuition, but just to work through a couple of quick examples here and I won't read every word here, but

uh just walking through the the schema ingestion and enrichment for in this case it's a home ownership knowledge knowledge base. I became a homeowner for the first time about six and a half years ago and uh it's quite a it's it's quite a task to stay on top of all the maintenance and upkeep and so forth. So um and it's also the case that uh I think

this is a software niche that it's not likely to be filled by any kind of you know mature software. This to me seems like uh chaotic enough that it's among the class of software problems that are well solved by um all of the new capabilities that the LLMs bring to the table if we can kind of put them in the right context of a of a knowledge

base. And then something that's a little bit more enterprise um you know enterprisey um thinking about a legal legal operations team at a Fortune 500 company. What are all of the the entities that they would want to track in their knowledge base? Uh and of course there's going to be a strong correspondence between the knowledge base and the graph. We'll talk about that throughout. Um so there's

uh a set of uh entity types that we would want to track a certain set of documents that you would uh invariably encounter in setting up a legal ops knowledge base and then uh you enriching or annotating uh those documents as they come in. So going back to that um just the basic definition of a wiki, why why do we use that word? Um this is the

as far as I can tell this is basically more or less the canonical definition is that it's a website that allows its users to edit its content and structure that seemed uh really well suited for the kinds of you know uh documentoriented um enterprise domains and other domains personal domains that uh we were looking to build build systems for. So uh this is a little bit of

a post hawk rationalization but effectively what what took place was last July uh I started building a a wiki um and then uh built in AI assistance uh like like document generation into it uh with the idea of of ultimately having the AI and the humans be architectural equals. Then we factored out the the event protocol. This has uh mostly been a solo project so far, but

factored out the event protocol and then kind of clustered them based on the user intent. So of all of the dozens of kinds of messages that take place in this system, um I I pulled out roughly seven or there are seven kind of knowledge workflow related uh uh activities or intents that I've identified and I'll talk about those in a moment. And then of course we wanted

this to be model independent, uh, cloud platform independent, open source, and something that could be installed quickly and easily. Okay, so speaking of that, here again is the QR code for the git repository. Um, I'll just pause there for a second if anybody wants to grab that or you can go to uh the- AI-alliance semiot. And uh now this is fairly new. Um I'm showing here uh

an Apple container invocation. The same should work for Docker or Podman. But if you just want to run that um that image, uh the the system was factored into a distinct browser front end and then a backend. So uh and as you can see in the upper right screenshot, there are links that find knowledge bases and create a new one. There are links to the instructions to

um to do those things to either find a knowledge base. There's only a couple there right now. Uh that list is going to be growing soon. Or that second link, create a new one, is the link to um and I'll show that in a moment. Uh that's the link to uh do a git clone of a template um knowledgebased repository. So once you get this browser up

and running, um you'll just hit uh localhost port 3000 and then you saw the the the panel that was open was asking for uh credentials to connect. Um in order to set up the knowledge base that would have those credentials and be the basis for uh the the knowledge base um the recommendation is to clone this template. Uh there are a couple of conf configurations that come

with uh that template. one is anthropic based and the other one is lama with gemma models. So then those will both show up if you do that list configs. Uh so what's in that that git repository that template? So this is not the main semi repository but the knowledge base itself. Um so I I've kind of marked marked up the uh directory structure on the right here.

So at the at the bottom your content um this is not a strict requirement of semionant architecturally but um in the last month I I've started to pair a repository with um a a a git repository that those are going to be onetoone. I think that works very well. Um but at the end of the day these semiant directory is appear to.git but you have full control

over all the other files in that repository. Um and within thatse semiant directory there's uh some operations related scripts and config files and then there is the event state and all of that event state that is durable state that we want to commit. There's a lot of other ephemeral state that gets uh stored um in various uh directories in the uh in the container. Um some of

those might be work that you'd want to keep around longer uh but they're not things that you'd want to So after you start the knowledge base um you provide it the first time you do that you would provide it with the the credentials of a user you'd want to create and again this is all just intended uh right now just for local development uh I'll probably stand

up a couple of ephemeral um GitHub codebased knowledge bases that I'll stand up occasionally for demos but uh but for now that we're going to treat them all as ephemeral things that um that let's say you within your organization might might stand So if you run that script um the it requires that you've got some kind of uh container technology either apple container uh docker or podman.

If you run start um you'll get these resulting five containers running and then if you happen to have a llama desktop running already it will detect that and use that for inference. And speaking of inference, um again the default is um to use a lama with those two Gemma models for inference. And there is an embedding model needed uh for the vector store. Um that is uh

that's going to be running for whether you go with the anthropic config or this Gemma config. And then um I happen to have uh put this together using quadrant as a vector store. So it's not an embedding service why we use a llama for that. Um this is just a simple vector store. And then um here on on the last slide save the best for last here

the the graph the neo forj uh and you can see um just a you know just using the the browser on port 7474 uh a very early small knowledge base. The green nodes in this visualization represent the documents in the knowledge base and um that light brown those are the annotations. So this is very small and not very sophisticated. Um and I'll I'll talk about that that

schema and where that's going to be evolving soon but um pretty strong correspondence that if you think of those annotations um for the moment think of those as the links and the green uh the green nodes are the documents in the knowledge base. Okay. So I showed you a little bit of uh of semion interface. Now, let's kind of walk through it a little bit more deliberately

and I'll explain uh what's going on. So, you can see in the upper right, um I've got a list of the knowledge bases. And in this case, I just for the purposes of this demo, I just spun up uh one of the ones I've got checked in. It's just got one uh set of foreplays uh that are 2500 years old and in the public domain. Um I've

got that checked in and then I just sort of added a few random things to it for the for the purpose of of this talk and this demo today. So you land initially on that discover page and then uh just like uh I think you'd expect from a wiki you can add resources. So this is how I added that hello nodes AI document uh right before the

talk. Oh and I should mention there's a format dropdown right there and it's it defaults to markdown. Um but it does and I I'll show this later on. It does support uh images and PDFs. um it does not support yet uh automated annotation of those things but I think um that's just a matter of filling a couple of gaps um architecturally uh we've already created the uh

the support for that we know exactly where where that code will land when we write that okay so uh just a little bit about the theory I mentioned clustering the events in the wiki by user intent so this is where we landed um and I've chosen I tried to to choose especially old and simple words uh for these. A lot of the a lot of the longer

words are kind of spoken for and have all kinds of connotations. So um these are the ones I I went with. So uh I'll talk about um most of these. So but let's maybe get some of the simpler ones out of the way. on the right hand side browse. That's just using some uh some materialized views that that contain the the simplest structure of of the wiki.

Um that that happens via the browse events. Uh mark is the category of events for annotations. So creating a link and resolving that link. Those are mark events. Uh Beckin is sort of a surprising one. I I was adding a bunch of visual feedback um to the wiki. So after you generate there are things that pulse and so forth and I thought you know rather than just

considering that a little UI flourish I think um I think there's space for that to become a legitimate kind of first class mechanism for basically saying to anybody um uh hey look here I want your attention uh and maybe you could imagine that that could be timed there's the immediate I'd like your attention look at this document look at this annotation or in a week from now

tell my collaborator to look at this thing and maybe that thing will be the instructions of an activity that I want that collaborator to do. So those are kind of the the simpler ones and then gather, match, bind, and yield. I'll show some more examples of that, but gather is basically context assembly. Match um you could think of as search, but search with the intent of resolving

a resolution of uh resolving a reference. Um and and bind is the actual action of resolving a reference. And then um yield is the generation or introduction of new resources slashdocuments into the Okay. So here I'm going to use a few of those verbs in a flow. So um and I'll show one or two examples of this in later slides but uh if we if we you

know mark a resource as let's say that Neo Forj link that was unresolved initially uh take take that hello nodes AI mark uh as a reference the word Neo forj then gather context um for that that was the first little popup that you saw and then on that I there were three options I could choose to match it to an existing resource source uh aka entity. Uh

I could either uh just decide that no, it's it's not in there and I don't want um I don't want this system to produce it. I'm going to to to type it in. Uh I didn't choose that either in this case. What I did was the third option just say hey uh AI you you whatever model I have configured uh between what the model knows and between

the context that I'm providing um that ought to be enough to produce a high quality um uh resource. So that's the option I took in that case. And then lastly the system took care of resolving that reference and the question mark turned into a link icon. Okay. So, uh, getting back to a little bit more kind of, uh, enterprisey examples. Um, uh, here's, uh, kind of a

run. I have this on the the, uh, GitHub repository. There's a screenshot of this letter unmarked, actually a little bit marked up. This is it in in raw form. Um, so this just gives me the opportunity to to explain a couple of the features. Um uh you had seen earlier I had I had already kind of used uh auto detection to to detect uh I think the

all of the organizations and it happened to pick up uh semiant uh and neoforj and nodes AI uh in this case I've already done that I've detected people and organizations uh and one place um and then uh I you notice um if you look at the left hand side there in the in the red box. I included the check mark to include descriptive uh references. Uh so

this would be like um anaphoric or cataphoric references uh but things but not necessarily pronouns. So uh we're not going to use the linguistics terms. Uh we're going to call that we call that in the UI descriptive references. So that picked up those two, the owner and the client. Uh which are interesting. Um the client um is possibly resolvable given this context. the owner if you look

at the context is not and in fact that's a point uh that that's uh that's an activity we need to undertake we don't know who that is and we need to confirm that person's identity uh some other annotations I mentioned that linking was one of the annotation motivations that's uh done in blue um if you look at the the the tabs in the upper left there um

there's some statistics in that first panel then there's the reference panel in blue and then by name. Um, yellow is highlighting, red is assessing, uh, the the bubble there is commenting, and then the last one is tagging. Um, I should mention that those are all defined by the W3C web annotation standard. Um, I have a a a reasonably faithful uh implementation of that standard as far as

the annotations themselves go. um that is considered the native uh representation and uh I initially found that because I wanted multimedia support and wanted to you know who had thought about doing selections uh across different media types and and and at that time I was doing uh references and and um highlights. But then not only did the web that web annotation standard give me um that way

to think about standards or standard selectors, it also had a bunch of motivations defined. Um and I'm not using all of them. I've got these five baked into Semiant, but adding others if they were of interest would be would be relatively straightforward. So in this case um the the instruction I'm giving to semiant is say hey flag using uh using an assessing motivation which was represented by

red underlines flag the flag anything that poses a scheduling risk. So you can see that's been done overlaid on top of the document and then on the right in the assessing panel you can see uh further explanation of of why um why each of those passages was flagged as a uh tagging. Um, I'll show an example of this, but I I have baked into Semiant. Um, and

I'll probably extract these and make these fully external, but there's a handful of I call them tag schemas. I think this is a a gateway to ontologies. I'm not going to use that word yet. Um, but um, there are a handful of of simple mechanisms I have that I I think are helping to set that stage. Uh, this is something that's very useful in in legal writing.

um when you're learning to write uh apparently I'm not a lawyer but uh I understand that they actually teach this way of thinking about uh legal writing and it's pretty straightforward. You can identify uh issues, rules, applications and conclusions. Uh and here is an example um where you can see on the right in the tagging panel we've identified um what's visible there as a rule and an

application and then um we've got a conclusion. I guess they're a little bit out of sync here. Uh I didn't have the conclusion scrolled into view but um that reversed in part, affirmed in part and remanded. That's the conclusion at the end of that uh Supreme Court opinion. Um so going back and just explaining that word um gathering gathering context uh we we talked about how the

owner was this um ambiguous descriptive reference and we can gather the context and generate u a document for that. In this in this case I'm showing the client is that resolvable given the uh given the context of this one uh email. Uh so the the way you pop up that gathered context modal is you click on that question mark next to the client um as it appears

in the the entry in that references panel in the kind of the middle of the page there. So click on that question mark and then this gathered context after a few seconds it will talk to the graph it'll talk to the vector store it'll talk to the materialized views and uh form this this context. Now this is often as I've seen AI systems built this is often

just sort of something that's done behind the scenes. I think that's one of the areas where when we say that we want um in this project the humans and the AI to be architectural equivalents. This is an example of that where um uh you could edit that. In this case, it's fairly limited. We can just um we can just add a hint. But over time, this can

be something that the human could just fully control. Say no, you that those those peer annotations are are not important. I want to strike that or oh, you forgot something. We want to go deeper. Let's leverage the ontologies that we have in place to to you know go deep on on some thread uh that's that we know is really important uh ultimately to uh either either you

know finding a match in the knowledge base already or generating a new uh resource that represents the entity that we're referring to. Uh and then this is the alternate resolution strategy I'm showing. Um, in in that case, it's the exact same workflow up to the point where you choose a resolution strategy. So, you would click on the question mark, see the gathered context, and then this time

click click match, and um it would use that gathered context to propose uh the the entities that you could um resolve the link with. I'll just mention briefly um it is uh it supports 29 languages. That's nothing that Semiant is doing other than allowing you the ability to select the language and it preserves it, passes it through to any generation um or yield calls. Speaking of generation,

um uh this is uh this is an example of generating the owner. So we were able to take that that vague reference to the owner gather some context you know and I should mention um let me go back here that those peer annotations that's one of the ways in which we're using the graph the use of the graph right now is uh it's I think there's a

lot of room for it to grow uh it's already provides a lot of value um so here again we see it's pulling in the peer annotations Michael Sarah and acne corporation uh that's uh something on the road map that I'm especially excited to build out, but there's a lot of kind of just this initial workflow that that needs to be established. Um, and then I mentioned it's

multimodal. So, one of the the personal reasons I I uh landed on this idea of a wiki was just doing family history and I've got a lot of, you know, old photographs and things like that. So, um, the ability to annotate images in this case or PDFs in a lot of enterprise setting. Um, I've got the ability to represent and annotate manually those, but we'll be working

on um auto annotating those things. Um, I'm just going to we're short on time. So, um, this this is up on the GitHub repository if you want to take a look at that. There's a system of record. If you look at, you know, the the second smallest yellow rectangle there, there's a system of record that corresponds with the Git repository. Everything else is derived from the durable

information. The other purple uh boxes, those are all derived from the uh information that's stored in the git repository. And then the green are the actors that are um are managed by the back end uh that the front end is talking to. Um I'll just gloss over this stuff. Uh there's a a lot left yet to do. Um but it's come a long ways since July when

it was started. and it is open source. The discussions are enabled. Would love your your questions and feedback. So I think I will leave it at that and um I'll take a look at the questions in the chat after this talk and uh try to respond. Otherwise, you know, please please go to that discussions um uh feature on on the GitHub repository. I'd love to follow