Automated development of hybrid graphical-textual modeling editors

This talk introduces Graphite, a framework developed at the University of York for the automated creation of hybrid graphical textual modeling editors. The speaker explains that hybrid modeling editors combine graphical and textual elements to represent domain-specific languages (DSLs), enhancing the development experience through features like syntax-aware editing, code completion, and error detection. Examples of DSLs, such as project scheduling and robotic systems testing, demonstrate the capabilities of hybrid editors, which include smart textual editors and consistency enforcement. The speaker details the workflow for employing Graphite, utilizing Eclipse plugins like EMF, Sirius, and Xtext, underscoring the necessity of both graphical and textual syntax definitions. The potential for further development and integration into web environments, as well as the open-source nature of Graphite, is discussed, inviting contributions from the community.

Hello everyone. In this talk, I will present a framework for the automated development of hybrid graphical textual modeling editors. And this framework has been developed at the University of York. For the agenda, I will first present what hybrid modeling editors are. And then I will present some examples of hybrid modeling editors. I will then show the framework called Graphite. And finally, I will show the generation workflow

for such hybrid editors. To understand what hybrid modeling editors are, you would have to understand what hybrid graphical textual DSLs are. First, a domain-specific language, unlike a general-purpose language, can solve problems for any field, for any domain, but it is tailored only for that domain specifically. And a hybrid graphical textual domain-specific language has a part graphical and a part textual syntax, meaning that some parts of the

syntax are graphical, whereas others are textual. However, generally, they are predominantly graphical, but they contain some embedded textual notations to define complex expressions and behavior. And here, on the left side, we have an example of such a hybrid DSL. This is a simple state machine language in which the states and transitions are modeled graphically, but the guards are expressed textually using textual expressions. And the nice thing

about this is that the textual expressions can be written using syntax-aware editing features, code completion, syntax highlighting, and so on and so forth. Here, I present an example language that will be used throughout the presentation. Now I present abstract and concrete syntaxes. I assume most of you are familiar with language engineering. An abstract syntax covers the concepts of the language, the relationships between the concepts independent of

any representation, whereas the concrete syntax is actually what the user uses and sees. And here I'm presenting the metamodel of a project scheduling DSL. It's used for modeling project plans. And the main concept of this language is a project. A project has a number of tasks and a number of people. A task has a name. It has a number of efforts, a leader, and a list of

dependencies. A person has a name and an effort is allocated to a person and it has a number of months. Now for this particular example, we will assume that stakeholders would prefer to have a hybrid notation. Specifically, they would like to have the tasks and the people represented graphically using this graphical representations. However, they would like the efforts to be represented textually because it may often be

difficult to represent in a diagram a large number of efforts. And in this case, the textual notation takes the form of key-value pairs in which the key is the person and the months is the value. And here we have a concrete example of an effort expression. We have three efforts, one on each line. The first effort is allocated to the person Alice and it has a number

of three Now, ideally, such hybrid editors should provide these capabilities. And on the left side, we have the minimum components that such a hybrid editor should ideally provide. First, it should have a diagram in which you can view the graphical part of a model. And in this case, on the left side, we have three views. The minimum views that this editor should have are a palette from

which you can drag and drop various graphical onto the diagram and this instantiates various types of model elements. Then a properties view in which you can view and edit the properties of the model elements. And notice here that there's a textual editor for the This is not a plain textual editor. This is a smart textual editor. Unlike the name, for the name, there's another textual editor, but

that would be a plain textual editor. And finally, a problems view in which one can view any errors or any inconsistencies reported from the model. And the capabilities of such hybrid editors are first, smart textual editors. I've already showed this. So, this is not a plain textual editor. It has syntax aware editing feature, code completion, syntax highlighting, error detection, and so on and so forth. Then textual-graphical

cross-referencing. Since we have some parts of the model represented textually, other parts represented graphically, to be able to have complex behavior, you may often need to have references between the graphical and the textual parts. So, in this case, the textual expression indeed has a references. It has references to Alice, Bob, and Chart. So, you can't write any person's name. It has to be a person that actually

exists in the graphical part. Then another feature that would be nice to have is consistency enforcement. As the model evolves, it would be ideal to enforce consistency between the graphical and textual parts. Let's assume we have this example and somebody renames Bob in the diagram to Robert. Well, one option would be to not the textual expression, but you may have hundreds of such textual expressions that currently

have a reference to Bob, but now you would have an inconsistency because Bob has the name Robert in the diagram, but there are hundreds of textual expressions that still reference Bob. So, ideally you should update all those textual expressions with the new reference. Then, another nice feature would be uniform error reporting if there's any error in the model, for example, if the textual expression cannot be parsed

successfully. For instance, instead of tree, you would write tree X, which cannot be parsed. It's not an integer. That should ideally be reported in the problems view. Integrated refactoring, this would be the capability of doing refactoring directly from the smart textual editor. Somebody could go in the smart textual editor, select Alice in the textual expression, then right-clicking and choosing the option to do integrated rename refactoring, then

refactoring would be performed consistently across the entire model, even if you have hundreds of textual expressions. And two other features would be tolerance of temporary inconsistencies and integrated abstract syntax graph. I will not go into them in this presentation, but you can ask at the end of the presentation if you're interested about these capabilities. Now, I show some actual concrete examples of hybrid editors to see how

they look like in practice. The first one, this is for the project scheduling DSL, the one that I've already presented the abstract syntax of. Here, we have a number of tasks, a number of persons. You can see an auto completion menu in the smart textual At the bottom, this is a DSL for modeling the testing space of robotic systems in simulation. It defines various fuzzing operations with

a graphical representation and the textual representation specifies when the fuzzing operation should be activated. There are some conditions and depending when the conditions hold the fuzzing operations get activated and deactivated. On the right side, here we have another hybrid editor. This is called the Structurizer DSL. It's It mimics the official Structurizer language. Uh if you're familiar with Structurizer, it's a software architecture modeling language with which one

can define various architectural viewpoints at various levels of abstraction. For instance, at the container level, at the component level. And here we have a tree editor which has and also some smart editors for inclusion and exclusion expressions. In Structurizer, you can specify what should be included in those views or should be excluded from those views. And here we have two other Here on the left side, we

have a DSL for modeling cloud services and infrastructure components for the software company NetApp. And we have various cloud services defined in the diagram and you could also see that there's a because there are some errors in the Currently CBO2 is referencing volume 4 but CBO2 doesn't contain any volume 4. It only contains volume 1, 2, and 3. And so you would get an error marker. The

error marker would be shown here but also in On the right side, we have another DSL and this is called the flow flow DSL. This is often used by Obeo in their samples. And this has been extended with a textual notation. At the bottom, you could see some routing rules that benefit from syntax-aware editing features. And this rule specify how information should flow throughout this system. Now

I will show a demo. This is the cloud services that I just presented. At the bottom, we have a textual expression for defining volumes. I show the auto-completion menu. Now I show a different textual expression, so you could have multiple notations in the same DSL. This is the flow designer DSL. At the bottom, we have a routing rule. And now I attempt to write a new routing

rule. I mainly use the auto-completion menu to write this rule. Now I show the structurizer DSL. Here we definition of the C4 model. In structurizer, you have a C4 model in which you define your various components. And here we have a system context view diagram, and you can open it directly from here. So that opens this view, and the textual expression is star, meaning that everything from

the C4 model is included at this level of Now if I remove the textual expression, everything would be removed from the diagram. Now I show that you don't can't use only star. You could use You could do filtering by elements, by relationships. Now I show what the container view looks like. Here we have a textual expression that does filtering by tag. And from this editor, I open

the diagram of this view. And here, filtering is done using this condition. It includes all elements that don't have the tag person optional. In this case, the person Bob was excluded from the So, there's only Alice, but now if I change it to person main, Alice would be removed and then Bob would be shown. in a way, I now I showed the semantics of the language, not

just the syntax. This is the fuzzing-based testing DSL. Here, we have the testing space of robotic systems in simulation. We have various tests defined. And from that diagram, I open a different diagram. And here, we have the results of executing this fuzzing operation. And to activate the fuzzing operation, this condition is used, which is shown in the Now, I go back to the first diagram, and from

it, I open a different diagram. So, here I'm just showing that it doesn't have to be one diagram. You could have multiple diagrams, and you can have multiple textual notations. Now, this is the project scheduling DSL, the one that I've shown as an example in And here, we have three tasks: analysis, design, and implementation, Alice, Bob, and Charlie. And we have some relationships between them, some dependencies

between them showed by the edges. And the first task analysis has two efforts, one referencing Alice, another referencing Bob. Then, I will go into the second task. And here, there are no there are no And in the implementation task, we have three efforts referencing Alice, Bob, and Chad. Now, I will show some of the features that I've just presented. Smart textual editors, well, this is the smart

textual editor shown here. And you could use auto completion, you would have error detection if you write something that is syntactically incorrect. Now, another feature would be textual Now, I'm referencing the persons Alice and Bob. They exist in the diagram. Technically, you could write any name, but you would get an error. So, from the textual expressions, you can reference elements defined in the diagram. And you could

also navigate to them. So, from a reference from the textual expression, you could directly navigate to the diagram in Sirius. Now, for error reporting, if you write incorrect or semantically incorrect, you would get an error marker shown here. You would see the exact error. And additionally, you would also see it Now, going back to the textual expression, you would remove that, and of course, the problem would

disappear from the problems view. would be integrated refactoring, doing refactoring directly from the textual expression. You would right click a reference from the textual expression. You would rename element, a pop-up window would be shown, and then you enter the new name. Well, it would be renamed in the diagram, of course, but also in all textual expressions that were previously referencing Bob. Now, they instead reference Robert. Now,

a final feature would be consistency enforcement, going directly in the diagram and renaming Robert into Bob, this of course will update all affected textual expressions. And even if you delete Bob, that this change would be reflected in the textual expressions. Well, the challenge in doing all of this is doing it efficiently because you may have that reference this model elements. Now I start presenting the framework. It's

called Graphite. It has been developed in academia. It's open source and it takes the form of an Eclipse plugin. It is based on EMF Sirius and Xtext. the Eclipse modeling framework. It is the modern framework used by Eclipse to define models, metamodels, whereas Sirius and Xtext, they are state-of-the-art graphical and textual modeling frameworks. You can define graphical DSLs, textual DSLs, and you do need domain knowledge of

EMF as well when using Sirius and Xtext. And it is available here in this GitHub repo. And by using Graphite, one could develop such hybrid editors almost automatically by using automatic code generation, but you would need some handwritten code, minimal handwritten code, and some configurations. Of course, you would have to define yourself the abstract syntax, meaning the metamodel, the graphical syntax with Sirius, and the textual syntax

with Xtext. This shows it at a high level what would have to be defined. Here we have the hybrid editor. On the left side, I'm showing that you would have to define a Sirius viewpoint specification model. It is Sirius way to define the graphical syntax for a specific metamodel. it would associate various parts of the meta-model with a graphical representation. In this case, uh it would associate

a person node and a task node with some images, person.png and task.png. Whereas on the right side, I am showing that you would have to define an Xtext grammar to specify to what notation this expression should conform to. And the grammar, of course, references the meta-model. I will present at a high level the This is the diagram for the entire workflow. I will not go into it

in the details, but if you're interested, you can ask questions. What is important is on the right side. you would define the meta-model. Well, I already presented this. You would define the various concepts of your domain, what concepts you would like the language to have. Now, once you've defined the meta-model, you would have to decide what parts of the language should be expressed graphically and what parts

of the language should be expressed textually. And in our specific example, I said that in this case, stakeholders would like to the efforts modeled textually. So, considering only the textual side, you would have to add string attributes and annotations to the meta-model for every property that you would like to be expressed In this case, we have just one property, the efforts list. So, for that, you would

add one string attribute to the meta-model and one annotation. This is not ideal. It makes the meta-model a bit larger, and in some cases, you may not be able to you may not be allowed to modify the meta-model. So, this is a limitation of this tool. Then, you would have to modify a bit the generator model. If you've used EMF, you should have an idea of what

the generator model is. It can be generated by using Eclipse's tooling from a meta model. Once you have a meta model, you can generate a generator model. However, it has some default settings that have to be changed. One specifically one property has to be changed because by default default EObject implementation. That default EObject input class provided by EMF. Well, with Graphite you cannot use that. You have

to specify a custom And this class is already provided by You just have to set the property that the value of this class. Once you've done that you can launch the EMF code generator. And then, this would generate three projects. They are Java project. They contain the model code, edit code, and the editor code. They can They contain Java code, and they define the modeling infrastructure that

enables EMF modeling. you can start doing the manual part. This is the largest part that you would have to do in this entire process. The concrete syntax definition. And for defining the concrete syntax, you would need domain knowledge of Eclipse Sirius and Eclipse Xtext. Which, even if you have domain knowledge it's not such an easy thing to do. So, it takes quite an effort to define the

graphical syntax and the textual syntax. And in Sirius, a graphical syntax is defined using a Sirius VSM or a Sirius mentioned this before. You would declaratively associate various parts of the abstract syntax with some graphical representations. Then for the textual syntax, you would have to define some grammars. And that may not be such an easy thing to do. And in the grammar, you specify how the textual

expression should look like, and you have to specify what model elements should be instantiated as you're typing textual expression. So, behind the scenes, you are writing textual expressions, but actually model elements are derived, and you could access them in model management transformations. So, if you would pass the model as input to a model transformation, it will not see just the string, not just the textual expression, for

instance, the effort. It will actually see the list of effort model element that can be iterated. You could do four statements, if statements over them. So, you don't have to parse again those in model management operations. You could directly access those underlying model elements. So, it doesn't just see as text, but it sees them as a graph, as concrete Now, the final step would be to launch

Graphite's generator. And uh you would just click a button. Now, now I've showed the process for doing this. The output would be a list of classes generated by Graphite, but also a list of modified classes. Specifically, Graphite would modify some classes generated by EMF in this step, launching the EMF code And what I want to highlight is that technically, you could implement everything yourself. You could develop

such editors with handwritten code. So, you would write manually the code for such editors that provide all of these capabilities. However, this is a significant effort and it may take a large amount of handwritten code and handwritten code may differ for every DSL. And you would have to do this for every with Graphiti, this entire process is streamlined. It should generally work for every DSL in theory,

but of course, there may be some examples of DSLs for which they may this may not work. For instance, you don't want to modify the metamodel. Well, in this approach, you actually have to modify the metamodel a bit. You have to add some string attributes and some textual expressions. And here I show at a high level the manual effort involved in doing this. You define the metamodel.

You've decided what parts are represented graphically and textually. Now, in blue I show the graphical We have the Sirius VSM. And for our example, I said that a person and a task should be represented graphically. And then for the textual syntax, you would decide that the efforts are represented textually. And in your grammar, you have to reference the metamodel and you have to instantiate effort model elements.

As you're parsing the textual expressions, you would be deriving efforts behind the scenes. So, uh this is the most difficult part in the because even if you have Sirius domain knowledge and XX domain knowledge, it can fail. So, this can be made invalid in various ways. You have to ensure that you're representing different things and you have to ensure that what you're referencing from the grammar, for

the person. This is a person element. Well, this is represented graphically in the Sirius SM because you define a person node. This is just to ensure textual graphical cross-referencing because if you don't do this, you wouldn't have references between the graphical and textual parts of the Now, another question would be how do you make sure that it actually supports references between the graphical and the textual? Well,

further validation can be performed to check that for whatever you're referencing in the grammar, you're defining in the graphical syntax. Now, it must be done manually, but by using model transformations, this can be further extended to make it automatic. Regarding the metamodel, I said that the metamodel must be modified. And here I'm showing what exactly would have to be modified for the metamodel. You would define the

metamodel and then you would decide which parts are represented textually. After doing that, for each property that that is expressed with a textual syntax, for instance, in this case just the efforts, you would add a corresponding string attribute and an annotation. And in this case, I show an example, we have the efforts property and I've added the efforts expression string. And efforts expression string stores the textual

representation of the efforts and they are bidirectionally synchronized. So, as you do a change in one of them, it is automatically reflected in the other property. But, there is support for some temporary inconsistencies. I've mentioned this earlier. Now, what does the annotation define? It links the added string attribute with the property that it represents. So, the effort's list with the grammar that it must conform to and

with the entry rule used by the grammar. What would happen? When the effort's expression is modified, it would be parsed according to this grammar by using this entry rule. You could use different entry rules for your specific properties. And the result would be a list of model elements, so a graph, an AST. Actually, a graph in this case. output would be assigned to the derived property. In

this case, it would be the effort's. So, as you're setting the effort's expression, the effort's would be set up After you modify the effort's, well, the effort's expression would be modified. So, most of the time they are bi-directionally synchronized, but it allows temporary inconsistencies. This is how you would trigger the process. After you've defined the metamodel, the generator model, the Sirius VSM, and the Xtext grammar, you

would select all four of them. In this case, this is the Ecore metamodel, the generator model, and the ODesign file is the Sirius Viewpoint Specification Model. You would select them. You would right-click in the Eclipse workspace. You will see here a Graphite, and you would select Generate Hybrid Editor. This would generate all the code required to develop the hybrid editor that I've shown with the capabilities that

I've shown for this particular DSL. So, if you'd have a different metamodel, a different different graphical syntax, a different textual syntax, it should work for any DSL in theory, but more testing is necessary. And also, there are all other facilities provided by Graphite. For instance, validating the grammar, validating the metamodel, and generating grammars. One of the largest efforts is the definition of the grammar because not many

are experts in Xtext. Well, the capability of generating a grammar simplifies a bit this process because it provides a starting point. So, from the metamodel, considering the structure of the the metamodel, considering the annotations, it would generate a grammar that is compliant with Graphiti, because the grammar cannot look in any way. It has to be defined in a specific manner when using Graphiti. Now, that's just a

starting point. It's just a skeleton. You can modify it according to your needs. Then, it also provides the capability of validating the grammar because considering the grammar must look in a specific way, it would be nice to be able to know that it it's actually works and to validate the metamodel. Now, what's next? Two other features that would be nice would be integrated search and find all

to be able to do search and finding references across the graphical and textual parts of the model. So, selecting something, then finding all references from that, seeing where that is defined graphically or textually, because you may have hundreds of textual expressions, and you may not know where a specific model element is referenced unless you actually go into the XMI format of the model, you do a control

F, you do a search, and you would see all the textual expressions manually. Well, with these facilities, you would see all of that automatically. Now, going forward, it would be very nice to develop a similar framework, but for the web. Right now, this is tailored for the desktop, and it doesn't have a language server protocol, a graphical language server protocol. Ideally, this should be extended for Sirius

Web or Eclipse GLSP or or very similar, and by using something else other than Xtext like Langium. And so you would have a language server protocol for the textual part and a graphical language server protocol for And considering that the biggest engineering effort is in the definition of the concrete syntaxes the Sirius VSM, the Xtext grammars, and even the meta model if you don't have EMF expertise,

integrate large language models. And in this way users with minimal expertise in Xtext, Sirius, and EMF would be able to use some high-level prompts to generate the syntax. The user would just define using text, using a prompt, and by providing an image what the Sirius VSM should contain and what the grammar should look like. one note, the current work this is obvious now, but the work doesn't

use any LLMs right now. So all of the code that is generated is not based on EMF, it's based on model-driven engineering techniques, models, meta models, you pass models as input to model-to-model and model-to-text transformations that generate all the code automatically. The difference is that it's deterministic. With LLMs you they are probabilistic and you cannot ensure that they work reliably. And it's open source, you are welcome

to contribute if you're interested. You can use it in its current form. So it should work, it should provide the features that I shown earlier, but if you're interested you can easily extend it. So it's very easy to extend Graphite and if you're interested it would be nice to contribute. the GitHub repo is shown here and that was about it from me. Thank you very much.