Project Lightning Talk: How To Add A New Language Feature To OPA - Charlie Egan, Maintainer

This talk covers the functionalities and improvements of Open Policy Agent (OPA) and its policy language, Rego. The speaker explains how OPA serves as a general-purpose policy engine that evaluates policies based on input data structured in JSON. A focus is placed on a new feature introduced in OPA 1.12, which is string interpolation, simplifying the way policies can construct messages. The speaker outlines the technical aspects involved in implementing this feature, including updates to the parser and abstract syntax tree. Furthermore, the talk emphasizes the importance of integrating these changes across the OPA ecosystem, including updates to documentation, linting rules, and tooling for various editors. The speaker encourages the audience to explore these advancements and engages with attendees interested in the subject.

Yeah, hello everybody. Um, I'm Charlie from the Open Project. So, uh, if you are not familiar with our project, it's a general-purpose policy engine. It's, uh, what can you do with a general-purpose policy engine? Sometimes people might ask us and even if you've not used our project, you've probably written policies in another language if not as, so whether a particular runner can deploy to a given environment

or who can make changes to, uh, a particular path or directory within a Git repo. These are the kinds of things that you can express with policies and they're the kinds of things that we hope that you might express with our policy policy language called Rego. Uh, the way that you use Open is you take some policy code that you've written, you load it into an instance

of Open Policy Agent and, um, then you provide structured information in the form of JSON. Uh, Open then evaluates your policy with that information in mind and produces a decision result for you. So, Open is responsible for making decisions based on your policy. Now, it's also responsible for reloading your policy as it changes and producing audit logs as well. we we've been trying to make Rego, um,

easier for people to get started with and more familiar to them. Um, so imagine you wanted to write a policy which, uh, you've got you've got a a user and they don't have the rules that are required and you want to return a message saying that this username, um, doesn't have, um, uh, it doesn't have a role that's allowed. Previously, uh, you would need to handle the

case where the username was undefined and you would need to use sprintf to build a string using formatting directives. Um, the username undefined handling is not explicitly related to your policy and so we've tried to make, uh, tried to make this better with a new feature called string interpolation. Uh, this same policy uh, would look like this now. Uh, so we have a new string interpolation syntax.

I'm going to talk briefly about how we've added that in Opa 1.12. We thought a long time about whether or not we should add this feature. Um all language changes have a cost, and we'll see a little bit more about that shortly. The first thing that you have to do if you want to make a change to our policy language is to update the parser. Uh the

parser is responsible for scanning um raw source code and breaking it down into different tokens. Uh we added two new tokens as part of this work. Um the first part is um any part of a string interpolated a string interpolated string, um and then there's another token to uh which we use to annotate the end of a string interpolated string. These are our new tokens that uh

our parser is outputting. The next thing we have to do um in the with or still within the parser is to update our abstract syntax tree. the abstract syntax tree is the kind of building blocks of different uh parts of the language modules, rules, values, and so on. And so we have a new AST node called template string, which uh is produced uh after the scanner has

gone through and created those those new those those new tokens. Um we have a new um template string AST node. And it has these different parts within it. The next thing we have to do is to update the compiler. The compiler is responsible for turning the output of the of the parser into something that we can actually execute. Um it goes through a series of compiler steps

to convert the uh the original code that was written and parsed into something we can actually run. Um some parts of the AST that come out from the parser make it all the way to implement uh all the way to execution, um but that's not the case for string interpolation. For string interpolation, the uh string interpolated um node is converted into a new built-in function called template

string. And this is this is how um template uh string interpolated strings are are evaluated in in Rego and OPA. Um, if you're interested to have a look at what your own policies look like, you can try out our new compiler explorer as well and watch as your code is adjusted ready for execution. So, we implement this function uh, this template string function. Um, this is uh,

implemented in what's called top down, which is our Rego interpreter. Uh, the implementation uh, as a summary looks a little bit like this. Uh, it recursively goes into each part of the templated string, evaluates it, ensures there's a single value, which is as required for um, string interpolated strings, and concatenates the result at the end. But, that's not all. Uh, OPA is not just one project, it's

a whole ecosystem of integrations and editors and uh, different tools which are related to working with Rego. So, we have to once we've made the change to OPA, we also have to as quickly as possible deliver uh, related change to all sorts of different projects in the ecosystem. Uh, think about grammars that are used in different text editors um, or the tooling that allows you to go

to definition for a given variable. We also have to update all of our documentation cuz string using string interpolation is a best practice now. Uh, and write linting rules to make recommendations and help people identify when they need to use string interpolation um, automatically. We want to Rego to work well everywhere and that's a huge part of the work uh, when you take on adding a new

feature like this as well. So, um, yeah. Thanks very much. Um, we've got We're around all week. We've got the kiosk in the project pavilion. We have the maintainer track talk as Uh, you can scan this QR code, it'll take you to a page on the OPA website uh, with all of the details and yeah, come and chat to us uh, if you're interested in this stuff.

Awesome. Thank you, Charlie. Appreciate