I started with reading a file.Now my code is inside Chrome | Jason Williams, Bloomberg

This talk focuses on building a JavaScript engine using Rust, sharing insights from the speaker's experiences with the Boa project. Jason Williams, a senior software engineer at Bloomberg and TC39 delegate, discusses the evolution of Boa from its inception to its current capabilities. He explains key concepts such as tokenization, parsing, and the creation of the abstract syntax tree in JavaScript engines. The speaker emphasizes the importance of community involvement, testing, and compliance with the ECMAScript specification. He also highlights advancements made in performance and feature support, including an interaction with V8 and collaboration on the implementation of the Temporal API in JavaScript. The session showcases both the technical challenges of developing a JavaScript engine and the value of fostering a collaborative development environment.

Thank you for having me here. I'm thrilled to be presenting this and being part of the event. I would love to be there in person, but digital is good, too. We'll be talking about building a JavaScript engine in Rust, lessons from Boa. So, we'll see how we get on. But before we do, my name is Jason Williams. I am a he/him. Senior software engineer at Bloomberg. And

I'm also a TC39 delegate. So, that is the JavaScript standards committee. If you have any questions from this talk, you can post them, I believe, underneath this this video. Um or you can also reach out. I'm most active on Blue Sky. So, you can catch me there. My name is my domain name, jasonwilliams.co.uk. I'm also in other places, but I'm probably most most active on Blue Sky,

I'd say. So, yeah, about me. I've been an open source contributor for for many years. 10 years ago, I was really interested in learning more about browsers, rendering, paints, layout, that sort of thing. So, I contributed to a project called Servo, which is a parallel browser engine project that was maintained by Mozilla at the time. And it was essentially designed um using a new programming language back

then called Rust to replace Gecko, which was the engine in Firefox. The the language was very new. So, actually, even when you were writing in that language, it could be changing underneath you. So, some of the syntax that you wrote 2 weeks ago was was no longer valid. And I remember we often had like formatters and tools that would just go around and update all the syntax

every now and then. I don't I don't think I made a >> [sighs] >> significant contribution to the engine during the time, but one thing that I did, I may have been the first person or one of the first to get it working on on Windows. It didn't work on Windows at all back then. Um So that was I was a Windows user, so that was my

main objective at the time. I was grabbing information from people's that was just in people's heads. Nothing was written down. And I essentially collated all of that together. I made a guide for myself. It was actually for my own personal use to begin with. And I followed that guide and managed to get Servo booting up on And really wanted to share this with the rest of the

world. So I was quite lucky enough to yeah, be able to do a blog on the Mozilla Hacks almost 10 years ago. As 2017. Um and I was very proud of this because it was not just showing off to the world that Servo is alive and working on Windows, but also allowing people to get more involved and contribute who who couldn't before because of the operating system

they were on. Another thing that stood out to me on Servo was the speed. It was quite performance. Um This is it running on Reddit back in 2014. I I'm sure it's even faster now. Um but the CSS engine that was written in Rust from scratch was able to make use of multiple cores. It was very concurrent. I think it was actually one of the first concurrent

CSS engines at the time. And it had also good memory safety. So we were all very confident in it. We knew it would work well. There was there weren't really any crashes. But it was also very fast. It was it was more than twice as fast as as what Gecko So basically I thought, well, can we do this with JavaScript? Why not do it in a in

a JavaScript engine? And the Servo team at the time were quite stretched, so um >> [clears throat] >> building a JavaScript engine was not something they were too interested in at the time. They were quite happy to continue using SpiderMonkey, which is the engine in Firefox. Uh so I essentially went off and toyed around with um some ideas, and eventually just uh started working on my own.

And that's where Boa came in. the naming, I I guess I was trying to follow Mozilla's theme of of naming projects after animals. I saw a a a snake in uh a zoo in Sydney, and I thought I I really want to name my next project after that. Uh so yeah, what I'll show you is how I got started really. Uh some JavaScript engine principles. Um it's

not exactly how Boa works today, but uh it's a bit of an insight really into into where things got started. Uh there is some Rust code in this as well. You know, the irony is not lost on me that this is a JavaScript conference, um but the principles are the same whichever language that you try to uh build a JavaScript engine in. I'll try to keep it

as high level as I can. Uh so yeah, one thing I learned quickly uh is that JavaScript engines are basically like pipelines or factory assembly lines or um a like a processing plant. The the work is split into stages. Each stage, you know, you have some input, you transform it, uh pass it along until, you know, eventually you end up with the the final result, which is

usually executing your code. I mean, execution even doesn't need to be the final out final outcome. Uh the final outcome could be another language, so a transpiler that takes a high-level language and spits out a lower-level language or assembly or something. Uh what I have is what we call a tree walker, and the name will become apparent uh in a minute. And so these are the stages

that I I started putting together. You have the tokenization stage. This is where This is where the engine takes the raw JavaScript source code and breaks it down into smaller pieces we're going to call tokens. We have the parsing stage, which takes those tokens turns them into a structured representation to reflect, I guess, the grammar of the language, the structure of the language. Uh that then gives

us the the abstract syntax tree or the AST, and that represents code in the way that the engine can actually work with, and you'll see that You'll see that in a few minutes. Uh once we have that, we can execute it, and then we get our final result. Obvi- Obviously, this is simplified. Uh you know, engines like V8 or even Boa today have a lot more steps

in between these, um but when you're starting out, uh this This is pretty much how I viewed Um but before any of that, we actually need need to read a file. So, um this is just uh some simple Rust code. Uh I mainly show this for two reasons. One is because it's a reminder that I also was learning the Rust standard library as well as learning to

build a JS engine. So, yeah, if you ever want to learn two languages at the same time, you can you can build one language with the other, and you get a good understanding of both. Um the second reason I show this, and by the way, this is yeah, if if it's not apparent, this is a standard library to read a a um a file into a string.

So, the buffer on the left is the string variable, and with Rust, you have to you handle your errors. So, the expect here is essentially printed out if something fails uh during the reading. Um and yeah, the second reason I show this is just to show that actually this was the first commit, um so I'm not um, if you don't believe me, you can see here uh

commit one August 8th, 2018 is is me uh taking the the JS file and just printing it out. So, start simple, you know, you can't really get more simple than that. I put that into a file, made a repo, and then started from there. So, we have our string. It's stored in a variable, and we need to tokenize it. And this is where the engine takes, yeah,

the raw JavaScript source code and breaks it down. Uh it's a bit like the sausage coming out of the factory, and you're you're chopping it up into into smaller versions of itself. Uh if that's an analogy to use. But we, yeah, have our let foo equals hello world. And uh we have our scanner or cursor, which moves along letter by letter until we hit a space, and

then that is our first token. So, you can see here we're sort of looking at this on a sort of character by character basis, hitting a space, and then creating tokens from it. So, here we've got a keyword. We have identifier. We have a punctuation or punctuator. And then we have a string literal. And you might be wondering, well, how do we know these are token types?

How do we know what token types to make? How do we know which one is which? Well, um luckily for us, we have the ECMAScript specification. Um I mentioned earlier I'm a TC39 delegate, so I'm very lucky to have helped also contribute uh to the spec. But this document is open for anybody. It's not behind, you know, closed doors or or payment or anything like that. Anybody

can visit the ECMAScript spec. Most engine authors do. And you can get a lot of information that you need from here. For instance, uh token names or token types. So, you can see here we have uh various tokens, identifier names, punctuators, string literals. And actually, if you scroll down a bit on this page, uh you can see we also have a list of uh keywords as well.

So you could you could take these keywords, codify them into your own engine, and then check against them to see if a if a token is is a keyword. If it is, you know what token type it's given. So back to our example, we have token struct, which is a bit like an object in JavaScript. It has It has two properties. We have token data, and you

can see we have all the different token types there. It doesn't fully match up with what we had in the spec, but that's okay. And then we have a position where it is or where we found it in the source code, useful for debugging later. And then once we've created our token, it goes to our array. So here we have four tokens, keyword, identifier, punctuation, and string

you can see just above I've put what each token is holding currently. So you can see let foo equals hello world. That's our array. Once you have that, you're done. And then that can be sent off to the parser. So the parser will go through these tokens as we have here from left to right. And for each one, we're not only asking what comes next, but we're

taking them and we're building a we're building up a tree, an abstract syntax tree that represents the structure of of what the program is. So we're going to go from this, what we have here, to something that looks like this. And this is a lexical declaration, so we can see the let at the top. And we say, "Okay, a lexical declaration must have a variable." So the

the variable is directly underneath. And even the variable, and and sorry, it's it's an array of variables, even though we only have one. So that's why we've got the square bracket brackets and variable. Because you can define and declare multiple variables in one statement. And each variable has a binding name and an initiator. The initiator is actually optional. You could just do let through and then move

But essentially, you get a tree that looks like this on the right. And when I was putting this together in Rust, uh it was actually something a bit like this. So, our lexical declaration is an AST node. So, what's on line two is an enum, and that's because both let and const are lexical declarators. Uh var is a little bit different. It has different semantics with function

scope, so it it has its own uh node. Um And yeah, you can see that it's just an enum with let and const, and each one has a list of variables. And to see what a variable looks like, you can see line seven, we have a binding and an This enum on line two, you can actually give it methods. Um So, we can create a parse method.

And because our first token was the let, we know that we're dealing with a lexical declaration, so we call parse on that, and that should deal with the rest of it for us. Um so, we we can see here on line four, for instance, we expect that the next token is an identifier. And the question mark on the end of line four is essentially saying, "Well, propagate

the error all the way up if it's not." So, essentially, we're ending the program there. It's a it's a syntax error. Don't need to carry on any further. Um For here, line seven, we're essentially checking to see if the next token after the identifier is an is an equal sign. Um If it's not, then we just say, "Okay, it's a binding with nothing defined on it." And

we just return none. So, we have enough information here to um return a variable type that has a and an initiator, or an optional It actually looks like this uh when it's all put together. Uh so, you can see our it's holding an array of one variable, which has an identifier foo and hello Execution is actually pretty simple because we've done most of the work up front.

So, as I said earlier with this type, coming back to this, as I mentioned earlier that we gave this type a um a method called pass, we can also give it another method called execute. So, you can see the impol on line one. If I didn't describe it earlier, I apologize. is essentially allowing us to add a method to that enum. And now we can execute it.

We've put the values in there. And now the values are in there, we can actually run it. And so, what's going on we do a pattern match first on the value itself. Is it a let? Is it a const? In this case, that the same happens regardless. We we pull out the variables and we store them on vars on line three. And then on line eight, we

we we recurse we loop through the variables. In this case, there's only one, so it's only runs we only have one iteration of If there is an initiator on line 10, we call execute on the expression. So, this is where the sort of uh the name, the tree walking, comes from because if if uh if the lexical declaration is is has an execute, that will then call

execute on nodes underneath that, which will then call execute on nodes underneath that until eventually we get back a value, which is what we have on line 10. So, it's quite recursive. we then declare our variable name into our context with our value. And um I haven't talked too much about scope and context on this. I think that's a bit a bit well, out of scope for

this. But um essentially, the scope is a hash map representing the current scope, and then those hash maps have parents. So, if a variable is not in there, um it will then recurse above and and and so on. But you can see here it's quite straightforward. What we're left with is a binding food that's been created uh with a JavaScript value uh which is a string of

hello world. So, that was a quick run-through of of the sort of things that I got set up in the early stages. It was quite naive as a as an implementation. I I'd never built a JavaScript engine before. Um but luckily uh it's had a few years to grow. And so today I'm pleased to say that we have moved on quite a bit from that, but it

was definitely a good start. We have a virtual machine, so instead of the executing as we go down, we now have bytecode generation. That is a lot more compact. It makes reusability of certain routines faster. More things are cached on the CPU, and we're doing less back and forth to main memory. So, so it means that general execution is a lot faster. Uh we have inline caching

as well. So, um property lookup on on various objects more faster than it was a couple of Uh we've managed to build and maintain a team. So, the things that I showed you earlier was was mainly just me. It was mainly just myself working on Boa for the first couple of years. We managed to have a few people join. They started off by doing one or two

PRs and then decided you know, they wanted to get involved in the project. So, we had a really good community of contributors over the years. I'm very proud of it. I'll come back onto that, but um uh yeah, it's it's it's helped us put together an open API that's quite stable. Uh so, we do have quite a few users. If you know of Biome, Biome for instance

are a user of Boa. They use Boa for their extensions. Um and we have support for various other things as well. So, I've put async support here, promises, and any asynchronous runtimes that want to use Boa, it works fairly well. I just want to highlight community. One thing that's really worked quite well for us is from day one when working on Boa, I wanted to make sure

that it was something that other people could contribute to and work on. I struggled to contribute to other JavaScript engines before I started Boa, so I wanted to make it as easy as possible. I knew that I would never be able to build a a complete JavaScript engine on my own and and it was definitely something that you would need help with. So, making it easy for

people to contribute to was for me was was quite important. So, with that said, I always made sure that, you know, code was quite easy to follow, you know, commenting on various things. If I If something worked, but I wasn't quite happy with it, I'd I'd often try to comment and and say maybe we can try this in future or, you know, to-do's and things like that.

Uh documentation, so in Rust in the Rust ecosystem, you know, we have docs.rs which will host documentation for you if you publish your crates um uh to crates.io. So, I always tried to make sure the documentation was good even though it was just me working on it on my own and there was no one actually else collaborating. I I still tried to make sure the docs were

good. Um good first issues, I always find these quite welcoming and useful. I personally like them when I see them on other projects, so I try to make sure that issues were categorized into easy, medium, and hard. Um and easy and some good first ones to to pick up as well. And then when people did pick those up that weren't sure about the repo um or the

project, I I I tried to help out on pull requests. Um I'm very happy to say we had a few maintainers. There's about five or six of us now, so we moved Boa into an organization, I think in 2020. Uh before then it was just under my own name, and then I also made sure that maintainers had access to, you know, review and merge pull requests as

well. it removed that bus factor of of of everything having to go through me, uh, wouldn't be very easy today. And then of course we had like, uh, Discord and and Matrix. So, for me yeah, building up a community was was quite important and and this really helped us a lot. With that said, testing. I I The The reason why I I mentioned testing after the community

aspect of this is because testing massively, uh, ramped up once we had people working on Boa. Uh, Nicolas Bequignon from, uh, SpiderMonkey said to me, you know, "Make it correct first. You can always speed up later." And so the order I've always done this in is a naive implementation that follows the spec and we can worry about performance later on. Um, we have someone called, uh, Eban

join Boa. Uh, he's he's been with with Boa since I think 2019 to And he helped build a little test harness that ran against the ECMAScript specification's test suite called test 262. Uh, in 2020 we had 18% conformance. You probably can't see it too well. But he also Eban also had these little dots, the red dots you could see it fail and I think it was one

you could just about see for pass. Um, but over the years, uh, people sort of treated this like a challenge and we had all sorts of contributors come to us and massively try to, uh, fix any test they came across. I really like this. This is, um, our test progress over the years. So I think the the left is around 2019, 8,000 tests passed. And you can

see the, yeah, you can sort of see the, I guess the burn up chart of the green going up. By the time you get to, I think it goes to 2024, um, we're at around 75, uh, passed tests. So So that's So that's really If you go into the website today, it looks a bit like this. So it's a bit more boring. We don't have that chart

anymore. We need to bring it back. Uh, we we revamped the website. And so now, uh, we're looking at around 51,000, uh, tests passed out of a total 53. So actually the number of total tests has come down since we since this and that's because test 262 removed a few tests, but we're we're we're pretty good actually on conformance. There's a website called test262.fyi and that runs

engines nightly and gathers the results of how they're doing on tests and we're we're fourth at the moment. We're actually above JavaScript core, which is the Safari JavaScript engine. So, we're currently 95.5% passing on the entire suite. Just before I go, I do want to quickly mention Temporal. So, the Boa team decided that we wanted to implement This was around 2024 when we got to work It's

the most anticipated feature in JavaScript right now. And for us, things really started to pick up around this time. But what was exciting is we we collaborated with V8 and Intel, the Google Intel team behind Chrome. And we decided to work together on a standalone library in Rust called Temporal RS. And over the 2024 and 2025, I put together a team. It's been a few of the

Boa maintainers, myself, Manish from Google. And we've had students from Bergen University, six students from Bergen University join us as well. And about Temporal RS has been actually running and complete, but it now passes 100% of tests for the Temporal suite in test262. So, that's roughly around 5,000 tests that Temporal passes. And so, because of that, it's it's arrived in V8. And it's actually shipped in Uh,

what you can see here, it's shipped in Chrome, 144, Jan 6th. Um, Node.js, which only came out recently, May 5th release. It's been in Deno for a while. And of course, Bower has had it for a while behind the flag, too. So, I'm really, really proud of of this work that we've managed to do. Um, for me, this just shows that, you know, things have come full

circle. You know, Bower was inspired by the big browsers, the big engines. And now, not only we're collaborating with them, where we have libraries and codes that's been contributed to those projects as Uh, so yeah, with that, I'd like to thank the core maintainers. This is not a one-man project. This is a a set of maintainers, but also the contributors. Bower has had a lot of I'd

like to thank everybody who's got involved. And hopefully, if this inspires you, please reach out. There's plenty of things that we want to work on and tackle over the next year or two. So, yeah, thank you. Thank you for listening. >> [music]