About this talk
In this talk, Andre Krastina, a developer advocate at CKEditor, shares strategies for improving testing processes in software development, particularly for rich text editing components. He emphasizes the significance of having a diverse test suite that includes integration tests, timeline strategies, and AI model interoperability. Andre discusses the methods used to analyze a repository with over 40,000 tests, highlighting the importance of broad coverage and deep validation in ensuring effective testing. He explains how to create an integration matrix considering various frameworks and how to handle evolving timelines in software releases. Finally, he addresses the challenges of testing AI models and the use of LLMs in verifying performance through precise prompts and thorough assertions.
Full transcript
Pretty good. Hello everyone. I hope you can all hear me. If don't, put your head headset up. We can get to it. As one of the booth owners told me, now it's a closing keynote. We are getting to the end. You can get ready for the party. So, in the following 25-ish minutes, I'm going to talk about how you can sleep better on the release date. It's
basically a couple of strategies that I found out in our code base. But, before before I get to it, let me tell you a little bit about me. My name is Andre Krastina. I work as a developer advocate at CKEditor. But, before, around 15 years ago, I spent my Thursdays basically clicking through all of the system, basically going through the test suites, trying to find bugs. It
was a very specific setup. So, I think if I remember correctly, it was the Internet Explorer, and the system was built on .NET, and I was I had version 3.5, I guess. But, yeah, it was the time when the unit testing are were starting to be used more and more. And the first 5 years of my career, I basically rebuilt the whole structure of the testing from
this Thursday nightmare to unit test, end-to-end test, integration tests setup, right? Then, I started to peek a little bit into the into the developer relations. It was around 7 years ago. And usually, I was handing over or I was handed over the SDK sample application and starters on GitHub. So, I was basically overseeing the the maintainership, the CI/CD pipelines, and basically was trying to keep them all
aligned. And of course, there were some tests in various technologies like Ruby, PHP, JavaScript, .NET, and so on and so forth. But, that's basically what I did up until a year ago when I started with the CKEditor. And with my roots in the testing, I usually not not I'm not looking to the product itself. I'm trying to end to the testing strategies because they are actually describing
what the software should do, right? I won't be going that much into the basics of the testing like unit tests, integrations, and end-to-end tests, but I would like to show you like they still have their place. Uh they also are not the only ones that you can have in your test suite. You have contract tests, you have unit tests, smoke tests, and acceptance tests. But, before um
uh before I started with the CK, I found out the repository which was called QA E2E tests. And it was very interesting because it has around 40,000 tests inside. They are not testing just the component itself, they are testing websites and other components, some deployment stuff. I was like, "Yeah, this is very cool repository. There's a thousands of tests in there. I need to have time to
research it." But, my yeah, I wouldn't really uh you know um got the time to spend a week on the repository. So, uh my plan was simple. I will send a CFP and if I if I get selected, I'll basically have time reserved to do the research because I need to do the research for the talk and that's why we are here today, right? So, just to
give you a little bit of the heads-up, uh what I will be talking about the CKEditor itself and the tests uh that I will that I have been researching is basically around developer component. It's a rich text editing component. You take it, you integrate it into your application, you have HTML editing, uh you have real-time collaboration. Basically, you have Google Docs in your application and you can
white label it. So, if I will be showing a couple of examples, just bear this in mind that I will be talking about basically HTML editing, basically Google Docs in your application. So, this repository is not about end-to-end testing only. It's you It's being used on a various use cases, and based on them I was trying to analyze it, right? So, I took the repo. I take
a look to to the runners because there was a CircleCI, it was something in GitHub Actions. So, I was analyzing what's in the repositories itself. Then I took the, you know, the practices and the patterns that I have been using on the various GitHub organization that I have have been managing before. Then I use a little bit a little bit of DAI agents to do the analysis.
I basically went through all of the issues and pull request six six months old and younger. And once I get them, I try to tag them with the patterns that I think they are important and they are interesting. And with this, I find out a couple of issues that were very, you know, picky or like interesting for me. Then I find out the owners. Fortunately, CKEditor still
has the testers and the QA QA team does not have a big of a fluctuation. So, these five people I interviewed, the old school, like we had a call, we talked about the issue, I got the insights, I got the context that is not in the in the in the issue discussions. And once I did that, I basically grouped them out into the three pillars. And these
three pillars are basically the strategies that you can apply when you would like to have the test that are not only end-to-end, unit, or integration. The first one is integration matrix, second one is timeline strategy, and the third one is AI model inter-interoperability because, yeah, we all need to talk about AI. There are much more pillars, but we just half an hour, so I just picked these
three because I think they are interesting. And we will go from, let's say, the least interesting to the most interesting because, yeah, now AI is the thing. So, integration matrix itself, you can think of when you're providing the developer component for web, you can have various frameworks. You can have React, Vue, Angular, Svelte. I'm sorry, Paulo, I didn't put it on the slide, but yeah, of course,
there is a Svelte. Next JS, Vanilla JS. You can have various frameworks, various combination of frameworks, right? Next and React. Sometimes, it doesn't really make sense to have a difference. We are providing the the service as a part of the CKEditor that needs this back end. So, you need to think about this dimension. There are various other dimensions. These four are basically just the, let's say, glimpse
and or the basic ones that you can come out with. You can think of a bundlers, right? You have Vit, you have Webpack, you have various custom plugins in both of these. So, you need to think of which ones you would like to put to your test suite to run it on. Then you have a delivery method. Yeah, I didn't have much space on this slide, so
I basically put put these on and mix them together, but the delivery strategy via the CDN or NPM, the import structures like UDM or ESM. You need to think how you set up your, let's say, final build before you're you're running the testing, right? And yeah, then you have the browser engines. We are not really focusing in CKEditor on the browsers themselves because we are providing the
component. We are very focused on the engines. So, Blink, Gecko, and WebKit are the main ones. There are some experimental ones. You can also combine it with WebAssembly, right? So, you need to really think which one you want to put it all together. And sometimes, this is the red part. And just don't take it as the final list. There are much more dimensions, but sometimes there are
dimensions that you don't really think of. For example, with us is a PDF. You wouldn't really think of that you need to think of PDF as one of the channels, but with us, we have exporting PDF and we need to think how this works and how this is being rendered because we are basically rendering HTML in the text area as well as as well as in the
PDF. There is a really nice quote that Kacper told me. It's our engineering manager in QA. We are doing these tests and these specific integration matrix tests with real clicks. We are not using API of our component to do to do things. We are using Cypress and Playwright to do the do the things as the normal user would do. And this basically brought brings me to two
more most important things when you are thinking about this matrix. The first one is broad versus deep because we had 40,000 tests. If you rendered a Cartesian product of all of the dimensions, all of the possible bundlers, frameworks, it would take ages. So, you need to find out the strategy how to split it out. I called it broad coverage and the deep validation. The broad coverage is
basically you are doing the combination of the all framework but just on the small set of the use cases. I call them happy path, right? So, for the CKEditor is like writing one character, writing a word, deleting the word, undo, redo. Basically, the things that you are doing all of the time when you are doing the rich text editing. For the deep validation, we have much more
detailed test. These are these 40,000 tests, but we are running them only on Vit and Vanilla JS. Vit because it's quick because we need to build the whole solution first and with Vit it's quickest and it's the most, let's say, recent bundler that most of the, let's say, modern modern application use. And we are using Vanilla JS because it's easy to read. We don't have the baggage
of the framework and we can just see what is what is out there and you can see it from the from the first glance. You don't need to have the knowledge about the framework itself. So, that's why broad matters and use deep where it's really safe. So, the basic basic setup with all of the functionality, but not be clogged by the frameworks and other things. And the
second things are the renders. This was the actual examples. We have a feature which is called page break. So, you are editing the document and if you put the page break in, it will do the page break, right? Nothing nothing super special. But as one of the target target outputs, we have the PDF. And if you have ever been working with PDF, it's hard. You can't really
run the unit test to test out the structure of the PDF because there are various versions. It's basically a very complicated and ambiguous XML. You can do some visual regression, but you need to use some browser or reader like Acrobat Reader or you can have a reader in your browser. And there are tons of them. So, you are basically ending up in the same problem as with
the browsers and their rendering engines, right? So, what we started to do with this PDF export and checking the page break. So, is we are just doing the we are just doing the exact match of these XMLs. So, we are doing the snapshot testing. We do snapshot one time. We check it manually. Once it's done, we are having the snapshot and checking the every single new export
is behaving the same. So, you have two renders. There are much more dimensions that you can think of, but always try to be out of the box when you are doing this this integration matrix. At the end of the day, it's just like a table or like list of attributes that you need to make a combination of. But the picking it to picking it right and running
the specific test set is the is the key in the in this case. The second thing is the timeline strategy. This was something that I think I thought it's it's already like mitigated because here you have your nightly builds, you have your beta builds, you have your stable and LTS versions. It's fairly fairly setting or like fairly stable setup that we are using for ages. There is
one more important thing that we are currently we are currently facing and these are the let's say dependencies that we cannot really affect. NPM security checks, NPM new releases, sometimes the CDN releases are making them major sorry minor changes in the versioning but they are providing breaking changes, right? The new version of Chrome is being shipped from beta to Canary so from sorry from Canary to beta
to the final release and it's very hard to have all the glimpse in place. How you can how you can mitigate it? The first one is fairly easy, that's how we did it before. Basically reactive. If you have a good set of tests and you run tests once the new version is out, you'll see if it's fail or not. But it won't give you any flexible time
in fixing it, right? If the for example in this case we were pasting from office so we have the integration for copy and pasting from Microsoft Word or Excel to the rich text. And if there was a table in there and if there was no right border yeah, it flush out some of the flush out some of the styles. It was border right style none. In version
147 it was something else. Visually it looked completely the same so visual test won't take it. Of course the unit test and assertions are. But if we were doing when we are doing the reactive reactive approach we just find out and we need to think of a fix, make a fix, and make it very quick to get out because sometimes these, let's say even non-visible changes, can
break customers pipelines. It took 4 hours to fix and make a release, but it was a stressing situation. So, how to mitigate it is to go be proactive, right? For this case, from Chrome is nice, they have Canary, beta, and final. Honest answer, if you want to test Canary, don't do it. They are doing magical things every single hour. They are releasing like crazy, don't do it.
We are doing beta and final. And in in between beta and final, you have 3 weeks. You had 3 weeks because before Chrome was releasing once in the month. Now, Chrome is releasing every 14 days, so you're on half of the time. So, still it's 1 and half of the week that you can decide which one to go, what versions you are going to release, and if
you're releasing multiple version LTS, long-term support, and the normal normal releases, you can choose on which part you can do it. So, this is this is the way how you can set it up. At the end of the day, we didn't really think about it. It is a per browser setup. We thought about it as the gate release because yeah, there are various browsers and they are
changing their release strategies every single every single half of the year, probably. So, we just run the tests with every single night. If we check or if you are if we find the issues, we know up front for what version of the browser on of what version of the engine browser engine is going to happen. And yeah, if it doesn't if it fails, we need to we
need to act accordingly. And the last part that will take probably the most interest of you is the AI models interoperability and how to test the AI models and how to test your product if they are using AI itself. First things first, yeah, usually if you have unit test or integration test, they are deterministic. So, you can have the you are basically checking the exact value and
you're done, right? For LLM, if you send it a prompt, the response is most of the time very similar, but not the same. And one day, if they are releasing a new version or if you change the way you're asking or if you just sleep badly or something or the temperature change or some parameters change, the the output is different. So, what you can do easy, make
it deterministic. What you can do is switch the API calls to the prompt and back and mock them up or record them and then use this set of fixture. This is nice because the testers are already used to that. The mocking or snapshotting approach it's very normal, but at the end of the day you are cutting yourself from the actual testing of the integration with your AI.
Maybe I should mention there you can think of the integration with AI like there is AI chat and you have a document. You ask it to hey, summarize me a document and make me a summary to the end. This is let's say the use case that you should think about when I will do the the examples because this is very this is very common setup that we
have in CK. So, for the deterministic is nice. Then we found out that if we make the prompts very strict like add me a paragraph numbers in this form, very specific prompt, we will get very specific answer and it will stabilize the test very and very very often it's fine. So, you just have your test. In this case is the prompt was add paragraph numbers. This is
very common use case if you have the the documents or something like that. You need to have the numbering of the paragraphs uh very strictly uh numbered. So, we just like ask, "Hey, give me the numbers." And sometimes it did the job, sometimes it did hash in in before, um Arabic numbers and so on and so forth. If you go specific and say like every single paragraph
will be called paragraph space number dot hash whatever structure you need to, it will very very much um you know, uh optimize and precise your honestly, uh surprisingly, sometimes it's enough. Because for very simple use cases, this is okay in even for changing the documents with like 20 pages and uh I don't know, hundreds of paragraphs. So, hundreds of changes in this case. Another bonus strategy that
I found out on purposely put the issues that LLMs are very good uh very uh well trained for. Like, make a typo in the text. If you make a typo in the text and have it analyzed by the LLM, it will give you the result. The uh the HTML around will stay the same, it won't screw up the structure or anything, even though uh we are focusing
to like test that as well. If you're focusing of checking that LLM is working and it's working properly for very small change, this is also a good um good approach. So, try to have your prompt as precise as possible. There is one anecdote, funny story that I found out, and this is the prompt inception. I know if you I don't know if you um if you uh
registered it, but it was a you were able to basically prompt the McDonald's app and use their LLM model for free, right? And this is something that you sometimes want to but for in our case, we would like to provide the customer also the way to ask this like little uh chat next to the document to allow to tell uh to answer like how can I do
the I don't know, nested list or even how should I configure this component for this and that? Basically, provide the access to the documentation, not provide the access to the whole internet. But, uh we would like to provide it in this case the answer if the AI uh itself uh access able to access the internet. There is one important uh part, and this the tool decisioning, right?
So, if the LLM at the big uh in the back end or the agent is deciding whether to touch this and that tool, you can a little bit point him to the direction. It's always better if you are setting up the tests that uh you are uh aiming for the happy path and telling the AI what it should do rather than what it shouldn't do. Because if
you say like, "Hey, don't use these free tools uh and answer me the question." it will probably think about other options. If you tell it like use this way and use it uh use it uh use it the way that I want to, in this case like use the documentation of this specific version and of the component and load the data. This is the way how you
can do it. So, in this case, you can uh you can allow prompt inception. So, basically, kind of misuse the AI uh that has been uh uh that has been designed for something else for the use cases that you're allowing to. I'm not talking about uh guard railing here. Of course, you need guardrails. Uh we don't want to end up as McDonald's, but uh you can sometimes
you know, uh direct uh uh the LLM uh in the tests that you wanted it to uh test it out and not cut the uh cut out the functionality. And the last thing which I thought would be the most interesting for interested for you is to use AI to test So first of all, you have this invariance. You can still check that the AT HTML is valid
after you ask it for the suggestions or like changing the numbering of the paragraphs or whatever. You can check that the correct tools were called. This is also important. Have the logging, have the tracing in the in the back end that you can check and unit test things under the hood. You can check that the response type is either modification or chat. This is very specific for
CK because when you are chatting with AI, it can give you suggestions to the content like you would do with the canvas for example or it can just answer you answer your question and help you with the ideation. So you can check whichever in whichever direction the AI went to. But the second part is use LLM as a judge. So you have your suite where you are
putting your prompt and your expectation is being returned, right? So if you are asking about add me two or three things to the list, you can ask another LLM is this result containing this specific amount sorry, amount of list items and that they are you they are containing the things but you're asking semantically. So you have your assertion to be text based or sometimes it's even image
based or sometimes even HTML based. Sometimes we have the test that we are saying like hey, this should be roughly the output. Is it? And it can give you even the percentage of the results. So LLM as a judge is good thing. On the other side, you are basically paying twice as much for the test because you are running the prompt, getting the answer, you are paying
once. Once you get the answer, you are sending it to an LLM for verification. You are getting the results. So you need to be a little bit careful about spending there. But as the output, you can get a pass and fail per model, which helps you, for example, if you are having the component or your application use is using various models in various versions, you can have
a history. And we in are using this information to decide whether we would like to upgrade or allow user to upgrade from version, I don't know, Gemini 3.5 to 3.6 if it's better. And we are also using this information, this historical information from these tests, to decide in our agent's workflow because yeah, the AI chat is just one of many functionalities of the CKEditor AI. So we
can have an agent mode, which basically decide which model it will use for each and each action. So this historical information can help you to power this auto mode as well. So when you would like to do the AI tests, first try to do try to do the prompt as precise as possible, and later on just go to do go to do the LLM as a judge.
There is one extra thing that I found out. Sometimes your tests are uncovering that is not an issue in your code, right? Your AI is not a bug in your code. One example, if you're a React developer, you base you probably are you are using Next.js, you probably know the error 418. It's the difference in between server components and the client component HTML. It's very cryptic error,
very hard to debug, and if you are new to Next.js and you don't know where and what is rendering on client and what is rendering on server, you will get a lot a lot of these errors. We found out that you can simulate or get to this error also with when you're using Cypress with a specific version of Next.js because the Cypress as a framework is injecting
to the HTML, to the front-end HTML, but don't touch the server component at all because it's Cypress it doesn't have an access to. So, we found out that this test is basically just the ghost in the machine, right? It's something that you can achieve to reproduce in a specific environment, but users will never get there because nobody is using Cypress to actually do the rich text editing.
So, yeah, you can have all of these matrixes, you can have all of these strategies in place, and they can still be the possibility that sometimes you don't have a test to test the test, right? So, if I wrap it up, we have three things, or I picked the three pillars that you should think of. First one is an integration matrix. Think of the things that are
the environment stuff, and let's say the dimension of your matrix that can affect your tests, right? And think a little bit beyond, like this PDF rendering. For your case, it will be something different, but think about the dimensions that are very very specific for you because they can they can be they can be important. The second thing is timeline strategy. Focus on the way of your environments
and dependency are being released, when and how, and proactively try to set the gates in there so that you have time if something fails to decide what you want to do with And if number three is AI interoperability, if you want to be in an enterprise, or you want to provide the access to the various LLM models, try to use LLM as a judge, store the results,
and try to be as deterministic as possible for most of the test, and if this doesn't work, use LLM as a judge, but be aware that uh the price can be a little bit higher for these tests to run. So, this was basically my research what I did on these 40,000 tests. I checked the runners. The one run for these uh for covering these things is basically
1 hour and uh 10 minutes. So, it's fairly quick. On the other side, if you would run all of the combination and you wouldn't uh care about the integration metrics at all, it's uh it would be hours, maybe even days. So, uh it can help you uh save some money, save some time, and get the results. And it's going to be it. Uh I hope that you
will be uh sleeping uh better on the release date once you apply these changes. Uh thanks for your attention. >> [applause]
