About this talk
In this talk, Simon and Dara from Evolving Web discuss the challenges and solutions related to maintaining Drupal projects, particularly focusing on testing and continuous integration/deployment (CI/CD) practices. They emphasize the importance of having a robust testing strategy to catch silent bugs that can emerge from seemingly minor changes. The presentation covers various types of tests provided by Drupal, such as unit tests, kernel tests, and functional tests, and highlights the significance of a visual regression tool like Diffy for identifying changes that may affect user experience. They also introduce Renovate, a tool that automates dependency management and keeps projects up to date smoothly. The session aims to encourage developers to adopt better practices to improve software quality and reduce silent failures.
Full transcript
part of every project, the one that the clients love to, you know, give attention. So, we're going to see if we can make that a little bit better, a little bit easier. So, my name is Simon or Simon. If it's easier for you, I don't care. I'm the director of technology for evolving web. Um, and I'm here with my colleague Dara, which I'm going to let her
introduce. >> Hi, everyone. My name is Disresa Spinach. I am a tech lead at evolving web and a lead trainer as well. So, nice to see you. So, I'm going to annoy you a little bit with a description of what evolving web is doing. Uh, you probably don't know the evolving web name yet, obviously. we are a major uh Drupal shop, but we evolved like we're now
like a a full full scale agency. We provide uh like specialty in Drupal, but we also do a little bit of WordPress. Uh we have training programs. Dariza is our lead trainer. Um we've we've trained thousands of uh Drupal developers and Drupal content editor over the years. Um we've been part and we're still part of the Drupal CMS uh leadership team. Uh one of our secret goals
for the next few months is to provide a better uh developer documentation for Drupal canvas. So I guess it's going to be helpful for a lot of you folks. Um all our projects are no strategy driven and most of our clients are like uh mainly North American but world class uh public sector healthcare high ahead um that that that kind of trend. You can you can come
by the booth and and chat uh if you want to know more. So here's a glimpse the the path we're going to follow. um a little bit of a context. We're going to speak about the different kind of testing that we think are useful in the context of maintenance. What Drupal provides uh out of the box to run tests. Um we'll explain how we do CI/CD at
evolving web like it's a little bit opinionated, a little bit interesting. So um I think uh I want to share that. Um, and then we'll see what we use to uh automate part of uh the user testing. And Tarisa will uh do a nice demo. There's no AI. I'm sorry. I thank you very much. Here we go. Okay. Hi everyone. So, let's let's put a story up
front. So it was a Tuesday, normal Tuesday for all of us. Very sunny hopefully. So it's nice. Um we have to do a we have the routine maintenance window, right? We have done it a 100 times. We have a module to update maybe a couple of modules to update. We update that locally. We draw sorry we we run OBD. We uh export configuration. We clear the cache.
We check the site. Everything looks nice, right? All good. We deploy. All good. Quickly check in production later on. Everything seems to be working perfectly. Okay. Normal, right? So far. Okay. It was a small update. Or so it seemed. There was just a minor version bump into the views reference module and it was all good. In theory, there was no breaking changes in the change log. except
that now there is actually a page that is no longer displaying the news releases of the site. And then what's we didn't really see it. Nothing really failed. There was just silence, right? But the bug was there. It was silent. It was invisible because we only check a couple of pages in our smoke test. And it was up until one or two days later that the content
manager came up to us, the client freaking out. I had to release something today. I created the content and it is not showing up in the news releases. And it is not just that that one is not showing up. No news releases are showing up. So this is this is painful, right? What happened? Well, we go we look at it. We checked some pages. It was working
fine. The cache we we reveal cache just in case. Everything is fine. Everything everywhere else except there. So what what really happened here? It's all gone. It is silent. It is expensive as well. There were no automated alert that was fired up because everything was working fine. So what happened here? Before we talk about that, raise your hand if you have ship code you knew wasn't fully
tested uh because there was no time, right? Okay, keep your hand up if you have done that in the last six months. And be honest. Be honest. >> We can be honest. There is no judgment here. It's perfectly fine. That's what we're here, right? Almost there. There there's a lot of hands that you'll be up at this point, right? Including ours. We have because there's no time
a lot of the times. So, we we just have to go with what we have time and resources that we have. This is not a talk about judgment. It is just that we have all been that team and we want to improve that situation, right? What do we do about it? We know for sure this is not a skills problem because the teams know how to actually
write tests for example. Uh the issue is that most of the times the maintenance work is underfunded, right? Money is there for new features. The new features actually get a sprint, they get a specs, they get all the butt, they get everything, right? We even have design reviews for them. But maintenance, we'll see an exporter and automated testing, let alone that. We we'll see when things calm
down except that things never really calm down, right? So the natural instinct is to say, "Oh, the problem is that we need more resources. We need to hire more people. We need more time for QA." And while that might be true in some cases, it is not true all the time. This getting more people doesn't really scale. As a site grows, the surface area for regression issues
also grows. So manual testing gets more expensive, it gets slower, and we still have those silent uh quiet failures happening. So a visual regression tool is something that will have caught the issue with the news releases page in under two minutes, right? So let's fix that. Let's actually see how the tools that we have uh help us right by the end of the session hopefully we will
have everything we need to go back to our team and build this. So now housing gone. >> Yeah. And we know there's no silver bullet for that. But what we know is like it's always going to be hard to explain what's the value of proper maintenance and continuous testing to our clients. Uh because it it's very invisible and like it's kind of on us. I understand that
it's it's put us in a in a better place on the long term, but it's really hard to sell the value. So the strategy is more about how can we do more with less and especially less time and more automated tooling so that we can spend our actual human time on the stuff that is really hard to automate. Let's let's just go quickly over what I like
to see about the three tiers of uh of like the different the three different areas of testing which I d the tiers of confidence. The first is like the safety net like everything you can check at the code level uh only without running the full system. So you can think about the the tools like PHP cod sniffer PHP stand all the llinters that are going to look
at you know your JSON file, YAML file, tweak file and quickly check uh for obvious mistakes um uh deviations from your coding standards and the likes. uh they're um you can run them early in the process. They're quick, they're fast, and uh as an example here is like quickly spotting that you're using a filter in your tweak file that is not declared or that your system doesn't
know about. Uh there's no need to deploy that can be caught up very very early. The next level is to test on an actual uh running system uh a fully bootstrapped Drupal including your uh code change and then on those an actual human can go and test uh so think your QA team but you can also start doing some automated testing like unit testing integration testing functional
testing we'll see we'll talk about it later but also visual testing mainly by allowing you to compare the state of the system you are about to ship with the like the the well-known working state like your production system for example. Uh one example can be that which is that that's an example of an actual test we run using playrite and you can play the game of the
end differences. I don't know how many there are. There's an obvious one but you can see that uh if a an actual QA resource like an actual human being would just stare at the right side the new thing it's very hard to kind of cognitively spot that something changed. If you have both side it gets easier. uh and we'll see that there's actually tool that automates that
comparison to just quickly spot at a glance what are the pages that changed and check if it's expected or not. And the last last year is more about monitoring the long-term trends. Uh how is the overall performance metrics or how they're trending over time? So for example, the page load time, the error rates, they're useful to detect like those changes in in in the trends and tie
them to specific changes you made to the to the code. So examples of that that category are for example New Relic that are going to be monitoring constantly your production system and by analyzing this data you're going be able to say oh something changed at that date I'm going to be able to go back at my change log and figure out if I change something at that
point. It's not going to be really the the topic of that talk called focus on the the first two. So I'm going to give it back to Dar and we're going to see what Drupal provides >> Thank you. Okay. So now that we have gone through that um the question is not really if the tool exists to fix this right but which one to use. So in
Drupal out of the box uh we have different tools. So Drupal the Drupal test is built on uh PHP unit and it is organized on three tires. So the first one is unit tests. Then we have kernel tests and then we have functional tests with a variant for functional JavaScript tests. For unit test these are the fastest. Um they test a single class or a single method
in isolation. No dependencies at all. No databases there. No Drupal is bootstrapped, right? Not at all. Then we have uh kernels test the test tests sorry these actually sit in the middle they actually bootstrap Drupal uh but it is just a partial Drupal environment we do have a real database and the kind of things that we can test with kernel tests are actually hooks entity operations maybe
some service integrations that we have but without spinning a full website right and finally functional tests these ones go all the Okay, so we have a full installation. We have real HTTP requests and we have real page responses. Right inside of this one, we have the functional JavaScript uh category. For these ones, we actually add a browser on it. So, we have a Chromium browser for anything
that needs JavaScript uh to run in there. It is important to know that each of these uh stages trades a speed for realism. So the trick is knowing where in the spectrum our problem lives, We know those are the three types, but when do we use each one? The honest answer is use the lightest one that can drastically catch the bug that you're worried about. So if
you're worried about a custom field for matter that needs to a string transformation, then write a unit test for confirming that that string transformation is done correctly. If what worries you is that a hook fires correctly after an entity is saved, then for this one you can go with a kernel test, right? You don't need this in isolation, but you don't need the full bootstrap. You don't
need the full environment. If what you are worried about is an editor filling out a form and then the right content showing up on the page that is functional, right? We want to make sure an action happens and we get search and result. So we go uh with functional tests and if any of that actually involves JavaScript interaction like we have a dropdown that is populated with
addex or something like that then we will go with the functional JavaScript. most of the times what the the failure that we see is that people is trying to reach for functional tests for every single thing. So you're killing an ant with an atomic bomb, right? That's not what we need. Uh so a lot of the times all we need to do is just find where in
that spectrum we are and then go with that type of test. The starting point for each of these um is specific. Okay, it will depend on the type of test that you're working on working on. Sorry. Every test type is going to have a different base class and that makes Drupal know what is the type of environment that it needs to set up for you. If you
extend a unit test case then basically you will get a blank PHP unit environment. No Drupal, everything just for you to test your isolated code. The kernel test base uh boots just enough of Drupal so that you can run what you need even with the modules that you need uh for that case. The browser test base goes the whole way. It has a full installation as we
have mentioned. It actually have a real test site spun up in memory and there are real HTTP requests that you can make to it. And finally, web driver base is actually extending the browser test base, but uh it has a chromium instance attached to it. So the pattern is consistent across all of them. And again, you don't need to configure all of these manually. You can just
get started based on any of those base classes. In some cases, we want to know how to get data into the tests, right? We want to make sure our tests are self-contained and repeatable and they don't change depending on our schema, right? So sometimes we want to have some background information, but we don't want that background information to be depending on the actual real data. So we
can set that up using the setup modules that all of those classes have that allows us to set up that environment. And Drupal also has traits, a lot of traits that allow us to create different stuff. For example, the node creation trait allow us to create nodes with uh based on on data in an array. Same with the user creation trait and many others. Okay, so we
don't really need um anything else. All we need to do is work with those traits. Set up the uh background information for each of them and run it. Great. We know what are the type of tests. We know uh how to work with them. But what do we test? Well, we will test our code. We don't really need to test somebody else's code. Only our code and
our team's code. Why? We do not need to test Drupal core. Core has its own tests. Contrive, same thing. They have their own tests. The community already does all of that. And if they don't, then that's a problem for the issue queue that we can help on, but it shouldn't live inside of our own project. What we are responsible for is the code that our team writes.
So our custom modules, our custom plugins, the business logic that we create because that's the thing that doesn't really have a safety net. So that's what we need to work with about the contrive end uh core thing. If you discover a bug in a contrive module and you fix it, that's great. But best idea will be to contribute that back to the community. And when you create
that patch, make sure you also include tests for it. That's actually the reason why Drupal has so much coverage everywhere, right? A patch is not merged until it has tests So um the contra updates uh are important to have them that way instead of having them just in our environment and also instead of having a fork because later on it it is difficult to maintain and nobody
remembers what the fork was for. We also need to think about this. We don't need to aim at 100% coverage, right? Not everything needs a test. The best way to think about it is more like um what is the thing that will be very problematic if it breaks or for example what are the things that we are doing that we are testing manually every single time before
the deployment. If there is a flow that we have to do every time before the deployment then chances are you should write a test for it. Okay, run the tests. And this might sound obvious, but there is a lot of cases where there is even a decent test to it in projects, but they're just not run. They haven't been run in months. So, it is important that
we actually run them. it is not that difficult. If you're using Dub for example, that is the command DV exec and then you use the PHPUnit um bin and just pass the path and you will be ready. Okay, but if you want to make it even easier and more accessible for your developers, you can even create a custom command in DD with that. And once we're sure
it is running locally, we can make sure as well we integrate that into the pipeline. And that's a part that we're going to go through next. >> Thank you, Darisa. So, we're going to speak about pipelines, which is a hot topic in the news these days, but I mean, I digress. Not those pipelines. Um, everyone knows what pipelines uh, I mean everyone knows like most of you
probably knows what a CI/CD system is, what pipeline are. Everyone is accustomed to the triple uh, def test production environment in our uh, hosting environments. That's uh, I'm not not going to try to convince to use that. It's uh, it's like it's well known. Just going to try to give you a little bit of what's our opinion on how to use those and how how we do
how we do that at at evolving. basically the the journey of the journey of a of a code change. Um what's really important like that something we really want to to to enforce in our projects is really to always test on a separate environment an ephemeral environment that is not going to be one of those three uh longunning and stable environment. Rule of thumb for me is
dev is not an environment on which you uh try to test unfinished work I might try uh raise your hand if you already pushed unfinished work to the dev environment and I'm going to raise my hand first because I've been guilty of that in the past. >> It used to be that but we now have the tools uh to do it in a more in a in
a in a cleaner way. Think of it like dev is where the latest version latest running version of what your dev team is producing lives. test is where the client sees the upcoming release and they can do something that is long like more stable and they can test it before it goes live to production. If your dev environment is broken because you're using it as a as
a like a a place where you fine-tune your work and suddenly a major security update happens, then you're going to be in a pickle because you're going to have to figure out how to roll back a bunch of stuff and have a clean and have a clean pipeline. Um um a nice metric you can monitor is what is the number of days you have between the different
releases of your last three pipelines. And you should try to have that like to shorten that as much as possible. Um we're we're using um GitLab mainly at evolving web. So I'm going to make all my example around GitLab, but all of that is applicable to GitHub actions, Bitbucket pipelines. Uh but that here we're using a GitLab CI. So I'm just I'm just going to get back
but our pipeline works with three different stage validate deploy and then a review state is more for testing. Um so for all the old-timers if you see validate and deploy you'll recognize that we've been using BLT for a long time. So everything is is tied to that. Uh if you still happen to be using BLT on the Drupal 10 uh website and you feel you're stuck on
one everything uh come speak to me. We have a solution for that to run VT on Drupal 11. Um, but that's what it looks like on a on a typical GitLab CI project at evolving web. There's multiple uh stages, multiple jobs, but those are basically the three stage that I was talking about. You'll notice that the validate stage is completely independent because that's where we run all
the tests that are not tied to a running environment. So, they're quick and they're always they're always run and we can always stop the deployment process in parallel because it doesn't matter. We don't need to wait for that validation test. uh validation step to be over to stop deploying the code. Um so deploy is all about packaging the code and then pushing it to uh the testing
environment. Um here you see Pantheon is is mentioned because that specific project leverages the Pantheon multidv uh feature which is a nice feature. Pantheon are nice people come and come and check their booth. their their sponsor of the events but that could be something like tugboat for example uh which also are very nice people that and you can go and check their booth and speak with them
um upon provide the same kind of feature uh they're also sponsoring the event they have a booth come and check them uh and it also could be aqua aqua also have on demand environment and uh someone's telling me they also have a booth you can go and and check them out >> uh but also if you do that also come and check out all booth obviously uh
so that uh we can get a little bit of traffic to. Um there's something very important um that we kind of changed across the years like once we started to have those automation in pipelines there was a tendency to run them all the time on every changes like every time a developer would push a commit on any branch in any circumstances it would always trigger that whole
pipeline. And we kind of scaled that down a little bit to decide that it would be only when we push code to the main branch that is supposed like is supposed to get to the last stage and goes to the dev test environment that we would run that or if an MR or PR is open like a merge request or pull request is open. We consider that
if the developer is not opening a merge request means that the the work is not completely finished yet. There's nothing to test. So there's no need to kind of clog up the CI/CD resources. Um so we'll see how we do that. Um is that moving forward? Yeah. So that's basically what I was saying. Thanks. So as I was saying, everything is around the the merge request. So
whenever uh the work is advanced enough and we feel like we we should have it running somewhere to test it, we open an MR even if draft draft mode. And that's the signal that the work that that we're we're doing on a specific branch is is good enough. Building like triggering the build or the deployment is still considered manual at at this stage. You you'll see on
that merge request, GitLab integrates the pipeline and you see all the different steps. We're going to run the validate stage because it's cheap and we can run it anyway, but the deployment is more is consider more more expensive. So it requires the developer to go and and click on a button to trigger it. Sometimes though, it's useful to be able to opt in to automatically do that
because like you're on the late stage of your like fine-tuning the code and you really want everything you push to be automatically uh triggered. So we in order to do that, we leverage GitLab tags on the merge requests. So we fine-tune our jobs. If a specific tag is attached to the merge request, then the deployment becomes automatic. So then the developers can choose between something that's manual
and on demand, something that's completely uh automated. So for that we use uh we leverage the GitLab CI rule systems which is the the most advanced way to configure when and how jobs are supposed to to run. So as I it defines when a job is actually activated as part of the pipeline. It also allows to refine the behavior of those job depending on on the case.
So it looks a little bit scary. Let me drive you through that. What you want to look at is the rules section. You see that the script section is like actually very simple because it leveraged BLT on the background. What's very important is like all the different rules and there's like each rule specifies a different case. The first one um is what to do if what got
what got pushed is a tag. The uh next one is whenever we push to the default branch and so on and so forth. The next one is like if it's a merge request. if it's a merge request that we want to run automatically and so on and so forth. You'll notice that it does two things. It set up the condition when the job should run, but it
also customized the behavior. Sometimes it's when is set up to manual, which means it's going to require user intervention to trigger the job. And some of the in some of the case it's on success, which means if the previous stages of the pipeline run successfully, then it's going to be triggered automatically. it also tweak a little bit the the the behavior of BLT depending on the on
the on the on the case. So that's that's interesting but obviously we want to scale that as an agency and many of you work in like agencies are multi- project at the same time. Uh we want to unify uh that that configuration. So the goal is to have the same on on every on every project we manage. Um, but we still need to handle the the slight
differences we have on different products. So there's there tend to be a an explosion of edge cases and different uh things we want to fine-tune. But we still want to have one reference configuration to keep it current and then avoid having like like bespoke configuration on each and every project and then when we go back to an old project, we we don't have the same tooling as
the the most recent one. So it tends to be a very big GitLab CI configuration. So if you already toyed with like we already started to play with the gitlab CI configuration and we all done that you start using the main gitlab CIAL and it tends to be uh to be big. So one of the solution for that is just break that in multiple files. uh you
see on the screen here it's broken down in multiple files in that that repo and then you use include to kind of uh consolidate everything and um you you can also use that technique to to pro like you you can also provide the default configuration using the the the variables settings here what we do in order to consolidate that's a nice uh feature of GitLab but it
exists also on on other systems we have a single project that's the reference repository for the CI configur iguration and on an actual project uh we're going to reference that. You see on the left the configuration the CI configuration of one project is just referencing our uh canonical configuration. We use branches and the ref keyword so that on a specific project we can target a specific branch
of our configuration that avoids like we can make some progress to our common conf configuration without introducing breaking uh changes. If we want a new feature on a pipeline, we're going to create a new branch and each project will opt in when they're ready to to integrate those changes. You've seen probably a lot of environment variables all the way. So that's the what we use to make
our configuration flexible flexible. So when we use the configuration of a specific project, we have to provide those variables. And in order to make it easy, it's just like just remember there's two cases. You either need to manage let's say system settings and secrets. And those are going to go in the GitLab configuration in the settings of the project. They're they're kept here. They're not revealed automatically.
Only project owners can access those. So they're kept safe. And the other variables that are more like project specific or things you need to fine-tune along along the life cycle of the project. For example, the PHP version or whether you want to add additional branches you want to see deployed automatically. you put that in the gitlab CI config so that you can track the changes or you
can have them specific to a have them specific to a branch. So having all of that like that's the the usual and well-known benefits of having a CI/CD but also what it brings us is the ability to to use the CI/CD system to spin up ephemeral environments and and also um avoid like the the thing we're using avoid clogging up the CI/CD resource that we have and
really make sure we're using it only when when when we we need it and exposing features such as creating multi-dev environments or tugboat environment in the CI/CD allows developer to do that autonomously without relying on DevOps people and even not even like they don't even need access to the actual hosting platform uh dashboard or access. So you can even keep that that aspect even a little bit
more secure. U there are cases where we're working on projects where there can be like 10 different developers. it tends start to be a little bit insecure if you let those 10 developers have access to the Pantheon dashboard or the Aqua dashboard and they can all do some different kind of things on on there. Uh um yeah and then by sharing the configuration you can apply that
on every of your projects on the platform. So that's about like the the underlying technical foundation that we can use to bring more tools to the developers. Now what about that boring maintenance process that we all have to go through uh every now and then? Uh what like let's let's talk about what's the anatomy of of that process. So what's great is that for a long while
already Drupal adopted composer. So we're in a better spot when it comes to do module updates or core updates. So, it's just about checking if there are any uh updates and then apply the updates, right? And jump down. All right. Uh not really. Right. After the code is updated, we need to go and run the update hooks. And then once the update hooks have run, if they
run successfully, we're going to have to check if the configuration change. And then if the configuration the configuration change we have to reexport the configuration so that we can then commit the change uh like the uh country updates or core updates alongside the configuration change otherwise you're going to just ship the code change but not the necessary configuration change. You need to push that and deploy to
the dev environment and start testing that in order to figure out like do some smoke testing. Remember what are the things that are usually breaking on that website and confirm everything you've need to do that every week or maybe every month or maybe every quarter because at that point you get a little bit too uh burned by the process. How can we make that easier? Let's focus
on the very beginning of that process which is when we check the update and we apply the updates. I want to I want to introduce you the tool I maybe may maybe some of you already know which is called Renovate. It's an open source project. It's on GitHub and it's awesome. Basically, it checks if there is updates on your project that needs to be applied and it
automatically open merge request against your project. So the only thing you have to do is review the update and merge it if everything's okay. It integrates with your source control manager. So if you're using GitLab, it integrates with GitLab. If you're using GitHub, same thing. Bitbucket. It supports almost demo. I've seen it running on um Azure DevOps. So it's really uh really supports all of that. It
integrates with um the package manager. So for Drupal, it's going to be composer, but it also supports npm if you're using that for your uh front end like theme front end assets. Um but it also kind of like I I yet to find a package manager, it doesn't support. So even if you have like very esoteric piece of code and I don't know using rust and you
have to man like you have to be integrated cargo and that kind of thing it supports that and there's no additional UI. It integrates with your existing GitLab and it does everything through merge request and the existing issues. So you don't have to kind of manage really an additional tool. You just give that script that runs every like 15 minutes or every hour access to your project.
You're giving an AP an API key and it each time there's a detected update, it's going to open an MR with the change uh ready to be uh reviewed. That's the example of an MMR that got open when Renovate detected that that project runs on 1133 and 1135 is released. Seems like we skipped one because probably we do it only every month on that project. Um, how
does that work? You just provide a renovate.json in the uh root of your project repository. You give an API access that renovate can uh connect to your project and open merge requests and and uh push commits. and you give rules uh in order to explain renovate how you want to group the updates together because otherwise by default it's just going to open one merge request for each
individual updates which can be Drupal contrib. The core uh core comes with multiple composer packages so it's going to start to be a lot and it's going to actually become it's going to be consume more time that it's going to save to to handle that. Uh so what we do is usually we group for core individually contrib like so core is going to be on his bucket
all the contribs on a specific bucket everything Drush is going to be on a different bucket everything nonruple is going to be in a dependencies bucket and whenever there's a major update it's going to be taken out of those buckets and be treated as a separate merge request so that we're never going to automatically introduce uh obviously a Drupal major release that's never going to be handled
automatically but also major contribs uh major release of contribu often time needs to be reviewed more carefully. So we put them in a specific match request. Um quick glance at the config file. So you see that uh renovate even uh allow you to configure a schedule when you think it's a good time to run that because if an update is released obviously renovate is going to start
opening merge request on all your projects. So you don't want that to happen at 10:00 a.m. when your developers actually need the CI resources to do actual work. I'm not implying that maintenance is not actual work. Don't worry. So that's the the general settings. And then you have the rules. So like looks a bit scary, but I annotated that. So you you you'll see that we have
a patent matching system to match what's core, what's contrib um and and then sometimes it's it goes to a group. there's a group name specified or for example for major updates there's no group because each package is going to be treated individually uh and there's that nice dependency dashboard approval which means those uh we're going to even wait for the developers to give approval before creating the
merge requests as I said there's no UI so renovate works by opening an issue in the issue list of your GitLab project there's going to be a specific issue for Renovate and is going to automatically maintain as long as it runs and it provides you at a glance the state of what's in your project. So what are the updates that are pending approval uh from the developer
to actually create the merge request? What are the updates that exist but they're going to run only when it's the the right time frame. So at night usually and what are the merge requests that are currently open and maybe you want to ask Renovate to rebase that because you didn't have the time to look at it and then it it got a bit updated. The nice thing
is like you just check the box. So it's okay. You just edit the issue description and when renovates going to next run, it's going to just see that you checked some boxes and it's going to run those uh those tasks. So once all of that is set up, that means you can even ask GitLab to list you all the merge requests that exist for all your projects
that are tagged with a specific tag. In that case, we're tagging our our projects with uh that that specific update dependency tags. And then you have at a glance all the maintenance work that pertains to updating core and contrib and and and you know what you what you have to do without having to clone the project on your laptop, update it, make it run and then run
uh uh the composer audit comment. >> Does it really work? Uh like of course it works but you have this like is is it really working at scale on all your projects? What can happen? Well, the renovate setup could be broken. Renovate could be stuck in a loop where I don't know there's something weird in your compos or JSON and you can't really figure out what's happening
so it fails. And another thing that's even even more pernicious is like you could be doing all of that job but everything is stuck either on the dev or the test environment and never goes to life which means it's it's almost like you've did that for nothing. What you can rely on for that is um Drupal has the status page and one of the item of that
status page is about are you up to date or not. You see on that example Drupal tells us that there's available updates for one or one or more modules or theme. So the idea is to try to grab that information automatically. So we have a different way of checking the state of the updates that is not relying on GitLab or the codebase or or whatnot. So of
course parsing that page is a little bit cumbersome but Drush to the rescue. You can get all the information of that page um as a as a JSON output and it's going to have the information about the the status of of your updates. So then what we do at evolving web like we have a set of recurring scripts that are going to connect to our hosting providers
APIs build an inventory of all the website we're hosting and then leverage Drush to fetch all those uh the information from the status as a JSON file and show everything in notion as a database um because it's just like the best way to build a dashboard these days basically. Um so you see here we have a column where we just show all the row JSON that comes
from Drush and then we use a couple of uh I think it's called formula columns with um specific um uh specific statements here in the like for notion formula language and then we have uh a nice status about for core or for of for countries what what is the severity and Drupal gives us a warning severity if there's there's an update or an error severity if there's
a security update. So, that's a nice way to spot at a glance if something's wrong. That's an example. We don't have more than 15 website with security updates pending. Um that that might be the the only example that actually leverages AI don't try to understand or to craft those kind of formula yourself. Ask Chpt. It's it's it's quer. Um well so now we have automated all of
that. We're capable of monitoring it and I'm going to give it last uh like one more time to Darza to speak about how we can test. >> Yeah. Okay. So >> good. >> Okay. Yeah. So we have all the automation for making sure that the uh updates are applied automatically, right? But we will still have the same problem as the beginning. So we will still want to
apply um even even with the PHP unit test and everything we talked about before, we will still want to have something else that sits on on top of all of that that allows us to cut regression like visual regression issues. So for doing this there are different tools out there. I'm going to talk about D specifically. Um the way in which it works is actually very easy.
The mechanics of it is easy. It basically captures a screen screenshots of your pages. It stores that as a baseline and then after a change it can make a screenshots about the of the new version of the site and actually compare that. So it does a pixel level div of the two screenshots and then it highlights whatever change that we can see. Right? Anything that changes is
highlighted. The reason this matter is that it sees the page in the same way as your users see the page. So that's great. It is not just that we see and we think that it looks pretty much okay, but that it actually compares pixel by pixel the differences. Right? Sometimes there are some changes that are difficult for us to spot. Maybe we saw it and we thought
it was fine, but maybe it was slightly switched or moved to the left and now that's not okay. Or maybe some text went to a new line or maybe a box got bigger than the other one. There are many things that could happen, right? So with this we can catch that. The nice part of it is because it is a software as a uh platform. Sorry, platform.
It is a uh service as a software as a service. I forgot. Okay. As it is a software as a service, there is no really a platform that you have to deal with that you have to maintain. Basically, you don't have to manage where to store all the screenshots either. That's also not happening. The only thing that happens is your pipeline makes a an API call to
defy. Diffy does the work and then it returns some report right that you can work with. Now that's but what defy is not. Uh we have been talking about different tools right do we need all of them? Well at the end yes you do need all of them. uh potentially uh you want to create all of the type of test. Again, you will try to see what
hurts more and then you will create a test for those and then you will build your test with over time. But all of these type of tests are doing different things. Playright scripts for example are going to check behavior. So click this, fill that form, check this result, right? You want to test that with playright uh PHPUnit it is for checking functionality in isolation as we discussed
but if it tests the appearance okay and it is not just does it look okay is it it is does it actually looks as it is supposed to look right and then something else that is great is that it doesn't really require a full QA team to operate it it actually uh allow the developers to catch those things before it actually goes to QA and they create
another bulk ticket and then it is sent back to the development. We can see it right in a moment uh when we create the merge request if we have it uh in in if we have it configured in this way. So we get to see about it directly from the MR from our pipeline instead of from the QA people or even worse from the client. great. How
do we configure it? How what to test? How many things? Okay, defy has a page where you can configure what are the pages that you want to be testing. the real question is that not every page in your site map should make it to the necessarily. Um, again, it is going to cause based on how many pages you have. So, that's also something to consider, right? There
are other tools that could work for scrapping everything, but uh we could use this for the top end pages. And this is going to depend on the size of your site. So remember, you're creating a set of pages that cover your site's visual surface area. So if you have a small website, maybe five to 10 pages is enough. You choose the most important ones, and that's good.
If it's a small site, 10 to 20 pages might be enough. A large or complex website, 20 to 30 pages will be fine. Now, how do we choose them? I like to think about them as templates. So, because it's it's Drupal, it is easy to think one per content type, right? And that's perfect. If you have five content types, you want to test one page of each
content type. You want to test 10 pages of each content type. Doesn't really make sense. They all follow the same format. So, you can choose it that But not just that, you also need to think about the other type of templates, let's say, that you might have because maybe you have some landing pages and they have different ways of displaying. Choose based on the type of layups
that you might have and select them there. You can also think about it as the most important pages, the pages that receive the most traffic. If those break, what will happen? Again, the client is going to freak out because the most important page in the site is down or has something that is not looking correctly. So that is another great candidate for adding to the top pages
that will be tested with every change. Um you can also check pages with the most layout complexity or pages with third party uh embeds. We had recently a case where we were updating the content security policy of the website and the form that was embedded for a subscribe uh functionality just broke basically because it was coming from a third party and when we updated the content security
policy it it couldn't land it couldn't be shown it was blocked. So this thing allow us to catch that kind of stuff as well. if it is set up in the list of pages to check. There are two moments that matter in terms of when to test this or when to trigger this tests. One of them is on every merge request. So um when a developer opens
the MR the pipeline is going to spin up the review environment. It will deploy the branch as uh Simon mentioned before and then it will trigger diff there's going to do a comparison against staging and if something is shipped visually it is going to highlight it. So the developer is going to see that before it is actually merged which is great. That's step one. The other moment
is uh right before we promote to promote to production. So here we're not necessarily looking for code regressions but for environment changes right because if we had it on MRS we already check for code regressions but now uh data is update that has haven't been run a switching environment that is the kind of things that we will be cut uh cutting here and all of these are
things that don't show up in a PHP unit tests or in playright so um this is a great tool So let's see the situation. It's Tuesday again. We're doing our regular updates and um we apply it. It's it all looks okay. We go to the state environment or local in this case, whatever we want. We see the site. It looks okay, right? Pretty much. Okay. We see
all the elements. Perfect. Just as an idea, we go and I I will check I will compare to production just >> it's there. >> Okay. Right. >> Can you spot the difference? >> I will say is it the same? Is it not the same? Let's see if it was not easy to arrive. It was the same page mostly. None of us got it. What happens here? >>
Oh. live demos. >> Okay, just wait. I have it open somewhere else. But there is way too many things here. So, give me one moment. Oh, there is it. >> Okay, great. Perfect. >> Can you? >> So, can you see it there? It's clear, right? The good thing about defy and I'm going to go back just to the whole thing here. Normally defy is going to show
you all of the pages, right? And all of those things that are in red or green normally show up gray. Just highlighting there is a change. Well, purple, sorry. So, you can get to see there what are the differences and work with them. Ideally, you are going to go through all of those pages. and you will choose whether the difference is expected or not. Sometimes it is
just a content change and it's all good. Sometimes it is actually a bug. For each of the pages, you get to see a comparison of production and staging. So you can move this to see exactly what's going on. So in this case, you can see there were some differences here, but it was mostly content. So it was marked as all good. And then down here we can
keep going to the next pages. For the example we are talking about it was this one. And the issue was here. The issue was the breadcrumb. What happened in this case? Basically the menu breadcrumb module got removed. Easy breadcrumb was added but easy breadcrumb was not working properly. So in this case the developer code a bug because we looked at this we had the comment in uh
GitLab saying that this is the this is the report. We were able to go and see it decide what was a bug and was not and then fix it. Right. We saved time before reaching QA. We eliminated the QA to dev uh bug fix loops we now are more confident about our production deployment. going back everything we have covered today from the Tuesday morning that went wrong
with uh to this incident with the easy breadcrumbs is really about one single thing. It is not about perfection. It is actually about the confidence and being able to ship the things um with the knowledge that our safety net is in place and that even those boring tasks have something that is automatically testing and checking things are okay from static analysis that catches the code uh mistakes
before the code runs. The PHP unit catching the logic errors in the custom code defy catching the visual regressions. renovate actually keeping our dependencies up to date and the pipeline making it all work together as a whole. Uh we have the full system. hopefully after today these all feel less optional and easier to approach. Thank you so much. I don't know if you have any questions. >>
Is there any those questions? We we ran a bit over so of course if you want to be on time for the keynote uh feel free to leave. We're we're around anyway. Uh but we can take some questions or you use Pantheon also in GitLab. Did you ever use the merge from like a multvon to go into dev or do you always go back through your pipeline
in GitLab and then >> Good question. So, so the question was do we merge the merge request at the GitHub level or do we merge from the multi-dev environment directly to the test like dev environment on pentail? >> Yeah, it's not as robust. >> No, we we us we usually go back to GitLab, we merge and we produce a new build that we ship and that's that's
why we treat dev as the first entry point as the production basically. >> Yeah. Uh it's mostly not a question but experience. uh so I started uh project with writing unit test ks functional test javascript test and so on and then ended up with uh that we don't need this kind of testing we need only unit tests unit and kennel mixed tests and functional tests with browser
actually for units and kennel test you need php unit and for everything else play right resolve the things but play right is still not integrated uh with uh Drupal. So if you interested in this uh there is an issue to migrate from network to play right and uh you can join them and move the things forward to make it work out of the box for triple core
and concrete modules because it's really big issue to support this uh different kind of tests. Yeah. And for unit tests like I have a module test helpers that tries to mix the unit and cannot test. It provides uh services that work out of the box. For example, if you want to create entity in the unit test level, it just creates this without any code. So check this
out and uh really you can narrow down all the test to mix it plus unit and play. >> Definitely that's good to know. Thanks. Uh I'm not going to be able to repeat all of that. So if you want to go tell in the microphone, but the the the real thing is like really make the difference on what like the test of Drupal itself and the contrib
like code that we spark. The the real the real goal of Drupal developer is to produce as little custom code as possible. So there shouldn't be a lot of code to cover in your own project. If they're lacking the test coverage of contributed upstream don't keep it for yourself in your own project obviously >> and as we mentioned before the uh you don't need to write every
single type of test you just need to know where in the spectrum your problem is actually located so that you can write that type >> It's also worth noting this is the nippy guy. if you want to talk about it and he's a little shy but >> well to this guy >> I was going to ask a question early on you mentioned get commit and recommit hooks
yeah >> and to go I didn't know if you had any tools or strategies for a team that's not using them yet to distribute or enforce that in the workflow so How far you get down? It's all separate. Each person has to do it. >> Well, so historically we've been relying a lot on BLT. So we were reliant on B the Git hooks that PLT provides. Uh
know that PLT was phased out. I mean we replace that with something we call mini BLT just which is basically the barebone that we need that's actually not dependent on anything else. And we we just keep that pre-commit hook that like it's just called BLT validate. And we we found that like just running those very like low-level sanity check on the code before pushing it, it actually
resolves like a lot of stupid mistakes basically. Um and but then like even if you have um common strategy like it's like you know dash no check or like it can be circumvented like quite easily. So the the the real layer of check is on the CI/CD. Anyways, I think it's more like provide where where we're going more and more and that's going to be the next
step at evolving web is to kind of try to consolidate everything around DDEV and basically move away from like having DD on the local and then the CI/CD does things differently. What we're going to move is like we're going to also leverage DDEV more and more on the CI/CD so that it's going to be exactly the same we run on both side. Uh so devs are going
to be encouraged to run it locally because it just saves time but the CI/CD is going to to be enforcing it >> Uh just like I had the same problem that you did and what we did was with our DD installation configuration we just had some bash scripts such as when you set up the project it puts all the hooks in place. We just had a directory
of all linting steps that we wanted to have and as long as everyone's pulling from the same thing and you can crack the whip and use >> crossplatforms. Another thing that I would have to worry about with my team is getting Windows working the same as our banks. >> I was getting there is like when you automate setting up those like it's great if you have an
homogeneous setup for your devs but once they're kind of ready to pick whatever they want >> and your team is growing it's like it's it's a little bit like yeah if it runs on Windows >> not with it. If you use the DX, it's doesn't matter. It runs on Lin container. So it works the same on Mac, Linux and Windows. >> Maybe you can run West directly
in the container. So it will run the server side and all the tools will be run inside the container. So it doesn't matter at all. And you even can run a local machine in the cloud and uh run local uh West connected to this machine. So if someone had have five hard times running the locally, we just launch for in the cloud and they work with their
Windows uh direct. >> Yeah. One more experience. >> The easiest start for testing is just to write a smoke test that opens the web page and clicks contact us or something like this in browser. So it's easier to understand for front end development, back end developers, for every apps. And if you start with this, this is already a good point to make the test grow and check
everything else. But the the hardest step, the first step just to run tests. And if you have QA resources that are not too afraid of technical stuff, you can teach them how to write playright scenarios themselves and they they love it at some point them play will create the script for them. So can create tests and developers will extend this test. So everyone should be there. Usually
developer just write codes and okay this is qu problem to cover this test and if you force everyone to contribute to tests this will be a collaboration that good for >> There was a question over there. >> Yeah. Uh most of our silent bugs are related to different browser versions. How do you handle automate testing for multi browsers? >> You you might know more. Let me go.
Yeah, I well that's a great question. I think we are doing mostly manual. The question was about uh how do we handle automated testing for uh multi browser. >> because most of the silent bugs are because of that. >> Yeah, mostly because of Safari. >> Yeah. Yeah. Yeah. So, great question about automated testing. I am not 100% sure if we're doing something specifically on automated testing on
cross browsers. I do know we have and we're using browser stack but that is more like manual stuff. So we use browser stack to get to find like testings in different environments and get it. Uh I think something that we are doing but I'm not 100% sure on this one. You'll you'll have to test. We do have other uh script that we run. It was we created
a custom script that actually uh checks all pages that we pass. Most of the times uh we're trying with full sitemap and then it does a diff between uh the pages on the baseline and the change version right that is run automatically and I think we are passing like some user agent that uh fakes let's say to pretends to be a particular browser. I think that's what
we are doing but I cannot tell you for sure. >> What about does support has support. It's in beta but people deploy it like maybe two months ago and you can try out >> we we built our own uh visual regression test framework uh that we're containerizing in Docker uh and we're planning on running that in AWS and what we found is um Chrome seems to have
some trouble rendering consistently from test to test. Is that something that anybody else has encountered because it uses like more font smoothing um anti-aliasing stuff like that. So it results in higher level gifts that >> we can for hours. >> All right. >> Thanks for staying to the very end. I think the the next keynote is interesting and there'll be some AI stuff if you if you're
missing out on. >> Uh but uh yeah, we can we can also chat later. Come see us at the booth if you have more questions. >> Thank you. Wonderful presentation.
