The Bug Stops Here: The State of Georgia Shifts Left

This talk discusses the evolution of Georgia's GovHub platform, focusing on the implementation of a 'shift left' methodology in software development and QA processes. Jasmine EPS, Director of GovHub, shares the challenges faced by the Georgia Technology Authority regarding their workflow, including manual testing limitations and delayed quality assurance. The session emphasizes the importance of automating QA and integrating stakeholder feedback early in the development cycle. The speakers introduce Tugboat, a tool that facilitates continuous integration and testing by allowing developers to test code in isolated environments before merging. The talk highlights the importance of establishing clear acceptance criteria, leveraging automated regression testing, and maintaining a collaborative approach with stakeholders to ensure quality and efficiency in delivering government services online.

testing. All right. Good morning everybody. Thank you for being here. First session really of the day uh of the conference. Everything was so much as yesterday. So glad you guys could make it. Glad you guys are here. Um this session is titled the bug stops here. The state of Georgia shifts left. My name is Jasmine EPS. I am the director of GovHub. I work for the state

of Georgia at the Georgia Technology Authority. Um, GovHub is our flagship product for our agency and is powered by Drupal. It allows our about 90 state agency websites to uh publish their web content. >> And I'm James Stanbury. I'm a CEO of Tugboat. I got the pleasure of working with Jasmine and the Georgia folks uh back when I was a architect at Lullababot. Uh so you'll learn

more about what Tugboat is as we go through the talk but yeah pleasure to have you all here. So just a overview of what we're going to talk through how this presentation is going to go. Uh Jasmine's going to start us off just talking about some of the challenges that the Georgia team had back before uh Labot started working with them and some of the the challenges

that we were working on together overcoming some of the solutions that we came up with. Then we're going to switch gears. I'm going to get up and we're going to get nerdy for a bit. So, but we're going to Don't worry if you're if you don't like the technical stuff. We're going to try to keep it fun. Um, and then we'll uh, you know, wrap it up

just uh kind of with key takeaways and that sort of thing. So, >> all right. Awesome. All right. So, first I want to start off with a few stats. I mentioned a minute ago we have about 90 state agency websites. Um, we have 600 registered content editors. And that's just I I preface that with register because you know some people get an account and then never log

in. They just want to have the ability to. Um we have usually about uh 45 million site uh visits uh a year. Uh yeah about 45 million site visits a year. Um and in that time we also have about 184 million page views. Um and one of the stats here I want to point out is the engaged sessions. So engaged sessions are those that last longer than

10 seconds or has some sort of conversion event um or has two or more screen or page views in that session. And a lot of that traffic comes from and 75% of those engaged sessions start through organic search. It's something that we noticed about our visitors. they're able to uh Google just like most people do and land on our sites with ease because of the work that

we put into our content model to make that uh content easily discoverable through Google. And so we house almost all of the executive level government agencies in Georgia. These are just a few department of revenue. Uh we have the governor's website and we always are involved in the uh the change of the guard if you will when the new governor comes along and getting their sight stood

up and getting their administration on board. And we have also you know department of public health uh we were working very closely with them when COVID was going on to make sure that they had uh dashboards and information on their website for the constituents to know what was going on. Department of Driver Services, Attorney General. These are just to name a few. And we have 93 websites

at the moment, uh, with the most recent one we launched just a couple of weeks ago. So, we're steadily growing, constantly growing. So, now let's get into the challenge. Ours being the bottleneck of our QA process, if you want to call it, at that time. So, when I started with the team that I'm with now, it was 2014. I started as a QA analyst and I was

faced with the challenge of standardizing our testing workflow and the tools that we use. Uh at that time when I joined the team it was a bunch of test cases in a spreadsheet. We weren't doing any automation. We weren't uh actually writing like user stories and test cases and things like that. So it was very uh primitive if you will if you will. And just like any

transformation story, we have to begin with a little bit of pain. Um, so I don't know if any of you ever been to Atlanta, but this is like um any hour of the day. We don't really have a rush hour. Um, and we have like like the worst rush day apparently is Thursday is what I just learned. And it's just gridlock all day. It doesn't matter what

time of day you're trying to get anywhere in the city. So it looks a lot like this. And so, and I I like to use the analogy for our legacy approach to software development. It felt like being stuck in Atlanta traffic. We had all of these things we wanted to do with a really small team, and it felt like we only had one little lane open to

get all that stuff through. We had one product manager, one engineer, and then one QA analyst, and that was me. And so this is you probably can't see all the things that's written there, but you can see that a lot of what is going on is over here on the right. This is our legacy software development life cycle. Most of the sprint was consumed with development work

and all that stuff in the front. Like we had a development partner at the time. Uh our work with them was very siloed. they would work on some things and we would work on some things and then we'd come together when it was time to push those things out to the customers and QA didn't really take place until the very end of the sprint one or two

days before uh making it to Pride. So, as you can imagine, that made things even more difficult um with trying to figure out um what broke, if anything. You know, um our vendor partners would work on things, like I said, in their own environment, and we wouldn't see what they worked on until the end of the sprint. So, everything was very much shifted, right? And it was

a lot of scramble at the end of the the life cycle to get QA done. And like I mentioned before, we had the one single sa staging server. So only had one place to really test things out before it got to prod. And yeah, it was just a little bit of a controlled chaos situation and it slowed down our development. It slowed down our our innovation. So

what is the problems that we faced uh with delaying testing to the very end? So, if we had some sort of deadline to get a feature out, a lot of times we would miss those deadlines. We'd have to push a feature or um some development work until the next sprint. Um there was a lot of burnout. You know, like I said, there was a lot of spreadsheets.

There wasn't any automation. And so, I was doing a lot of manual testing, which, you know, that just breeds u um issues, breeds mistakes, user error, you know, if I'm just doing all of the manual testing. Uh it was like I mentioned difficult isolating which pull request actually broke the rest of the code. Um if we're just waiting until everything is merged and then testing it, it's

really hard to identify like which piece of code actually broke everything. So there was a lot of cherry picking. I learned that term from my lead developer. Okay. Cherry picking. All right. Um, so and it was expensive to have to do all of this rework at the end when misunderstandings between the QA team or the product manager or the developers surfaced at the end of the So

this is a uh depiction of me almost uh the time at the end of the sprint very stressed out you know very uh worried then hoping that I didn't break our entire platform hoping that I didn't push a code or you know contribute to pushing a code uh or bug out that was going to you know bring down all of our and then we were faced with

this we need to move at the time about 80 sites from Drupal 7 to Drupal 8. And those were the famous last words before the dread set in because I'm thinking, okay, it's going to be a migration and we don't really have like a solid QA process. How am I going to know, especially going from Drupal 7 to Drupal 8? You guys know how big of a

jump that was and how differently everything worked. like what kind of methodology am I going to apply to be able to ensure that these sites make it to Drupal 8 and their content and everything is intact. So with you know already the the difficulties we were experiencing you know we were faced with this uh migration and and the life support for Drupal 7. So what did the

challenges what challenges were presented with the migration right we already had the challenges with our QA process but then that was compounded when we were faced with this uh migration so you know the lack of testing protocols and tools you know the challenges that we faced quickly uh presented themselves. So the complexity of coordinating large scale changes you know that like I said from seven to eight

huge breaking changes how are we going to manage that um stakeholder review you know we had to go and meet with all of these state agencies and sit with them and say hey we're about to move your thousands of pages of content how do you feel about that and we really need you to like look at it all to make sure you actually need it. you don't

need to move, you know, a bunch of garbage into the new house. Like, please can you look at your thousands of PDFs and let us know if you need those. So, again, that was a challenge. Um, and then risk management concerns like we weren't uh sure what the risks were going to be until they presented themselves because we were this was a new territory for us as

well. So, now let's get into the solution. You know, we saw this migration as opportunity though, even though it was very um daunting at first, uh considering the infrastructure that we had at the time and the and the workflows and the processes. We saw it as opportunity for improvement though. So, what is shift left? You know, a lot of people, do you guys raise your hand if

you know knew what shift left was before got here? Okay, a handful. That's cool. Um shifting left is just literally shifting a lot of processes earlier or left in your life cycle. So certain processes like backlog refinement, stakeholder sign off and a lot of the automation and manual QA can happen sooner rather than at the very end of your sprint. And that stakeholder sign off is like

the biggest thing that Transform does. Like I mentioned before, we were working in silos with our development partners like that we told them what we wanted but then they went off and made those assumptions about you know what was the best way to implement and it wasn't a lot of conversation to understand what the acceptance criteria was for that. So you know what happened what are the

opportunities that we gleaned from shifting lift. So involving all the stakeholders sooner through multiple rounds of backlog refinement and confirming the acceptance criteria early on it really prevented a lot of the rework that we had to do after a deployment. So the bugs reduced dramat dramatically. Performing QA sooner and incorporating automated testing as well as manual testing gave us the opportunity to identify and remediate bugs really

early on. And that way we weren't pushing a lot of bugs into, you know, our production code. And of course, which we'll get into in a minute. We uh implemented tugboat. Tugbo for the win. Tugboat. And I can't praise Tugboat enough. Wherever I go, I'm like, do y'all have tugboat? Cuz if not, you need it. Um testing each PR, which was really cool. U as they became

available. testing each PR as they became available rather than waiting to test until each PR was merged like we used to do drastically improved our production uh development workflow in our CI/CD So this is how we started feeling. Everyone gets a staging server and everyone theoretically could be involved in the QA as well. Like if we have a UX person who uh helped us make some changes,

they got their staging server where they could test their uh UI changes and make sure that they were good while I'm over here testing some functionality like we don't have to wait on each other. There was no more bottleneck. And Tugbo was really the game changer for that for our very small agile team. and it removed that bottleneck and eliminated the congestion of waiting for every bit

of code to be merged before testing. So as you can see now in our diagram things are shifted left a little bit and it really changed the way our team worked. Tugboat again was the catalyst for not just improving our development but just improving like our processes like us thinking about our processes more uh intentionally like this is a way for us to just change the way

we do things all around. So we were able to get more work done within a sprint and being able to deploy each PR in its own tugboat environment. And so the devs could create PRs and get immediate feedback again rather than waiting until the end of the sprint in order to get allow QA to test and give that feedback. And QA was able to test immediately. you

know, they didn't have to wait after PR was created and give that timely feedback and giving the devs time to remediate those things. You know, when I worked at my first job, um, big phone company, I won't say their name. Um, I was a QA analyst there as well, and that was the most stressful environment I'd ever worked in because the de the developers would literally give

me a day to QA things. And I always felt so scared to be like this doesn't work. Um like like and that's like the QA's job, right? Like to break it, to stress test it. And I didn't I got a chance to do like the happy path, you know, just if the user did exactly what we intended for them to do that works. But of course, a

user is going to find a way to do some weird stuff. And if we don't test for those weird things, it's going to it's going to come out eventually. So like that was something that was totally eliminated in our process. Um and we were able to create real continuous delivery. Um an environment where you know work was being deployed incrementally and improved upon more quickly compared to

how we worked before. And it totally changed our ideology thus positively impacting our sprint velocity. We were able to get way more done in our twoe sprints. So here we are like Usain Bolt off to the races and the shift to involving our dev partners like I mentioned in the backlog refinement performing QA earlier. It allowed us to deliver code faster with less bug. We it was

less going back and having to remediate things. It was just total forward momentum. All right. Um just quickly here a couple of the QA strategies that we deployed in in improving our QA uh process was selenium driven automated regression testing. Um this was at that time but now we're doing more uh B hat and playright um testing to to test those things which is awesome. It also

offers visual regressions but tugboat visual regressions are tried and true and what we rely on uh especially for um end users who don't necessarily have access to the code base or access to like our repo to be able to see the visual changes in Bhat or playright. So this is this helps if I need to send it to the our UX developer or researcher she we can

give her a tugboat link and she can look at the visual regressions to make sure that things look as expected. um manual of course QA of each PR. So uh we have that time to do that, right? If a a developer is done with the work and we have a PR for us to manually QA, we have that time rather than having to do all of the

manual QA at the end. So that was great. And establishing acceptance criteria. I cannot stress this enough. Like if you don't know when a ticket is done, you're kind of just wandering aimlessly. And and it really helps the developers from what I've been told. If they know exactly what constitutes an issue as complete, it helps them in their prioritization and getting their work done. So establishing that

acceptance criteria is really important and you know we wanted to make the migration an like I said an opportunity for us to shift left and in doing so it improved our processes and test coverage. It improved our collaboration across teams. Things became much easier and the path to migration became much clearer. It was just like the aha moment as you can see here between these two gentlemen

and we are now you know way more confident in our ability to migrate our platform from Drupal 7 to Drupal 8. And just a quick little recap uh what was the challenge you know poorly executed software development life cycle I caused our product growth to be very stagnant. We couldn't innovate. We couldn't uh implement new features fast enough and it just bec our product started to become

a little stale and the lack of processes and automation and stakeholder involvement made it nearly impossible to scale. Right? So the solution was for us to uh adopt a shift left meth methodology and moved all of our processes sooner in the software development life cycle and incorporating tugboat into that uh into our DevOps tool set was the catalyst we needed. All right, and with that I will

hand it over to James to get down to the nitty-gritty of things. Yeah, as uh as mentioned, we're going to get nerdy here for a bit, but yes, we will we will try to keep it fun. There's a lot we could go into, but I'm just going to try to hit some high points and some of the things that we we focused on for for Georgia. So,

first, let's set some context about the Georgia stack. So, as we've already talked about, Georgia, the GovHub uh platform is Drupal multi-sight. And we're going to get into this a bit more, but uh it's really important to establish build tooling. And the build tooling that we chose for the GovHub platform is Drush uh along with some of the fancier features in Robo. Robo's built into Drush. The

poll requests are over on Bitbucket. We've got some private docker images. Those are uh stored on Red Hat key and we'll get into what those are for here in a bit as well. For local environments, uh developers are using DDEV. CI/CD is performed on CircleCI. And as Jasmine has mentioned, Tugboat is where a lot of the QA and UAT is done. And finally, the production sites are

hosted over on Aqua. So this is another way to kind of visualize uh all of those different components and the context for what we're going to get into. So before you shift left, what uh there are some important foundational technical steps that we did for Georgia that really eased that shifting left And what I'm going to talk about first is data handling. One of the important foundational

things in shifting left for the state of Georgia was that data handling. We're going to look at this from three different priorities. First and foremost, security. Of course, we needed the process to be uh to strive for the utmost security no matter what. We also wanted to avoid those it worked on my local moments that can occur when different environments have different data in them. So, we're

going to talk about how we did that. And we wanted the process to be as streamlined and as efficient as possible for all of the developers and the entire So, first up, let's talk about that security bit. So, on the security side of things, one thing we see a lot is that teams will just sanitize data on their local environment. So they'll be pulling down um you

know unsanitized data to a local environment and then running sanitization scripts hopefully. But if the point of sanitization is to protect sensitive data from ever leaving a secure boundary that process isn't sanitary. So the way we architected this for Georgia we wanted to make sure all that sanitization happened inside of the secure boundary. In this case that's aqua. So we've got some CI/CD process that runs on

the Aqua side of things, sanitizes a database dump, and then exports that to a place that developers can then have access to. Another thing we often see is that teams are not developing or testing with real data. And the thing about real data is that it's messy. Uh, and when you develop and test with messy data, you find things that you otherwise wouldn't find. You're not just

testing that happy path like Jasmine was talking about. So, for example, if you're writing a migration like we were working on migrations, right? I if I'm a developer and I'm working on a migration, I could test everything out. I could be like, "Yeah, okay. Users are coming over, nodes are coming over, taxonomy terms, and I migrated 10 PDFs and it works great." And then you deploy that

to production and you realize it's going to take approximately three years to complete the file migration with the way you wrote it, right? Because there's, you know, half a million PDFs in production. So you got to think about things differently when you're working with real data and you're going to end up making different choices that way. So the way we accomplished this for the Georgia folks is

we created those uh once we had those sanitized database dumps, we created Docker images with that sanitized data available already in them. And if you're wanting to go this route, uh just a a quick little pro tip, you can use the Tugboat Docker images. They're already uh pre-built to be able to like save the the database images within the Docker image itself without you having to kind

of muck around in the Docker image itself. So the other important part of this is making sure once you have that sanitized data, make sure all the environments that are using Docker are using all they're all using the same Docker images. Um they have that data available to them. So uh as far as the data side of things, laying this foundation, this ensures security by keeping that

unsanitized data inside of a secure boundary. And that also means you don't need to provide access to prod infrastructure to different engineers. You know, sometimes it can take a while to get someone onboarded because they've got to go through this process of getting access to uh some of this data. And if you have this set up, you don't need to have those uh that red tape that

they need to go through. and getting that consistent data across all those different environments, you're going to catch things that you wouldn't have otherwise caught uh the the things that sneak through because you you weren't work you weren't working with that messy data and it's going to you know I have I have said this so many times in my career so uh that it worked on my

local um you're going to catch those things when you're using consistent data across these different and once you have all this set up it makes the whole team that much more efficient. So new folks can join quickly, just give them access to those private Docker images. They're ready to, you know, run their DDEV commands and get a local setup right away. And it just simplifies that whole

experience for everybody. The other foundational thing that we worked on for the Georgia folks was that build tooling piece. And the most important part of this is just standardizing on something. uh we spent quite a bit of time you know trying to decide which taskr runner uh to choose from from a whole long list of them. Um in the end like as I mentioned we we decided

on Drush and we decided on Robo and uh and the reason for that was that meant that all of the scripts that were written were written in PHP. They were written in the language that the team already knew. Now, I'm guilty of in the past having chosen as our task runner, let's say a make file and me being the only person on the team that could really

edit that file, that was a mistake in retrospect. Um, so choosing something that the team can understand, there are tools out there that are, you know, just a simple YAML file that's a that's a taskr runner or something like that. It doesn't have to be Drush. I'm not saying that you need to choose Drush, but just choose something that the team understands and can maintain. And one

of the things that we see that really screws teams up the most is that they have a different set of scripts that they run say on prod than they run on these other environments. So make sure you standardize your scripts across all these different environments and you're using those scripts across all the different environments. So now that we've uh worked on these foundational things, we can move

on to beginning to shift left. So we're going to talk about the engineering testing strategy and we're going to talk about it from uh three different environments. We're going to look at local We're going to look at the CI environments. U in this case it was CircleCI. It could be GitHub actions, Bitbucket Pipelines, GitLab CI, whatever it is, it doesn't matter. It's just wherever those automated tests

are running. And then we'll also look at the tugboat side of things. Now, you may think since we're shifting left that we would just start, you know, by taking one step left, right? But what I'm going to recommend is that when you approach this, you actually go all the way left. So, let's start with local. And the reason for this is that we see this happen a

lot where um you know the tests are broken on the environment and it's like why is it broken? Well, I don't know. I can't run the tests on my local. I can't figure it out. It's not working on CI/CD. And uh if you start with local in mind and this goes back to that build, you know, standardizing on build scripts when you're setting up your CI/CD uh

automated tests, make sure you're using standardized scripts as well and start on local environments. Debugging tests in CID can be a real headache and not only that, it's expensive. So develop those automated tests in such a way that any developer can run them. Then they can if something breaks they can get a step debugger on their local walk through try to figure out what happened. Right? A

great way to get started with automated tests on your local environment is with tests that run super duper fast. And what I'm thinking of immediately is linting tests. And you can actually set up get pre-commit hooks to test linting right away. Um so anytime a developer tries to create a commit it's going to do those tests right there. Um a great on the on the Georgia project

we used a great tool called GrumPHP for this. So the way this works is you just run a composer install of this project and it will automatically set those uh git pre-commit hooks up on a development environment. So there's not like any hoops that you need to jump through for that. It's um it's it's really handy. There are other tools, you know, if you're if your build

pipeline is more on the Node.js side, there are of course npm packages that do a similar thing. There may be other PHP packages that do a similar thing. Um this is the one we chose. And really just lint all the things. It's you'd be amazed at what you catch with just linting on even like a composer lock file for example. There's so many times I've done a

bunch of work and I just forgot to run composer update and update that lock file. And so I push up this all this code that I've done. It creates a pull request and I realize immediately that it failed and it's but we're just burning CI minutes, right? Because I've already created the pull request and if I had had this get pre-commit hook, I would have caught it

right away. Um, so yeah, just turn on all the things like there's no there's no hurt in in uh linting CSS, JSON files, YAML files. You'll be surprised at what you start catching when you have all these things turned on. Markdown as well. broken links, for example. So, here's an example of what GrumPHP looks like. I apologize, this is very tiny, so you probably won't be able

to see it, but I will try to describe it. So, up at the top, I'm running a get commit message, and then the lines that you see below that are the actual linting tasks that are being run. So, it's running PHP lint, it's running YAML lint, composer, JSON lint, PHP CS. These only take, you know, a few seconds to run. Um, and if any of them fail,

it's not going to let me do that commit. Um, in this case, since I'm a Mr. Rogers fan, uh, I put this little message in there for the Georgia team. So, if it everything passes, you get this awesome you are special, uh, in your terminal that shows up. And likewise, if it does catch an issue. So, here we have the YAML llinter fails. I know that's really

tiny and also the contrast isn't great, but then you get another Mr. Rogers quote that says, "If it fails, be brave, be strong." So, now we've got um you know, like I said, this this is going to catch a ton of stuff. If you're not already doing this, this is an easy win. Just go ahead and start, you know, doing this where you've got these pre-commit hooks

hooks set up. And now that we've got that, we're moving on to the next step, which is our CI/CD. So there's not much to shift left here on the CI/CD thing, but if you if you if you're new to automated testing, again, start with linting. It's pretty easy to just run all those same lint tests in a CI/CD server. And then next, if you are new to

automated testing, some some folks kind of get overwhelmed with like, oh my gosh, I've gota we've got to do automated testing, which means I need to figure out how to test everything. And it's like, no, just when next time a bug happens in production, before you fix the bug, write a test to verify that bug. It's as simple as that. And AI is super helpful in this.

Um, actually if you're able to use AI, AI is really good at writing tests and you can describe the bug and have it spit out a test for you and then run the test, see that it fails, then fix the bug and verify that the test passes. You now have automated There's also if uh on the end test side of things if you're not familiar there is

a project called automated testing kit which gives you a starting point for automated endtoend test with playwright or cypress so that's a good place to and uh Andre Angelani is here at Drupalcon he's the maintainer of that so if you happen to run into him be sure to ask him about that moving on we are Now going to talk about tugboat and specifically you know since this

is the nerdy section we are going to look at the configuration side of things. Um tugboat config is very similar to docker compose if y'all are familiar with that. So there's a yaml file that lives inside of the repository that describes you know what the infrastructure looks like for a particular preview. There are three differences that we're going to talk about from say a docker compose that

tugboat has. So Tugboat has the concept of different build stages. We'll look at that. Tugboat also supports multi-sight which is important for the the GovHub platform. So we'll look at what how that's configured and then we'll also look at how the visual regression testing uh configuration So yeah, we're going to look at some, you know, some configuration here. Hopefully your eyes won't cross. We're not going to

get too uh into the details about it, but you know, just kind of fly over things. So, here's an example uh Tugbo config. This is actually a a little screenshot of a portion of the GovHub config file. And um let me see if I guess the the cursor is not super easy to see, but these are the different build phases. And I'm not going to go through

um all the lines and stuff in here, but we've got init right here. We've got the update, and we've got the build phase. Um the init phase is like for setting up what all the OS level dependencies basically for um for getting this service ready. The update phase is where all the data kind of gets pulled in. So databases um static assets whatever. And that in this

case there's not a lot because we've got those Docker images that are already pre-populated as we talked about. And then the real the real um important section here is the build phase. This is those commands that run anytime code would say get deployed to production. And again, as we've talked about the importance of standardizing, you're going to be running the same steps here that you would run

say in a production site. And we've got lots of documentation as well on our docs.tugboatqa.com site as well with different config boilerplate that you can look at. Next up is the multi-sight config. So, uh, the way that you set up multicight in Tugboat is with aliases. Think of an alias as a different URL that you get. Each alias that you add is a new URL. So, what

Tugboat does is once you add these to a particular service, then you get another URL. And again, this is kind of I can't even read this from here and I'm standing right next to it. Um, but we've got the Department of Revenue is the do. It's the second one in that aliases list. And then over on the right, we've get a URL that has the do prefix

at the front of it. Um, so this is this allows you can then take the uh if you're familiar with Drupal Multicight, there's a sites.php file and then you can map these little prefix URLs to a different Drupal multicight. So, you got one pull request, but you can test however many multi-sightes you want to um and and actually be able to test out that pull request against

different sites. And we've got the visual regression testing that we've talked about. Um this is pretty straightforward as well. Uh you just list out which different URLs you want to test and you can list out different URLs for different multi-sight um aliases. So, in this case, it's uh we're testing the test site. And this is actually a good tip for you. So, the GovHub folks, what they

have, they have a site called test that um is it has all these different landing pages that have all of their components on them. And that way you can run visual regression testing with more static data uh and be able to you know so that you're not running having a a false positive with a failed visual regression testing because the the underlying data has changed for this

component. You got static data and you're also able to run visual regression tests on components that maybe aren't used as much. you know, maybe those there's a component that we have that just is rarely used, but we still need to make sure we didn't have a regression with that. Once that's set up, uh this is the tugboat interface here. We've get um again very tiny, but um

the blue text there, we've got 120 tests that run. There's 116 passed. And what we're looking at here, the active tab of that blue text, there's four failed tests. So the typical workflow is once these uh once the pull request is created these visual regression tests are automatically run and a QA engineer or project manager or maybe it's another developer says hey looks like the visual regression

tests have failed. Let me go in and check this and see what's going on. So they click on one of these and here's what they would see. So down at the bottom we've got a before screenshot. So this is a screenshot taken of this site uh before the code has been merged in before uh any of those build steps have run. Uh and then over on the

far right we've got the after screenshot. So all the the code has been merged in. It's been uh all those build steps have run and then another screenshot is taken. And in the middle we've got a pixel by pixel comparison of what has changed between the two. And in this case we see that there's you know 92% similarity which is a failure based on the threshold set

which is supposed to be 95%. So then they might click on the after screenshot over there on the right and see hm the Google map Google map integration is failing to load. So this could be indicative of a legitimate bug right a regression in this pull request where the map integration legitimately got broken somehow. Or this could be a false positive where the Google map integration is

just broken. Maybe like an API key needs to be rotated or maybe Google Maps had a networking issue at that time the screenshot was taken. So at this point um you know maybe if it was the latter case where the there was just a networking issue you could click on the regenerate button to the far right. But if not, maybe it's like, "Oh, this was actually we

knew we knew this about this issue or we this was actually intentional." So, we're going to go ahead and click approve. At which point, we then get a happy green check mark. We love happy green check marks. The happy green check mark would also show up over on the Bitbucket side of things. Okay, so that was that was a quick flyover. There's a lot more we could

go into. So, if you all have any questions, you know, of like specific things that you're interested in, feel free to, you know, after this, we'll um I'd love to I love chatting about all this stuff. So, um but let's do a quick recap of what we did talk about. So, lay that foundation first when you're when you're getting ready to do this. So, that means, you

know, creating those sanitized database dumps, doing this in a secure way. So, make sure you've you're creating sanitized database dumps in a secure And then you're going to catch so many more issues if you're using the same data across all of your different environments throughout your software delivery life cycle. Standardize your scripts. Make sure you're running the same scripts across um local, you know, stage, tugboat, CI/CD.

You're going to you're going to catch so many issues just from doing And then we talked about you know once you switch into uh actually shifting left into the on the local environment set up some sort of getit pre-commit hooks run those fast tests. If you have fast unit tests add that in there too. It may take a minute but like you're going to save so much

time. So on the CI side of things uh run those linting tests there as well. unit tests, end toend tests, integration tests. If you haven't checked out automated testing kit, if you don't have automated tests already, that's a great place to start as well. And then on the tugboat side of things, you're going to be amazed once, like as Jasmine talked about, once you shift QA and

U toat left that bottleneck, that you know, Atlanta traffic jam, we're going to be able to distribute the load of QA and UAT across the entire sprint. you're going to catch things earlier and when you do catch things, it's not going to block all of that other work. >> Awesome. Yeah, James definitely needed to be the one to explain all the technical parts of it. Um, so

real quick, just a few uh key takeaways that we want to leave you with today. Uh, the first two for me. Um, involve your stakeholders early and often. uh a lot of what we do as far as our roadmap is stakeholder driven. We have advisory committees that we uh are that allow us to engage with our we call them power users but we want um diversity amongst

that advisory committee. So some power users, some new people, you know, we want those the the fresh perspective of someone who just joined GovHub as a content editor and what they see and as an opportunity for improvement. So involve them early and often. And speed doesn't have to compromise quality. The thing that Tugboat allowed us to do was to quickly add new features and innovate, but we

didn't have to sacrifice the quality of our code or the feature itself and how it was actually enriching our content editors. So just because you go a little faster doesn't mean you need to compromise on how quality of the code that you're producing. >> Yeah. And as as we talked about on the technical side, you know, I just want to emphasize again the importance of laying that

foundation and testing everything you can before merge. But also I one thing I don't have here is just start where you are and take one step. Like sometimes if you're not uh if you're looking at this and you're feeling overwhelmed, just take one step in this direction. Um even if it's like maybe it's that pre-commit hook or something, you know, just start taking chipping away at this.

So all right. All right. So are there any questions? This is our QR codes if you want to connect with us on LinkedIn. This is our uh my agency site, digital.jorg.gov. The Department of Digital Services and Solutions is our website. You can learn more about GovHub and the other uh services that we offer. Um and of course tugboatqa.com. Any questions? Yeah, >> I have two questions. So

far the doctor that has database and files that are sanitized, how do you manage and update that? And then similarly for the visual regression testing in total how are the baseline images where they stored >> yeah okay so the first question was the uh the docker images that have sanitized data in them how are those created is that the question like or or >> I'm sure you

have the plugin but say you want to do a new data set >> yeah okay so there is uh for the GovHub team, they have a CircleCI job that runs once a week that kicks off the the job on the Aquia side to sanitize those database dumps and then it imports those database dumps into a Docker images and commits the Docker image and then pushes that to

Red Hat key. So once a week um the team can you know run their DDEV commands to pull down fresh images. Once a week, the tugboat is getting fresh data. You know, all these different environments that are relying on Docker are getting fresh data once a week. >> All 95 databases are being each week. >> There's not all 95. I don't know how many. There's like It's

kind of similar to like how Jasmine was talking about which stakeholders to get involved. They have their sites that are like the important sites they need to make sure. So, it's um >> Yeah. So for a tugboat in particular, we for a tugboat in particular, we test I think eight or nine sites out of our uh our our infrastructure uh because most of them function the same.

There's a handful that have stuff like the the maps API that we really need to make sure is working. That's Department of Driver Services. They mostly use maps and locations content type on our sites because they have a lot of different locations where you can go take your driver's test. So we want to test that and tugboat to make sure that map is working because people rely

on that. And then you know department of revenue, the governor site, those big sites that we just want to make sure their features are working. So we just have like a sample of the 90 sites that uh represent the majority um that we make sure we test and tugbo. >> Yeah. And one thing uh during the migration process one thing we did is we had two different

database images. We had the source database image and the destination database images. um and the source database images. We would take you know say six to eight sites at a time based on the whether the importance of them or the you know how much data they had and we would u bundle up docker images uh with source data that the team could then say work on that

next phase of migration of these next six to eight sites. So um the other question you had was around visual regression testing. It was about the before screenshot. Where does that come from? Is that right? >> The baseline images like where it's stored is tugboat. >> Yeah. So the the baseline images are stored in tugboat. Um and they are created from in tugboat what we call a

base preview. The base preview acts as the starting point for any preview and it should mirror as much as possible that production site. So screenshots are taken of that base preview. Then anytime a new poll request preview is created, all those same screenshots, you know, same paths are taken screenshots of and then they're compared uh at that point. >> So it's all cloud. >> Yes, it's all

in tugboat cloud was what he said and I said yes. >> Further recording >> talked about databases. What about like the the images and all the supporting assets and the code base and all that that's also stored in the image >> the uh the what was the question again >> database files codebase right so are the images just including the database or what's comprised >> so the

the question is the are the the docker images that we have are they just the database or are they all the files and everything I don't actually know the answer into that anymore. Um, I will say though for the migration we did have static files in the images themselves. Um, but a lot of folks I will say will use something like stage file proxy so that they're

not having to pull all of the assets down into those images and they'll just only on demand will need those will get pulled down. >> Yeah, over here. >> I was c curious about the visual Um, and can you mark sections of the page to be like video? >> Right. So the question was uh is the visual regression testing uh does it work with responsive different responsive

layouts and then also can you mark different portions of the page as like dynamic or you know and the the first part we do have we take screenshots at three different break points. So we take a desktop, a tablet and a mobile screenshot for each different path that you um that you specify. uh those are hardcoded for us. Uh there's not like a oh we want this

breakpoint. So um and then also right now we do not have a way of like marking out a dynamic page. So that's one of the reasons why it's really helpful to have that um those component landing pages. So you can have like all of your components with that static asset static data in there. Um, there are folks that when they they do need to get a little

more fancy, like let's say, um, I want to click on a menu and then hover over this link and I want to take a screenshot at that point. There are other tools that you can integrate with Tugboat like for example, Diffy is a good one that folks have used that can do some of those fancier interactions. Um, so ours is pretty pretty basic in terms of like

we take a screenshot, you know. So yeah. >> Okay. Love this. This rules >> this is wonderful. Uh we're where we're coming from. We're not multi-sight. We use Pantheon. We've set up a very close set of stacks to you guys only using Pantheon and Percy and uh Cypress. >> So the thing that it had a lot of great benefits, but and I want to know what your

process was for this. So all the feature branches get pulled in. Everything's merged. You cut a release. How are the QA people doing the verification on releases while all of this is happening? >> Um yes. So first I want to say that he said we were awesome. Put that on the on the record show. Um, but the question was once all of the feature branches are merged,

what are the how's the QA process taking place to test the all of the merged code? >> For like the release candidate for for deploying to production unless you guys do true continuous deployment. >> Yeah. So we for each PR so of course we plan the sprint and we decide which issues are going to go into the sprint and as the developers complete the work they'll merge

their PR to tugboat and so that at that point the QA takes that tugboat preview and only tests what is uh identified in that issue for the test cases. Once they say this is great on tugboat, good. We actually merge to like a master tugboat and once um all everything is tested on tugboat and merge to that master preview in tugboat, they then do automated regression testing

on that tugboat because at that point you've already tested the code for each of the PR. So now we're just doing regression to make sure that all of this stuff didn't break, you know, what's already in production. So at that point, we really don't do a whole lot once it goes to staging. We feel very confident at that point that everything that we've merged is ready to

go. So we'll do another round of regression testing on staging, maybe do a little bit of smoke testing on staging with each of the PRs. Um, and then we push to production. And I will confidently say that we rarely push bugs into production at this point. And if we do, it's really easy for us to identify which PR broke it. Yep. >> Thank you. >> Yeah, absolutely.

>> Um with this shift left, do your stakeholders preview earlier or what is their involvement to say if they ask for a feature change? >> So we actually the question was do the stakeholders preview earlier with the shift le methol methodology. So, we actually do not like to tell our stakeholders that we have Tugboat because because if we do, they're going to want us to send them

a Tugboat link every time we do something and or or they're gonna want to use Tugboat to play around with their site. So, we actually give them they have access to their staging environment. So, if they want to stage content and see how things look, they have access to that and can do that all day. Um so the way that we engage with our stakeholders when it

comes to feature development and building the road map is like I mentioned through the advisory committee we have these discussions about like what works for you what doesn't work for you or you know our hypothesis is this can you help us prove that otherwise you know we have like deep discussions with them um maybe like it typically once a quarter but if we're trying to really like

you know build out some new features we'll meet with them more often but that's like the main engagement we have with our our state agency partners. Um this the stakeholders that we work with regularly is our partners with Lullabi. We have a um a sprint schedule where we have standups twice a week and we talk with the developers. We do backlog refinement with their development team. Make

sure they understand what we're trying to do. Let them ask questions. So we do that every week. Every week we have the same schedule. We meet and do backlog refinement, sprint planning with them. So they're very integrated into our product our our life cycle, our software >> Does that does that answer your question? >> Well, it raises more questions for me. I want a hold of you.

>> Okay. Okay. Yeah. >> Uh so kind of circling back a little bit to the the database images or the images you're using. Um I mean it's you're talking about 8 n 10 sites. Sometimes databases are hundreds of megabytes. just have small sites or does it take you a really long time to generate these? >> Not a week. >> Yeah. So, the question was, you know, if

we've got eight to 10 sites that we're pulling in database dumps for, sometimes these databases could be very very large. Um, and specifically what was the >> so saying is there are these just very small sites overall or >> are these very smallites take a really long time to be download >> or does it take a really long time? So uh I it's been a minute since

I've looked at how big the images are. Um Darren's in the back. He might know Darren works at Lullab. Um, but >> they're not huge Drupal sites, but Yeah. So, I don't think I mean when we were doing I will say when we were doing the migration, it would take a while. Like if I was pulling down on my local and working on a migration, it might

take you know 20 minutes to to to do a Docker pull of some of those because you're getting both source and destination data and you need a lot of that data there. Um, but otherwise it didn't end up being too much of a problem honestly. But I would say if if your site does have a large database, you know, maybe that's not the right approach and there

are other ways to go about that. So there's, you know, some folks will will pair down that database. >> Oh, built-in coffee break. >> There you go. Yep, that's right. >> Docker pull is the new compiling. you know, you pull out your sword and um >> so I know we're we're running out of time, but we really loved your feedback. So, if you scan this code, you

can you can give us feedback on the presentation. Um >> and and hit us up on LinkedIn if you have any more questions about what we've talked about today. >> Yep. Swing by the Tugboat booth, too. >> Thanks, everybody.