Streamlining Content Relevance, Translation, and Migration with a Content Audit tool

This talk focuses on streamlining content relevance and migration with the introduction of a content audit tool in Drupal. The speaker, Marcel, who has over 20 years of experience in web development, emphasizes the importance of proper content management from the project's inception. He discusses the challenges involved in migrating outdated content from existing websites, highlighting the significance of conducting a comprehensive content audit early in the project lifecycle. Marcel outlines the features of the custom content audit tool, which includes modules for analyzing redundant or outdated content, checking translations for accuracy, and facilitating a smooth migration process. He also elaborates on how the tool addresses common hurdles like client engagement and content review tracking, ensuring that all content is relevant and ready for migration by the time the new website is launched.

Hey everyone, thank you all for coming to my session streamline content relevance translation and migration with a content audit tool. Uh my name is Marcel. I work at Nord as a Drupal solutions lead. I have more than 20 years of experience with web development both building projects or teaching new developers. And as you all I share the same dream. We get a new project. We plan everything.

We build a safe website. It has a good design. It's fast. It perfect the way we wanted to build everything. And then content. We need to bring content inside our website. If this is something that the client is starting from scratch, that's fine. The content will be built around our website. But most of the time that's not the case. Most of the time they already have another

website or perhaps multiple websites they are that they are consolidating inside a new build. And now we need to bring into our new project a content that perhaps is outdated and perhaps I had a lot of duplicated pages. Translations are not accurate. Now everything is a nightmare. the website no longer looks good and we need to make sure that we fix that content in time. Now we

all know that the content is the purpose for any project. That's the reason people go to our website. It doesn't matter how nice it looks. It doesn't matter how fast it is. If the content is not good, people will not come back to our website. So we need to make sure that the content is good enough for our project. And the issue is that we don't pay

enough attention to the content until it's too late. Usually we get a new project, we go through a discovery phase, we get excited about what we are building, we focus on the build and after everything is done, this is when we start worrying about the content and how we are going to migrate everything. But by then it might be a little too late because we need the

client to review their content. Remember they are the content the the subject specialist for their company. They are the ones that should be telling us what is still relevant, what is still up to date. So we need the clients to go through all their content, make sure everything looks good before we copy that inside our project. But that could be time consuming and if we leave to

do that near to the launch time then we are going to have some delays which is never nice. So this is why we try to bring this content audit to the beginning of a project to make sure that this review is going to happen at the same time that we are building the project just so whenever we finish the building the content is also ready to be

included inside this website. But how we manage this review? That's a question that we've we've been trying to solve for a long time. In the past, we use some spreadsheets. They give us a lot of flexibility. We just copy a site map all the page that a website would have and then we create everything that we need inside that spreadsheet and then the client can go through

review But that's not easy because the client also has some freedom to make changes to that spreadsheet and then things could just start getting confusing. A lot of different copies, a lot of different people doing whatever they want inside those files. That's never easy to manage to merge everything in the end. This is why we came up with a content audit tool where we can centralize everything

inside Drupal. make sure that we have a source of truth. All the uh our content review would go through this and we would have a place where we can even track if the client is doing what they are supposed to. It's always frustrating. We ask the client to review their content and we trust that they are doing their their part. But once we get to the end,

once we are ready to migrate the content, perhaps they have not finished everything yet. So this tool help us to keep track of everything that is being reviewed. If the client is on time with everything just so once we are ready for migration the content is also ready. Uh this tool provides us with three different subm modules. One is for what analysis. I will talk more about

this soon, but the idea is just so we can make sure that the content is still relevant, that we can make a clean map on anything that is no longer needed for a project, that everything is up to date. Same goes with translation. If we have a multilingual website, we need to make sure that our pages are translated and that the translation is also accurate. And finally,

if we are migrating everything into a new project, then we need to go through a migration review and just decide what we are bringing. We don't need always to bring everything inside a new build. We can just bring whatever is relevant to the new project and just make it clearer. So the main idea is that we have different subm modules for our tool just so we can

choose the ones we need for a given project. It's not all projects that we need all of them. We just choose the right one for the right project. But one thing that all these tools they have in common, they all need a source. Usually we are migrating from elsewhere could be a WordPress website or perhaps an older version of Drupal. We need to get that source and

make sure that we make a read of that and we can just review all the content that we have there. This is why during the discovery phase, we start talking to the client about migration, we start talking about this content review. Once we get to the end of the content, the the uh discovery phase, we have a plan for the build and we have a plan for

migration. At this point, we can start both tasks at the same time. We start the building and the client starts reviewing their content. For that we need to set up our content to uh just bring one or more sources for our project. Most of the time it's going to be only one but sometimes we are consolidating multiple websites so we need to include multiple source just set

up everything that we need and make sure that the client starts their review. So for the source whenever we are including we just need to choose where this information is coming from. It could be a load version of Drupal. could be a different version of WordPress. Sometimes it could even be something or it could even be the project that we're working on right now. I will talk

more about this project. So but once we select what is the source we had some settings where we can just uh tell our tool what's the URL where we are going to get the content what are the languages that is available for that source what is the default language just so the tool can prepare everything and then we can just synchronize everything that we need like we

are not going to just copy the entire content inside our tool we are just creating a list like a site map just so we know everything that is available on the uh current source and we can review everything and make sure that everything is ready for migration. Now this synchronization what that means whenever we set up a new project we need to set up what is the

source and for that we need to ask the client for their database. So they will give us their database could be one, it could be multiple. We just set everything on our local environment. Uh if you have worked with migration before, you know that setting up multiple databases that is pretty much easy inside our local environment. But we also need to give the access to the client

to that. So they need to have access to a hosting environment. We need to make sure that we send this to an environment where the client can just use our tool and the hosting environment will not have access to multiple database. So what we do is we have a dash command that we just create that list. We set up everything inside our Drupal database and from there

we can use uh config export and config import to make sure that everything is available inside our tool and that will give access to the client just so they can start the review. They will have a list of all pages they need to go through and we can just keep track of of everything. So in the end this is what we are going to have after we

set everything up. We are going to have a list of different sources that the client needs to review. And if I open any source, I can have if they're doing rot analysis, what is the progress for that? Or if there is a translation review, what is the progress for that as well. This is really nice because during development we are going to have some meetings with the

client usually weekly meetings where we can talk about the build where we can show them what we have been doing. We can ask questions. This way we can keep track of everything that we are doing. The client can hold us accountable for everything. But at the same time we can use it this and bring this information to our readings to make sure that the client is also

doing their part. So every meeting we can see what the client has already done reviewing their content. If they had questions they can bring their questions but this will help us to keep track that they are doing everything that they are supposed to. They will review the content. We will build the website and once the build is done the review should also be done and we should

be ready to run migration and all the content should be ready to fit the build that we just created. Now once we have set up the source it's just a matter of selecting which tools we are going to use for our project. As I said, it's not endless that we need the three of them. But if we do, we would go with rot analysis, then translation, and

finally the migration review. Now, going through each one of those, first we start with a rot analysis. And if we're not familiar with the concept, it's just a process to identify everything that is redundant, outdated, or trivial inside a website. Basically, the client needs to go through all their pages and just check if everything is still accurate to whatever they are doing. If perhaps something changed inside

their company, they need to make sure that this information was updated on their website. So, they need to make sure that the content has a good quality before we bring that into the new build. Now usually how we do this we have a series of questions that we are going to set up for each project. Depending on the project the questions will be different questions like what

is the importance of this page or what is the primary audience and what is the quality for the text that we have right now. All of these questions will help the client to identify issues that they might have with that page and therefore fix that later. So all these will bring them to one decision. What is the action that we need for that page? Do I keep

the way it is? Do I update the content or do I simply delete that page? So that tool also provides us ways to set up some owners so people know who is responsible for updating this page. Also to add some notes about what needs to be updated. What is important to keep in mind is that during the rot phase we are not worried about updating the content

itself just identifying the issue because right now something could see that I just need to update but later I identify another page that has a better quality that says the same thing then I just can come back here and say no I can delete this page let's not waste time on that. So during the watch phase I will check all my pages and identify what I need

to do with them. Now, as I said in the past, we use spreadsheets for this. We get site map. We create a list of everything that needs to be reviewed. We go inside the spreadsheet, create different columns for each question, everything that the client needs to answer. This will give us a nice picture about everything that needs to be done. However, although the spreadsheet does give us

a lot of freedom to set up this rot analysis the way we want, it also gives the client some freedom to change in any way that they want. Usually, they are going to have multiple people reviewing this. So they're going to create different copies and each one will start making changes to that copy and then things could just overlap each other and some pages might not even

be analyzed because one is just waiting that another one is going to take care of that. So once we get all those spreadsheets from the client once we try to merge everything it becames a case. So for that we create the rotten subm module which help us to keep track what is being analyzed inside our drupal build. This will help us to identify everything that was already

checked a red analyzer avoiding people to do the same analysis twice and just following everything that we need. Now that's what we get with this tool. It will be just a list uh basically a Drupal view with all the page that needs to be analyzed and a column with the action that they chose for that page. Initially all pages will have no action set up but over

time they should have one of those three actions keep update or delete. So they can go through each one of those pages and just run that analysis. Now this is a mail work. uh the tool as I said it doesn't have the content itself it's it's just the information about the page what is the link to where they can get the information where they can get to

the page and then they can open their website check everything come back and just fill up the form with all the questions that we ask here and for this we have created a content a field entity where we can just create different fields therefore we can customize that for each client. We can include different questions. We can change the options for each question which will give a

custom build for that client. Now, we also included here some bulk operations. Since this is a Drupal view, we can include some different filters which will allow the client to easily get a group of pages and just check things uh in groups of whatever they want to do. So, I could filter content based on a content types. that say I want to get all the news that

are 5 years old and I want to say all of that I do not need to migrate. So they don't need to go through a bunch of pages that is no longer relevant for them. That helps to speed up the process. And the goal is that in the end we are going to have all the content either set up to keep update or remove. Once we are

at this stage we know that the ro analysis is done. Now what is important to keep in mind is that we did create this initially for migration to make sure that everything that we copy is uh accurate is up to date. But what analysis shouldn't be something that we just do whenever we have a new build. That should be something that the client does uh once in

a while. Every other year would be a good time for them to go through their content and make sure that everything is still relevant. This is why we had this option to set up this project as the source and we could perhaps just install the rot analysis module and the client could review their project uh their current website without migrating later. It just make sure that everything

is accurate. We also include some information in the dashboard to help the client to keep in mind what they eat to check. Uh, we usually like to include a chart like this that tells how long your content was last updated. If we had a lot of pages that was last reviewed more than two years ago, perhaps this is the time for them to go through a water

analysis and make sure that everything is still relevant. Everything is still accurate. Now, once I have gone through the rot analysis, once I know that my default content is accurate, now I need to also check translation. if my project is multilingual which is another issue on its up because whenever we create a new page we translate that page everything says both pages says the same thing everything

is accurate but then over time uh someone could make changes to my original content and they might not update the translation and now I have different languages saying different things about the same information. So we need to make sure that everything is accurate. Now there are different tools that we can use for our website inside the Drupal but remember this is an external source. We need to

make sure that we can check that external source. So this translation review subm module will use that same source and try to identify what should be checked what the client should go through and see if the translation is still accurate and then come up with a decision. Do I keep the translation the way it is or do I update So first thing it will just display a

basic report for everything that is not translated on the website. So the client knows they need to check those pages and make sure that they translate those. But then there are the pages that were already translated at some point. But we still up to date. And the way to do that basically is just the client needs to open both versions, the uh English version and the translation

version and just make sure that the content is still right. But it could be time consuming and the client could feel that it's a waste of time because they will review 500 pages before they can identify one issue and sometimes it doesn't feel worth the time. So we try to create something that helps the client to identify what could be an issue. So we should start with

these pages and pay attention to that. Uh we came up with some different ways to check that information. Start with how many days apart we had from uh the last time the original page was changed and the translation was changed because if uh my original page was changed, my translation should have changed as well. And if there is a long time between then perhaps I should check

and make sure that it is right. Uh al is a matter of other things that we can help the client to identify if the text is still correct or perhaps the elements on the page. So for that what we have is an option for them to edit it each page where they can get links to the original language and translation. They can open both and check everything.

And then the tool just help the client to identify which ones they should start with by just making two different checks. One uh basically just checking the elements that we have on the page. If my English version has three images, two links, I'm expecting my translation to have the same thing. Three images, two links, and if they don't match, perhaps I included a link in my English

version, but they didn't include that in the translated version. So they should take a look and see if that is still correct and if it is they just flag that we should keep the way it is otherwise they just update that. But we wanted to do something more we want to make uh to find a way to help to identify have the client to identify the text

is still good enough and this is where we started using AR to make sure that we can check the text and point the client to the right direction. So what we do is we set up everything to get both the original text and the translated page, send that to AI and we just ask to get a score for that translation how accurate it is and give us

some basic notes. So the client will have a score for each page and every page that they get with a low score they can check the notes from the AI and then they can go check those pages and make sure that everything looks good. Again in the end we are just pointing the client to the right direction. You have different pages. Uh you need to make sure

that all your pages are correctly translated but you should start with these pages. Now going through this we would have a what analysis saying that our original content looks good and that the translations are also accurate or we just need a few updates and if we're working with migration this would be the time when we need to review what we are bringing to our website for as

developers that would be a simple answer each page do I migrate this page yes or no we just needed to load that if we don't need it to migrate, we don't worry about that. Otherwise, we are going to write our scripts to make sure that we bring it. Now, it's not so easy for the client to decide that. This is where the rotalysis helps them because based

on all the rotis, they will know what is the quality of that page. Is this important for the website or not. So, this will help them to decide if they should migrate that or not. Again in the past we were using spreadsheets for that. They would go through the rot analysis and we would just include one column in the end for them just to say yes or

no. And that was not always the case. They would not just say yes or no. They would come back to us and say things like okay let's get all the news that are five years older or more and let's just not migrate those. Easy. We can write surfy all the script and make sure that we are not bringing this. But then we go through the spreadsheet and

we start finding some exceptions where there is this new that is 10 years old but I should bring this one for some reason. So now we need to start coding exceptions inside our system. But even worse, the client will keep changing to this spreadsheet and then we have many different copies and then one of those copies had one note that we didn't catch and now we run

migration and we missed something and the client is not happy. So we needed to find a way to avoid this and again we created a module for that where the client can review everything inside out to and we can keep track if that is happening or not. Now the decision is simple for each page. They need to go through those and just say migrate or do not

migrate and they would have the entire watch analysis here which would help them to decide about that migration. We also have included some bulk operations. Again this is a Drupal view. They can use different filters for each project. We can even include more filters and they could just select things. I want all meals. I can filter by dates, just select everything to not migrate and then the

client can go through all those exceptions and just start changing. For this one, you can migrate. So for us, that would be just a flag about migrating or not. And once the client goes through everything, once the client completes the entire review, we would have the migration review also done. We know that we are ready for running migration. So at this point the content was reviewed, the

translation is good and we are read to a migration because the client has already decided what to bring or not. This will give us a clear decision. We don't has a multiple spreadsheets where the client would say oh but there is this spreadsheet where we said something and you didn't follow that. All decisions everything's in the same place. The client can even export that if they need

it. But we always get this from that list. And even better that is connected to our migration script. So whenever we run migration the code itself would just check everything that should be migrated and bring into the new build. Which means if the client make changes the day before we r migration are a problem. I don't care. You can make changes 5 minutes we run migration because

everything would be automated. We run migration, we bring the content and then we can run some reports to make sure that everything is correct. So the report will check uh the content audit what was flagged to be migrated. It will check if the content is inside the new build and everything that was checked not to be migrated if we didn't copy anything by accident. So we would

have this report to give to the client. But more than that, this is not the end of the content of the tool. Once we run migration, the client can go back to that list and now they would have a new column which would be the destination for that content. So everything that they have flagged to be migrated should have a destination link saying this is where we

migrated at. So they can click on the link they can see the page inside the new build or if they are just browsing around checking the new website checking the content at the bottom of the page we would have a block the yellow one where they can simply check where this information came from. So if something looks off to the client they can click on the link

see the original page and just compare everything if everything looks good. So now the client has reviewed everything. We had a way to keep track of this review. Make sure everything was happening in time. Once we run migration, we have the content inside the website and we have ways to track everything that was migrated and make sure that everything was bought the way it was supposed to.

And this is important because in the end we will have a new website which has a better content quality. We had reviewed everything. We had made a clean up. We have a lot of less manual work because we don't have a lot of different spreadsheets that we needed to consolidate in the end. The decision is clear. Everything comes from the same place. So when we run migration,

we are confident that everything will work as the way the client is expected. So we have done this for a few clients for now and we had really good results. The client was happy. The migration was the way it was supposed to be. I'm not saying that we didn't had some complaints about something that was brought by accident, but then we went to the report and we

showed the client it was brought because you chose to be migrated and now the client knows that they are to fault. So it's much clear what was decided. Uh and this is a module that we are still working on. We still have a few different ideas. Uh the main one is that we do want to make this into a contra module. The hardest part is about the

source. If you have worked with migration before, you know that each source is different. Each project that we are migrating is different. So there is still a lot of customizations. We do need to make some improvements to make sure that this is more plug and play or if I need to customize something that we would have some hooks for that. Uh we also intend to use more

AI like we are doing this for the translation but perhaps we could use this for the rot analysis as well something that could give the client an initial report not that it would be the final decision but once they start they would have a blank page they would have a basic information there that AI has already reviewed. So they just need to correct if that information is

right or if they want to make changes to whatever AI told them. Uh we also can use AI for validating everything. We have been playing with playright where we could get both pages from the source and get the page for our new build and just use playright to compare everything and AI could tell us all the content is there or perhaps there is a broken link or

perhaps there is a broken image. So that is also really helpful or even create some editorial workflow because right now we do have the tool where the client can log in they can just review everything but we cannot uh actually assign anyone to work on a group of content or a few pages. So we do have some plans to to improve this tool but this is what

we have right now. This was a really good tool that we started using and it did help us a lot with some migrations. Make sure that everything was migrated the way it was supposed. Yeah, that's what I have to say. Do you have any questions? Yeah. >> Are you sure? [applause] Do you have any kind of analytics integrations for this? >> Separate manual. >> Uh if I

have any kind of analytics for that, we do have some reports that we run but that's for our tool itself. We are not using any external tool for analyzing pages or anything like that so far. >> Yep. >> Have there any issues with your module >> what? uh it just in Drupal allowing uh like unique content types not being able that is not able to interact with

the tool is that the question uh so far we didn't have because we are not bringing any content type or anything the tool itself is just a kind of a site map where we keep track what word was analyzed or not. Uh so we don't need to bring the content we don't need to worry about the content itself. But during migration yes sometimes when we are running

the migration script if we are working with components that uses paragraph types or blocks we need to make sure that sometimes we break the HTML and include that in the right place inside uh our new build. But that's outside the tool itself. We are just writing the migration script for creating those new pages. >> Yeah. >> Any previous module inspired by previous >> if that was built

on any previous model or inspired by any previous module? Not actually. We had those spreadsheets and we start with the idea of getting rid of those. So we started from there and then yeah maybe there was some inspirations here and there uh where we could improve things especially when start working with AI just getting a few ideas here and there but not a base for anything >>

yeah we can set up reminder so it's take over it's the message six months for months Not yet. That could be something interesting to be done because usually when we set everything initially it will create a lot of uh nodes inside our page a lot of different entities. So what we do after we migrate everything after we review everything once we are ready for launch we just

remove the tool which would just delete that shrinking the database and then later we would install the module again and just set everything up for that project just so they can go through the rot analysis but that could be something for the future. We just keep that installed and we start a new rot analysis so we just remember the client they should start and that could be

an automated process. they could start that themselves, but that could be a great idea for the future indeed. >> Yeah, >> this is a stupid question. So, this isn't a public module. So, if I wanted this for my website and I'm that client that I need all these tools, >> am I do I have my developer build this >> or >> Yeah, right now we just created

these. Sorry, the question is if this is already public available or if people would need to copy the same idea. We do intend to make this into a contra module. Right now it is a custom module that we are just using for our clients. As I said, we did have some good cases. So we do intend to make this into a into a country module. We are

just working out of this source idea how we can uh customize the source for each project. Once we have finished this, we would make this public available. Sure. Any other question? Are we good? Great. So, we can wrap up. Thank Thank you all for coming to my session and have a great day.