About this talk
This talk addresses the challenges of managing small data sets within small teams, emphasizing the importance of structured approaches and best practices for data handling. The speaker, Oleksii, a lead data engineer at Brown Developments, shares practical experiences from their work on computer vision tasks, particularly in managing and updating data sets efficiently. He highlights the need for proper data analysis, export, annotation, and validation processes to avoid common pitfalls. The presentation also discusses alternatives to large-scale tools that may not be suitable for smaller projects, advocating for local scripts and organized workflows to simplify the data management process. Through several project examples, Oleksii illustrates how a well-defined structure can save significant time and improve the reliability of model training, ultimately enhancing productivity.
Full transcript
Is it's Is microphone on? Yeah, great. So, I would like to talk today about um uh managing small data sets and how to do that for the small teams if data set is not so big. So, here we won't talk about the AI and agents in deep like in other talks, but rather focus on the basis which is a data. Of course, we use data actually to
train any system like AI or deep learning or any other. And actually, we have a gap because there is there are a lot of tools for big tech and big companies which allows us to manage the data, but there is more to nothing which can help work to help working with small data and for the small teams. Uh so, my name is Oleksii and I work as
a lead data engineer at Brown Developments company and our team builds um segmentation, OCR, and detection tasks. And currently, we host few millions of uh images per month in Australia and New Zealand companies primarily in freight and car rental sector. Um what we would like to cover here. Uh like to talk about how a team sets up a data set and what goes wrong when it evolves
in time. Um how can we do that? And I will be practical here, so I will show you a cycle where we update the data sets, we evaluate uh how that update affects the model. And this examples will be mostly uh tied to the computer vision, but they the approach is universal for any data set like a video, audio, or any other. But, my primary experience is
computer vision. let's take an example. Um in our team, we uh created uh our first solution, which was container number detection, and each update took like a 5-7 days. I will cover that in more in in deep. And each time we did some update, we get same errors, which were not documented because data it was very diversive and different. here is the example where we started from
just 2,000 of uh images, and it was simple. We created a YOLO data sets and trained the system, and it was working well. But, sooner or later, we got more more errors from the production, and our data set grew to 10,000, and we started to have problems. So, I believe any who works a bit with the data can see the diagram like that when you have a
lot of entry points, and when you try to create a code, and you would like to manage the data set from the code, but you something to fix the label, something to update the class. you can create something to start the model training and model evaluation, but the data begins to be very diversive, which is the nature of the data, and you got another portion from another
source, and in this case, you will uh add script, which will create a chaos again. now we can ask, what solutions do we have here? Um there are big tools like Airflow and DagsHub, which are leaders in the data management. And big companies use them heavily to handle all that stuff. But if you have a data set from 2,000 to 200,000 images, setting up Airflow with all
the databases orchestration and all of that is just an overkill. And it's has a really steep learning curve. So, probably your team will need to hire another professional who will just manage that platform. And here we say that we can do that locally and it still can be very efficient. there are also all-in-one platforms which allows to manage uh the data like Roboflow and Computer Vision. And
there you can annotate, train, and deploy everything in one place. Uh but it's actually limited because uh sooner or later you will got some specific task like I would like to filter something by that class in the data which is not more than 1 year ago. And in this case, these tools will not help you. So, you will have to export data locally, filter it locally, and
then you will have to import it again. So, in this case, we started like that, our team started like that, and we found out that we spend more time on import-export rather than on actual work. So, what we can do here? Is there any solution? Like should we start big tool or can we try to do a local setup? Um the idea is simple. We can create
a scripts each script is actually a stage. And um each stage has um its own scope. we give it a number, and when we need to um navigate, we always know that we go just from 1 to uh 9 in this case, and really nice because when 1 or 2 or 3 months passes, you know that you just can open You can still open from and go
from the first till the ninth. And the more important that we talk here about the data set, not about the script. So, I would like to cover uh some most interesting stages and how we do the data set updates. So, how it goes, and I would like to show some war stories which our team had over the last 4 years working with the data. First stage is
analysis, which is obvious but often uh overlooked because people say like, "I just take a new data, add it to data set, and that's it." It's not correct approach. Uh we have to how we uh get the data. For example, our database production server collects data into the SQL Lite database with images, and we export them just as one file, and then we try to see uh
what is going So, for example, we had a project with sea freight containers, and one client started to use tanks. And we got 2,000 images of tanks, and most of them were not correct because we didn't train them on tanks. And if we didn't have analysis stage, we will just take all 2,000. And in this case, we see like there is a single source of problem, so
we take just 200. And we can save time on annotation. then we export. So, export is extremely important thing because we need to put uh data in a correct format which is our data set format. And it can be quite different. For example, our team is using internal data set format with multiple folders. And when we take a new data from server or we lower data from
internet, we uh transform it on export stage. And here is the entry point where we can add more like transform passes. So, we can apply filtering and like take some percent of errors and we isolate all that stuff just in one this stage which is which is quite important. then comes annotation which is most of you are familiar with. It's a human labeling where you create or
verify that the labeling is correct. And here are a lot of challenges for the local teams. Because there are external tools like CVAT, Roboflow which are used for uh labeling. But many teams don't invest time in making that process automated. So, we uh load images into annotation platform, then we do a fix. Then we load back to our local data set. And then we found one error.
So, in this case we have to do that process again and again. We have to import, export. And if you do that like three, four times a day, you will end up uh actually in exporting instead of doing some real work. interesting breakthrough for our team was ability to create own annotators which great if you have um um like some specific task. And in our experience, all
tasks are specific. So, it takes just few hours with agents and AI to create annotator based in OpenCV or even web-based. And for example, in our case, we for us was extremely important to to zoom over the pre-annotated region because in CVAT and Roboflow, we have to zoom all over again. So, we created a segmentator, which can have arbitrary arbitrary points. And when you press a hot
key like a space, uh you can zoom out, which we show here. And then we press space, and we can zoom on another annotation. So, the idea here is not that you need to build something like that. The idea here is if you can invest a little bit of your time and build the annotator, which is ideal for your data. So, for example, in this case, we
saved maybe hundreds of hours because we have 200,000 of images with such pre-annotated data. So, it's it's it can be different in your case. So, we finished annotating the data. And we know that data probably is correct. At least it's correct for us as humans. And then we say, "Okay, should I add the data to the main data set?" Let's say we have 20,000 images, and our
new data is 1,000 images. it's a bad idea to add the data now because if you add data now as something goes wrong, like you will see that model is not improving, or you made some mistake on creating this data. Uh then you will have to restore the data set from the backup and do all that stuff. And it's hard to see where is your new data.
It's all mixed. our team lost like 3 days of work when uh merged their own data into the data set. And then we restored from the backup and actually we erased what we added. So, to prevent that, the key idea here is to use um a Git analogy staging folder where we create a separate folder and put their new data. And this new data is not yet
validated. So, we just add it to the main data set as a separate folder. And we can easily go back to it. We can iterate it. We can check it again and again until we are sure the data is great. So, we will cover that a bit later. uh we would like to data set formats. It makes sense to create own format. And even here when we
talk about staging folder, it's actually a different uh data set configuration where you have a main data set and the staging. And on train prepare step, you can convert your data set to any format you need to train the model because different models require different data also you need to for example, merge the staging into the main data set just for the train time. So, we treat
here the train data set just an artifact which is used on the server, but it's not a source of the truth. another thing to cover before the training as we talk about the small data sets exactly is the ability to make a correct train valve split because often teams do two things. They create train valve split randomly all the time or they create a complex um folder
structure to store train valve split in into the in the folders as files. random split is absolutely no for the small data sets if you have around 2,000 images because on every training you will depend on the how lucky was the train valve data split. For example, it can be unlucky and the valve can represent something uh which doesn't include all cases and model will be not
trained correctly and you will see that like we added new data but model is worse. What's What's wrong? And the wrong is that you have a different train valve split. Idea here is that we fix the train valve split and when we uh add a new data, split only uh new added data to the train valve and keep in file like a split JSON file names which
where some files go to train and some goes to the valve. Uh in this case, we can be sure that we added new data and we know like only that data was the source of model become better or worse. then we train the model. We transfer data set. We use a library like YOLO like um RF data or any other if you work with video, audio, or
text data. Um we do it locally on GPU server. So, services like Vast AI where we can spawn like 10 GPUs and we pay just a few dollars. And we we are not affiliated with any of the services. But, it works really great. Uh, when model is ready, when we trained, it makes sense to give it some name if you don't use a model registry and small
teams usually don't do use them. Because our team, maybe it was not so smart, but we use so bad names and 6 months, it was really not possible to understand what model did. And when you give a version number, date, and few words of description, it's way, way better. when you finished with the training, we have to ask a question. So, if the new model is actually
better. Um, does it improve over the old data? does it has any regressions? Because model can have regressions if new data collides with old data. And we should see we we can calculate numerically and we can uh, see visually if the is the issue gone. So, for example, in this case, uh, the number was not segmented well enough. But, after the training, it's segmented good. So, we
can see that for this example, model worked. If model is not working or if it has some regressions, we should go back. We use the concept of the staging folder which we discussed before. We can easily iterate the edited files. We can see what is wrong. Maybe we need more data. Or maybe the classes are wrong. Or maybe we did some mistake and exported data has zero
bounding box sizes. It can be variety of reasons, but we can catch the issue now. Don't send that model to the production hoping that it will be good and we will see how it works. We can catch the issue now and worth to mention that we always take like 10 20% to hold out test which we use to do that evaluation. Uh, so we assume that model
was better. And now there is another very important thing. let's say we have 20,000 images and this images or data set entries, they are valid. We know that they are they have no mistakes. And we are adding 1,000 of images and we don't know maybe we still do some mistake during the annotation. If we mix these images now, we will have to later we will have to
reiterate the whole data set and pay annotators much more to find a small mistake. So here is a trick that we can take a trained model and run it over the new data which was also part of the training such approach can catch interesting errors which are insane for the model, which are which can be not noticeable for the humans. So for example, here we catch small
annotation which was not filtered by the entire annotator actually by us and it helped us to prevent mistakes and especially it's useful in OCR systems where even one symbol mistake can poison the whole data set. It's it's very important. So when we when we finish uh checking errors, we come to the decision gate. we can ask we can ask ourselves our metrics improved and are we confident
in the data? Are we sure the data is okay? We can see that model was improved. We can see that um, we checked like data again and again. In this case we say yes and we merge In case of no, fix annotation, retrain, or maybe even discard the whole data set. Maybe we are we were wrong and our model cannot adopt new data. It's also okay and
it's good. It's all experiment. let's say we add data. In this case we merge. So, in simplest case we just copy files. We can use hashes as file names. um, incrementing numbers and we take last number and merge other numbers to And we do the backup. It's the must and we can use tools like heuristic for incremental backups which uh, send data to the S3 or any
other source or we can use a DVC which is control version system for the data do mostly the same in this case. >> [snorts] >> Um, say here, just by adding such simple structure we our team was able to save a lot of time. Like we reduce adding data from like from 7 days just to 2 days and what is more important the process started to be
enjoyable. Cuz now it's not a pain like we work with all that scripts and we don't know what's each script do. We just uh take the new data. We go by the by the sequence from 1 to 9 and see the result. And we can process much more projects and much more data sets than we ever could before. We already tested that on for 2 years in
production six projects and we reached 2 100,000 images as the biggest data set and I believe this approach can work even more because you fix the data set and you add new data. So, what we can say here is that the proposed solution is not only about computer vision. It's about any data set if you invest your time in the structure and into the stage sequences and
strictly defined protocols you can save a lot of time later. Like because it's easy to start with wrong structure. It's much much harder to fix that structure later when your data set grows. And that's the biggest problem. So, it takes just 15 minutes to set up some basic scripts and even the first four stages and they can already greatly improve your workflow if you like work with
uh small data sets and you are a small team. thank you for the thank you for the attention. Yeah, it was a pleasure to talk about that.
