Teaching PyTorch To Read Your Worst PDFs With Docling - Mingxuan Zhao, Peter Staar & Carol Chen

This talk introduces Dog, an open-source Python library developed by IBM that focuses on efficient document processing for large-scale applications. The speaker explains that Dog simplifies the preparation of documents for large language models (LLMs) and vision-language models (VLMs) by automatically identifying data formats and providing consistent data structures for downstream use. The library supports low-latency and batch processing modes, enabling users to convert documents like PDFs and Word files quickly and effectively. Additionally, Dog has integrated various OCR models and aims to provide standardized outputs for diverse applications, emphasizing its community-driven growth and continuous feature enhancements.

Do a quick introduction. With that, hope everyone is in the right room for the Dog presentation. If you're not, this is going to be the Dog talk specifically. My name is Ming Zhao, a developer at IBM. I'm here with Carol Chen. Um, and I know many of you probably came here specifically to hear me speak when you read the schedule. However, in a last-minute change of fate,

Peter started actually the IBM research team out of Zurich who created the Dog library. So, I graciously allowed Peter instead to do Dog. Hopefully, he'll be able to answer more questions than I would be able to, but off to you. Yeah. Thank you. Very very Thank you very much. So, very Maybe let's get a few things out of the out of the way. So, we will be

together with with our friends at Human Signal and uh we will do a swap. If you haven't signed up yet, I mean, it's free of charge. You can just do that. But, you have to sign up to find out the location. Yes. Uh and there will be I'll put There will be many other uh presentations that are done uh by the by our colleagues Ming and and

Carol over the years. I'll do that in towards the end. For those that haven't started yet with Dog or that maybe have vaguely Who knows what Dog is or has already worked with it? Okay, so we have a few people. Nice. Super. Okay, so let me start with a very quick um Yeah, nothing to stage and then you know, feel free to always interrupt me at any

time. Um I'm here for questions. Okay, so what is Dog? Dog is really at its core currently a Python where we focus on a few core principles. Uh one is that it has to be open source. It has to be super easy to use and has to be able to run locally very very efficiently. So, that's kind of where we And then, in order to make it

really useful and very very adaptable, what we effectively are trying to do is to make sure that you as a developer don't have to worry about running any models or taking care of any type of formats. What you really want to be looking for is I just want to take the content that is in this document make it ready in for use for LLMs or VLMs. That's

the bottom line. And so the way we do that is that you can just as a user of the library you can just point it to any type of format with some blobs of data. It will Docling itself will figure out what these blobs of data are, which format, it will provide it the right pipelines, will use the right models under the hood, and it will always

produce you the same by identical data structure that you can then use downstream in whatever application you want. Be that rag, be that anything else of this. Okay? So that's kind of where we sit. Now, we have been running and we will maintain um a library, but we of course we've also started to think about how can we also make it very very easy to use. So

there is also a version of Docling which is called Docling Serve. That's also fully in the open that you can just download as a container and where we focus on a few different components. And I'll get back to that one. Speaking of the thing about this this two-by-two matrix. There are really two types of modes and there are two types of applications that you can do. The

two modes is like low latency. So imagine that you have and you can super fast convert hundreds of pages of PDFs. We currently have in the background and we can basically do a 200-page document in less than a second locally. So ultra ultra ultra fast. These type of applications are going to be super nice for any type of of of agent where you will have a loop,

it will stumble upon some documents, maybe PDF, maybe Word documents, maybe PowerPoint, it will automatically be Um on the other hand, of course, you want to be able to do batch processing and this is where you can then scale out and this what we're working with our with our Red Hat team um is to make sure that if you really want to have like large-scale document uh

processing, so think about like billions of PDFs, so we'll get to that one. You can also do that in a very efficient way. In this case, you don't very fast converter, but you care about you get as much as possible converted in a certain limit of time. Uh so that's kind of like a tool, I would say, modes. In terms of like tasks, there's two different tasks

that actually Docling is is is attacking. One is um the full document conversion and there you have to target. My target is going to be like a document database or ultimately maybe even or or vector database and that's going to go for any search-related task or rag or whatever variation of that one. Um there's another task where, you know, because you don't In search, you don't really

know what you're going to search in in a file, but you kind of know what you already want to extract in in the first place. And in this case, you typically are in a in an information extraction field and this is also another task where you not only provide the document, but you provide directly the schema of the type of things you would like to extract and

you don't care right? So those are like two different tasks that you can do in two different modes. You want either want to do that super fast or you want to do that in bulk. Another way of looking at this matrix is kind of like where does the data has to go? If you do conversion, you basically typically want to put that in a like a vector

database or a document database. If you do like schema put that in a structured database or like an SQL database. Okay? So that's kind of like the mental picture you have to have and as long as you're base basically working on any of these use cases, Docling can be of use Now roughly speaking, I mean, this is like a an older video that we made, but it's

a very good No? It's going to be a bad video. It's going to be a bad video. All right. All right, good. Thanks. Okay, good. Maybe it is. It is. It is. It is. Okay. Uh okay, good. So, this is basically a um a very quick video that we made. It's actually already a bit too old, but okay, it So, So, this is basically the docling in

action with the CLI. We actually have a very nice CLI just so that you can effectively very quickly experiment with it. Of course, you can just use it as a library. So, most people that should use it should really pip install the library and actually use the programmatic API. But, if you want to test the quality, then you can do that. You can You get the the

the tables out very nicely. And these days, and I'll come to that a little bit later, we can also like do image classification. And for the charts, uh line charts, bar charts, and and pie charts, we can actually get the data out as tables. So, if you're doing like ragging complex documents, which a lot of quantity you can get all of this data out in nicely formatted

tables. So, you your rag is just going to work just like any case, be it on tables or be it on complex documents. Um we do a lot of work also on lists and nested lists and all of these things because that's extremely uh important for uh legal purposes. self. That's kind of the team. It has grown a little bit, but not that much. But, actually what

is even more important is like you can actually see like a lot of uh contributions from the field. And that's actually what we are really uh very much very proud of. Like, we're having a a lot of uh different features, fixes that are coming in. It's starting to become quite a bit of work. Um I don't know if because of the agents or because of popularity or

a combination of both. But, it's actually quite nice that we can actually have this continuous flux of new features, new fixes. Recently we included like a complex LaTeX conversion, so that's actually coming straight from the from the community. taking off quite nicely, and we have a lot of contributions from many, many different companies, and I always want to thank everybody for that because it's it's quite quite

interesting to see. Now, this is a little bit of a humble brag, but okay, fine, I'll do it anyway. So, we were a couple weeks ago at GTS uh GTC. We and Jensen basically showed on one of his slides where structured and unstructured data is coming from, and so we've been working a lot with Nvidia, integrating with Nvidia on how you can actually use Docling at scale

in order to do document processing. Um and we're doing a lot of optimization with Nvidia to Docling up and up to speed with the latest with the latest GPUs. Um there was some work there, so this it all comes from his from his keynote, but where we were, for example, showing how you can actually use Docling with with Nestle, and we're working with a bunch of other

companies to really accelerate massively um not only the but then actually also at the same time reducing the cost of the actual um conversion. There's another use case is what we did last year with with with Hugging Face, and this actually is where some real numbers are coming in, not just some power slide PowerPoint slide. So, this is the work that we did Face, where we basically

processed around a billion billion PDF documents from the web. Um and this kind of if you do a little bit the math, and I'm not going to go through it, if you run, so basically on this billion number of PDFs, if you just would use a general VLM, it would be ultra ex- Um but if you actually run it through Docling, so basically what they did is

they said, "Okay, Docling for programmatic, you run Docling for the pure scan, you run a VLM." Of course, you could also run the VLM on all of the different PDF documents, the programmatic ones, but it's around 50 times more expensive. So, of course, that makes a huge difference, right? They did around 350 million documents, and that was a cloud bill of 750 for programmatic, and they did

around 3/4 of a billion documents, or no, close to a billion documents, 918 million documents with Docling, and that only cost them 35,000. So, that kind of gives you an idea. And of course, now you can say, "Well, more for the same cost, or you can just reduce your cost, basically. But that's kind of like the kind of cases where Docling shines. I hope I gave you

already a little bit of interesting things, and now the question is, "Yeah, but why can you not just use normal, the very, very cheap PDF scrapers?" And I'm going to give you very One should be very careful on not just randomly picking different type of capabilities out of the of the web. So, what you will find is you will have a lot of low-level PDF parsers. We

also wrote ourselves one, Docling Parse. The problem is when you do that, when you scrape, and this is you will see maybe some example of that one, line, you actually get no structure out. So, basically, PDF ultimately is like a printing language, and so basically, when you extract all of the data, there is absolutely zero structure. And so, that has a lot of consequences, and people were

in the in the beginning using that essentially in order to get training data, to get tokens for their On the other hand, so this is like, I would say, the cheap version to very quickly parse. On the other hand, you also have the very expensive methods, which are the VLMs. And here you have actually the opposite the opposite problem. So, you will have a document here on

the left-hand side, you use some type of VLM, and these are now older models. remains. Um I don't know if you see it. It looks actually all good, but actually what the problem with most of the VLMs is actually the recall. So, they will do very well, but because they're autoregressive, the moment you predict a token that maybe skips the paragraph, you are stuck with that. So,

there is like there from time skip paragraph, skip text, skip something, okay? Now, you can do that, and you can think, "Okay, overall, the the performance is is quite good. My desk metrics are quite nice." But if you bring that a little bit into a business environment, some people get quite squeamish, okay? So, that's something that you need to know go with super cheap, but then you

have zero structure. In the case, you can go super expensive, but there you might actually miss things. And both cases are not very satisfactory, okay? And so, if you run that, for example, here, and there's actually use cases where this was, I don't know if you have seen this use case of vegetative electron microscopy. Any that that use case? No? Good. This is an interesting one. So,

basically here, there is some dude that found out that all of a sudden, there is on on on Google Google Scholar, this vegetative electron microscopy popping up as a term. And these are actually terms, and they figured out where this is actually coming from, namely because again, they use probably one of these models, use a very cheap model, very cheap um PDF scraper, and it was going

this vegetative electron microscopy was going across the the the columns, which for some reason got into the and started to actually spit it out, and you we have now all kind of papers. This one paper is cited by 100 people, okay? holy Um but it is what it is. Now, of course, if you would have done document, it would have been no problem, but I think these

are the type of cases. So, that is maybe their problem with the model, but the the the fact is for our system where you want to quickly, you know, enhance the context in your own rack system, you will have exactly the same problem. Okay? So, this is These are things you need to be very much aware of. Now, how does actually Docling work a little bit under

the hood since we are a bit here in a technical conference? Essentially, we have these different We have these different pipelines. So, what we essentially do is we say, "Okay, we're going to try to detect on the fly what the format of your files is, be that like an MP3 or MP4 or a PDF or an image or docx, whatever. And we have these dedicated optimized pipelines.

And so, basically, Docling is a place where, you know, documents are. It's going to identify what type of document it is, not always working directly on the on the extension, but you actually look a little bit at what's under the hood because people can, of course, change extension and that doesn't work. And then we pipe it in the right pipelines. Um you see that there's actually for

PDF two pipelines now. We have an ensemble pipeline. This is super robust. This I would still say considered our standard one. That is the fastest. And we have now the other one, which is basically uh visual language model based and where we are actually um developing our own models up to the point that we at some point are going to exchange that with the standard one with

Um and the whole the whole point of Docling is at the end of the day, as you are doing all of these conversions, you end up with this Docling document that then can be enriched. So, this is where not only do we do the conversion, but where you can start doing funny things such as enrichment. So, you can start adding summarizations. You can add entity detections. You

can add classifications. You can add all of these type of things. Also, for example, extracting content from the image itself, like um chart understanding. And then that gets all mapped into a docling document, that's a pydantic data structure that is always going to be the same, so independent of an audiovisual thing or a Word document or a PDF, you will always get this pydantic data structure, and

then you can just do stuff with it, like chunking, like transformations, etc., etc. So, this from a user perspective should make it very easy. You just have a pointed to something where there is stuff, data, and you will always get the same representation on the back end, and what it does in the middle is none of your concern. It just just should work. That's really what it

is. And that's how we are thinking about it. Now, from an ecosystem perspective, this is why we started to build a little bit on this. We have many different components. We are build but we are also very flexible in actually running other models. So, we're continuously We have already merged in like Falcon OCR and GLM OCR and Deep Seek OCR and all the different OCR models. Honestly,

we don't really care, so we were very happy if you can use different models. Um what we want is that you as a user experience. Blobs in, you know, pydantic data structure out, and whatever model you want to use is totally fine with us as long as it's as long as it's um reliable, and it always gives us the good the good content, right? So, that's kind

of the key part. We have on the all the way on the top, we have the doc you have done the conversion, maybe you just don't want to import anymore the docling, you want to actually just use um the library, and that's why we have actually the docling core, so that's actually where all the type definitions have been super light. And so, if you want to do

downstream operations on docling, then you can actually just use docling core, and you don't need to care about docling. And then downstream So, we're going to start rolling out towards middle of the year um a full evaluation framework where you can do really one-to-one comparisons as well as a ISO standard for for unstructured data which is going to be important kind of the equivalent of for actual

content and with that we're going to have some partner companies that are going to allow you to as a company you're going to be able to just use their platform to do annotations but on the other hand also have access to annotation campaigns where you can just pay a few a few bucks and just get your own data set fully on standard so that you have once

and for all a proper data set and not you have to use some other data set from the field right? This will give you a lot of confidence in actually like are the models that I'm using in Duckling getting the performance that they need to be and if there is new models coming out that are going to be released are they going to set so is your

performance going to increase so gives you full control over the entire over the entire system. We have extensions to different languages that will extend a lot and then as I said yeah we have the MCP if you want to use it as tools we have Duckling serve if you want to use it as a as a docker container etc. etc. On the innovation of models talking a

lot about the framework and Duckling is not a model it's it's actually really a framework where you plug in the models of course we're building models because you know we think we can do a good job with that and we've been rolling out already two different models so one was a small Duckling which we did last year around March time frame which became number one September 17th

we actually rolled out last year the Granite Duckling I mean both became number one in Hugging Face which was a good sign and at some point we we were number one on the data sets as well as on the model side so we were very happy. We are going to roll out later this month or maybe early next month another model I which I think is going

to So, we've been working and, you know, training our muscles for quite a bit on how to make a really good model. The key part here was to make it super small and super portable and super fast. So, we didn't want to do like 8 billion. Maybe 1 billion, 2 billion will be the most, but the whole point will be like super super speed extreme cost performance.

So, that's going to be rolling out pretty soon directly in the new ISO standard so that you can actually verify and validate what the output is. The main problem with all of the models that I see currently is they're all spitting out some variation of markdown, which is kind of, you know, concocted and so they're just kind of shifting the the problem. On the other on one

hand, yes, you are doing the OCR, but on the other hand, you need to write all the regexes, all the parsing, all the validation and that just doesn't really make a lot of sense. So, we're trying to do that in the model. And you know, I think this is a small video which I can So, this is a kind of like this is a kind of what

Granite Duckling can do. And as you can see here, it is already doing this location stuff. It can do The current models can already do like math identification, location, etc., etc., etc. It can For example, charts. But what we really want to do is, you have that now in the full uh Duckling, this new ISO standard, and be a lot more Another main problem with all of

the VLMs is that typically you end up with like infinite loops. So, there's a lot of work that needs to be done to make sure that we don't have these If we do, it's it really hits the performance. This is a funny one. So, we do also the multilingual, so also Arabic. Um one of our main developers is Arabic. Uh he's actually No, not Arabic. He's Egyptian,

so uh he can see if this is all correct. But it since you read from right to left it's actually nice a nice use case. Let's see here. So just this is a kind of like a sneak peek in what this actually does. But really what we're doing with with the granite with a the new ISO standard is to really make that all LLM friendly. So you

have like the green tokens which are the semantics. You have location tokens, you have the text content, etc. etc. etc. has been going into that because once you have a very good representation this will then actually lead to a much better a much better model. And what we are now doing also with other frontier labs is trying to experiment like if we can actually train these models

straight into the doc line into that ISO standard can we the same model size and with the same approaches can we actually get way better results out because now we're not just training on strings, we're actually training on a on a standard that we can represent. And then a few small things that I just want to highlight because I see it's 3 minutes left. So we have

a Java extension actually this is not I'm not a I'm not a Java person but some of the finance guys they really like Java and they're still supporting Java. We did a lot of work with Nvidia and are continuing to do work with Nvidia. So keep keep your eyes out on that one. We had one the lot of parsing I wouldn't say it's a like a huge

uh technical thing but what I really like the community so it's not driven and then we have the chart understanding there's a PR for a new chart model where you can actually take any type of chart like a bar chart, line chart and and pie chart translate that either to a table or even to the code that would generate the plot etc. etc. or a description new

PRs that are going to come in. There is already a model out there just to translate it to a table but um yeah, there's a lot more coming in there and all of that is to actually start extending essentially the capability that we that we can do. Um yeah, this was the same the same slide that I did before like where we definitely go there. There's a

nice demo um that is going to be done, so make sure you you attend that one. And then if you're in other type of uh you know uh venues, I mean Ming and Carol will be also there, so feel free to maybe send your colleagues if you if you're if you're interested into Yep. That one. Yeah, and feel free to tag me. I'll be I'll be walking

around here, so happy to to talk with you to see what uh what problems you have. We have a lot of feature requests, and you know, most of the time if they make sense, we just do it ourselves pretty basically uh in no time. Go ahead. And there's stickers, yes.