Your Users Are Not Just Human: Designing Systems for People and LLMs - Guust Ysebie

This talk addresses the need for optimizing PDF documents for both human users and large language models (LLMs) in the context of accessibility. The speaker emphasizes the importance of making documents accessible, highlighting that approximately 16% of the population may have disabilities that hinder their ability to extract data visually. They provide insights into the iText SDK, a leading open-source PDF toolkit, discussing how to implement accessibility features in PDFs. The session includes a crash course on PDF structure, focusing on its fixed layout and container format, and explains how to add semantic information to enhance document processing. By leveraging accessibility standards, the speaker demonstrates how organizations can significantly reduce data extraction costs and improve efficiency in document workflows using metadata. The talk concludes with a call to action for attendees to consider the implications of accessibility in their future document processing strategies.

Your users are not human anymore. Um, a couple of years ago, let's say 10, when you were developing applications, you were targeting mostly human users. This means uh you can rely on human vision to actually analyze PDF documents applications and to understand what they need to do. But currently, in the age of AI, um we um we need to design for both people and LLMs. Why? Because

LLMs get used quite a lot and it means that you need to make sure that your PDF documents, that your data, that all your stuff is optimized so they can also work with it. So, you guys already had an introduction of me. So, a couple of more information, um working at iText mostly on the iText SDK. It's one of the leading open-source um PDF SDKs, meaning that

um we offer solutions to work with PDF. This means going from PDF creation to PDF modification, digital signatures, and PDF conversion. So, today, as I told you, I'm developer for the PDF SDK, so we're going to mainly focus on the document format PDF. Uh and what do I want to try to teach you today is the case or accessibility. I'm not going to be the person that

going to say, "Yeah, you have to make your document accessible." Uh no, we have like other uh people doing this for you, like governments. Um but I'm saying documents are becoming accessible anyway, so you can leverage the information now included in these documents to actually um significantly improve your data extraction models and things like this. So, how will we do this? First, I do a quick review

of the accessibility. Then, we go on a bit a quick PDF crash course. Um then, I show you how we implement accessibility with N PDF. And then, I'll do a quick demo um how you can leverage um and how you can uh optimize your own internal and external document workflows. So, let's get started. The case for accessibility. Um why should you even care? Um most of the

time, accessibility is an afterthought. So, let's take a PDF document. Um most of your users will be able to actually read the PDF document. Why? Because we have a supercomputer into our own bodies, which can quickly look at the uh visual document and analyze the content within. But, we have to remember 16%. So, one in six people has has a dis- disability. Um this disability means that

in some way they are limited on extracting the data visually or with the help of tools and understanding what your PDF document is about. 16% um it might not seem a lot and justify the cost of adding accessibility in your document workflows, but you have to remember 16% would be if you take the global amount of people, it would be um all of the people in Europe

and all of the people in America that would not be able to access your document. So, it's quite important that you make it accessible for those as well. Again, I'm not going to be uh the guy who said, "Yeah, you make need to make every um document accessible." I have like a lot of government and laws who do it for me. So, if you want to do

business here in India, in Europe, or in in in if you deliver PDF documents, they should adhere to certain accessibility standards. Meaning to be compliant and to be legal, you have to make them accessible anyway. So, your PDF documents are accessible now. So, you can just leverage the data the data that is used and included for accessibility and just like kind of hack it, so you can

also improve your own So, what are the principles for accessibility? First of all, it perceivable uh which means if you look at it, you should be able to extract the content quite easily. Operable, we're developers, most of us or some of us will not use a mouse because it will be too slow. So, we want our reading orders and our tab order to work quite efficiently. And

also, we want to make it understandable that um for example, the reading order, we can read it easily from top to bottom, left to right. Those are all things that need to be included within a PDF to make it accessible for both users and LLMs. So, PDF, I uh promised you guys a quick crash course. What is it? It's a portable document format. What do you have

to remember about PDF? It's actually a container format. So, it means that we include uh different kind of um files into it to generate one document which looks the same on all devices. This means if you open a PDF document generated and opened in a web browser, it will look exactly the same as you open it on your Android phone or on your Windows phone. This is

very important in cases um for government and contracts and invoices and stuff, so people always get the same accessibility and viewing experience. Um fixed layout, as I said and I will go deeper into this, a PDF will always look the same, and that's because we use a certain kind of programming language which allows us to do it. And it's widely used in both archiving, printing, and sharing.

So, what is a PDF? A PDF contain is a binary format, and you can see it contains uh three big parts. The first part is the header. It's just some basic information about the version and a magic number, so it's easily identifiable. Then we have a list of objects. Objects you can just think of it the same as in JSON. So, they just metadata information. Um we

have a bunch of different data types within this these kind of documents. We have numbers, streams, references. These are all things you probably know from JSON or XML, so I won't repeat them here. If you have any questions, please come after the talk, and we'll then discuss this in detail. Uh but the what is most important? These are the drawing operators. At the top, you see um

like some kind of Q, and then some hello world, and some numbers. These are our drawing operators, and those are the bits that you see when you're opening a what is it? It's actually an embedded programming language which which is stack-based. So, it means that it gets executed sequentially. Um and because it gets executed sequentially, and it's like always the same, um you can always have the

same rendering of the information. So, let's take a look at the example. We have hello world with an underline. So, how would it be translated within PDF document? Well, it's quite easy. Uh we have the first one on the right on line three. we select the font for this um, text. And this F1 is just some standard font. And we set the font size currently at 12.

Then we say move to location with the coordinates 36 uh, 780 and start drawing the text. Then on line uh, five we have the actual text drawing instructions. And then we just end our um, instructions. And now hello world is drawn. Of course, we also need to draw the underline. And this you can easily do by just um, drawing a rectangle as you can see on line

nine with um, line width of 0.75. I know this is quite low level. So, let's approach it from a higher point of view. what did we see? On the left side of the screen, you see a satellite image. And this satellite image is um, not annotated with any metadata. So, if I would ask to you, um, could you um, provide me with a way um, where I

could can go from my hotel to this center. Then what would you have to do? You would need to identify all the streets, intersection points, and start and end location. This is how a PDF is currently displayed. So, we visually see it, but we still have to extract the information ourselves. On the right is um, the satellite image with the metadata included. So, you can see intersection

points, and you can see lines. This means that a lot of um, this data can be used, this metadata for um, running pathfinding algorithms and other uh, good stuff. So, how do we add semantic information to PDF? Well, um it's quite easy to do. We can just start by adding labels. And as you can see in the example here, we have a heading. And then we have

some text. So, again, they are just those drawing instructions and we take those drawing instructions and we put them within some metadata so we can actually provide semantic meaning to them. And as you can see big on the right-hand side, we start with creating a kind of tree of all the elements with their sub-elements. This means our data, which was now only used for viewing is now

also programmatically accessible. um this is how one would add semantic information on a low-level um for PDF And that's a rule-based system. Rule-based system, just think of all the rules you have in HTML like header tags uh H1, H2, divs, paragraphs, tables. Um we just took them over from HTML and put them into the PDF format. And we have one additional rule, which means uh artifact. And

artifact means that it's not something that's useful as content for our end users. It's just there to be visually uh better and improved. So, how would one add uh this this semantic information to your uh PDFs as well? Um it's quite easy to do. Most of PDF SDKs actually provide a quite simple way to enable this. Um if you want more details about it, we can look

after it a little bit later or you can can come by the booth. So, how would we leverage accessibility for processing? Well, um because now we have the tags and we information about our documents, then we can easily start by uh building on top of this low-level information. For example, one of the things we can do is assistive tech. Um everybody knows it. Um it requires um

the structure, so the reading order, um and things like that. For search, we want to uh be able to extract all the textual information to be able to run our um data on Also, we have extraction, um reflow, and conversion tool. All these um can be made quite easily because now we have additional meta information in our um in our document. Um All right. So, this is

the important part for all of you guys how uh we are trying to get the costs down for your um application and data workflows. So, let's say you have two pipelines. Pipeline one is OCR-based, so we'll take a PDF document which doesn't have any metadata and extract the data from it and put it into markdown. Why would you put it into markdown? As you guys all know,

uh tokens are quite expensive in large language models. So, if you would need to parse the whole PDF document, then it would mean that you will lose a bunch of tokens. Um and pipeline two processes a PDF um but it relies on the metadata. This means that it's far easier to extract the metadata and convert it to markdown. On the left, pipeline one, we will use the

a tool called Docling. It's industry standard for data extraction. And on the pipeline two, we will use the iText as the gate the data itself. So, let's have a look. Um as you can see, the PDF we're going to try to extract data from looks something like this. It's quite easy. Um it's English and it contains a lot of information. So, we want to convert this one

to markdown and uh contain all the information. On the first on this side, I'm going to run it within Python. We have this document. It's a quite um commonly quarterly uh or operations review document which I generated and it contains a lot of tables, um a lot of lists, and some additional uh difficult to OCR data. This all means that um OCR engines will be quite hard

to process it. Also, I started uh running the Python script with Docling, um the main open-source tool. And as you can see, it takes quite a bit of time to load the weights. This, of course, will be only done once, but then to extract all the data from it. So, now the warm-up has been completed and we're starting the time conversion. as you can see, this page

is only four pages long. And what to keep in mind is that it's um an English document. So, if you're using documents in Hindi, it will take a lot longer. Why? Because Hindi is a lot harder um to extract data from because all those models are mainly trained on uh English Um so, on the right hand side, now, this one, um, is a application which does exactly

the same. It extracts the data, uh, but now uses the metadata included. And then if we execute this, we get down to 100 milliseconds most almost. So, for four pages, you go down from 12 seconds to 100 milliseconds. That means if you have to do a bunch of document processing like millions of documents, uh, your processing time would decrease by 100%. This is not the only thing

to take into account, um, because it's, um, quite hard, uh, to do. Um, but you also have to take into account the quality of the conversion. Again, this was a quite easy document to convert. but even you can see if we open the Docling next to, um, metadata extraction, uh, smart data extraction itself, it already contains some differences. Uh, mostly here we can see differences within the

formatting itself. So, Docling didn't recognize the sublists of this document. But also, what's quite important is we see the watermarks here. And the watermark is not available in the smart data extraction. Why? This this watermark was considered a rendering artifact, so not really required content. And this also means that it's, uh, smart document extraction improves the safety because if I can inject a watermark that your Docling

OCR engine would be able to read, I can probably expose the data that you don't want to, which might become a security risk and issue. so, as you can see, we reduced, uh, the run time drastically, and the result is almost the same. This code will all be available for free open source, so you can play with it yourself. So, what do you have to remember? Accessibility

is mandatory anyways. So, if you're targeting European countries or you're targeting um PDFs um or big enterprises, you have to comply with tagged PDF documents. So, this would mean in the future there will become more and more tagged PDF documents, which means that you have to alter your document processing capabilities to include them and to more easily process them, so you can actually uh have better results.

Um and those improved capabilities are quite important because even in OCR, let's say you would have an accuracy rating of 99.999%, this 0.1% can make issues, which are very dangerous for legal implications and such things. All right, um this was all about the PDF data extraction. Um I thank you all very much for coming. This talk was mainly focused on the smart data extraction. Um because we

are in India, my colleague also has a talk at the end of the week specifically targeting data extraction for Hindi and other engine languages. So, please check it out. If you have any questions, please ask them or come to the booth. Thank you very much. >> [music] >> Oh.