Keynote: Orchestrating Document Data Extraction with Dapr Agents - Fabian Steinbach

This talk covers the importance of structured optical data for manufacturing precision lenses and the challenges faced when dealing with unstandardized documents, such as handwritten notes and typed forms. The speaker explains how modern models, specifically Dapr Agents, are utilized to automate the extraction of critical data, emphasizing the need for control, reliability, and flexibility in the process. Control is achieved through a durable workflow that prevents unpredictable results generated by autonomous agents. Reliability is ensured with state persistence after each processing step, which saves time and resources. The speaker also highlights the flexibility of the system, allowing for easy integration of new models without code changes. Key takeaways include the benefits of constraining AI, using appropriate models for specific tasks, and decoupling model management from code.

Hi, I'm Fabian, software architect at Zeiss. So, to manufacture precision lenses, you need structured optical data, but sometimes all we have is a photo of a hand-scribbled note. So, these documents we receive aren't standardized at all. Think about handwritten notes, typed forms, and everything across multiple languages and writing systems. Basically, everything you get from your doctor. But, extracting the data accurately is critical. Wrong data, wrong lens.

So, we needed a way to extract the data reliably and automa- automated process. Turns out, modern models are surprisingly good at interpreting these documents, but to bring that capability into production, we needed three things: control, reliability, and flexibility. That's what Dapr Agents gave us. First, control. In our case, we didn't want some kind of fully autonomous agent generate unpredictable results. So, we constrained the AI using a

durable workflow. Preprocessing, OCR, and then some generic LLM or agent calls. Second, reliability. With Dapr Agents durable workflow capability, um state is persisted after each step. If something goes wrong midway through, the process doesn't start over again. So, an expensive, already completed OCR task doesn't run again. That saves time and cost. And third, flexibility. Even when the model landscape changes, the workflow should stay stable. With Dapr

A- so, if a better or cheaper model appears tomorrow, we can use Dapr's conversation building block and just change the configuration. No No code change required. So, what are our three key takeaways from that? First, constrain the AI. For us, it worked way better constraining the AI with a workflow instead of just letting it run wild. Second, don't use a hammer for a screw. In our case,

we mixed specialized models, general LLMs, with deterministic logic. Don't use an LLM where a function could do the job. And third, decouple. Change models via config, not code. That allows you use always the newest and best models. So, and that entire thing allowed us to go from prototype to production in two months zero label training data required. And having comparable recognition results, on par recognition results, with

a specialized machine learning system. Thank you very much. >> [applause]