Keynote: Orchestrating Document Data Extraction with Dapr Agents - Fabian Steinbach (ASL)

This talk covers the challenges of extracting structured optical data from various unstandardized documents, such as handwritten notes and typed forms, which are commonly received in the manufacturing process of precision lenses. The speaker, a software architect at Zeiss, discusses the importance of reliable data extraction to prevent errors in lens production. They highlight the implementation of Dapr Agents, which provide control, reliability, and flexibility in automating this process. By using a durable workflow, state persistence, and configuration management, the speaker demonstrates how they achieved a transition from prototype to production in just two months, ensuring access to the most effective machine learning models without the need for label training data.

Hi, I'm Fabian, software architect at Zeiss. So, to manufacture precision lenses, you need structured optical data, but sometimes all we have is a photo of a hand-scribbled note. So, these documents we receive aren't standardized at all. Think about handwritten notes, typed forms, and everything across multiple languages and writing systems. Basically, everything you get from your doctor. But, extracting the data accurately is critical. Wrong data, wrong lens.

So, we needed a way to extract the data reliably and automa- automate that process. Turns out, modern models are surprisingly good at interpreting these documents, but to bring that capability into production, we needed three things: control, reliability, and flexibility. That's what Dapr Agents gave us. First, control. In our case, we didn't want some kind of fully autonomous agent generate unpredictable results. So, we constrained the AI using

a durable workflow. Preprocessing, OCR, and then some generic LLM or agent calls. Second, reliability. With Dapr Agents durable workflow capability, um state is persisted after each step. If something goes wrong midway through, the process doesn't start over again. So, an expensive, already completed OCR task doesn't run again. That saves time and cost. And third, flexibility. Even when the model landscape changes, the workflow should stay stable. With

Dapr I- so, if a better or cheaper model appears tomorrow, we can use Dapr's conversation building block and just change the configuration. No no code change required. So, what are our three key takeaways from that? First, constrain the AI. For us, it worked way better constraining the AI with a workflow instead of just heading it letting it run wild. don't use a hammer for a screw. In

our case we mixed specialized models, general LLMs, with deterministic logic. Don't use an LLM where a function could do the job. And third, decouple. Change models via config, not code. That allows you to use always the newest and best models. So, and that entire thing allowed us to go from prototype to production in two months zero label training data required. And having comparable recognition results, on par

recognition results, with a specialized machine learning system. Thank you very much.