About this talk
This talk discusses the critical importance of infrastructure in enabling AI models to access fresh, real-time data. The speaker, a product manager at Oxylabs, outlines their experience in developing a robust video API and search data tools that support AI training. They emphasize the shift from static knowledge to dynamic data access, detailing how they adapted their offerings from basic video downloading capabilities to a comprehensive suite that includes transcripts, subtitles, and metadata. The speaker highlights the necessity of under-one-second response times and explains how achieving speed and scale transformed their architecture, allowing them to handle significant increases in data requests. Ultimately, the talk illustrates that innovation lies in the ability to adapt quickly to changing client needs and market dynamics, underscoring the evolving infrastructure required for the next generation of AI.
Full transcript
[applause] >> Thank you. So, hello everyone. So, most talks about AI starts with models. This one starts somewhere less glamorous, with the infrastructure that decides whether those models get fresh, usable, real-time data at all. So, I work at Oxylabs, and Oxylabs was established in 2015 and describes itself as a premium proxy provider and web intelligence platform that allows companies to extract data at scale. in simple words,
we uh we let companies to connect the open web and operate it also at scale. And public data, theoretically, is available for everyone. But in practice, when you need to access your web databases, tools, and increasingly the AI systems, you need an infrastructure to do it. And that's the layer we built. And this is actually matters more now more than ever. because the industry shifting is away
from static knowledge, training still matters, of course, but training alone alone is no longer enough. To stay useful, models need access to fresh, usable, real-time data. And without that access, even the smartest is limited by the things he already knows. And this is where my story begins. As mentioned, I'm a product manager at uh but I started closer to engineering. I was leading the core squad working
on core initiatives, mostly dealing with the servers. Uh but one of the teams that I was deeply involved with, was what we called UX. And what is UX? UX is user experience, you are completely correct. But what UX actually meant for us at that time, that there is no ready-to-use product, that client actually needs something extremely unusual, um that we have a extremely painful deadline, and please
build it fast and make it work beautifully. So, that that's what taught me thing one thing really early, that innovation almost never comes as a neat roadmap. Um beca- it arrives as a pressure, as a deadline, as ambiguity, and sometimes as as you will heard now, and sometimes as a trip report from San Francisco. And this is exactly what happened. One day, our team came back from
San Francisco and said, "There is a demand for video API for AI training." And there is one question that engineering teams are extremely afraid to ask. "What's the timeline?" "2 weeks." Okay. And we never built that before, so what's the scale? "It's 5 petabytes per month." And this is the moment when the feature request start stop sounding as a feature request and sounds more like an infrastructure.
Uh so, we started from the scratch. Because the client was asking actually not to download some videos. They were asking to build a pipeline to give them a storage. So, we started from the scratch, and we actually made it. In less than 2 weeks, we we built it a brand new square, but with brand new logic, with new storage integrations for something that we have never built
before. The third part, the client didn't even tested the first version. But the real product worth came just along. A new client comes and ask, "Great that you have downloader, could you support transcripts, too?" Of course, we say yes. We build it. Client tested. And we're seeing that a lot of a lot of requests are failing. But they're failing not because the system is broken. Because client
actually didn't need a transcripts, they needed subtitles. And actually, there is a difference between those two. So, we adapt again. We build the subtitle support. Next request comes in. We're struggling to gather videos in languages that we need. Could you build us a search layer that we actually could gather those IDs? So, of course, we say yes. We do that. Another one. What about metadata? So, we
add metadata. And this is a such a honest moment that describes uh the product development. Because first, you think that you're building downloader, then you suddenly adding transcript support, then subtitle support, then metadata, then channel information, and you ended up even adding additional internal library that glues everything together because external dependencies are actually changing too fast. in roughly 3 months, we end up having the whole video
API suite that supports um channel information, search layer, subtitles, transcripts, and probably leaving something out. after all of this, came the final twist. Because client actually gathered 30 petabytes of data, and I hope that he's sitting somewhere here, because it's 2026, and we still haven't received the payment. yeah, the first lesson is quite technical, but also very human. Uh that actually innovation is just a repeated adaptation
under high pressure. Because once you understand that client doesn't buy the first version of your product, they buy your ability to adapt. Then the next question becomes, can you do that at extreme latencies constraints, too? And this is the part where I tell you a little about about search data. And search data actually mattered at all times, but AI changed the world of place. Because before that,
search data was usually used for analytics, SEO, monitoring. But uh now it's a part of AI system, because it grounds pipelines. It powers the systems. It helps agents to connect to live information instead of stale training memory. And that shift is not hypothetical. Google's grounding documentation explicitly positions Google Search as a way to connect models to current public knowledge. So, in other words, modelers is increasingly expected
to work with live retrieval layer around it. And that's why the next request mattered so much. it's 2024, client comes and asks for sub delivery under 1 second. At that time, our search scraper was delivering average 4 seconds. So, as you can see, the gap is huge. But we we decided to go for it out of curiosity and because no one from the industry had something like
that. And actually we did it. once again, client disappears. The product was never tested. But to be honest, uh the market wasn't ready for that, too. So, we put it on a shelf. But what became clear, that was never about doing a regular scraper faster. Because what does regular scraper do? It focuses to extract as much data as possible from the search page. So, we're talking ads,
widgets, full different layouts, and a full layout of the whole When it comes to the fast search API, it takes a different approach. It focuses on the things that matters for AI assistants. So, it cuts away the heavy layout that is more relevant for human browsing instead of machine retrieval and focuses mostly on things like top stories or organic results. it's 2025. and the ask was simple.
Zero data retention, sub-second, What that meant for us, for us that meant that we have to support different geolocation and query parameters, to have a system that actually is capable to deliver results under 800 milliseconds P90, and to have a testable solution that is ready in less than 2 weeks. we started from the scratch because when your starting point is 4 seconds, we are we are not
talking about optimization. We are talking about redesign. after less than 2 weeks, we got it. And we actually achieved uh we actually built a scraper that was performing performing and delivering results under 650 milliseconds P90. And God, that alone would be a great story. But the real story was happening on the next call. So, we're sitting on a call with the client getting ready for them to
test it out our new product. we got blocked. And we got blocked really bad. And this is actually OG alert from that day. this is also a a a great reminder that there is a huge difference between the system that works in development, the system that works in a test, and the system that survives reality. And once again, we started from the And this part was really
really hard because we had to rely on browsers. And browsers are amazing. They are extremely useful. But you know what browser also is? It is slow, it is expensive, and it's complex, and it's deeply incompatible with your dreams about low latency. So, at this point, there is no way Oh, so at this point, we actually had a contradiction that client needs speed. So, our reality needed browsers,
and what browsers offered for us, it was 4 So, at this point, there is no no magic trick. You just go hunting for a time. So, you review everything. Layouts, proxies, parsers, sessions, the way it delivers results, the way you gather the request, every place when you can cut off second or three, as in our case. And this is how system becomes fast. Not by giant breakthrough,
as we thought at first, by small steps that adds up. And that work paid off, and actually evolved to something that we called fast search API. And fast search API delivers results under 550 milliseconds on average, directly into your AI pipelines. And our scale grew from 400 daily million for 400 million daily requests to 6 billion daily requests. And that numbers matters, because when you're growing from
400 daily requests, it's not just a growth. It's a change in operating model. It changes the way you think about costs, observabilities, failures of the main, and everything around. this So, a lesson from this part is that in AI era, speed is not just performance. Speed defines what product can actually exist. So, at 4 seconds, you have a slow pipeline. In sub-second, you have something that can
sit in interactive AI system. when speed becomes product, what's next? I can tell you what's next. Then the scale becomes the real test. So, the first stop the first story was about adapting product scope. The second was about adapting architecture for latency. The third one is going to be about adapting systems for scale. And at scale, infrastructure becomes really humbling. Believe me. this is how it went.
So, at one point, demand forces to scale up our web and blocker quite aggressively. I decided a slide that you'll be familiar what is that because names doesn't give you information enough. So, it's similar to a scraper, but it has a proxy interface and uh instead of it. we're working our way around 10,000 requests per second, which is completely normal. But, we had to find our way
to 60,000 requests per second. And what about the timeline? Less than 2 months. So, that number alone sounds impressive, uh but it might be misleading if you're thinking just as a simple HTTP call. In our world, that means end-to-end scraping job. That means routing, rendering, browser's execution, parsing, retries, normalization, delivery, and everything else. So, when you scale up at that much, even adding adding 2,000 additional servers
doesn't solve the issue. You need an architecture. You need a central components that actually remains reliable, you need that observability that still tells you the truth. And you need a test that can resembles a a reality close enough. And this is actually where our bottleneck showed up. Not in dramatic outage, in load testing. So, the hardest part was not generating synthetic traffic. Synthetic traffic is easy compared
to reality. But when we talk about organic data testing, meaning generating and processing data that behave enough like real client user to give us something useful, that was the tricky part. And one of those and during one of those load tests, we hit a wall at 20,000 requests per And that's actually a really uncomfortable place to be because there is no longer a question is the system
is working. It is working. Now the question becomes do we actually know that it can go further? So, that uncertainty was the true bottleneck. Also, the pain points showed up. Logs, metrics, and operating both at So, everybody loves observability, but when you observability at scale becomes a true work because collecting logs is hard. Processing logs is harder. Extracting signal from exploding volume while the system is under
stress is harder still. And same applies to metrics. They are essential. But when you scale up system enough, the telemetry itself becomes a part of the complexity and part of the load. So, we have to scale up gradually, but eventually, we also had to accept the that the real test is going to be in production. And I know that sounds dramatic, but thankfully, that part actually went
completely fine. Uh the real drama is happening right now. Because internally, we were uh internally, we called this project 60 because the initial request was to scale up to 60,000 requests, and we renamed it to project 150, just joking around. But as we speak, like today, now, we are finding our way into 100,000 requests per second. And that number matters, and the lesson is simple. The scale
is not the finish line. Well, at least not for us. Uh and probably as soon as you reach one number, the next one will appear. So, what the hell Oxylabs do in this whole context? So, I guess from all the stories, one thing is clear. We are not just proxy company. We actually focusing on building an infrastructure helps companies to extract data at Because that means reaching
an open web, collecting data reliably, handling browsers, dealing with antibot systems, and doing everything in a way that AI systems can plug into their into their systems. And this is exactly why it matters. Because the best things thing that we can offer to them is not just infrastructure, not just data access. It is this. So, you build the intelligence and we take care of the messy maintenance
underneath because the messy part is real. Target change, layouts change, detection changes, volumes change. Client niche change and also market shifts. So, this is not build once business. This is an adapt forever business. And honestly, that may be the most um definition of innovation that I know. That innovation is the ability to adapt fast enough that the changing requirements become the new if I need to leave
you with one thought I will probably get back where I started. That the next generation of AI will not be powered by better models. It It's going to be powered by better access and infrastructure around those models. Infrastructure that can connect to an open web to a reality. Infrastructure that can pull web data into your pipelines, databases, tools, and agents. Infrastructure that can actually evolve from single
downloader to multimodal product suite. Infrastructure that can scale from 400 billion daily requests to 6 billion. Because this is really the story. Not just scale, not just speed, not just scraping. Adaptation, adapting systems, adapting architecture, adapt adapting products. And this is what it means to me in this in in What And this is what infrastructure means in this AI era. That the model is not alone anymore.
It has a bridge to it. >> [applause]
