Loki's New Architecture: 10x Faster Logs at Petabyte Scale | Grafana Labs

Loki's original architecture was designed for one job — and it excelled at it. But petabyte-scale structured logging demands something fundamentally different, so the Loki team rebuilt it from the ground up.

In this GrafanaCon deep dive, four Loki engineers — Poyzan, Ben Clive, Trevor Whitney, and Jason Nochlin — walk through every layer of the new architecture. A Kafka-backed write path decouples reads from writes and cuts infrastructure costs by up to 30% in large deployments by eliminating duplicate data and deduplication overhead. DataObjects, a new columnar storage format built specifically for logs, stores each field in its own column so queries can filter early and scan only what they need — resulting in 20x less data scanned and 10x faster query execution for analytical workloads. For needle-in-haystack searches (finding a UUID across 3.5TB with no label filters), a new object-storage-native index — developed at Logline and brought in via acquisition — shrinks the scan from 3.5TB down to 8GB with less than 20% storage overhead. The single-binary monolithic install stays intact; Kafka is only required for distributed deployments, and a new Query T component lets teams migrate safely by running both query engines in parallel until old data ages out.

0:00 Introduction
0:17 Why Loki Is Changing & Evolution of Logging
1:54 Loki's Original Design: What Worked
3:57 Four Bottlenecks Explained
4:33 How Loki Works Today
7:04 New Write Path: Kafka for Scale
9:32 Write Path Benefits
10:53 The Query Problem: Structured Analytics
12:33 Why Chunks Hold Loki Back
14:15 DataObjects: Columnar Storage for Logs
15:01 New Query Engine: 10x Faster Execution
20:01 Columnar Query Execution Walkthrough
23:15 DataObjects: Object Storage Design & Compression
28:01 Open Challenges: Detected Fields & LogQL
32:11 Loki: From Search to Analysis
33:30 Needle in the Haystack (Jason Nochlin)
37:27 New Indexing: How It Works
40:37 Results: 3.5TB → 8GB
42:09 Migration: Do You Need Kafka?
43:42 Dual Architecture & Query T Migration
45:47 Wrap Up & Try It Now

Links/resources:
Learn about Loki's new architecture in InfoQ: https://www.infoq.com/news/2026/04/grafana-loki-ai-agents/
Get started with the Grafana Cloud forever-free tier: https://grafana.com/g/cloud
Have a question? Ask Grot, your AI helper: https://grafana.com/grot/
Reach out in our community forums: https://gra.fan/communityyf

---
Thanks for watching!

👍 Was this video helpful? Like and subscribe to our channel for more videos.

Connect with Grafana Labs:
X: (https://www.twitter.com/grafana)
LinkedIn: (https://www.linkedin.com/company/grafana-labs/)
Facebook: (https://www.facebook.com/grafana)

#Grafana #Observability #Logging #Loki #Kafka