"How I Built a Kafka Plugin for Grafana" | 4M+ Downloads

One developer. One gap in the Grafana plugin ecosystem. Four million downloads later, Kafka data is streaming live into Grafana dashboards around the world.

In this GrafanaCON talk, Hamed — open source contributor and data engineer — shares his journey building the Kafka data source plugin for Grafana, starting from a simple observation: you could monitor Kafka cluster metadata in Grafana but had no way to visualize the actual data inside your Kafka topics. The plugin connects Grafana directly to any Kafka cluster (cloud or on-premise), streaming messages in real time without page refreshes, and supports JSON, Avro, and ProtoBuf formats with nested schema handling, schema registry integration, and inline schema upload. A live demo shows a payment analytics dashboard streaming live transaction data across all three formats simultaneously, with autocomplete topic search and flexible offset options. On the roadmap: historical data queries for time-traveling through Kafka messages, Grafana alerting on Kafka topic patterns, and additional auth methods including Kerberos and OAuth.

0:00 Introduction
0:42 Hamed's Background: From Data Scientist to Contributor
1:10 What Are Grafana Plugins?
2:50 First Plugin: Node Graph API (2021)
3:33 How to Become a Grafana Open Source Contributor
4:49 Apache Kafka: Architecture & Key Features
7:05 The Gap: Kafka Data Invisible in Grafana
7:57 Kafka Data Source Plugin: Features & Capabilities
9:26 Use Cases: IoT Monitoring & App Analytics
11:08 Live Demo: Installing & Configuring the Plugin
13:15 Live Demo: Real-Time Streaming Dashboard
14:34 Demo: Avro Schema Registry Walkthrough
15:37 Future Work: Queries, Alerting & Auth
16:48 Wrap Up

Links/resources:
Kafka data source plugin for Grafana: https://grafana.com/grafana/plugins/hamedkarbasi93-kafka-datasource/?src=yt
Get started with the Grafana Cloud forever-free tier: https://grafana.com/g/cloud
Have a question? Ask Grot, your AI helper: https://grafana.com/grot/
Reach out in our community forums: https://gra.fan/communityyf

---
Thanks for watching!

👍 Was this video helpful? Like and subscribe to our channel for more videos.

Connect with Grafana Labs:
X: (https://www.twitter.com/grafana)
LinkedIn: (https://www.linkedin.com/company/grafana-labs/)
Facebook: (https://www.facebook.com/grafana)

#Grafana #Observability #ApacheKafka #OpenSource #GrafanaCON #DataEngineering