NODES AI 2026: Lakehouse + Graph: Delta Lake, Unity Catalog and Neo4j for Governed AI Pipelines

This talk focuses on integrating Neo4j, a leading graph database, with the Databricks Lakehouse platform to tackle the challenges of managing both structured and unstructured data in enterprises. The speaker, with extensive experience in ERP systems, explains how a graph database can enhance contextual analysis, especially with the rise of generative AI. He delves into key components like Delta Lake and Unity Catalog within Databricks, highlighting features like time travel, schema evolution, and governance capabilities. The presentation also covers the new Neo4j Connector for Unity Catalog, which facilitates metadata management and two-way communication between Neo4j and Databricks. Use cases discussed include customer 360 solutions, supply chain optimization, and applications in healthcare and IT network management.

[music] >> Thanks for your patience. So, I'm going to talk about Neo4j graph database and integration with Databricks Lakehouse platform. Now, the challenge in today's world is uh how do we manage and build graphs and do a contextual analysis on unstructured data? And that is where we believe that Neo4j plays a pivotal role in integration with Databricks Lakehouse or something similar. Uh just a little bit of

background of myself. I've been in the industry about close to 30 years. My background had been in primarily in ERP systems such as SAP. And in the last 5-7 years, I have moved into more graph technologies and AI-centric transformations. >> [gasps] >> Uh what problem we are trying to solve? The biggest problem is the enterprise has data primarily stored in structured and unstructured data. And uh structured

data is easy peasy. We manage it through SQL databases. But unstructured data is where the challenge comes. Now, in the last 5 years, the generative AI has taken the center stage. And that has compounded the problem further. now, with the Lakehouse architecture, that solves significant problems that we faced in the past. this is where Neo4j's graph technology helps in in understanding the relationships and the connections between

our entities. And if you look at on the right-hand side, the third bullet on the right-hand side, that really sums it up basically. The true intelligence requires understanding connections not just isolated data points. So to give you a paradigm, if I place an order, uh who placed an order, uh when was it placed, uh where is getting shipped to, what were the order items, uh where is

shipping from, where is shipping to, there are a lot of relationships and connections in that just one transaction. Entities here are the orders, the the address, the customer, but rest all is all relationships and connections and that is where most of the technologies that we use in our enterprise fail. And this is where the integrated platform, uh something like uh Databricks plus Neo4j's graph database comes handy.

The core components that we are going to do a deep dive on is two components of Databricks, that is Delta Lake, Unity Catalog uh from Databricks side, and then Neo4j graph database. Now, Databricks I'm using specifically because we specialize in Databricks as partnership in Honatech, uh technologies used in Databricks are pretty open source technologies and any other product such as Snowflake or Microsoft uh Fabric can also

be replaced, but the use case example here is primarily on Databricks with Neo4j uh because Neo4j has been uh going full stream with Databricks to my understanding. little bit on Databricks, Databricks storage platform is built on uh Delta Lake. Delta Lake is open source technology. Uh it's a very reliable storage platform. It's uh underlying format is Parquet file format. It is a columnar file format. Those who

do not know uh or are new, typically we store data in row format. And Parquet format uh is columnar-based data uh base. It's very similar to SAP DB also or SAP HANA. Where data is not saved in rows but in columns. So, it provides the full asset. It has full transformations on each and every column that you can do. Uh performance is much better compared to row-based

databases. Um and some of the other features I'm going to talk in future are basically time travel and undo. And it's open format. Unlike some proprietary formats like Oracle, uh this is open format. So, like we talked, it's a Parquet file format. Uh the each transaction you do in Databricks is stored in a transaction log. So, you make a change, you update, you do delete, each one

is gets stored and tracked. So, it provides single source of truth as well as atomicity. Uh we talked about asset transactions. It also does uh schema enforcement and schema evolution on the fly. So, anytime I need to add a new column, uh I can do that in Delta Lake file format. Time travel. There's a key feature or the highlight of Delta Lake which is leveraged in Databricks

or in any other technology that uses uh format allows you to undo any transactions that you have done or any operations you have done on on Databricks or any other solution. In a parquet file format, each transaction stored as a separate file and the previous state is stored all the time. So, unless you run explicit transaction to delete those older records or old files actually I should

say specifically, uh you can go back and do a rollback very easily, very conclusively. So, that's what it's called time travel. And it has been very handy most of time whenever we do debugging or do uh issue resolutions in the replications. Now comes the elephant in the room, the governance in any company. Typically, we use 10 20 30 systems and customer master or customer information stored in

20 systems, same with sales information, same with employees. Maybe to a limited extent, four five systems. Governance becomes a big challenge. Every enterprise application, you take SAP, NetSuite, Salesforce, they have their own way of doing the governance. The biggest challenge comes in when you are trying to analyze data in a central repository through a data warehouse or through some kind of analytics platform, how do you govern

it? >> [snorts] >> And every system does it differently. And this is where I think Databricks shines with this Unity Catalog platform In Unity Catalog, you can do the governance on each object on each entity type uh user, customer, uh the access, the objects that you have created, and you can propagate down the line uh across different systems. And again, this is also an open-source solution. Uh

it helps you do a central metadata management. So, for example, uh if I create a notebook in Databricks, uh I can maintain and govern it centrally. In fact, I can move that notebook to another application, uh and I can still track and manage its metadata. Uh I have, let's say, uh table in SQL Server uh on prem or on cloud, uh I can access it in Databricks

as a managed table or unmanaged table, and I can govern it. Uh if it's uh let's say, if it's a table that's still sitting in the SQL Server, I cannot govern it completely, but I will have access to all the fields and columns. And I can control it within the Databricks platform. Now, the other uh good feature Unity Catalog allows is automated data lineage. The moment you

make a change to any attribute, it propagates throughout the Databricks platform. And I'm going to talk in future uh next slides uh how it propagates into Neo4j as well. Uh graph database, I think this is the key here. Graph database allows you to store the relationships and the entities and their connections. And Neo4j has been the leader over the last 15 years in bringing that to the

mainstream. The the language that is used in Neo4j Cypher is very robust. that is what has been used in integration also, which I'm going to shed light on. knowledge graphs knowledge graphs we know on Neo4j is the real world entities nodes and relationships that connects them in a structured query format. In Neo4j it allows you to ingest, model, enrich, and results are visible in a graph format.

Uh graph data science library, okay, this is some a new library from Neo4j. It is a basically enterprise grade algorithm that allows you uh to visualize large data sets in memory actually and the relationships. And it has about close to 60-65 algorithms across multiple categories. And the the benefit is that it allows for the scalability as well as integration across multiple systems. GDS has a central principle

of centrality algorithm. So, basically it allows you to do the page ranking. Uh there are special use cases that has been built into this library. And some of the use cases are customer segmentations, fraud detections, and analysis of research topics. Uh there are two connectors are [snorts] used typically to do the integration between Data Bricks and Neo4j. And we're going to touch on those two briefly. One

is the old traditional connector that has been in existence for a while that's called Neo4j Spark Connector. That's the primary bridge to connect to between Neo4j Databricks. It's a two-way connection. And it allows you to query Neo4j using Cypher. And it allows you to access the Spark data frames and see the Spark data frame results. And as you build your solution in the uh through the medallion

architecture. those who don't know medallion architecture, it is basically bronze set of tables, silver set of tables, gold at different aggregation levels. for example, gold is the final version from which we do the business level reporting. And bronze is the raw layer from where we uh bring all the raw data from different sources together and do in a ideal use case scenario, we use the gold level

tables uh and access that and use that to feed to Neo4j. The connection allows uh the read, transform, and write uh from lake house to graph. As well as it allows uh from the other side from graph to lake house, the query and to pull and to analyze. Uh as of today, uh Neo4j has uh provided a new connection uh called Neo4j Connector for Unity Catalog. Now,

this is something uh that has uh been made available from Neo4j Labs, well, fairly recently. earlier the Spark connector that we talked about in the previous slide, that did not had access to the Unity Catalog, the the catalog that is used for metadata and data governance. Now, this new connector from Neo4j the metadata governance of Unity Catalog as well. So, right now it is it has been

released, the new connector. It is available. The link I've provided is on GitHub. Um but I would play a little bit of caution to use it straight away into your production landscape. I would advise to use it in your test environment first to smooth out any um discrepancies, and then perhaps slowly move it into production. I have summarized uh some key differences between those two old connectors.

right off the bat in the first table uh at the top table in the third line, you can see it's the lab in beta. That's my personal assessment. Um I mean, uh different people have said that it is almost ready. Uh I personally felt that it is still in a uh beta stage per se. uh the main difference is the Neo4j connection connector to Unity Catalog is

SQL based. The old connector was using PySpark, so it was Spark based technology. So, that's the main difference. Uh and with the access to connect to Unity Catalog in Databricks, that exposes you all the metadata and the catalog information of all your objects that you have created in a data bricks and that allows for complete two-way communication. Governed data pipelines there's a new paradigm basically governed AI

pipelines means you create the policies and standards at one place and cascade it throughout your landscape between the data bricks as well as Neo4j and it is important primarily because of the transparency and regulatory controls. Recently EU launched a new compliance on AI and that has kind of necessitated companies to start investing in the governed AI pipelines where the governance is done and managed throughout the data

pipeline that is used for AI or any kind of LLM activities. is a topic where there a lot of ways you can skin the cat but at the end of the day it has to provide end-to-end data lineage it needs to be synchronized throughout the landscape. we believe that this new connector is achieving close to Nirvana as far as the governance is concerned and as this connector

matures this will provide full end-to-end data lineage and complete synchronized across the landscape. >> Now in a with LLMs in the play you can use LLMs in multiple ways, with Neo4j. Uh there's a graph LLM system and graph rag wherein I can have user query converted in knowledge graphs, pass it on to LLM's, and that generates the response, or I can do vice versa. I've done a

comparison here. There is a basic lexical graph and then the parent-child retriever. both ways based on your business needs, so you can approach it. part two is more prevalent, so I'm going to spend some more time on that. In text to Cypher, basically you pass your query LLM. The LLM basically understands it, converts it into the Cypher that Neo4j understands, goes out into Neo4j gets the information,

retrieves retrieves it, and converts it back into user-understandable context, and presents it. This is the most common way how graph rag has been implemented with LLM. There are a lot of use cases I'm sure with throughout this this sessions and conferences you have heard all the different use cases. Not going to touch about it. The four use cases where it's been used extensively is the customer 360

and personalization. Customer 360 has been in the center stage. More and more companies has started implementing customer data platforms to better understand their customers. There are a lot of products out there, Segment, Salesforce, that offers customer 360 solutions, but almost all of them are are having relationships, but they store data in RDBMS format. Uh I believe Neo4j is in a very good place in position to transform

this and create a new paradigm with uh graph-based customer 360. Uh last year I presented a complete solution on that. In the same uh Hunitec uh and Neo4j's presentation on customer 360. So, that's a huge one that I believe is going to come in the next 3-4 years where companies are going to build custom solutions on customer 360 using some kind of graph technology. Supply chain optimization

is also has also picked up graph-based solutions quite a lot. Some of the solutions uh such as a Blue Yonder in supply chain, they have started using graph to good extent. In a supply chain, your key objective is the route optimizations. And almost all the supply chain uh solutions use 3PL third-party logistics for deliveries or for actual shipments, and that is where uh graph technology comes very

handy. Healthcare and life sciences here uh there's a big uh What's the right word? There's a big use for uh graph database to integrate with uh all the relationships between the genes and the drugs and the research that is happening. Uh it has come to light uh in some of the researches uh any for example, uh medication or therapy that works in one part of the continent,

let's say Africa. I'm just giving an example. It's correct 100%. May not be replicable uh to another part of the world because of the way our heredity has evolved. And that is where uh graph technology helps us uncover what are the discrepancies and outliers where one kind of treatment or solution in the medicine world uh does not work or work differently. IT network management is an area

also where graph technology has started making inroads. If you look at your uh enterprise landscape, there are hundreds and thousands of servers. Uh think of them as nodes, and then there are connections all the the networks and LANs connecting all of them. And each line is uh doing a communication or transmission of IP packets of different kind of data. Uh some data is on uh CRM data,

some data is on ERP data, some is on uh sensitive PI data like HR data. And that is where where uh graph technology has come into very handy. Uh some of the best practices uh on lakehouse side, uh there's an optimization uh command and then Z-ordering. That is used uh on uh the Delta Lake file format we talked about that uses the parquet file format for compression

and to accelerate the queries. So, that is used extensively on lakehouse. On lakehouse, there's a new engine called Photon engine that is built on C++. By default, uh earlier it was not uh the default engine in Databricks. Uh so, some of the old customers, they continue to use the old technologies. And now, it's always encouraged to use the Photon engine on Lakehouse on Databricks. On graph side,

uh the some of the big improvements and best practices you have is to create indexes frequently. We know that. and keep the data model clean and simple. Overall, the architectural practices that we use is to focus on caching, continuous monitoring, and emphasize on the serverless computing. That is it for 30 minutes. To summarize, I think two more minutes. To summarize, there are two connectors right now in

the play to connect to or integrate Databricks platform with Neo4j. that is built on Spark, but it does not do or it does not exposes the Unity Catalog for metadata and governance. The new connector introduced by Neo4j is is for Unity Catalog. It also supports two-way communication. However, it is being developed and actively being evolved right now. So, play caution with that. going forward, definitely I would

recommend using the Unity Catalog connector uh connector for the future. That is it, guys. Thank you. Feel free to connect me on LinkedIn. Um I am a good advisor with HunarTech. Thank you. >> [music]