Fost Singapore 2026 - Intelligent Data Platform : Powered by Semantic Layer, MCP & Agents.

This talk discusses the concept of an intelligent data platform that utilizes a unified semantic layer to enhance the trustworthiness of AI agents. The speaker, Sabir Farooq, explains the evolution of self-service analytics from traditional BI tools to the current era of AI agents that can answer complex business queries. He emphasizes the challenges faced when agents are directly connected to data layers, leading to issues of accuracy and context. The introduction of a semantic layer is presented as a solution to provide essential business context, thereby reducing ambiguity and enhancing the agents' ability to deliver reliable results. The presentation includes a reference architecture based on a lakehouse model and a demo comparing two AI agents: one with access to only the data layer and the other integrated with the semantic layer, illustrating the dramatic difference in their responses.

Thanks for tuning in to listen to my presentation. So, I'm going to present about a topic intelligent data platform powered by semantic layers, MCP, and agents. So, the talk is about how a strong semantic layer can really make your AI agents trustworthy, right? So, that's the crux of this talk. So, let me quickly introduce myself. I'm Sabir Farooq. I work as a principal engineering architect at Credit

Agricole CAB in Singapore. I'm really passionate about data architecture. And nowadays, I'm also riding the GenAI wave like many of you. Um where I, you know, try to leverage the GenAI to optimize what I already know, right? For example, this talk is also a manifestation of um you know, exploring that. Apart from the tech part, I also love traveling and exploring new places. you can follow me

on LinkedIn. Um like send the connection request. I have shared the uh QR code here. So, in terms of agenda, so, we are going to talk about the self-service analytics stream, right? you know, what's a naive approach of achieving it by uh you know, connecting your AI agents directly to your data layer. That's the naive approach. And we see like what are the challenges, and how bringing

a semantic layer can really address these challenges, right? We will also see um quick demo with a lakehouse-based reference architecture. So, uh let's see how the self-service analytics itself, right? The dream itself is uh evolving uh over the years. So, we started off with BI tools era, where uh we had drag-and-drop dashboards, uh which provided uh much better uh experience than uh you know uh doing SQL

for the uh analysts, right? But, uh still they cannot uh they they the analysts cannot actually um uh you know, or the business users cannot ask natural language questions, right? Uh using this method. They still have to uh look for um you know, help to build uh reports actually, right? Um to to achieve any custom requirements. Then came the natural language era, which is actually powered by

NLP. So, which enabled uh you know, the business users to ask questions in natural language. But, um it's it's closer to the dream, but still fragile, right? The moment you throw um a complex question or an ambiguous question, it breaks, right? So, that's the issue with this uh NLP-based um BI. Then today we are living in the AI agents era, right? And for the first time we

can say that the dream is, you know, uh very very closer, and um you know, like the gap between a business user asking a natural language question and getting accurate, trustworthy answer uh is really close. Um but, still uh I want to reiterate that like if you connect the agents directly to your data layer, it doesn't work the way we expect. And there is no doubt that

uh it you know, the AI agents are transforming the data and analytics space, right? Uh it offers uh capabilities like natural language queries, autonomous data analysis, self-service analytics, and conversational interfaces. So, there is opportunity for the business user to ask valid business questions directly. So, like show me revenue trends by region or you know, why did sales drop last month, right? Valid business questions. But, the challenge

itself is building agents that are accurate, consistent, explainable, and trustworthy, right? So, these are the four um tenets uh which are very, very important, but uh it remains uh you know, a challenge to build an agent which ticks all these boxes. So, then let's let's talk a bit more about this dream, right? The dream is for the user, the business user to ask uh questions like, you

know, show revenue by region for the premium customers, right? The dream is uh in a dream scenario, like the agent should be able to understand this instantly and provide the response within seconds, which are grounded in terms of correctness, right? In terms of the definition, explainable, and trustworthy. But, the reality is that when the user asks this question, the agent uh might need clarification, it's not sure

what revenue you mean. It can use multiple different formulas for revenue, which may or may not align with your organizational definition, And even uh it can finally generate a black box SQL, right? Which seemingly is correct, uh but, you know, it may give uh you know, wrong answers at the end, right? So, that's the reality. So, the gap here is uh you know, your agents needs more

than table schemas, right? Okay, you expose your table schemas to the agent for it to work out things, but uh really what it needs is business context. Now, let's understand this you know, the hallucination problem a bit more in detail. So, when a user asks the question like show Q1 revenue for premium customers, you know, there is ambiguity in this formula, right? Uh for the agent, right?

Are you referring to Should it use total price or it should use extended price or extended price minus the discount, right? So, that is this ambiguity. And even the business term called premium customer, how do you define a premium customer? Are you going to define it based on, you know, creditworthiness of the customer or is it based on, you know, a market segment where the customer operates,

right? So, there is ambiguity again in the business term. On top of that, the agent also could use wrong join paths. Even if you expose the agents with your schema details, you can still figure out some join paths, but there is no guarantee that it's accurate. So, at the end, what the agent offers is a contextual blindness, right? Where it could really return you a inflated amount

for this question, and you know, it it sound it it it turns out to be a very confident wrong answer, right? So, that would be the situation. Now, what's the fix familiar fix for that? Do you want to bring another agent to you know, as a agent to review the first agent's you know, SQL query, right? So, that's an option. Like in that case, like how many

such agents you will bring in? Like, you know, you may want an army of agents to you know, finally get it and it becomes it goes into infinite loop. The simple fix is to introduce this unified semantic layer, which really provides this business context, which is, you know, lacking so far. So, now let's understand what is the semantic layer. Right? So, it's not a single tool, it's

a unified framework of components. So, it's a standardized framework that organizes and abstracts organizational data, thereby giving machines a way to understand your data in context. More than accessing it, it also needs context, right? So, that's what this same semantic layer brings in. And now, you know like standard capability map like in terms of you know the fixed set of capabilities the semantic layer offers. Um there

are some overlaps, but most of the industry players would agree on five different capabilities which we will see now. The first one is a metadata. So, metadata is nothing but a context about your data. So, in terms of lineage, in terms of quality, ownership, and freshness. So, this will enable your you know choose the right data source, right? For example, uh should it choose the data source

for historical you know a historical data source or a you know streaming data source to get you know real-time data. So, this kind uh you know you know decisions it can make easy with this metadata. Next comes taxonomy or business glossary. So, these are shared and you know agreed uh vocabulary or term definitions. So, this will really help the agents from you know guessing things, right? Um

like instead of guessing it can go by the real term definitions. Next comes ontology. So, ontology is a formal model how concepts relate to each other. So, this enables agent to understand the rules of your domain. And the next one is knowledge graph. So, these are entities and relationships applied to real data at scale. So, it means like typically you know if you apply ontology concepts for

a real data at scale, that's what is your knowledge graph. So, this will enable agents to navigate your real-world relationships you know in a efficient manner. The last one is metrics or KPIs. So, these are executable formulas, right? Which can help the agents to get accurate values directly instead of you know generating itself some SQL. So, that's the metrics and KPIs. So, overall you know like don't

see this uh five capabilities as just mere boxes. Rather, uh think of it as organizational lenses, right? Different organizational lenses through which uh you know, like you you um discover your organizational And obviously, uh there are some overlaps between these capabilities what is listed here. So, that's something which uh to remember. I'll not get into the tools. There are a lot of tools which offers um these

capabilities. Again, um same tool could offer more than one such capability. Um and and in a typical organization, um these capabilities may be there already, but in fragmented uh fashion. And the question is uh it might not be exposed to your agents, right? So, the capabilities are there, but it may not be exposed to the agents. Now, let's see this uh semantic layer with some with a

example, right? We'll see um you know, like if let's say customer called Bob who places an order number 123 for an amount of $240 on a particular date, and his account balance stands at $8,500, for example, right? Now, let's see some of these um lenses, right? The capability we talked about, you know, how it manifests based on this scenario. So, let's see uh the glossary and taxonomy.

Um let's say we have a definition of a customer as an account with at least one completed order, for example, right? And there is also a definition uh of premium customer saying it's a high-value customer. So, based on this customer definition, Bob qualifies as a customer here. And in terms of ontology, uh as we mentioned before, it establishes a relationship between business concepts. So, here, if you

have a customer and an order, like we can establish the this relationship as customer places order, and order in terms in in turn contains line item, right? It can have one or more line items. So, and also premium customer is a terminology and we can say that premium customer is a type of customer, right? So, that's also a relationship. And customer has an account balance, that's another

relationship. So, these are some relationships with respect to this scenario. And in the knowledge graph layer, let's say there is a premium customer definition, concrete definition available to define saying that premium customer has account balance greater than or equal to $8,000. So, since the account balance of Bob is $8,500, we can say that okay, Bob qualifies as a premium customer. In terms of metrics and KPIs, like

we might have definitions, right? Metric definitions within this layer. So, for revenue, it's defined as extended price into 1 minus discount. So, that's the revenue formula here. So, now this particular order, we can say that okay, contributed to revenue upwards of plus $240. And the gross margin for this order is 38% for example, right? So, these are facts from this example. Now, a unified semantic layer is

trust layer, right? For every consumer, whether it's a human or agent. And overall, a unified semantic layer breaks the silos and it provides a single source of truth, right? And it also brings the much needed deterministic behavior for your agents. Of course, agents are not fully deterministic, but it brings a degree of determinism when you have a semantic layer. And it also provides full auditability. And the

most important point to highlight is it also provides governed access control, right? Your semantic layer can have all your access control definitions so that there is no inadvertent leakage of PII data for example, right? It can have those guardrails in place. Now, why are we talking about the semantic layer right now? Is it something new? Obviously not, right? So, it's it's a concept which has been there

for ages. Uh it was never missing. Uh in fact, it was actually embedded deeply inside your BI tools. Or you can also say that the semantic layer was also scattered across your different wikis and conference pages, for example, the thing is the agents were never able to tap this knowledge, right? Um since it is uh trapped inside your BI layer, it was the agents cannot tap onto

this knowledge, right? So, that was the issue. And the architectural shift is to detach this semantic layer from your BI tools and create a unified layer, right? Which is platform agnostic and which which can be exposed not only to your BI tools, but also to agents, uh your search engines, or any type of new consumers, equally, right? So, that's the idea. Now, let's see how your semantic

layer capability can be exposed as an API. So, first of all, the semantic layer of an API brings or establishes a contract between your organizational knowledge and your agents. So, there are typically two patterns in which you can expose the capabilities of your semantic layer. The first pattern is a context-assisted pattern where uh you know, you your semantic layer can just simply return some definitions, relationships, and

join paths, um right? Which the agents can use as a context to create a well-formed SQL, right? Rather than guessing SQL, it can ground the the SQL generation based on this context which the semantic layer provides, right? In terms of uh these definitions and, you know, term definitions, relationships, and so on. The next pattern, the second pattern, is a fully mediated pattern where the in fact, executes

against the data source. It means the semantic layer is attached to a data source, so it will execute the queries the agent need not have to create an SQL. All agent has to do is, okay, find out all the metrics exposed from this layer and you know, just call the semantic layer for a particular metric with some dimensions and filters, and the semantic layer will return the

value directly. So, all agent has to do is just synthesize these results and reason about it, right? So, here it presents a zero hallucination um uh solution, right, with this pattern too. Now, the complete solution involves, of course, uh like when you want to expose this as an API with uh two agents, the one of the solution is to expose this as MCP capabilities or MCP tools,

right? So, that whenever a user ask a natural language question, the agent can plan about it, reason it, and it can look at the all the available MCP tools and can choose the right tool to answer this uh and finally, once it gets the answer, it can just synthesize this results. And the unified semantic layer can be connected uh or integrated with uh you know, different data

platforms, be it the Snowflake, Databricks, BigQuery, or your Redshift, or your lakehouse. We are going to see a demo based on uh the lakehouse architecture. Now, let's since the um the solution involves MCP, let's also quickly discuss about MCP. I'm sure like you know, uh many of you are already familiar with it. So, the MCP or Model Context Protocol is an open standardized interface, which actually enables

the LLMs and agents to interact uh seamlessly and securely with your external uh uh systems or your enterprise tools, You can think of it as a USB-C for AI where it provides this much-needed interoperability. So, this is uh me my wanted to just share. I mean, I I think it's true anymore. Um it's uh it's something which was floating around in the early days of MCP where

we had a lot of builders, MCP builders, right? Like you name any tool or any any platform, the creator of MCP tool. And there's a lot of MCP tools and there are less and less MCP users, but the scenario has changed now. I can say based on what I have found. Like there are also a lot of usage of MCP, but then of course there is still

we are not sure like you know where it is heading, right? In terms of its trajectory, right? Future trajectory we are not sure. Today at least it's used widely by many consumers. Since the demo involves a lakehouse architecture, so I thought we'll also share a bit about the lakehouse A very quick recap. So lakehouse is a open architecture without a vendor lock-in and it offers the best

of functionalities from your data warehouse and data lake. So it it brings in you know, features like transactional guarantees, performance and governance right from your data warehouse as well as the scalability, flexibility and cost efficiency of your data lake. In terms of the layers, you can see the different layers of a lakehouse So you can bring your structured or semi-structured or unstructured data. You can stream it

or batch it into your You could use different engines, right? Like Flink or Spark to perform this ingestion. And the data itself is stored in an object storage, right? So that's where storage is. And the data will be stored in file formats like parquet. Of course you could perform like when you land your data, right? You could do further processing of your data again using engines like

Spark or Flink, and you can even organize this data into what we call a medallion architecture, right? With a um bronze, silver, and gold kind of layer to represent your raw data, clean data, and aggregated data. And the next layer is a metadata layer. So, this is very, very important. So, this is the layer which differentiates a data lake from a lakehouse. So, metadata layer is powered

by open table formats like Apache Iceberg, which brings in data governance um and the data management capabilities. It also provides this SQL uh query capabilities on your lake uh so that engines like Trino or Spark Thrift Server or like other tools uh engines, right? Which can perform queries and serve the work consumption And your consumption layer is typically your AML or BI reporting tools, which can um

query the data through your um serving layer, which is made of Trino or, you know, Thrift Server. So, that's a very, very high-level uh view about a lakehouse architecture. So, now we have reached the uh demo, right? So, let me explain the demo architecture first. so, we have a Docker Compose setup uh with uh which represents an Iceberg So, what we have is a MinIO, which provides

this object storage. And we have Trino as a query engine, which can operate on your Iceberg uh lakehouse. We use Tabular Rest Catalog as the Iceberg uh catalog, right? Um And we are also using a TPCH data set um which has tables like orders, line items, customer, uh there are eight tables there uh representing a retail use case. Uh TPCH data set is a performance benchmarking data

set, which is quite popular. So, what we're doing is like we on startup, we're ingesting this data into Iceberg um through Trino, right? So, we we are ingesting some good amount of data into Iceberg so that this data on this data we can perform some business queries later. So, we have a Lakehouse MCP server. It actually connects to Trino. And exposes capabilities like executing on your Iceberg

Lakehouse. It also exposes you know, tools to get information about your schema, information about tables and columns and so on, right? So, it exposes set of MCP And we also have a semantic layer MCP Which again is powered by two tools. Okay. The first one is DBT metric flow, which um provides different metrics. It exposes different metrics through MCP. Which in turn connects of course through the

DBT Trino adapter to Trino so that it can perform queries on Iceberg. And the second one is Ontop, which is a virtual knowledge graph. Which also connects to Trino using JDBC. So, it can basically expose SparkQL queries for knowledge graph operations. Both this approach uses the pattern two which we have seen earlier in our slides. So, pattern two is where the semantic layer is directly integrated with

your data layer, right? So, that agent can just query only the metric. It needn't have to um you know, um perform any you know, like construct any On the agent side, like we we have two agents. Like these are nothing but agent.md files. The first one is a data layer only agent, which means it has access to only the Lakehouse MCP server. The second one is a

data plus semantic layer agent, which has access to both your Lakehouse MCP server as well as the semantic layer CP server. And this can be executed from Cloud or Copilot or Cairo or any other AI tool of your convenience. So what we are going to see is in the demo we will see the difference between you know how you know these two agents behave in terms of

you know real business So this is also available in GitHub so reference this codebase from GitHub and you can also try it yourself. Now we'll see a small demo, quick one. So this is the Docker Compose setup I have. So with all the components which I was explaining to you now, on top DBT metric flow. So these are the different components. So the for the DB2 DBT

like we have the semantic models defined for all the different tables and you can also see some measures or the metrics which are also defined, right? Um Um which maps to an expression or a particular column. Including some aggregations. And then we expose an MCP server for the lakehouse operations like to execute a Trino query or to get some information about you know your Iceberg tables and

schemas And then we have another MCP server like which has two tools set of tools like one for the metric flow like which lists a particular metric definition Uh, you can list dimension for a metric and finally to query a metric, right? So, these are the functions it exposes. Yeah, this is another thing, explain metric query to explain uh metric itself is calculated. On the other side

on the on top side, yeah, we have a knowledge graph, which also exposes some knowledge graph queries. Um, and also it defines the definition for premium customer, right? So, it defines premium customer as a customer whose account balance is greater than 8,000. Now, we have the two agents like we were talking about, right? One is the data layer only agent, and the other one is the data

plus semantic layer agent. These are agent.md files, which have instructions in terms of the capable capabilities, right? Tool calls and all that. You can see that the containers are already running for the setup. Now, we'll go to Cloud, and then we'll use the scenario one, the agent which has only access to the data layer, right? We'll ask um Yeah, first we'll start with some question, which is

um you know, it is not having a lot of business terms, right? It's seemingly okay, right? Uh to be dealt with a normal data layer only agent. I'm just asking, "Show top five nations by total order value." Let's see what happens. So, you can see that it finds out, "Okay, what are the different tables?" And for each of the table, it tries to understand the schema, and

it's based on that, it's able to, you know, construct an SQL query uh based on the schema information what it got, and so it's able to execute a query with sum of total prices and perform an aggregated query, right? Which seems to be okay. So, it gets the results. And, uh, it can just, uh, it will print, uh, you know, visualize this and this is the results,

right? So, it was able to find it and there's no worries about it, which is okay. Now, we'll put a little more business-oriented question, right? Show me revenue for premium customers for Q1 1995. There's premium there's premium customer. Let's see how it handles. So, it doesn't find anything, uh, definition for the premium customer, right? So, but it goes ahead and defines, you know, its own version of

premium customer. So, it sees, "Okay, I'll go and take the, uh, maybe, you know, top 25% of customers having account balance." And it arrived at a particular, um, value of the account balance. And if you see, it also used the revenue for the revenue it is using, uh, what you call, um, still the total price, right? You see it's the total price, not the extended price minus

discount, which typically is my net revenue. So, it went ahead and uses again, total price, which could be, you know, gross and it gives some results. But this is not in line with, you know, my business, uh, definition within my organization, probably. So, now we'll ask the same question using another agent, but this agent has access to both the data layer and the Let's see, you know,

how this deals with So, now here it loads the tools, the semantic layer tools, because it knows that business terms. So, it already used the KG premium customers with the knowledge graph query I mean the the endpoint and it got already the customer details who are the premium customers. Then it goes and finds out, "Okay, what are the metrics available for me?" And since we're asking revenue,

so it found that, "Okay, revenue is a metric, but it finds out, okay, for this metric, what are the definitions and filters or dimensions and filters it can use?" And it tries to pass all those details. And uh you know, it performs a query to your metric uh layer semantic layer, right? By asking, "Okay, I need revenue for this this customers for this dimension or this period."

So, it's grounded based on what we define in our semantic layer. And then it plots uh you know, once the result comes, it's easy for the LLM to the age uh you know, just um map it into a visual uh layer. So, yeah. So, this just you see the revenue difference between the other one and this there's a huge difference and even the number of premium customer

it produced 100 plus and here it's only 10. So, yeah. So, this is how, you know, um the diff we can demonstrate the difference between, you know, a semantic layer using a semantic layer and not using one. Um so that's that was the demo. You can go ahead and um check out this code repo for to try it uh we are almost at the end like just

to summarize the key takeaways from this session. Uh direct agent access fails systematically, right? So, if you connect your agent directly to a data layer, which can produce confident wrong answers, especially with respect to business-related queries. So, the solution is to use unified semantic layer, use it as a meanings API um so that uh and this meanings API or semantic layer can be exposed to your agents

through MCP tools uh so that the agent calls meaning rather than, you know, just simply on the tables. Uh in terms of your adoption, you can start small. You can build your layer incrementally, right? The semantic layer incrementally as you grow with your agent journey, you can also, you know, grow your semantic layer. So, to summarize, the data layer stores your data, the semantic layer stores your

meaning, and you should expose both so that your agents become trustworthy. So, to conclude, intelligent data platform is a data platform which has a strong unified semantic layer with MCP agents so that it enables business users to ask real business questions which are trustworthy answers. So, thanks for listening, and have a great day.