Define Once, Enforce Everywhere: Model-Driven Design Using Legend - Sumit Rastogi Deepika Srivastava

This talk explores the concept of domain models and their significance in contrast to data models, particularly in the context of large-scale data migrations. The speaker describes various architectures that impose complexity on data storage, including OLTP for CRUD operations, OLAP for analytics, and data lakes for enterprise data sharing. A key challenge discussed is maintaining data integrity during these transitions, emphasizing the importance of consistent attributes and policies across different systems. The presentation introduces Legend, an open-source data governance tool by Goldman Sachs, designed to help manage data models and ensure compliance by facilitating documentation, data lineage, and policy translation across various databases. The speaker highlights the benefits of a model-driven development approach, including consistency, transparency, reduced risk, and platform portability, ultimately stressing the need for a durable data asset that aligns closely with business needs.

What do you would understand in today's session is what is a domain model? How is it different from a data model? And how enriching the domain model can help you maintain the integrity of the data or any large scale data migrations. Okay? And we'll try to understand with an example Now this looks like a very typical, you know, system design in in today's world. And we developers

have made this awesomely complex and for a good reason. You can see the variety of database that we use to store our data. Right? Right from the start, right? We use OLTP for most of our CRUD operations where it is user facing. Now we want to do analytics on top Perfect. So we transfer our data to an OLAP store, right? Now if you're in a large enterprise,

most likely you would like to share your data or join with the enterprise data. That means you copy your data into a data lake. You want to stream your data over Kafka to share real time with someone. That is again your data is getting copied. That's not all. You have search engines to have a free text search and then if you want to make your queries faster,

you have a caching layer on top of it. Right? Now very quickly I just want to what is the challenge we get into if you if you have this kind of architecture. I just want to hear one or two. Latency. Okay. Data integrity of the data. Integrity of the data. Perfect. Redundancy. Redundancy but for a good reason because every choice of database is for a good Right?

I think one thing to note out here is the intrinsic value of the column or an attribute never changes. What is changing is underlying technologies, the way we are storing. That means all the constraint that you want to have on a certain column does not go away just because you're copying data. It has to be consistently applied. The challenge that we're trying to you know talk about

here is integrity. Is my data safe when I'm copying it across the world? We'll take an example here. Let us say we and we I I I I'm very sure most of the modern enterprise, you don't have data in one place and at least two places. It's pretty natural, right? Assume that you're trying to build a new data store. For a good reason, maybe you want to

put into a lake. The first challenge that you will see is there might be a drift in the whole configurations. For example, when you're creating a schema, the schema might miss some critical things. Probably you miss a column or maybe a validation rule. That is still easy to catch. The trickier one is the next. Suppose you have a sensitive attributes and you mask in your databases. What

if you mask it inconsistently? For example, you're using a star somewhere or a dollar somewhere. That is cosmetic. Worse is you miss masking in one of the That means you're suddenly exposing your data unmasked. Entitlement policies. Now, this is the very big problem in industry because there is no consistent way to apply, you know, entitlements across systems. What will lead to? It will lead to over over

restricted data or over exposed data across systems. Right? Now, summarizing the challenges out here, it's mostly we are talking about is a configuration drift. Inconsistently applying the policies across the systems, which will lead to the security gap. Assuming that you have a sensitive attributes and then it is not handled properly, compliance risk, and you don't want that to be caught into an audit for sure. And then

poor transparency because there is no single place where you understand how your data is managed or treated, The cost of migration of your data from one database to another is very high because we don't have a single spec, right? And this leads to a vendor locking because developers have huge resistance to go going to a new database because there is a lot of unknown. They don't know

what is behind the database, what all things have been applied over years. They don't know about it. And that's the challenge that we face in the current world. This is the slide we'll I'll try to cover twice, but I just want to introduce to a tool which is called Legend. It is a Goldman Sachs contribution to Fin OS, which is a Fintech Open Source Foundations. This is

a data governance tools end-to-end. It helps you define your data, map to a physical store, helps you share your data with other teams, define governance on the data, and the most important is the lineage. How is an attribute that is right from the source till multiple systems across the enterprise, how is it is used or transformed? That is the lineage that you get out of it. But,

we'll be come with more details around it, but let's now cover what are the ways to fix these kind of problems. Over to you. Thank [snorts] you. Okay. So, whatever we just talked about, right? Like it all boils down to kind of a single problem. Basically, we are trying to club our intent with our implementation. Right? And a show of hands, how many of you have actually

experienced this problem? Let's say you have a data sitting in let's say Postgres, and suddenly, you know, you're for analytical purposes, you're now required to migrate your data into let's say Snowflake. What do you end up doing? Do you create your schemas again, DDL statements, uh create your indexes again? Can I get a show of hands, please? How many of actually done that? Right. All right. Great.

Um so, yeah. So, here's where our solution comes in, right? Now, what we're trying to say is you should uh separate out the what and the why of the data from the how of the data. What that means is you should have a logical model that talks about your business domain. What is the data that you're trying to model? And why do you want to model that

data, right? The business relationships of that data. All the different aspects, the security aspects of that data, the different masking policies that you want to apply on that data, the way you want to share that data. That's all very intrinsic to your business, right? So, you define the what of your data in a single place without worrying about the underlying tech stack that is needed to actually

implement this data, right? So, the logical model captures the what, the why, how the data has to be uh you know, modeled, and how it needs to behave, whereas the physical model, it will capture where the data is stored, how the data can be optimized for a particular system. And how it can be implemented in any particular database or any data system for that matter, right? So,

we need to separate out these two. Now, moving on. Uh what that gives us, what is the benefit that gives us? Now, technologies can change, right? Data platforms evolve. Cloud vendors will come and go, but what remains consistent is your business domain and your business knowledge. So, the way the data modeling should work is that it should outlive any other data ecosystem. It should be a durable

data asset. If I tell you to, you know, migrate to a database and you end up creating a different model for that database, you're not really doing a correct data modeling in that sense. You do not need to do that. And that's the idea behind what we're trying to present here, Um now you can tell that, "Hey, great. We've been doing this since ages. We have ER

diagrams, entity relationships. We model the data. What new are you talking about?" Accept accepted, but we're covering more than that. What we're saying is once you have your logical model, there are these special constructs that you can apply on those logical models that talk a lot more about just the entity relationships. So right here, I have given some examples. These are some internal examples that we used

to define our data. So as you can see, there are a lot of annotations here. So at the rate doc, right? Now let's say I create a big schema, multiple tables, multiple attributes, and then my manager comes and says, "Hey, where's the documentation?" Now I end up creating another file writing out all the documentation over this field does that, that field means this. No, we should not

be doing that. We have come up with an annotation where we can define the documentation and the meaning of every attribute in the model itself. Anybody who needs this data can come to this model, read what attribute this is, what it means, and just subscribe to it if they need it, right? Similarly, we have searchable. Now I think Sumit mentioned about indexes. Now these databases have been

living in your enterprises for what, 5 years, 10 years, 15 years? How many people have actually gone and, you know, ran that create create index or alter table statements? How do you get to know that what people have done? This is one-stop place where you can manage all of those things. For any indexes that you need, you define them using the searchable attribute. For any primary key

constraints that you have, you use the state key. Data privacy. Now, we are coming from a fintech firm and data privacy and sensitivity is supreme to us, right? So, um there are a lot of GDPR and HIPAA requirements that a particular, for example, PII information, any SSN, any passport number, whatever we are capturing is highly sensitive. Great. I captured it as sensitive in my system, but when

my consumer is consuming it, they end up exposing it. How How do I solve that problem? I want them to use the same model to consume the data. And this is where I have defined that this field is sensitive. It has a data privacy class of DP10 for your compensation, it would be DP30, and so on, things like that, right? So, now everybody who is consuming this

model needs to follow the same data privacy classification. Um there are a lot of challenges when, you know, uh in production data, as a developer, when you're trying to work on this data, you need to sort of import this data in a lower environment. Let's say a dev environment or a QA environment. But there are sensitive fields which obviously you cannot import, right? So, what do you

do? Uh you might end up masking, uh you might end up encrypting certain fields. All of that can come here. So, you define your masking policy on certain attributes. And whenever you do a data migration, those particular attributes are automatically masked for you. So, yeah, just uh some of these annotations that we are using to talk to sort of define how uh the data is and how

we want the attributes and everything to behave. Now, Great. Now, we have defined the model. Our model is very rich, awesome. What do we do now? Do we go and manually implement this model in all of these different databases that we are interacting with? No, that's not the point here. So, there comes our policy translation engine in the middle. What this does is it parses through all

these annotations and then it tries to generate executable artifacts out of these annotations. And these artifacts can be related to your physical systems. Whether you are interacting with Snowflake or with MongoDB or with Postgres, any database. You can have mappings specific to that database, configure rules specific to that database and the translation engine will automatically generate those artifacts for you. So, it can automatically generate as you

could as you can see on the right hand side, it can automatically generate those DDLs for you. It can generate those index statements for you. Um it can even generate a nice JSON runtime documentation which we're actually using, you know, as a context feed to our AI systems and things like that. So, all of this is auto-translated and auto-generated for you, right? So, now moving on. Um

just to cover the benefits of what I uh talked about. Now, what you get by using the model-driven development? Consistency. You define all the attributes, you define all the rules, the entire behavior at one single place and you implement it any data system that you want. Transparency. I don't need to go and ask somebody that, "Hey, who owns this particular attribute? Where is this coming from? Which

system is sending it?" That ownership exists within the model itself through the at the rate owner tag. Compliance. As I mentioned, any sensitive attribute, any attribute that needs masking is already covered. Similarly, we have um you know, uh roles. Uh basically, when we want to give access, we we are we can also define roles on certain attributes or certain tables and we have an we have hooked

it with our entitlement system that ultimately defines who get access to which data. Reduced risk. As I just explained, faster delivery. Now, you don't have to sit and write your DDLs and your indexes and everything again and again. Just come to the translation engine, hook in the end database system, and then just get started, right? And of course, platform portability. This is something that we have personally

experienced. Like our data uh sitting in and then when we had to migrate to Snowflake, we literally did not have to do uh a lot of uh you know, configurations and everything. Just uh immediately run the translation engine and things are just auto-created for Awesome. So, here are the key takeaways uh that we want to share today. Uh defines once and deploy everywhere. Eliminate any kind of

configuration AI-ready data ecosystems, because that's enriched model can serve as a one-stop single source of truth for your context generation. And governance travels with the data. You don't have to worry about your consumers not enforcing the same rules or the same governance. It's all embedded in the same data model itself. Any questions? So, this is a wrapper layer that integrates with all the databases? It's a It's

not a wrapper layer. It's more of a mapper. So, basically, it's an engine. So, you have your model. So, uh what Summit talked about the Legend platform, it's a platform where you define all these models, and then the It's It's kind of a mapper or an engine that will read that mapping, read those annotations, and generate those constructs based on the database. Can this constants be specific

to data? Specific to databases. To do it manually? The modeling is done in the Legend. Yes, the modeling has to be done manually. I mean, write that code and You have to list No, it's a It's a click of a button. So, you kind of model Are you asking about modeling drag and drop? No, no. Legend uh you have to It's a code-based platform. UI on which

and craft your model because it does not know your business domain. So, you can then understand the business domain, but you can craft the whole graph. So, define the business rules and then Yeah. Yes, and then what we did is declare that the data artist Yeah, I can open that slide. Yeah. Yeah, so the data case is Sorry. So, Legend is a tool that Goldman Sachs has

built for data governance and they have open-sourced it under FinTech open-source foundations. It is available as a hosted instance, but only for the testing purpose. But, if you want to use in the enterprise, you can take the binaries and actually deploy it for your own use. So, what it helps you do that? It helps you define the logical model. Logical model is very close to your business.

What is what you define out there. You can enrich that model with annotations which are pre-provided. Plus, you can handcraft your own annotation depending on the business need and give a meaning to that in the output. It helps you define a physical model where exactly physically the data is available. That you can model out there. Now, you can create a mapping between your physical model and the

logical model. And you can actually expose your data from one or more databases, but your face of your domain is always the same. Client always sees the exactly the same model. Depending on what use case, you can choose the mapping that you want to have it. Right? You can build governance around it. If you are sourcing upstream data, you can put your rules on the What I

believe the data should be, is it passing all my validation rule before I can consume it. You can expose you can analyze the data by drag and drop and doing aggregation Once you're happy with your structure, you can expose it as a service right on the platform. So, suppose you drag and drop in group by, you liked it, you want to hit the single button, you can

expose it as a service and end users can start using it. Key thing is what is exposed is your business domain, which should not change over time. Your underlying technologies can always change. That is what it will And the final thing is, not final, second last is discovery of data. If on the single platform all enterprise data are listed, it's a single place to discover the data

that you need. In one of the biggest problem in in the industry is like, you know, we don't know what even our own enterprise is producing. We have so many divisions. We keep We talk to people and get it, but over here it is all discoverable. You can discover it. If you like it, you can request for an entitlement, get that element and start The last and

most important thing I would cover is lineage. Suppose there is a stream of systems right from the consumer till it goes to balance sheet. There are multiple systems. How an attribute from any systems till the last, how is it used, how is it transformed, how is it used in different logic, is all covered in the lineage. You can always understand how the data is consumed across the

systems. You can also find out if I get delayed, what is the ripple effect? Not the worst level, second level, third level. You can go to end level and try to understand. Right? I would I would recommend it it's all over the internet. Just look for legend. Hit. Try to download, try to read about it. It needs an installation of our enterprise. And if you like it,

I would recommend to use it. It's a logical model. What about the projected data? It applies to schemas. See, when we say logical model, we always think it how is my business seeing this information as. See, I I'll tell you, if you have a database, you can and you're using a mobile database, you will always think like a document. When you use analytics, you will think all

about group buys and large filtering. But that These are like one of the aspects of the business. How does your business looks at How does the business see your data? That is what your logical model is. It might not be exactly how you store So, that's the difference. Because what we have observed is the business structure does not change drastically. You can always add to it, but

they don't change very heavily. With the new technology coming in, databases change. The underlying technology changes, but your business does not change that often. That's what our observation has been. So, if the data is not quite that up to date, what happens then? Correct. How do you manage that? So, basically, let us say that you have let's say trading platform, right? Trading platform, you get a trade.

You have a logical model of the trade. Now, there is a compliance team who wants to monitor your trade. So, what it will do is it will import your and use the attributes of your model to write compliance rules. That lineage is already captured because it's all in the legend ecosystem. And also, just to add to that, like if you notice here, right? Like the owner app.

So, let's say I I've taken an example of like name here, but um let's say you have any other attribute, right? And different teams can be sourcing data or is sending their data into a common ecosystem, right? Let's say I'm a consumer. I'm consuming this information, but I see a drift in the information. Like I see, you know, that information seems to be wrong. Now, as a

consumer, who do I reach out to? Now, I using the owner app, I know that, okay, this particular team is the producer of this data, and I can always reach out to them and say that, hey, can I can I understand how you're sending this data? That is one. The other pieces we have noticed that, you know, there are a lot derived attributes also. So, for example,

I'll just give a very basic example, let's say your full name. So, your first name and last name together can make full name, right? Now, people may be actually having this logic in, let's say, their code layer or Java layer and whatever, but as a consumer, I am getting this attribute independently. Full name is independently. But I see if I see any issue here, I can come

here and try to understand that, okay, I see that this is a derived function. And right here you can see, the function name also, like it's a derived data full name. So, I understand that, okay, this is a derived value which is coming from these source attributes. So, that is another aspect of lineage that we mean, that where is this information traveling from and coming to us.

So, that's the second piece of lineage. Similarly, like we have a lot of validation rules also. to give you an example, I think I might have here. So, yeah, um, yeah, you you can notice it on the name. Like, we have a validation saying not blank is true, meaning that it can never be blank. So, even if I'm in or even if I'm in Snowflake, the moment

I'm trying to ingest this data, if I send null, any of the downstream systems will send an error. So, I I really don't have to, you know, care about that, hey, is every system implementing it correctly or not, because I have just defined it here and every system will sort of take that validation rules also. So, Wait, wait, wait. That's fine. I Wait. That's fine. Can we

get a mic for him? Yeah. So, basically, right, where exactly this data length will match? For example, in the architecture, right, for example, in SQL Server, if you have 50 characters, that same length it will not fit in Snowflake because in Snowflake it occupies as bytes. So, whereas in SQL Server, it will occupy as characters. So, the same data might not work when we create the DDL

with 50 Snowflake. So, where exactly those things will be handled in your architecture? Yeah. So, let me ask you, like like exactly the kind of the statement that you made, right? Like when you are thinking about storing this data, you're immediately thinking about database, like varchar 50, and you know how What we're trying to say is let's not think about it in that sense. Let's think about

what data are you storing. So, if you if you don't mind, you can give me an example, maybe. Yeah, for example, any any Let's consider name only. Yeah. So, let's say name. So, name is a data that we are storing. Now, in our code world or Java world, we would, let's say, use string maybe for name, right? And then we know that, okay, we we don't have

control over like how many characters really a name might be and things like that, right? So, when we are translating it into our policy translation engine, there are two ways that we resolve it. Either we ask the developers that, okay, hey, do you have a configuration in mind for that particular database? So, let's say you have your Postgres mapper, there is a configuration which you can override,

and you can say that, hey, Postgres, please use 50, but Snowflake, maybe please use 100. I'm I'm giving an example. That is one way. The other way is actually going and looking into your data and then trying to resolve that what is the max length and then adding a little more to it for buffer of the future and then trying to come up with those rules. That

is when you're doing an existing data migration. So, that's the other piece of it. Now, one thing the intent here is any shared governance rules or any shared policies need to sit here, but we completely understand that you know there are certain rules or constant that are very database specific also, right? So, that extension is allowed. Like in our engine, we have our base policy set up,

but you can always configure certain database specific policies also if those are needed. Okay, thank you. What if it's a different database? So, we have a choice of the database this your policy. Yeah, yeah. Basically fish. Oh, this is Why do that that? Fresh milk man. So, there is a hosted environment, of course, but it's only for evaluation and testing purpose, not designed for the production use

case, but it is also available as a binary which you can download and host it on your own. It's not a SAS. It's a product we can download and use it in the enterprise. >> It has to be still on the Exactly. So, that IP file This Arab artist I have to mention that the field of Austin Colorado Yeah. Correct. It cracked down. So, that Uh, I

will still use my data Yes, yes. It is agnostic, so it'll create a connection. For example, you you have to draft what how to connect to a Snowflake or a MySQL, those concepts you have to define and then you will start using it. That's the mapping part of the world. So, do I have after these does order by or once I got got No, no, it is

the APIs are also hosted out there. So, if you want to expose an API from the system, the system has to be up because you want to share this API with end users. So, so so so you can think of it like a code, right? So, like for example, uh you have your code, you check it into GitLab, and then your code is sitting out there, and

then you obviously go and deploy the code. But for maintainability for usability for future purpose, you always keep that code in a good shape and form, right? And any changes to the code are also version controlled. So, this is a similar concept that, you know, you maintain this in sort of a repo which is like legend itself, and then anybody coming and changing it, it's all Git

controlled, version controlled, so you know who changed what. Let's say tomorrow somebody deletes a column and nobody knows who deleted and all of that. All of that is captured here. So, from that maintainability perspective, it is always good to have it. But I think to your point, once those DDLs are created, and once you go and deploy it in any database, it's up and running. You may

not need to come back, but the intent is that you should come back. Um What's all Yeah, so basic databases that we use, we have sort of tried to, uh you know, incorporate, but you can extend it, Folks, we are out of time, but we have a booth on the ground floor. If you want to have any questions, maybe I will have to take it there as

Thank you. Yeah, got it. I thought this. >> [music]