James Bowkett: The 12 Factor App for Data

This talk, presented by James Bulett, focuses on the principles of resilient data architecture using a concept called the 12 Factor App, adapted for data-centric applications. The speaker draws from his extensive experience in data-centric applications and emphasizes the importance of treating data structures as code, advocating for better management and traceability. He discusses various modern data architectures such as data mesh and data fabric, explaining how they can enhance resilience and facilitate continuous delivery. Bulett highlights the significance of separating data from logic, emphasizing strong data types, and ensuring testability and auditability in data operations. Additionally, he addresses the importance of treating data as a product, promoting clearer communication of data quality, service levels, and the operational implications of data management.

[Music] hello we're ready for our second talk for the day on the data track and I have James bulett with me who is uh an experienced professional with more than 20 years uh doing data stuff and he's going to share his wisdom with us on um the um foundations of of what makes data architectures resilient and how to make the best out of your data he's working

at open credle um a consultancy that um provides data platform and developer platforms for organizations uh James without further Ado over to you to take the stage great thank you very much and uh welcome everybody I'm glad you could join us so the uh the title of the talk is the the 12 Factor facts WP for data so I'll start I'll kind of tell you a little

bit more about me and and about my experience um and why that how that's may be relevant to the uh to the talk so uh as we said in the the intro my my uh my day job is a as technical delivery director at a a a Hands-On technical consultancy called open Credo um so what we do is we do you know the three things that says

there sort of cloud native architectures platform engineering and and sort of data and data mesh kind of platforms and the area of expertise that is is most um is most relevant for for for this talk is the the data and data meas Solutions and I've spent about 20 something years of of of my career just in and around data Centric applications basically where you draw a database

in the middle and you just have loads of lines coming in and out of it so it's kind data getting loaded in and data getting pulled out and doing something interesting for it and probably stuffing it in another database somewhere elsewhere um so if I think about that career in data and uh I think about all the different kind of you know technology is is techn not

many people tell this but technology is a fashion industry yeah so so um different architectures come in and out of fashion so maybe you've got a a Lambda architecture or you've got a a kapper architecture um maybe you're doing a medallion architecture you know with bronze silver and gold or you're a a bit more even more modern you know you've got a data lake and a data

or a lake house or maybe you're one of the one of the cool kids and you've got a data mes and a data Fabric or a data fabric so all of those different kinds of architectures and there you know Snowflake architectures and and star schemers and things um all of those different architectures and everything in between they all share one modern requirement a nonfunctional requirement in the

modern age is that they have to be upgraded in a safe and controlled manner they need to be deployed quickly and they need to be testable and well tested now this is what our stakeholders demand of our software and quite rightly so because that's what they're starting to get used to you know um with the Advent of microservices and and uh proper continuous delivery this is what

they've come to expect this is how we deploy modern applications uh but the problem that we generally see is that actually we've got these great big databases and these data architectures um uh that that stop that that sometimes become a problem become a roadblock to continuous delivery so how can we add that resilience to our data Centric deployments so the world is increasingly data Centric I got

this great statistic off the internet so it obviously it must be true um even if we don't realize it the amount of data that we're generating so there's this stat that says each internet user is personally generating 1.7 megabytes of data per second you know you think about it yeah you're on your phones you're on Facebook and Google and maps and WhatsApp and you've got adverts and

this and that and behavioral analytics and all of this data is getting collated in harvested if you've ever ever looked at the the you know browse through those cookie policies as to where your data is going your data on that one page is going to several hundred different cookie providers and they're all kind of generating data and and and doing things with it and who knows what

and that that's that's a subject for another talk because that's that could be a scary world so there's also this other really great quote that I love it's a guy by a guy called Steve McConnell and he said that the problem with quick and dirty is that the dirty remains long after the quick is forgotten now hold that in your mind if you think that the norm

you know normal operations is going to be dealing with terabytes terabytes or maybe even petabytes of data and maybe if you've got small problems or Nickles with that data because you've done the odd quick and dirty fix and let's face it everyone loves a quick and dirty fix you always have to do it sometimes you have to do it but they have to live long they have

to live long enough the is with data is that they live longer in that data because at terabytes or petabyte scale those small little problems that you're coding around then begin to multiply out and they sit there they're set in concrete unless you're going to do something about them so it's much cheaper I forget what the the theory is but it's much cheaper to solve bugs and

solve issues before it's left the developer's desktop rather than after it's gone into production and you've loaded petabytes of data because you're changing data and all the while that you're changing data and loading more data your problem is getting worse and worse it's becoming exponentially more hard to change so let's focus on doing things right first time if we can so if I think back to my

career um I remember when I I first read about the kind of the 12 Factor app um and that really helped me mold and helped a lot of people mold they think about Cloud native and server side development especially in terms of kind of observability and deployment you know one artifact several deploys this kind of thing so we thought why not have a similar set of guidelines

for the world of data Centric applications a nice sort of book of 12 things that that actually that we can uh we can think about and we can kind of sign up to and they're nice and easy and hopefully they feel nice and familiar as well as sort of almost like Common Sense things like oh yeah I really should about that but it's something you can put

maybe on a junior developer's desk and kind of go hey look think about these things these are the things I'm going to be looking for because they will save your bacon a little bit later you know future you will thank today you yeah and as I said before data heavy applications have never I've always really suffered from uh dragging this great big database with them um so

this is a guide for Designing data Centric applications and also some recommendations for how to approach some of the kind of socio Tech technical challenges with becoming a a data Centric application a data Centric organization the the 12 factors I've divided them into into four areas Unfortunately they don't divide cleanly into the four so it's a little bit lumpy so I do apologize for that the the

the database purist in me would love them to be three in each but uh I wasn't able to do that even if I bend my uh my definitions but um I still managed to get 12 so I'm Happy from that perspective at least um so diving into the first category then is sort of architecture and design so the first one on my list is uh data structures

as code if you're using a structured data store don't rely on the UI to change your structure you've got to use tooling um if you if you do use the the UI to change the data structure you've got no traceability between the data structure shape and any application changes that rely on those you've got no you've got no thread other than you the Flesh and bone in

the middle who clicked you did The Click Ops on the on the UI um you've got no Common Thread you've got no one artifact that you can um promote and deploy because the same piece of manual uh editing has got to done to other uh to the other environments as you promote um you know there are tools to do this you know Flyway sketch liquibase terraform um

terraform is not brilliant at this but and terraform is great at many many things um but it is at least infrastructures code they are good things use them you know the world has moved on everything should be code not just infrastructure or Secrets or anything just everything should be code so for the sake of your cacy and those of your fellow humans don't make manual changes to

the data structure ever and I like to include server or database configuration in that as well because uh the database configuration things like isolation levels that can actually change your application Behavior so you want to make sure that is absolutely locked down as well so you can create a an ephemeral container a little Docker container which will boot up migrate the database as you know as code

you know and the container has come out as the docker image has come out as code as part of your build process so it migrates the data structure logs the result and then it dies and it's super simple it's super super simple um and it applies to big data source not just relational because you can take that that config and make any config changes as code as

well and it there's really no no excuse not to do this and it just creates a mess if you don't and it's very cheap to do and as I said before future you will thank today you so moving on append only data structures so one thing that I I probably should have said in my bio is is that my background is in finance as well now U

Finance you know it's guilty of a lot of things um but we we really like a pendon data struction I know we don't have the Monopoly on that and we didn't invent them or anything like that but actually it's used quite a lot especially in kind of things like double entry uh bookkeeping and things like this um but if you have an append only data structure it

gives you the full history and it allows you to time travel through that data you know what is it that I knew back you know a week ago versus what does the data look like now yeah um so something to bear in mind is that is that the date that the data is loaded is not equal to the date that something happened in the real world so

if you take maybe it's a transaction maybe it's a user click maybe it's something happened in in an upstream system the date you loaded it is not the date that it happened because also you may want to restate that data if you're doing an append only you may want to restate that data that you can actually load the same data for a business date or real world

date you want to load that in another time because you want to maintain that history because you allow that time travel go back in time now why important if say for instance so I I set up a hedge fund years ago and one of the things that we do is we wanted to back test back test the model uh run a simulation what did I know at

that point now if I went back and I changed the data afterwards if I mutated that data afterwards and I ran that model scenario again I'd be then skewing that model based on what the data really is but it's not what I was able to do at that point what did what did I know at that point and as we move into a world where models are

going to be doing everything from uh you know uh chat gbt to uh figuring out you know how best to cook my dinner from my intelligent fridge um the the thing is there's going to be models in everything so actually getting your data house in order and understanding this distinction between those is going to become really important because then you get a a thing called Model over

um and then the model is actually kind of weak and brittle around those around those pieces of data um so if you you do have an append on your data structure then your your data always leaves a footprint so it makes later debugging a lot easier you know if you if you have a a new piece of data coming along and you've overwritten the old one you

don't know what the system was thinking at that point and so therefore you know you can't actually then uh restate that data you can't do anything with that data you're like actually that that piece of information is now lost so um this is another example where future you will will thank today you now moving on um optimizing for uh access and retrieval now the value from data

is not just from collecting it and hoarding it the value is what you do with it the value is getting that data out again if there's not all we'd be doing is we would load all this data and we just stick it in a hole in the ground there there's no point otherwise if we're not going to get it out we're not going to do anything from

it there's no value in just collecting it the value is in actually what you get out of it you know reports display aggregation models but modeling data naturally is not always the best way of doing easy and quick retrieval you know dis is mostly cheap so if the data won't change denormalize that data you know you I'm talking mainly about relational stores here but you know store

those dimensions on the fact table if it's not going to change um because that's especially true if you're following the append only data structures because then you know what the reference data was at that point you don't need to keep a separate copy of the reference data and things like that and do expensive joins and stuff you actually bake it into that row and you've got everything

that you need at that point um alternatively what you might want to do is provide summary cubes ready for Access now the tradeoff that you've got for that is that any bugs in that Upstream any any data that's coming through from that Upstream will then need rixing in any Downstream data cubes or summary uh summary uh data structure created um now there are some systems that are

optimized for rights but you still need a plan to get at that data so you take something like S3 so S3 is everybody's Golden Child hey we'll take the data out let shove it in S3 well awesome great idea S3 is a great technology and you can put a theer over the top to make it look like a database place all good but you've got to be

mindful of the schema of that data underneath as well you've got to be mindful of how you're going to get that data out again otherwise you've just created a graveyard and you've just sent that data off to its death so um sorry that wasn't was very cheerful um but so I shall move on uh quickly over to um to separating data from logic so this one is

a little bit of a um this one's a little bit of a a sore point for me because I I worked on something that that didn't have this and it's uh it can be pretty painful so um we'll spend a little time looking at some examples on this so uh one of the things that good software Engineers know a lot about is leaky abstractions and allowing your

abstractions to leak out of a certain piece of code or a certain component in your system um so you you don't want that that's a bad thing now separating data from the logic you if you bake those two things together if you make them intrinsically linked then that piece of logic that decodes that data has to then be replicated in everywhere that that data is read so

let's have a little look so uh uh this is a this shares a remarkable similarity to A system that I worked on about 10 or 15 years ago uh so it was in SQL server and uh we had some uh company values and we had a start date and an end date so start date was 1753 and the end date is uh 999 you de deck 31st

99999 um now these were the largest and smallest dates that you could represent in SQL Server 200 I want to say 2003 um now anyone coming along looking at that go W that's that's a really long validation date that's like almost 8,000 years now is that a Bonafide in date and start date or is it some kind of special value now you don't know from just looking

at the data and so therefore the code doesn't know either so any piece of code that accesses that needs to have uh some recollection or some knowledge of what that date is um so the Boolean logic also won't work on the end dat col you know if it's still valid you know if it's still valid in the end of the future leave it as a null null

is a good thing you know use that um so if you want to get all of the current records you might put a a guard condition in there it's like well where end date is null or end date is December the 31st so you've already got a little bit of technical debt just in a little bit of uh SQL there without even knowing that that's the the

the final value the Special Value the magic value um let's look at another example um so I've got some uh some contracts here so as I said I was a I was a I was a finance guy for a while so here we've got uh some kind of contract here so the it might be canceled because it's in a breach or it's it's the cancel the uh

the the the contractor has actually matured now um when um so in that bottom example what what if both the cancellation reason and the matured date or the matured is not null so if I want to find all of the contract breaches you know where the cancellation is a breach that's great find that but then is that row there is that a contract breach or a matured

contract don't know so that means I have to bake that logic everywhere I'm looking at this you know it's another leaky abstraction um or and so let's at another example so so there are many different ways to interpret this uh particular one of the um so consider this this is a this is a a a set of Trades and you've got an option in there and you've

got an equity now um if you're warehousing trades you probably want a trades table that's great that's a good idea but the thing is not all trades look the same in finance so you might have an option you might have an equity and they they will have different um they have different uh uh qualities they have different kind of facets of data they entity relation model it's

different now if they're shoved together in the same table sometimes what people will do will reuse a column for something else okay or it will only be null under certain conditions so take this for example so if you've got an equity if you've got an option you'll have some kind of underlier you'll have an equity or a bond or something like that you can uh price and

you can buy at that price underneath if you've got an equity you don't have an underlier because that's not how they are structured now this is a real problem because I need to know when I'm accessing that if uh have I got another slide no um I U what it is I'm looking at what is in the instrument type depends on what I what columns I look

at later into the row you know now consider this what happens if the instrument type changes I've got to update that data and I've got to update all the places in my logic what about if we start trading swaps what are the new values possible in the instrument enumeration so it's not just in the database table that's in the logic as well and you know what swaps

are a wee bit more exotic as well they might need more columns now in a relational database this starts to play real Havoc with the query Optimizer as well so you'll end up with chained rows uh change Pages rather if you've got if you've added to the data structure over time and not updated all of the not reloaded all of the data so that that drags on

your performance um the query Optimizer will take sampling as to where the data is as to what the optimal path is when it's trying to find the data and if it's sparely populated that's something like that underlier price is sparely populated it's not going to know where the data is and so you're going to it's your your performance is going to suck so if those values change

over time you'd best hope you can track down the code that generated the value in the first place you know but all of these examples mean that you either need to bake some logic into a view or into everywhere that the data is queried and that's just the path to the dark side because you've then got to track that down as well which is painful moving on

sorry I've spent a long time on that as I've said it's a it's a it's a particular hobby horse of mine so strongly type your data FS now this is just sort of good discipline really you should be strict with the kinds of data you let into your data store and that allows you to maintain the data quality if not it's yet another leaky abstraction yeah so

this applies to no SQL stores uh as well as stores with SE uh with schema as well you know so what if you store a number in a varar column or a string column yeah the sort will be based on your collation on something that's actually within your database rather than it its natural ordering of uh numbers yep so it's another leaky abstraction and worse that's being

controlled based on the uh charet or the the colation of your database as well um so this is especially important if you're going to follow uh number 12 as well where you're treating data as a product um because you want people to be able to selfs serve and kind of understand and contemplate your data um now if you've got that kind of varar in a if you're

storing a varar in a string column you're going to need to convert that column wherever you access it you may also need to convert it when you join it or filter it um so just be consistent be consistent of your data types and your Precision across the tables especially if those are uh columns that you're joining on um and be consistent with your precision as well you

know is it a double is it a real is it a Char because otherwise that will leak into your retrieval logic as well um so if we look at an example so I was talking about sort of stuffing numbers into a varar field so that's a string column there the user ID there now if I've got that table there that has user ID as five and we

read it as five but it's the string value five then maybe I've got three different processes in three different languages here you know I've got Duke and we've got the goang Gopher and we've got the p py now each of those pieces of code has to pass that user ID and turn it into a number now that's just code that shouldn't be there and code that shouldn't

be there is technical debt so because the design choice or the lack of design choice is leaked into all these different code bases you've instantly cre technical debt as soon as you release that uh user user table with the with the string column yeah now this is a fairly nauy example but there are other examples out there and there that this is just something to be on

the lookout for because what seems like a fairly innocuous thing in a poll request turns into technical instant technical debt on the part of the retrievers and also it means that code isn't behaving the way people expect it to behave either so that you know nobody likes surprises in that way um just looking at time I'm probably taking too long um okay quality and validation so we're

coming up to about halfway points here um architect for regression testability so we talked about doing continuous delivery regression testing is a prerequisite for that you need to be able to make sure that you can say hand on heart that the um the data that you're um that your pumping out or the new version of that data is just as good as it was before um now

there are some ways and means to do that actually so making it friendly for regression testability so let's take a kind of a a bit of a kind of a a fairly vanilla example here I'm loading some data into prod now if I want to then look at uat for a new version quite often what will happen uh especially in the banks is that they go okay

right we'll do a dump of prod or we'll give you the latest backup of prod which might be last weekend it might not be the the nightly or whatever it is and then you run your new version against that yeah it's okay but you've got to ask for someone else to do something as soon as you ask for something else especially in a bank that will drag

on your Dev experience and your Dev uh time yeah now a much better way to do this so let's reset so we've got our data coming prod now consider putting an immutable stream in the place to carry the input you know um maybe something like Kafka I'm a big fan of Kafka so if you're using Kafka you might be able to use red panda um but you

could use uh Flink or Pulsar something that doesn't change so you can make guarantees about that data now then what you can do is that you can listen in on that data and load it into uat at the same time so you can keep your uat purely in sync with prod um now this is great this means that actually you don't have to wait for somebody else

you've got the data that you need and then you can even um act on that data uh in uat and go Yep this is fine I know exactly how it's going to behave in prod and and then you just promote to prod now actually there is another alternative you can do blue green straight into prod now that is a little bit more Brave and requires a little

bit more mechanics I did a I did a conference talk about that a couple years ago so um maybe have a look at that if you're interested or or hit me up on LinkedIn or something um but you need to make sure that all of the processes that look at the data understand that there can be a blue version of your data as well as a Green

version of your data um but anyway uh let's not go into that now um this one's really important as well in the spirit of kind of continuous delivery it's tracking changes in your test data now we all get given a data file and we go Ah that's great you give me a data file I'm going to put some test cases into it I'm G to bake it

into the into the data and then I could have run it up in a unit test and just check that my code works with uh all these weird and wonderful um use cases that are in my jir stories awesome the right thing to do but if you just take a breath as soon as you um are given that test data check it into Source control when it's

just clean and it has no changes applied to it reapply those changes elsewhere you know as you add a test case if you add a unit test or whatever it might be as part of that check-in as part of that uh pull request do the changes to the underlying data as well because what will happen later in the project someone will go at the Upstream system go

ah right okay that data that data file I gave you before I need to change the format because I need to add a column into it or something like that go all right cool that's okay if you've changed if you've checked in all of your Deltas into Source control it means you can take that new file and you can just um apply all of those changes to

that you've got you've got some kind of lineage with that um and it means that actually then you can apply your test cases to a new data file which will give you extra resilience or extra confidence in your releases later so switching to a new category um audit and explainability so we're on number eight so um so there's a couple of metadata ones in here data cataloging

um now spreadsheets don't cut it anymore and there's loads and loads of tools out there so there's amonson Atlas open metadata um use them you know if you don't if you have nothing think spreadsheet is a good starting place but there's no audit there's no change control with it if you use a system like open metadata then you can access that data uh that metadata rather programmatically

and you're then at a point where people can start to search that metadata as well and then it means you don't have to talk to people let's face it with techies we don't like talking to people so and because the reason being is that that slows us down every Interruption costs us extra time doesn't it so then uh making sure that people can self- serve that's a

really good thing um ideally you want your cataloging tool to support branches and release tags so that the catalogs metadata can be altered as part of an atomic release role out so as you promote between environments you're actually promoting the catalog information as well you know really kind of keeping that as one artifact several deploys yeah um uh alternatively what you might do is you have a

a separate instance of your catalog for each shared environment you know you might have a prod a uat or a Dev Um this can all be supported by Tooling in a in a a CD pipeline or whatever it might be um but you really got to think about how that roll out works and how that metadata is going to going to use uh just looking at my

notes here um um for additional points track the lineage in your data as well so it's needed for auditing and compliance you know if we're going to have explainability in models we need to understand why our model was uh sexist or racist or any kind of had any kind of prejudicial behavior so being able to trace where that input data came from that's going to be super

important I think that is a really coming uh that that is a coming requirement um and in fact sort of some kind of law enforcement actually requires that already um so the the lineage is more complex than data source a goes to data source B the field level or even at the row and field level can change as well so some some records they come from you

know they Munch together from several data sources and sometimes they say oh well if it's if it's a Tuesday then I'm going to take the data from data source a but if it's a Wednesday I'm going to take a dress from data source B so therefore just saying that data sources A and B go to C that's not good enough your individual records are going to have

their own kind of lineage which this all helps um when you need to restate data because you can walk that tree as well um this is another one that I have another uh I have another talk on so if that's something that you're interested I can I can direct you to that there's some there's some Talks on YouTube on this um so please get in touch um

and the other thing to note is that the data lineage is not static over time that might change so actually your catalog or your lineage catalog will change over time as well so you need to be mindful of that if you're going to go back if you're going to do time travel and fixing old bugs um code traceability so this one I don't I've never seen anyone

do this properly to be honest with you but the data that tracing the data that's loaded back to the code that loaded it that generated it that's super important and it's super easy as well you know you can't just do it time based stamp the records with it um because you want to be able to then uh Trace that back you know for several reasons maybe there's

a regul reason explainability in models audit fixing bugs in up Upstream systems covering your ass with Downstream system and say hey look this this this came from that lot up there and and we we haven't changed our code yeah maybe you want to roll back and roll forward and you want to see what the uh what the implication is of that as well but it gives you

a Fighting Chance of debugging an issue because actually the code that's in source code today in Source control today may not or most likely will not be the code that that loaded it you know few months ago few days you know few days ago few months ago whatever it might be so code isn't static um and but the data kind of is so you've got to know

what code was code version was that that that generated it in the first time um right moving on to consumption now um so this this one is is kind of um this is all tied into sort of some of the more kind of the the data mesh and the data product world of of thinking is is we want to make the domain clear to the reader we

want to separate data from the inside data that's private to us you know maybe it's code metadata like the the um the code version that that loaded the data or generated the data from data in the outside data that other people can access that we allow them access to yeah um we want to make that domain clear to the reader yeah because again we don't want to

talk to them at all and unless it's really necessary in which case yeah we'll never meet in and we'll do something yeah but don't disturb me I'm coding right um and we want to have both policies for both both those in in data on the inside and data on the outside you know uh retention periods because you'll have slas to data on the outside but the stuff

on the inside that's yours yeah you want to keep that um now one thing is if you don't Define apis you know what some people do is hey it's okay let's create a database link you can come in you can you can create data now don't allow external uh systems directly into your data sort just don't do it because if you do congratulations you are now married

may you may you have a long and happy life together I mean marriage is a fine institution you know as my wife tells me um but between two consenting adults absolutely between two databases is absolute hell because if you do get married to another system and you allow other systems in you're you've got unnecessary unpredictable load on your platform you've got a risk of Deadlock you've got

it's difficult to audit difficult to secure um and it's difficult to evolve your own schema as well because you need to sync your release schedule with their release schedule and you're like why am I Consulting somebody else I did them a favor and now my releases are at risk because of them so no good deed goes unpunished as as if you've ever worked with me you'll hear

me say several times a day um so your SLA guarantees are affected by the coupled system so just don't let them in provide them with an API some kind of clean AP for getting at data um you want to Define slas and slos for the data as well your new you need a kind of a new product mindset to keep yourself honest on those things you know

and those are slas for accuracy and quality referential Integrity the timeliness you know how how quickly you going to publish that data after it happened in the real world if you ever need to restate your data what's your time window for doing so how much time do you need how much time is uh critical for your consumers because if you bubble that up into an SLA you

can have that conversation rather than having and you have that conversation beforehand rather than having that conversation when everyone's running around with their hair on fire why you you know why haven't you given me this data I need it for the report for for the regulator well you never told me that you needed it for them I said I can't it's these guys up here and so

you know that's never when you do your best work trust me um so having that conversation ahead of time that's that that just saves that grief and then it also gives you a business case if you say well actually I need a bunch of more infastructure to be able to meet that or I need to be able to kind of back this up or I need to

I need to design around it to enable to meet that SLO um um that gives you the kind of justification and you say well actually no we're not paying for it well actually well that's the risk I'm not going to be that um and you know another thing is what documentation will you provide what guarantees do you make about keeping your docs up to date you know

you've got this set of customers you know you're defining a product you know you've got it um then what are you doing about kind of keeping them up to date because if it's not up to date bad news they're going to phone you up and they're going to talk to you still we still don't want that okay now moving on to the last one is to treat

data as a product so um this is courtesy of the data mesh crowd really this is where it really kind of came to prominence in my mind is you want to introduce product thinking for data it means your consumers will get a decent level of service and quality you know how quickly do they need that data how accurate do they need that data to be um what

would be the impact of restating the data on your consumer so lots of things start to kind of drop out of treating data as a product lots of good things start to drop out of that so and the more the data is procured by others the more it be relied upon so the more dependent upon that will be and so then the more they will create derived

data from that and the derived data can be relied upon as well and that is really the path to becoming a data driven organization so that's all I have for the for the 12 I have a little recap slide here for for all of those there um but um I'll probably leave that there but actually as has anyone got any any questions and by all means please

do reach out on LinkedIn I'd love to hear your your feedback and and comments um but yes if you've got any questions then we can we can answer them if not I can I can give you some time back and give you a chance to for a coffee before the next uh before the next session thank you for the great session James I found it like as

an operations person I I love hearing about how to make things more operable so one question from me is do you think the people who design these systems and who have to make these decisions are the same people who then suffer from the consequence of whether these 12 principles were followed or not oh it always depends from from um from from from from place to place really

you would you would hope I mean if you take a lot of the um a lot of the big Tech you know uh people like um I was reading a book by the uh by the engineering I think it was VP of engineering at stripe recently and he kind of said well look you know um if there's pain in the system cords you know let's you know

remove that toil and the the only way that that you know we're going to make that economic if you like is by the people that design and build the system actually supporting it for a bit maybe there's some kind of warranty period before it gets handed over or something um because they need to fill that time uh they need to feel that pain rather and I think

Nothing Stops people from feeling that pain or or doing something about that pain than being got out of bed at 3:00 a.m. um you know I I think that's that's kind of important that kind of um I I hate the phrase the kind of eating your own dog food because it's it's um uh there's a nicer phrase um drinking your own champagne but I think in this

case you want people to eat the dog food because you're like actually they need to feel the pain and actually say well you know that was a quick and dirty fix and yeah the Quick's forgotten but the dirty is still there y yeah exactly because I I love this quote by Kurt vut um that the pro one of the problems with the world is that everybody wants

to create and nobody wants to do any maintenance and what you're talking about here is exactly that more maintenance kind of thing like how to think in perspective about the way your architecting system so that down the road you prevent problems that will require uh reactive support yeah and yeah in yeah go on well I was just going to say and I think actually some of these

I think that feeds totally with what you're saying as well because some of these you want to avoid the maintenance so let's just not be lazy right like programmers are the most lazy people in the world but you want to be lazy in the right way you want to be able to push a button and the script does all the stuff you know you don't want to

be lazy and sloppy you know you want to be um lazy on a beach yeah yeah exactly so the architecture maybe is going to require a little more time to do so that you have less time for to to operate it after um I'm really interested in your experience with um data as a product uh do you see a lot of uh organizations adopting that or is

this still relatively new as a concept um actually I think it I think it's kind of old actually sorry don't don't don't tell thought Works um because because um there are lots of places you know um that have been um that have been doing that for a long while so for instance we uh so places like people that sell data for a living people like Reuters and

SNP and Bloomberg and things like that they wouldn't be those data powerhouses if they didn't have some kind of product mindset they might not call it that but they will have that data C loing they will have that kind of that I I think you're right but I'm I'm talking more around uh like people are starting to look at their own data uh and creating uh internal

data laks and things like that and if you're a large enough organization a data as a product makes a lot of sense and the whole data mesh construct but do you see people who are not in the business of selling their data Implement data as a product um yes I do yes we've got we've done that with one C customer at least that I can think of

off the top of my head and um they what they wanted to do they wanted they wanted the shiny stuff they wanted to do machine learning models and things like this and we said look you know you you're only going to scale to do that they had lots and lots and lots of developers and lots of data scientists they were out in India so they had a

very big Development Center and um we said to them hey look let's do this as a data product and then people can selfs serve in that data and they were like yeah cool that makes a lot of sense for me because um if you don't have some kind of way of self-serving people coming to get your data and and and doing self-service on it you know it

is going to drag on on on everybody else in the in the organization I've seen it where people are doing extracts and they're doing five different extracts for five different consumers and that you know it's like each extract is some kind of snowflake kind of thing sorry not in the not in the not in the database sense but in know it's it's an individual sense and it

it just drags on the productivity of the dev team as well so so I think the product mindset is a fight against that definitely yeah and it makes a lot of sense even if uh yeah if you're doing it for internal use it's just a very good practice okay I see that uh we're running out of time so we can um give a few seconds back um

we also have had a comment uh uh from one of your the audience great session so um uh people have enjoyed it as at least as much as I did um so thank you again for the lovely talk and have a great day bye that's great thank you and um thanks to everyone that that watched the watch the video as well Cheers Cheers thank