Effective Data Modeling for Document Databases // Anna Henningsen

In this talk, Anna discusses effective data modeling for document databases, particularly focusing on MongoDB. She emphasizes the importance of understanding how to properly structure data to take advantage of NoSQL database features, which differs significantly from traditional relational databases. Anna explains that a developer has more control over data layout in NoSQL, which allows for optimized performance by grouping frequently accessed data. She highlights common misconceptions about NoSQL, including the belief that they are merely JSON storage solutions and the notion of being schema-less. The session also covers practical examples, such as modeling customer addresses and order data, to demonstrate best practices for efficient document data storage.

Uh, all right. So, yeah, hi. I'm Anna. I briefly going to introduce myself. So, if you want to find me online, this is my handle. I'm not going to try to pronounce it cuz, you know, uh, I'm going to get it wrong. Um, I have, uh, links to my Blue Sky account and the slides if you are curious. Um, but yeah, so I I >> [laughter] >>

I used to be a full-time Node.js core contributor. That's kind of the thing that made me a tiny little bit famous. Um, used to be on the technical steering committee before Node.js itself. Uh, that has been quite some time now, though. So, last 5 and 1/2 years I've been working at MongoDB, which is a company I'm sure a lot of you have heard about at some point.

Uh, we also have three cats. If you want to see pictures, we have thousands. >> So, yeah. So, so, just, you know, that's me. but yeah, so before we get into the core of this talk, I want to justify a bit why I'm talking about this, why I picked this particular topic. And, uh, then after that we'll actually get into, you know, how to do effective data

modeling for document databases. And then finally, I'll I'll wrap up with, you know, uh, a a myth or two that still persists about, uh, NoSQL stuff. All right, but yeah, so first of all, I I know it's not 2012, right? It's >> it's not the time of the big NoSQL hype, uh, when when also MongoDB was really really popular or started to really take up speed. Uh,

but it it's like when I go to conferences, I usually do that because I want to share something that I learned and then I am excited about, you know? I I I want to have other people know the same things cuz, you know, that's cool. Sharing stuff you learned. Um, and so this is like I this is the first time I'm actually doing a talk that's related

to MongoDB, even though I've worked there for 5 and 1/2 years. But anyway, so this is my personal graph of like, you know, the time I've spent working there versus how well I would be able to explain to people why they would choose a document database like MongoDB. I think it's crazy that I've gone like 4 and 1/2 years into my career at that company without really

being able to articulate that well. And then last year I get sucked into a project that was around like, you know, how we communicate data modeling to our users, to our customers. And that's when I really when it clicked for me, when I got the, you know, the the light bulb popped over my head and started to shine. And yeah, it's it's just weird that you can

go for this long at a company that does this thing uh without really grasping that. And I to be clear, I was still doing good work. Still putting in effort into making uh MongoDB easy to use for for people who use it. Um but yeah, so that's one thing. The other thing is so a lot of the time when you go on online forums, go on Reddit

or something and you read about developers talking about what you see a lot is you see people trying to use MongoDB like they would use a relational database and then being frustrated that it doesn't work as well. Which is like, you know, if you want to use it like a relational database, you might as well use Postgres. >> Um and it's it's okay if you hate MongoDB,

it's okay if you hate no SQL databases. I this is not something I care about that deeply personally, but if you do it for the wrong reasons, if you do it because you don't understand how to use it well, then that is a skill issue on your side. >> Um okay, so uh one more thing before we get really into the meat of this. So when I

was little, right? I was my parents were working in an office and they had these drawers. They had these drawers with lots and lots of files which, I don't know, I I I feel like I'm getting old enough that I can say like a lot of you look like you don't remember this time anymore. Um but, you know, the this was something that was a big part

of my childhood. >> And um I it it is a good mental model to have in mind. Like when you talk about performance of stuff, it usually helps a lot when you have an analog equivalent in mind when you have the ability to say, "Okay, this thing that I coded, it represents the data like this thing does in the real world." And so, when we talk about

databases, doesn't matter if it's NoSQL or SQL. You can imagine this to be like a drawer with files. Like you have the different drawers in in these boards. Uh each drawer corresponds to a table or a collection. Each of these folders and a relation where those are rows and the NoSQL way, they might be documents. And these little label cards that stick out at the top here,

right? These are your indexes. And honestly, a lot of the time this is a really good comparison and it really tries to drive the point home of like, this is how this works and this is how you can expect things to be performing. Like if you have an index that describes well, I mean, I don't know, your customers' last names or something. Um then you as a

human being in front of a drawer, if you have these index cards, you can find the corresponding file really quickly. Cuz you know where to look cuz there are these little cards. So, that's exactly the same thing a database does, basically. All right, but yeah, like I said, let's let's get into the actual core idea behind NoSQL or behind document databases specifically. So, MongoDB is one example.

And just to be clear, what I'm talking about here is not super specific to MongoDB. Like not it's not going to apply in full full depth to every NoSQL database, but we have competitors and everything I talk about here basically applies to those competitors as well. It's just not my background, you know. >> Um but anyway, so the core idea of NoSQL databases is actually just instead

of having the database being charged of the data layout, you as the developer or you as the database admin are being put in charge. You are being put in control over what the data layout looks like on this. That can be a good thing. Doesn't always have to be. The The other thing is like this enables you to do something that just makes sense, which is that

to store data together that gets accessed together. And access can mean both read or write. It really depends on what you're you're talking about here depending on your application. But these two quadrants, they have consequences. first one, because you are in control, that also means you have to think, which I know that's annoying. We we as developers we try to think as little as possible and put

guardians in place to make sure that what we do cannot possibly go wrong, right? But at the same time, sometimes you want that little bit of extra control over what you do, over what your application does, over what your database does. you have to learn how to properly data model for your document database if you want it to be efficient. If you don't care about that then

you you know, fine. Very concretely speaking about the access stuff that is accessed together gets stored together. That means yes, you depart from what the relational model looks like. You might not have a separate collection, not a separate drawer in the file drawer model for every piece of data, right? Uh you might want to store things together, maybe put them in the same file. Maybe some sometimes

you want to do that, sometimes not. And sometimes you want to duplicate data, which is, you know, also something we hate as developers. So, we want single sources of truth. You know? Um but sometimes you you may want to give that up in exchange for performance. And so, I'm I'm going to try to visualize this by using a classic example that we've all been through or seen

on talk slides a million times, which is a web shop. You know? if you haven't built one so far, um you might be even in the minority here. If you've never seen it on a talk slide. Uh well, yeah. >> [snorts] >> Uh but anyway, so we have the classical whole types of, you know, the all the entities that you typically have for a web shop. You

have customers, items, orders, reviews, everything that goes with it. And in a relational world, this is easy to model cuz every type of entity just gets its own table. As, you know, straightforward. And again, for document databases, we have to think about more. Before we start determining what our collections would look like, we think what do the read and write patterns for the application actually look like?

going to I I wrote down four examples here. Obviously, a real-world example would have a lot more, but this is something that can get it started in terms of trying to, you know, put together an efficient model. So, I think all of this kind of makes sense. Customers have addresses, but they don't have a lot of them. Typically, maybe two or three, in an extreme case, maybe

a dozen, but it's never going to be a million. when you try to display an item to the user, you do that, but you also display or a summary of them. old orders are very rarely accessed, Mostly care about the most recent one or two and you want to give the user the ability to look at all of them, but you don't need to access all of

them as frequently. And some items they may have variants, others don't. All right, so let's let's try to use the first one and try to come up with a good data model for that. So the fact that we have multiple addresses, right? In a in a relational world you would have these file drawers and you would have one for customers and you would have one for addresses.

And when you look at the customer, you would look at the customer document and you would have a join with the addresses table and you would find all the but that's [snorts] not all that efficient, right? If you were to actually be a human being with a file drawer, with an analog thing, you would probably try to put it all in the same file, right? Cuz it's

maybe an extra sheet of paper or two, it's fine. Um so yeah, we can just store these inline. And the fact that we can just basically do this if we say, you know, we have a maybe one to two, maybe one to 10 at most kind of relationship between those two, this enables us to say, "Well, I'm just going to embed this other type of entity entirely

into the primary entity, in this case the customer or user." I haven't been super consistent about what I call it user and a customer even though I find the distinction very interesting like in in terms of what languages people use uh cuz a lot of the time it's it's the same thing, but not always, right? anyway, so that would be one example. That is like one of

the core things you can do with document databases. Um take this a bit further. So like when have items and you want to display them, you might want to display a summary of the reviews, right? Uh and one easy way to do is to, you know, store the reviews separately, have a separate file for them. But then next to the items keep track of how many people

gave how many stars to this item. Uh, and this like this has an obvious downside, has a really obvious downside, which is you have to go to two different file drawers if you wanted add a new review, right? You have to put in the sheet of paper or the file for the review, and you have to go update the item and increase the counter. And that's inefficient,

kind of, right? But it also allows you to retrieve the item summary, like the the review summary, much more effectively. and so you you have this option where you say, "Okay, I accept that writes for reviews are a bit slower, but because items are looked at so much more frequently than reviews are written, this actually gives us a net performance benefit." Um, by the way, whether you

want to do this in a transaction or not, these two updates also choice that you can make. You can just say, "I don't, and if the server the the application server that handles the write goes down in between those two, that's fine. You might not care about that." but you might also. Depends on your Uh, another thing. So, like I said, older orders really rarely get accessed.

And so one way where you can, you know, take this even further, take it to a bit of an extreme, is you can say, "Okay, well, for a given user, for a given I will just store the most recent orders and the most important information about these orders. So, like I select fields, I select maybe the most recent five or and I store them next to the

information for the user itself, or maybe even in a separate drawer. So, this is like the strongest example of duplication that you could basically give. But it's silly, you know, it kind of makes sense cuz imagine you're a file clerk at a bank or something and you you want to you you're entirely analog. You want to look up something for a customer that just walked in. You

want to be able to just grab the for that customer and have all your relevant information there, right? This means that if you want to display the customer's orders, the most recent ones, you only need to do a single document look up. You don't need to fetch more than a single row. This is something you would never do in a relational world. Um and it does come

with a very clear downside, right? We don't have a single source of truth anymore. Or rather, we have to declare what single source of truth is. We have to say, "Okay, this order is for this order's collection. That is the source of truth and everything else is a cache." And in this case it's probably fine cuz orders very rarely get updated, right? You know, once order has

been submitted, you might update the delivery status or something, but for the most part it is just what it is. And so, you could take a transaction to update the status field and that's about it. All right. I I had a fourth rule on there and this is just like, you know, a short example. Um so, there might maybe items in your web store that have different

variants, right? You have something like clothing items might come in different colors and different sizes. Other items don't necessarily like a handmade ceramics bowl. It's probably just going to be a single size single color. if you wanted to have this ability in a relational store, right? You would have maybe an items table and a variants But that can be quite inefficient cuz like every time you look

up whether an whether an item has variants, you would have to go to the variants bar and look if there is an entry. There might just not be one, but you would have still have to go and check. Here in a document database, this is actually quite a bit simpler. Cuz you can just say, "Okay, this is an optional field. Sometimes it's there, sometimes it's not." Um

can also take this to quite an extreme if you want to. Uh didn't have a good example for this for web shops, but you know, generally speaking, you can your your data can be very polymorphic if you want it to be if you're okay with that. Like you can have very different item descriptions for different types of items. And that's kind of just fine. getting closer to

the end of this talk, um I wanted to talk about like a myth or two about NoSQL databases. Cuz again, sometimes people are wrong on the internet and I don't like that. And by the way, I don't know why this is so loud when I laughed, but uh I I don't I don't feel like I have control over this, right? The mic is here and I I

I don't know what to do. Um anyway, so for so a lot of people think of NoSQL databases as just being JSON storage. that for some NoSQL databases, this is true, right? Like some are really just optimized for sending JSON blobs there to live there and expecting them back. But um that that that is not true for all of And this is where I, you know, again,

MongoDB is my background. It's it's the company that sent me here. but this is stuff that we support. Like you can do joins like you would in a relational world. You have views where you aggregate your data before you process it. You have full support for transactions. This is like something MongoDB has supported for, I don't know, 10 years at least or so. we have integration with

full text search or vector search for AI integrations. data encryption and query search, I put this on the slide and I I personally really care about this. This is a project that I was quite involved in at MongoDB. And this is like, you know, if I ever were to give like a product talk where I actually just start to advertise one of our features, that is what

I would pick cuz it is so cool that you can encrypt data and still search it. Um I don't know, come talk to me after the talk if you if you want to chat Uh but anyway, so and typically this is all in our open source community edition, right? This is like this is not pay pay gated obviously there there there are some improvements if you pick

our cloud hosted platform or something, but this is all part of MongoDB's community edition in the most recent versions. Uh and obviously, you know, again, competitors have some of these things as Uh the other thing that I wanted to talk about is like people try to think or, you know, claim that no SQL databases tend to be schema-less. And, you know, obviously there is some kind of

truth to that, right? Cuz like the examples I just gave we just put together a schema by showing examples, right? You know, there there I didn't go and create a collection with a specific schema in mind and I uh I didn't have to do that. I could just insert the documents that I displayed here on the screen as they are. But, for those databases that supported schema

validation is something you can do, you should do if well, if you want to, but if in doubt, do it. And it is something that can be accurate. Like it can accurately describe every document in your database. so sometimes I use MongoDB for custom hobby projects of mine, right? Cuz this is again, that's the database I'm most familiar with at this point. And it it works fine

for my use cases, which is like, you know, three users at most cuz it's my hobby application. And so one thing that I really wanted at one point that I felt like, you know, "Hey, this would be cool to have." is so I Who here uses Zod? Yeah, yeah, nice. Um no, I I love the library. I know this also got named out before. But it is

like, you know, the best way for me at this point to describe the schema of objects in JavaScript. And so this is something I used for this application that I recently built for my Dungeons and Dragons group, by the way, as like a helper tool. Um but anyway, so something that I thought would be nice it This was just like super easily integrated with the schema validation

that the database provides, right? And I So I I actually put this little library out there, the MongoDB GS {slash} MongoDB Zod one, uh which is just like, you know, "Hey, if you want to use this, feel free to." Cuz you can This is all it takes to add for uh your collection to the database that actually is being enforced. Um And by the way, this like,

you know, if you A lot of you use Zod, so you know this, but this integrates really well with TypeScript, too, right? This is just like going to give you both validation on the database side and type safety on the in your in your application code. Cool. Um Just to finish this up, a couple things that I wanted to mention that we're working on that my team

at MongoDB is currently working on is uh so we want to give you tools to visualize schema, to improve your schema based on those visualizations, based on this analysis that you run on your database. Uh we want to have automated analysis in place to give you better recommendations for how you can improve your Uh make things like schema validation and easier to integrate and make migrations easier

if you do want to change your model at a later point. And we want to document these best practices for our data modeling in a central location. If you go to our documentation page and you go to the data modeling section, there's a lot of advice and a lot of that is good. It's way too much for 20-minute talk. Um but if you if you want to

learn more about anything that you just saw, that's a great place to go. And again, kind of also applies if you're using one of our competitors. Um all right, so this is what it currently looks like if you have already installed the Compass, that is part of the tool suite that I'm working on. and so this is for example you can you can kind of analyze your

uh MongoDB schema and create an ER diagram like what you would be used to uh if if you've ever taken, you know, computer science classes around databases or anything like that. Yeah. Uh thank you for your time. And um yeah. >> [applause]