Freedom Through Boundaries: Building Configurations That Age Well - Bogdan Stancu, Adobe

In this talk, Bogdan from Adobe's observability team discusses the importance of building configurations that remain manageable over time. He uses his experience with Helm charts for deploying OpenTelemetry collectors to illustrate how neglecting configuration management best practices can lead to complex, bloated configs. The speaker emphasizes the need for clear boundaries and rules in user management to avoid chaos and maintain usability. Key topics include the need for a solid specification, the significance of user-centric design, testing practices, and maintaining backward compatibility while encouraging manageable upgrades. The session highlights that with proper structure and governance, teams can foster user trust and flexibility.

Hello everyone. I'm Bogdan. I work for Adobe in the observability team. And today we're going to talk about building configurations that age well. Uh I would first like to ask who in this room does this does uh configuration management for other users. Can you Yeah. Oh, a lot of people. That's great. Yeah. Uh just so you know this talk is uh we'll present a case that is

a bit extreme and most of us when we think about what I'm going to present will think that everything is pretty normal like you you think about what we're going to talk about but if you don't actively keep thinking about the rules that we're uh things can kind of shift now one of the questions I guess you all have is who's a dog and he's my dog.

He's 3 years old. I have no creativity. He's a terrier. So, his name is Terry. And we've been training for 3 years almost. Like right as I got him, we started training. And I constantly keep hearing people telling me that he's just a dog and I shouldn't train him. He should be free to do whatever he wants because it's better for him. But that's not really the

case. And I found out that whenever I go in the park, for example, everybody has their dog on a leash, but Terry can be free because I set some boundaries. I set some rules and I know that he's going to come back whenever I call him. We go on hikes. I feel like he has a lot more freedom because I set up some boundaries. And this thing

made me uh realize that it's kind of the same thing with user management, kind of configuration management. You have to put rules so the users are more free. They cannot really break stuff. They can't really go wrong. Some prerequisites. I guess most of you already know what Helm is because uh this is going to be uh based on a Helm chart and yeah well Helm templating for

for Kubernetes. You give it a template which is the top part with that values uh thing. You give a values file and then it spits out uh the combined configuration. What what's important here is that what we will talk about the API is the config file the that you can't see it uh the thing on the bottom that's that's what users see that's what they should care

about. Then again, uh, open telemetry. I guess you all heard a lot of open telemetry this week. Uh, so I'm not going to go really into what the collector is, but it's an agent. It has a configuration, and this is what uh, our well example will spit out. Okay. Now, we're going to imagine that we are an observability team building a helm chart for our users to

deploy three sets of collectors. And we'll start by putting this mask on that we don't really know what we're doing. We're going to first try to uh see the experience of not knowing any best practices in doing configuration And yeah, as I said, it's a small Helm chart. We have to deploy three sets of collectors and our various files file looks like this. We have a global

field which is going to change all three sets of collectors and then one for metrics, one for traces, one for logs. Again, this is just an example. Uh I guess what we're going to talk about applies to all kinds of configuration management, but this is just an example. So we have something to talk about and it's it works at the beginning but then uh people start using

it and we get one person one team that wants something for their specific use case. Uh they want to deploy their they have another system of deploying their own secrets. So they want to be able to use that already deployed secret into this helmchart that we built. So, of course, we say yes, one just one team using something. Yeah, we say yes. We're going to we're going

to put it in not thinking much because it's one user and we want everybody to be happy, right? Then security knocks at our door. Uh, a bunch of secrets ended up on Git and we have to not do that anymore. We have to keep our secrets in a secrets management solution. and another team already deploys the external secrets operator everywhere. So we add that functionality as well.

So now we have three ways of ingesting well of using secrets into this. Everything that we're going to talk about is only about like secrets because uh it's it's an easy example. Uh and also now we have external an external dependency that external secrets operator. Uh yeah, so we have three ways of doing one thing. Uh and we're fine with it for some reason. Uh after this

we want to always build thing. We want to make stuff uh we want to enhance it. We want always to make our things better. And we are constantly working on uh doing stuff better in the back like people shouldn't when we do stuff well talking about this exact example uh we're building something that people will not see. It's uh it makes stuff in the back uh work

faster or something like that and we go in the in the office or something like that. We talk about it. We ask for opinion and some user hears about it and they know what it what we're talking about what what it means. So they want to use it even though it's not released and we have not planned this and it's not uh part of the actual configuration.

It's an internal thing. They want to use it and again we like that people kind of are interested in what we do. They know about what uh what's going on with this product. So we add another flag uh so they can use it. And what I've uh kind of another thing about this what I've seen is that when we meet colleagues or well users that know what

we're doing know uh how things work behind are not just the normal kind of user. Uh we start treating them better because we feel like we have a better connection with them. So we they when they ask something we tend to say yes easier than a user that doesn't really know what's happening. They just use the defaults and that's kind of it. Okay. So this is uh

yeah we we added another another configuration a user is using it so we can't really we will not be able to remove it that easily and this shouldn't have been there anyway. And doing this for one year constantly saying yes ends up with something like this. uh huge config. I've seen I've seen 1500 line configs. It's it can get crazy and well yeah again this is just

one example. Uh can we get can it get any worse? Yeah, it can. But not just adding stuff in the config. We can also have uh configurations that are not uh well can't really work together. we have the uh autogenerate secrets and also existing secret name. What that means a user doesn't I mean we gave the users two options and if they set both of them they

have no idea which one is actually going to get applied. Are we going to are is the chart going to autogenerate the secrets or is it going to use the secret that it's already deployed? This should fail, but it doesn't because in our uh release notes, we said, "Hey, now you can autogenerate secrets. Set this to true if you want your secrets to be autogenerated." We never

said that you have to disable anything else. So, uh now we have this which is wrong. Also because of what we kept doing and people have been using our product for a long time now we have like all all of the configurations that we talked about somebody's using them so we can't because we want them to be happy we can't really just remove them right uh so

we never go out of v1 we're not going to go to v2 we we're never going to uh do a breaking change cuz that way some people will have to do work and people don't want to do work so they're going to be unhappy. then we've heard that it's good to test your products. So we added tests. What tests? Well, the the default configuration that we give

to people should be enough. So uh yeah, we have tests now. we it's just the default but it's enough. That's what we think. Another thing that we we keep doing is that we never do uh any change log, no documentation, no release notes. We do PRs. We explain what's happening in PRs. People can just go in and read what has changed, right? We we don't really need

to explain anything else because we know what we're doing. We know Helm. We know we expect people to know that and it's pretty easy. Like all the changes that we've done are pretty easy. So people will understand all of them. We don't have to do anything else. And if we do have to say something, we're going to do it in in a comment because that's where people

are, right? uh when people change the configuration they would want their instructions to be in the same place uh and if anybody ever doesn't understand the uh the configuration if they don't really follow this because as I said we have a 1500 line config uh we're going to blame them right because all the information was there it it it's easy Right. Okay. Yeah. The So, this this

I'm going to take the bad guy hat off now. So, uh let's see what what we can learn from all this experience. First, boundaries. We have to create boundaries. We have to write a spec. Every product has to have a spec. And the spec should be updated anytime we want to make a significant change. One example of this uh is the uh open telemetry OEPS. Uh there's

a talk on this uh I think it was at the maintainer summit in Atlanta uh last year. Uh they have a pretty good uh process there. But the idea is that whenever we want to make a big change, we have to create a file well a spec for that change and then merge it into the current spec into the current uh description of our product. So we

don't do do just one spec once and leave it there. We reject anything else that's uh that's not com well we don't agree uh meets the spec. After this uh anything that is mutually exclusi exclusive should fail when you try to apply it. You can do this in uh uh in many ways. Again this helm thing is just an example. Uh but I guess we we could

all just add some uh I don't know checks. It depends a lot on what you're trying to build. Uh but yeah, never fail silently. That's pretty self-explanatory. Okay. Design for the user not maintainer. Now uh users when they use a configuration they are changing the outcome. they want to look at the config and have options about the outcome. Nothing internal should be uh should be in the

in the configuration because then what that causes is that you cannot change anything that uh is internal that doesn't actually have an outcome. Uh you cannot change it because users will set that. So now you you can't change it even though setting it doesn't really do anything. Uh this is yeah uh an example about uh okay well we'll uh go gone the other uh phase of what

I just said is that uh ex uh well defaults should be explicit anything that has an actual uh anything that changes the outcome should be visible to the user should be part of the default config. They should see it. They shouldn't we shouldn't just assume that the user wants something happening in one way and uh not letting them see it because whenever we will change that uh

they will see the change but not understand because their their config didn't change. Uh so same config should be same outcome or it's a breaking change. So now we have to increase the major version do a breaking change but for the same uh same major version we have to keep the same config same no matter the internals. Uh if we have breaking changes we have to have

documentation on that. We have to uh tell people announce them early have a path uh have a migration path. So whenever a user has to do something they have to follow something they should have that thing to follow. So they they they don't nothing happens that they don't expect. Uh and yeah there's that line that treat backwards compatibility as a budget. I like it because uh it

it also means that whenever we add something we have to remove something. Backwards compatibility should be seen as a as a budget either like a number of versions either at time but whenever something gets old enough and people don't use it because we have new ways of doing that thing we should deprecate it. Uh now this QR will lead to a talk that a friend of mine

did uh while ago on how we do tests for our Helm charts. And it's like it it goes pretty deep into uh how we did that. But having unit tests, having tests for every single possible combination of our uh of our configuration has to happen because you don't want to just assume what people will do. You don't want to uh have a way of things breaking that

config wise is correct. And uh another thing, CI can take as long as it wants. Don't ever think that you have too many tests. Think that you have too few tests because if it takes 5 minutes more to merge a PR, it's never a problem. But the other way around it is I recommend that uh watching that talk. In that talk, one of the points uh and

a pretty important one is that Helm doesn't care about typos. If you just have a typo in your config, Helm will just ignore it the default to the default value in the Helm chart. Uh so when it's it's a good practice whenever you have a a flow of deploying that uh product that you have if anybody does anything that is not expected it should fail. For example,

we have here a typo. That thing should fail. It's not it's it's a field that is not it's not expected people have to have a way of seeing the final result. In Helm for example, it's usually it is Helm template like it it just does uh it shows you what the the final config is. But uh in anything other than Helm, you can you can do this

either in the product itself or in the CI. So people can I don't know just add a comment somewhere in the in the like in the PR add a comment like hey here this is what's going to change so people know the full extent of their change even though they only change one line a bunch of things might uh might and uh make upgrades boring. What I've

seen is that uh we had struggles with convincing people to upgrade uh to the new versions because they they didn't really want uh well that's what I thought what what we thought uh they didn't really want the new things as long as it worked it was fine but uh what we found is that one reason why people don't upgrade is that they fear change they fear the

breaking changes uh so you have to work on making those upgrades as boring as possible. Make the migration paths paths. Uh and also what we what we did is this uh compatibility metrics. We we have a tool that exports uh which version of the chart is uh compatible which with which version of the config. So if some people are on some version of the config, they know

that they they don't have to change anything to uh to upgrade keep upgrades boring. Uh and yeah, the the conclusion is that mostly uh will never allow less freedom. They allow more freedom because people will have the trust to use your product and know that nothing will ch uh nothing will break. They will be able to try stuff and they can uh not have the fear of

breaking anything. That was it. Thank you. And if anybody has any questions, I think we have a bunch of time, right? Hi. So, uh basically nice presentation about good practices we should all follow. However, there is one fundamental reason why people don't follow these practices and it is that there is cost to them. writing those tests, writing the these boundaries, defining it through validations and so on.

So what would you recommend to uh justify these costs to setting up these boundaries? What is the best solution? Because it needs to be understood across the whole value stream, not just the developers themselves. Well, I think just justifying it is uh what what what worked for us was trying and learning and seeing that uh when something breaks it's bad. So we have added all of these

best practices at as a fear of not doing something. >> So basically governance at the end. >> Yes. >> So the general architecture and >> yeah just just write docs and enforce them as best as you can. >> Okay. Thanks. because uh exceptions are easy to come by always. You can just say ah it's fine or something like that but you have to have the strength of

not doing that and it's it's hard. >> Yeah. >> But yeah >> thank you. >> Uh thank you for the talk and a lot of amazing insight. >> I I I can barely hear you. >> Yeah I understand. Uh so the question is along the lines of how do you find that line uh where there is too much configuration or not too much configuration because when you

are determining the API well configuration in this case is an API right uh you can either provide a good like set to configure it very flexibly or you can provide a very high abstraction level. So what's your opinion like how do you determine that just right level of flexibility? >> I think that the just right level of uh detail in a configuration is that people should be

able to do whatever they want not everybody has to have an option. uh anyway for example uh this the example that I gave is a pretty pretty similar thing to what we have and what we have in open telemetry I I are you familiar with it u there are other components than what we kind of have as default let's say somebody wants to use a new processor

or something like that we have extra collector config as a whole new field and we clearly say hey if you use this we're not going to we're not going to uh like provide support because you're in no man's land. But it works and it just kind kind of copy pastes whatever users put there in the uh in the collector. >> So the people have to have a

way of doing whatever they want, but it doesn't have to really be supported. You have to focus on the defaults. You have to focus on the masses. But anybody that is more, I don't know, advanced has to have a way of doing it. So they don't just do their own stuff. Uh so in a way I I think someone mentioned a very cool uh idea that I

picked up. Uh a snowflakiness index should allow for those tail ends like the very flexible kind of configuration almost verbatim probably but at the same time like expose that welloughtout interface. That's basically the recommendation. >> I I can barely hear you. I'm sorry. Uh so like if you think about it there is a certain level of snowflakiness right >> maybe uh if you think about it there

is a certain level of snowflakiness when people get to configure whatever they want right that's basically one of the points of your talk like you you do want to establish the b the boundaries there uh but basically the recommendation is think of your API to provide most of the use cases like cover that snowflaking index like try to reduce it as much as you can but allow

for the tail end for configuration that is almost verbatim to whatever you want to configure >> as you make sure that uh it's clear that you're not supporting it like it's possible but you're not going to offer >> like filter approach like this is experimental if you do it you're >> hi um so how do you actually say no to the users without them maybe finding another

even worse way of doing the thing that they want to do. >> How? Hard question. Uh well, I think it depends case by case. Uh usually have you have to have an explanation why you're saying no. And in most cases that that works. If you don't find an explanation why uh you you're not doing it, maybe you should do it. I think that's the answer. like it

it should be pretty obvious why you don't want to do something and it should be good enough for them as well. Also, uh what what I just talked about like people should be able to do anything they want. uh this is money talking about stuff that we support like hey we can say like hey what you're doing is possible this is the way of doing it but

don't don't come to me tomorrow when it doesn't work cuz it's just you so uh I'm not going to like we're not going to fully support it. I think that's that's make sure that everybody can do anything they want but have have clear boundaries where you're going to step in and when you're going to not. >> Thank you. And I would love to get the PDF uploaded

to schedule if that's possible. The talk slide. >> Oh yeah. Yeah, I yeah I will. >> Hi. So you talked about being explicit and not setting up defaults for people. But doesn't it uh contradict a bit like the length of the config you want to provide? Like for example, I have some Helm chart and I enable pot disruption budgets when like replicas is more than one, right?

And I can provide do it automatically or provide uh like a line in config enable PDB. So one is explicit but it getting uh the config lengthy and the other one is like implicit and just happens. So like how do we balance that? Should we provide like everything to be configurable by user or like sometimes we just make decisions and that's okay. >> I think sometimes making

decisions is just okay. Hard question. Uh I mean it's a balance. Uh you it depends. >> It exactly it depends like every product every configuration is is a back and forth with the users depends a lot on what they constantly want to change. Again kind of I have the same answer. uh what we're talking what I've been talking about is is the stuff that you kind of

actively uh manage what you actually actively offer support to uh and if like half of your users want to change something it should be part of the config and you should uh I don't offer support for that to work but if just one user wants to change something then they can even fork the Helm chart and do it that way if if they want to go >>

so deep that it's not uh the the default config is not enough uh they they can they they should have options uh but yeah focusing on the 95% of users it's uh uh the way to go I think right thanks