Tom Sadler, Joel Keers – A fork load of maintenance - forking a key dependency #FOSSBack

This talk explores the challenges and strategies involved in maintaining and upstreaming changes to an open-source project at the BBC, specifically focusing on DASH.js, a JavaScript library that enables DASH streaming on web browsers. The speaker, Tom Sadler, and principal engineer Joel discuss their experiences with the internal fork of DASH.js, highlighting the significance of following open standards like DASH and DVB-DASH for media delivery. They emphasize the need for effective internal collaboration and engagement with upstream maintainers to minimize divergences and promote contributions to the open-source community. The session also covers the crucial role of legal processes and internal advocacy to ensure successful upstreaming efforts and the long-term sustainability of their forked projects. Additionally, best practices for engaging with open-source dependencies are shared, underlining the importance of modular software architecture and cultivating a supportive environment for developers.

[music] [applause] >> Thanks everyone for being here this this late on the second day of the conference. So yeah, I'm Tom Sadler senior principal at the BBC. I work with a number of teams to enable open source inner source and industry engagement. I'll speak be speaking with Joel today who's a principal engineer specializing in streaming on connected TV devices. So Joel and I both work in the

TV and devices part of the BBC. So that's the group that builds amongst other things the connected TV applications. So iPlayer being our video streaming service, sounds being our audio streaming service. And yeah, we're going to talk about an internal fork we maintain of one of our key open source dependencies for media streaming. So I'm going to talk broadly about some of the policies and strategies and

Joel's going to share some experiences of working on the ground with this fork and with the upstream community. I'm going to talk about actually relevant to the last talk for anyone who was in the last talk an open standard called DASH. So DASH enables delivery of media via chunks instead of having one single stream and having chunked media enables things like live streams where you know you're

continuously getting more content on a live stream and also adaptive bit rate. So you know when based on bandwidth you'll get a higher or lower quality of video or audio. And DASH is under the SVTA streaming video technology alliance which is a foundation that the BBC is a member of amongst our partners like the European Broadcasting Union. So yeah, makes a lot of sense to build on

top of open standards and that's kind of why this is our preferred mechanism for streaming. So the the actual code dependency that we're going to talk about is called DASH.js. So this is the DASH Industry Forum so DASH Industry Forum is yeah, the body that looks after the DASH standard. So they supply this JavaScript library called DASH.js. So this uses a web technology called Media Source Extensions

to implement and enable DASH streaming on web browsers. before connected TV started using Media Source Extensions for playback, our colleagues that built the BBC websites and web streaming were already using DASH.js. So we had so the BBC had involvement with DASH DASH Industry Forum and DASH.js for a while and then yeah, when when it came to implementing Media Source Extensions for connected TVs as well as web,

we sort of followed in in web's footsteps. So yeah, as I said we're going to talk about this fork. So just to be clear this this isn't like a fork where we want to go off in our own direction. We want this fork to be as close to main line as possible. We want to use that fork to contribute upstream but what our web colleagues and our

R&D colleagues before connected TV got involved, what they and we found found out was that there are some BBC specific weirdnesses in our DASH streams. So being able to modify DASH.js so that it supported those BBC specific things was very helpful and it made sense to do that in the fork rather than trying to either make a contribution or ask DASH Industry Forum to make those changes

for us. Part of the reason for this is we're talking about open standards. So we actually use an extension of the MPEG DASH standard called DVB-DASH. So that's Digital Video Broadcasting DASH. So it's as I was saying it's an extension of the core standard that broadcasters tend to use. And actually I mentioned R&D so they actually have yet another fork because they want to do more experimental

things that wouldn't be suitable for for production. But as I say this fork we don't we don't want it to be massively divergent from main line and every divergence we have is more code for us to maintain that isn't being maintained by the wider community. It also means that when we take a new major version of DASH.js from from the upstream and this is what Joel's going

to go into more detail on, every divergence needs patching separately. So it just makes that maintenance overhead much higher. So yeah, upstream contributions we in TV and actually R&D and web before did identify that we do want to upstream as much of our patches on DASH.js as So yeah, this reduces the amount of divergences and it's also part of being a good open source citizen. You know,

we don't want to just consume DASH.js and not give anything back. especially especially outside of R&D so R&D are a bit more mature with their open source engagement than the sort of audience facing service side of the BBC. We found that yeah, making arguments for getting team capacity to upstream could be difficult conversations with managers. Training developers so you know, we know that there is a big

open source skills gap and we I think we felt that when we were trying to do upstream contributions and also the legal policy and process. So when I hand over to Joel he's going to talk a little bit about the first two but just to before I hand over just to talk a little bit about the legal policy and process. So um it's really important to work

closely with your legal team. Yeah, they are there to help you. They're not sometimes it feels like they're there to say no to you and block you but actually they are there to help you add value to the to the company you work for. It's really important to have both lawyers that are willing to speak to developers and developers that are willing to speak to lawyers because

lawyers don't understand code and coders don't understand the law generally speaking. So the process we ended up with was we needed a value statement. So why do we want to contribute upstream? We need to identify the license or the CLA information. So DASH.js specifically we had signed the CLA. I think I mentioned our R&D department. They'd signed the CLA and we couldn't find a record of that.

>> [laughter] >> So we eventually managed to dig it out from some and actually openness helped us. I believe there was on on DASH Industry Forum's GitHub there was a reference to us signing the CLA in that meeting minutes. So yeah, openness is great. senior manager sign off. So just make sure that there's sponsorship from your sort of senior engineering manager. And [snorts] then yeah, once you've

once you've got all those things then you can go to the legal department and say yeah, here's the license, here's why we want to upstream, here's the senior manager sign off. Are you happy from a legal risk point of view? And they will ask questions around like patents and who's the company behind why you're upstreaming and things like that. And then so this is a this is

a process for ongoing contributions. So where so in in the DASH.js case, you know, we're intending to use that for the long term so we expect to make many contributions over many years but to avoid having to go through legal and senior managers every time, we nominate lead BBC contributors who can sign off junior developers contributions which is really helpful. And then yeah, the other thing that

legal want us to do is just keep a record of it. >> [snorts] >> Heard a good talk from Michelle on yesterday. They were talking about they do this with GitHub app. So might have to look into that. Sounds like a an easy way of doing it than manually updating a conference page. So I'm going to hand over to Joel to talk about some of the on

the ground experiences. Oh, right. So just a bit of a recap of what Tom was talking about. So Media Source Extensions is a extension to the HTML media element if you're if you're into web standards. DASH.js we use as a library that sort of wraps up Media Source Extensions. It gives us some levers to our playback quality sort of options. Gives us options to go and experiment

with things in the wild. yeah, just recap on that. I'm going to be talking about the major version upgrade that we did a few years back but to begin with I kind of just want to cast cast your time back cast your your mind back about eight years. So long time ago but a long time ago we began with um everything in silos. So inside the BBC

we have different divisions that do different things. Um web happened to be in the technology division. We're in this little thing called iPlayer for TV just carrying on around there. And everything kind of operated in a silo. Web looked after the fork at this point. So when we came to integrate this everything was owned by web. From the TV point of view, we're thinking right, okay, this

is a great fork. Great, web will look after this for us. We can just take it, we can use it. They'll handle all the upgrades and all of the upstreaming. No, everything was sporadic. That there was a single trunk-based development branch which did our upgrades. No shade on the web folk, but it was their world and that's how they wanted to run things at the time. So,

we we had to change that when we wanted to start contributing to dash.js itself. Um and sort of get that open source engagement that we were really after um with with both the maintainers and with the other teams that we worked with as well. um it was about 2023, TV took the initiative. We were thinking how do we how do we really sort of bring our teams

together internally and then actually give back to the open source community as well and try and minimize those divergencies that we built into our fork. How do we get back to mainline as it were? Um so, we started by figuring out like what can we upstream? What makes sense? Where where have we gone off the beaten track here and we could just contribute back to the community?

Well, it's kind of having your cake and eating it to some to some extent. Yeah, so that's that's where we went for we went we went down the path of can we upstream? Yes, we can. Tom alluded to that with the really hard work that he did working with legal and the other stakeholders to get get us to this point within the So, I'll talk about some

of the project phases that we went through. we um we were looking at engagement with the mainline um project itself to begin with because we thought if mainline won't take any of our our upstream contributions, like we're we're kind of stuck. We can't reduce our our technical debt that we've built into this fork that we've had going for years. How can we get there? We can't get

there without engagement with the mainline dependency. And [snorts] I'll talk a bit about how we went about upstreaming and some of the outcomes we've had since um since we did this project. Um [snorts] How it's been a good I got a thing for both insides and like sort of BBC. So, I'll talk about some maintainer engagement. Um so, Daniel Felhauer over at Fraunhofer, he he's one of

the main maintainers of dash.js. Um Great guy. We had to um well, we have to. We we engaged as much as we possibly could with him to try and get to the point where we could um get get some of our features upstreams. So, this was kind of meeting face-to-face at dash.js to a face-to-face in in Berlin at the Media Web Symposium every year. Try and go

there. They have weekly meetings online. Slack space. Um you know, good engagement on GitHub, GitHub issues and pull requests. So, we're meeting the maintainer in every place that we possibly could. Um just to get that sort of engagement of will you take our patches? Can we can we get back to mainline? Can we help out? So, that we thought that was very important to call me out.

>> Um and then there were sort of the the internal challenges of trying to do this as well. I talked a bit about um web and the sort of ownership of our internal A- A- Again, the main problem with that was trying to get alignment across the organization about can we create this feature? Can we put it in the fork? How about how do we do the

releases? This is all internal things that that are a challenge and we thought, "Well, we can kind of solve a lot of these issues by being more open from the beginning." >> So, we came to that we came to that point um like Tom was mentioning by having a kind of decision record of all of our divergencies from the mainline and getting everybody together who knew about

what these divergencies were and creating records. Um And and managing to managing it through [clears throat] this looks a bit of a mess, but I can hopefully explain it. >> So, um I was talking a bit about um web. Web's there at the top. You've got the tags, you've got the releases there. Trunk-based development, so everything happened on their release branch. When we came along, we were

going to do a major upgrade from V3 to V4 of dash.js. We were thinking, "Right, okay. We want to just have feature branches. We want to work on a feature after having identified it and we want to you know, have that around so we can upstream it later on." So, we we came up came up with this sort of branching solution. We came we still had our

Um same name convention. Do a feature and then hopefully sometime later. Now now this is important there. This is about a year later, right? About a year. Um Yeah, um eventually go back upstream just in time for version five having taken V4. we got there in the end. >> [clears throat] >> Um in terms of executing the plan, talked about porting features, having that decision record of

what we actually uh changed, what we could upstream. Um so, that was one thing. Another thing that dash.js gave us was the feature flags um sort of settings they have in there. They they made sure that we could um contribute our features back not being enabled by default, so useful for people who need them. Um if you don't need them, they're off by default. Great. Um obviously

with a um project like dash.js, it's modular architecture, it lends itself to to being able to be built upon using the feature flags. It was great for our process. >> Um so, right. I'll talk about some of the contributions. I already kind of described this feature this sort of process of we port the feature. Is it useful to upstream? If it was, we document it and then

go through the process of upstreaming it. You can see this this one here. This is the one that took a year. Um I started the process in the fork and one of the grads on the team at the time, um he'd just rotated rotated into the team. Uh we have quite a strong inner source process where people are used to contributing to other projects, so we just

brought that along for our open source contribution guidelines. And um there he is contributing that feature back upstream uh with us having battle tested it through its release in our player in the time. Um Yeah, everyone was using the process um from grads rotating through the team to the juniors on the team. It's great. okay, so some more of the outcomes from running this process of you

know, having the the decision record, being able to document the features that we've got. Are they useful to upstream? Engagement with the maintainer. Um It interestingly improved our inner source mindset by opening up. We were contributing back to the mainline project before consuming it back ourselves in many cases. And one of the things I wanted to talk about was something we've been doing with R&D recently. We

had R&D embedded in our And we delivered a a public low latency trial at scale. Um they did a talk recently at um some other conferences um that they're just discussing this. But, what I wanted to point out was that they were both upstreaming their patches to the mainline and then we were consuming them straight back, popping them into our fork and and and running the process

both ways. So, we're we're seeing the benefit of this contribution um sort of process coming back to us time and time again. And then web are going to be taking the same approach to their V5 upgrade. We're a bit late. But, um yeah, it it we we think it'll it'll work just fine just have improving that engagement. So, I'll pass back to Tom now and you're going

to summary summarize what we've been talking about. >> Yeah, thanks, Joel. Um so, yeah, just to summarize what we've covered. Um so, yeah, um there are sometimes reasons whether or not you think they're good reasons or not is a a debate to be had, but yeah, where you've got divergent requirements, where you want to iterate faster than the maintainers can um review and release pull requests. Um

Or I suppose this is where the the sort of bad reason forking is where you don't have a process to contribute upstream. But, if you do choose to fork, there is an overhead. It can be quite expensive. Um and there are benefits of engagement with your open source dependencies because it can reduce the maintenance of forks. And as Joel was just saying about our inner source approach,

it can actually improve internal collaboration. Funnily enough, that's what led us to inner source was we one of our one of our own open source projects called Tal, which is deprecated now, but we were getting some really good internal engagement on Tal and we went, "Oh, why can't we do this for internal software?" And that kind of led us to inner source, but that's that's a different

talk. Um [snorts] so, just to sort of give people some takeaways, these are some opinionated bits of advice for maintainers. And then I'll go on to opinionated advice for forkers. So, if possible, avoid CLAs. like as I say in I get sometimes your intellectual property department will will make you do this. As I say, there are I'm not going to make a judgment on whether there's good

reasons or bad reasons. Sometimes there are reasons. We were lucky that we were able to dig out the the old CLA that had been signed. But yeah, it it can be a roadblock. So, if you're looking for contributions, if you can avoid them, I'd recommend you do. Use a recognized open source license. Again, legal departments will be funny about contributing to anything that isn't an OSI approved

license. And actually to to continue on from the CLA point, there are certain licenses like Apache 2 which do have they do have contribution clauses in them. So, at least our lawyers kind of like the fact [snorts] that there's a clear agreement in the Apache 2 license that says, "If you're contributing, unless you say otherwise, you are contributing under Apache 2." And the developer certificate of origin

is a a nice lightweight add-on which essentially does the same thing. It says, "I'm signing this commit. I agree I am contributing under the existing license." So, I'd yeah, recommend those as alternatives to CLAs. Yeah. I think I think people in this community probably get it. But yeah, if you have contributing docs, code of conduct, good community. As Joel was saying, like we had really really good

engagement from the dash.js maintainers. And just wanted to call out Chaos. They do some really good work on metrics that can show your community health. I'll say Daniel from dash.js was really good to us in terms of investing his But yeah, if maintainers can invest time, then you will get more from your adopters and your potential contributors. >> And yeah, architect your software in a modular fashion.

So, it's interesting actually that dash.js is a reference player. In some ways, our contributions have made it less of a pure reference. But then because of the feature flagging, it the default is still very much a reference player. And then there's feature flags to make it work on weird TV browsers, things like that. And yeah, also what we found is sort of senior managers and and people

in the legal team feel more comfortable where when an open source project is underneath the an industry body or an industry foundation, especially if it's one that you already have a a preexisting relationship with. So, I would say that is if you are thinking about that, then one of the benefits of of putting your project under a foundation is big corporations will probably easier to work with.

And yeah, finally, advice if you're thinking about forking, really think about the costs and benefits cuz the costs are high. And the benefits sometimes there's other ways of getting the benefits like contributing upstream or maybe architecting in a composition way rather than a modification way. Yeah, minimize if if you do need to fork, minimize the divergences. And yeah, contribute upstream. Hopefully we've made a good case for

why why you want to do that. Yeah, it it helps keep the upstream fit for your use case. So, I mentioned like the weirdness of connected TVs. Those those bits of dash.js behind the feature flags mean that dash.js is more suitable for for our target our target platforms out in the wild to our audience. >> Um yeah, I said maintainers should invest time with their contributors. But

the other way's true as well. You know, do go out and speak to them. Merko in his talk yesterday used the phrase PR and pray which I really liked. Don't Don't do don't do PR and pray. Speak to speak to the maintainers of the project you want to contribute to. Um yeah, invest time with legal and policy colleagues. As I say, they are your friends, not your

enemies. They're not just saying no to be awkward. There is risk to be managed. And you do want to work with them. And yeah, very grateful for having some great legal colleagues at the BBC that have helped us get to this point. Um and yeah, invest time with your own developers because remember there is an open source skills gap. A lot of people feel uncomfortable raising a

pull request internally, never mind externally. It's great that we've got juniors and grads even willing to do open source pull requests. And yeah, shoutout for inner source again which can be a good way to to train people up. Um again, if anyone caught the is the inner source commons good for open source talk yesterday, some data that certainly says that suggests good. Although yeah, data's always a

tricky one with things Um and yeah, when when you are I suppose this is before you're thinking of forking, but when you're thinking thinking about adopting an open source dependency, think about whether it whether you can contribute to it. Um cuz if you can't contribute to it, then you might find yourself blocked. So, yeah, if you can think about that's not a word, but contribute contributability. Anyway,

you get the concept. So, yeah, do think about that when you're adopting open source dependencies. And that's it. Thank you VERY MUCH. >> [applause] >> HI, THANKS FOR THE PRESENTATION. UM YOU mentioned earlier on that you wanted to be a good open source citizen. Um something like that. But apart from apart from wanting to be a good citizen, is there any legal obligation for the BBC as

a public institution to do things in open? I looked at the the charter right now and there is R&D clauses referencing open standards, for example. Friend Since you mentioned R&D, it feels like the primary work of what you're doing is outside of the R&D. So, I couldn't find the parts in the agreement that somehow regulate what you're doing. And follow-up question, do you anticipate any change for

the upcoming agreement? I think there is one for 28 scheduled. So, is there anything you wish to change for the as a basis for how you work? Yes, as you say, R&D is the main area where we're fulfilling that part of the the charter obligation. I think that the want within the audience-facing product side is more just like we have we have enough enough engineers and enough

engineering leaders that believe in open source and just want to do it right. Um off the top of my head with the the charter, Yeah, some some of the ideas that are in the next charter I think do include you having a positive impact on the economy and on the market and I think there might be something to do with yeah, digital which could then give those

of us that want to involve in open source. We can then point to the charter and go we should be doing this. So, it'll help us advocate for that internally. But No, this this is a bit more of a we just recognize this is the best way to do software development rather than it being any sort of public service remit. But again, that that is part of

the argument as well that will convince some people when we say like, "Well, actually as a public service organization, this is really within our our core values." So, Yeah, it all helps, but no charter obligation, at least not yet. Hi, thanks for your thanks for your talk. I have some advice for your list to add to if you're up for it, but also a question. In the

in the slide, the one back for the peer advice for maintainers, I would love you to add also a security.md as part of the docs. You know, maintaining a fork that is like that does have some separate stuff. A way for folks to be able to report in a vulnerability is going to be very required very soon. So, practicing it now. And then my question, it was

brought up by yourself when looking at when you're developing a feature and deciding on whether it makes sense to contribute that upstream or not. And then you'd also mention how you're working with the maintainers of the original project in general. How are you making the decision whether what you're doing is only going to be relevant for you, BBC, versus it going back upstream? Do you want to

take that one? We both take it. >> Um security.md, that's >> Oh, sorry. Yeah, the security.md, yeah. Good shout. I'll Next time I do this presentation, I'll include that. Um in terms of deciding whether a feature should go upstream, sometimes it'd be just really quite naughty feature naughty things like, "Oh, I'm going to be exposing a different set of a different set of source files or something

in my in my fork compared to main line. That's not something we'll be contributing back and expecting that to be taken upstream. There there are some things that Tom mentioned. Um features that were very specific to BBC streams or some features we'd identified that fit particular use cases, maybe on connected TV that that were not useful upstream. And we would approach maintainer with those those features and

say hey we think this is very much in our domain and we we don't see any point in it going upstream do you agree and just get that agreement we won't just chuck the PR at at dusty s and say hey can can you just take this we'd have a conversation about it Right now we need the round of applause for Joel and Dom. >> [music]