Buildpacks: Towards 1.0, AI and Other Things - Aidan Delaney, Bloomberg

In this talk, Aiden, a cloud native build packs maintainer, discusses the journey toward achieving a 1.0 release of the build packs specification. He explains how build packs facilitate the transformation of source code into OCI images, emphasizing their applicability across various languages including Java and Python for AI and machine learning applications. Aiden performs a live demo using a Java application to showcase how the build tool simplifies the creation and deployment of production-ready images. He highlights the importance of custom builders for organizational needs, the integration of security controls, and the benefits of managing a consistent development environment for multiple programming languages. Additionally, Aiden describes forthcoming updates and enhancements to reach a stable 1.0 version, ensuring compatibility and support for the developer community.

Welcome to the 2026 uh build packs uh maintainer track talk. Um I'm Aiden and they gave me a clicky thing. Look at that. It's brilliant. Um I do tend to walk around so I don't know if this is a good thing or a bad thing but um why listen to me? Uh I am one of the build packs maintainers. Um Joe over there is also a build

pack maintainer and there's a couple of others around and we're always happy to talk build packs with folks. Um suppose I I had an academic background and then I have 20 years of industry experience which just tells you that I'm old and maybe you should listen to old people. Um uh and most of my work in the Bilpex project has been in helping and creating the documentation

and we were lucky enough last year to receive the CNCF Laura Mipum award for that work which was fantastic and thank you to the CNCF for that. Um, I'm also going to put a plug in for my employers who are Bloomberg um, who give me time to work on this. Um, and particularly in Dublin, we're hiring right now. We're hiring globally, but I put up two QR

codes. Uh, the left one is to uh, early career hires. So, I know at the moment the job market is tough, particularly for early career folks. Have a look at it. And then the other one is just for general uh, positions, particularly in Dublin, uh, which is where I work, but we're we're hiring worldwide. London, New York, San Francisco, Frankfurt. And in this talk, I want to

talk a little bit about two two kind of threads. The kind of road to 1.0. Uh where are we going? Uh what do we want to achieve out of a 1.0 version of the spec and then you have to have AI in your talk title these days. So, uh going to talk about AI and machine learning a little bit. Um, and what I'm really going to show

you is that from a a build packs perspective, it does not matter if you're building a fast API application or if you're building a machine learning training uh model. It it's the same process. So there's nothing scary from a build packs perspective uh when we try and do AI. Um, so along the way here, we're going to do a live demo. In fact, I'll do a live

demo as quickly as I can because people don't believe it unless you actually show them, do they? Um, and I'm going to use a Java application. I know this is heresy at CubeCon, but I'm going to use a Java application because a lot of us may be infrastructure engineers, but we end up supporting Python or Java application developers um because they are very very popular programming languages.

And I'll yeah demo with a Java application with a custom builder which allows me to introduce a bunch of build packs topics. But first, who here has experience with cloud native build packs? Oh, about half the audience. Okay, this is growing year on year. Joe, this is a good thing. Yeah, when we did this about uh four or five years ago, maybe 10% of the audience. Now

we're half the audience. So soon we'll get to 100% of the audience. This is going to be good. If you have experience with cloud native build packs, it might be via our CLI tool pack where you just did pack build and built an image out of your source code. Or it might be even be from uh through uh the spring boot project they have. If you've got

Java Spring applications like the one that I'll demo today, if you do spring boot colon build image with Maven, it will build an image and under the hood it uses build packs. You might not know it's using build packs, but it is. or maybe you've used our um our our KPAC uh Kubernetes operator uh which is written by a gentleman there and uh you know so there's

many ways to think about build packs um but today I'm going to demo a productionready Java application image and I'm going to developer think about it from kind of two perspectives the first perspective is as the application developer so I wear two hats I am an application developer and as an application developer what we'll see is that we'll use the pack CLI tool to build an image

out of the the Java codebase. Um, and then we'll run it on Docker or Podman or deploy it to Kubernetes. I'll just run it on Docker on my laptop today, but you could just as easily deploy this to Kubernetes. Um, and it is a liveish demo. So, it's live, but it is presscripted. So, if you see typing and I'm not typing, it's because it's it's presscripted. There

is an automated GIF, an animated GIF available if you want to have a look at it. Um, but what I'm going to do is pack build an image called example from the source code in this particular directory. But what is this source code? And if I come back over here, right folks, pop quiz. What programming language is this written in? >> Java 25. There's no public static

void main here. There's no classes. I've just got a void main at the top. I'm using io.printl print instead of system.out.print. I'm a Java guru. I'm not, but I got to pretend to be today. Right? So, I've got my folks who are Java gurus and they are doing things like this. Java 25. We're up to the latest specs on all the Java stuff. You'll notice we're using

VAR all over the place instead of the way some of us might have learned it in university in the last millennium. Um, I'm speaking for myself in this case. I know all you folks are much more modern than I am. But it is a complex application. it is doing something with numbers and then it's running it for 500 somethings epochs and then it's going to print out

some numbers at the end of the day. So we're doing a fairly straightforward machine learning task. Um what we're actually going to do is try and learn the equation that um the linear equation that fits these numbers as as pairs 1 3 2 5 3 7 and 49. Okay. So that's what it's going to do under the hood. As an application developer, I'm concerned about the logic.

I'm concerned about this deep learning 4j stuff. I'm concerned about using Java 25 and all the knowledge that I need to build up to use that. And it's using Maven as a build system to manage my dependencies. The last thing that I need to care about as an application developer is how do I build this into an image? So, how do I build this into an image?

And that is what our pack command line tool does. pack build example will build this current source code in this directory, this complex AI or ML application into an image. Because we're at CubeCon, we're going to turn on revoke flags. And because we're at CubeCon, we're going to use my custom builder, which is going to build all this stuff on top of the the CUDA base libraries.

And we'll have a look at that in a bit. But you can see that the build just happens. And it prints out a lot of stuff because I've turned on verbose mode. What you're seeing here is the standard build packs um uh uh analyze, detect, build, and then export. It's actually building the application under the hood, and it will kick out at the end an image called

example. And we didn't have to write anything more than pack build, which is nice. just to prove that that works when it's finished. Oh, I will uh docker if I can spell run-m So, I'm running the exam um example. It is going to run it. It's got to print out a whole lot of stuff after it's finished running successfully. But it has run and it has successfully

printed out some numbers at the end. Congratulations folks. We have a linear equation. Yay. So what we have done is built a fairly complex application using a custom builder. Um and the experience from the from the application developer is pack build. It's pretty straightforward from their perspective. However, oh, let me first explain what cloud native build packs are. This is why it's hard for us to explain

what cloud native build packs are. We provide this pack build tool, which is great, and a Kubernetes operator called KPAC, which does the same, but on Kubernetes. And because it's Kubernetes, it does it better. But hey, um, we maintain a specification at buildpacks.io. Uh, and we use this pack build tool or pack tool to allow the composition of different build packs. I could talk about those in

a bit if people want. It transforms your application source code into an OCI image and that's all it does. There's nothing more. It does one thing and one thing well. But it does provide a multi- vendor uh s structured repeatable build process and we are a CNCF incubating project. What do I mean though by multi- vendor? Well, as a project crazily because we're called billpacks.io, know we

don't actually provide any bill packs or not any production level build packs at least the production level build packs are provided to you by Heroku or there's an open source project called Pquetto which came out of the VMware Tanzu platform previously uh came out of Pivotal Google provide their own build packs for for Google Cloud Run um and you or me you know you will have internal

technologies in your company that you might want to write a build pack for and use and combine with all these existing build packs that you pull down from upstream Now um the application stacks that are supported by build packs are the application stacks that either Heroku or Pquetto or Google will support. These include kind of.NET, Python, Ruby, Java, PHP, TypeScript. I haven't seen a Forran build pack

yet, folks. So if any of you are passionate about Forran and want to get a FRAN build pack, please talk to us. Yeah. Uh but the uh they're interoperable. Um and what I'm going to show you what I showed you in the demo is me using the Bequetto set of build packs because I wanted to show you a production level uh example, not just a toy example.

I would be happy to that that image that I've just built. I would be happy to deploy to production if it went through our CI system and if the image was signed and all that kind of stuff. But again, it's easy. it's straightforward to to um to uh embed pack in a CI system. But the other kind of hat that I wear is as a platform operator.

So I own the Kubernetes cluster that these images run on. Um and you know you might call it DevOps in your company or dev sec ops or platform engineers, but we have control over the I can happily pull down Heroku's builder. They have really great build packs and they provide a builder. But in my case, I wanted a custom builder. And I wanted a custom builder because

I need to build my Java applications that do machine learning on top of a CUDA base image that's provided to me by Nvidia. And my Java CUDA builder is a or Java CUDA builder is a custom builder. And the kind of questions are from a platform operator perspective, you know, why do we need a custom builder? Well, a custom builder is is kind of an answer to

some of these questions that you see on the slide. What is our corporate base image? Right? In most companies, you will have a corporate base image where you buy buy it in from Red Hat or whether you use Debian as your base image or even as an open source project. In the BuiltPax project when we provide our examples, we base them on both the Alpine and the

Yunu uh base images. So, we have corporate base images even as an open source project. You know what language stacks do we support in production? Now just because Bill packs support net and Ruby doesn't mean that every team has the capability of supporting those in production. So you can pick and choose which language stacks you allow on your custom builder. Yeah. What are the runtimes that we

support in production? Again, just because I support Python doesn't mean I support Python 2.7. In fact, we don't support Python 2.7 for good reason, right? Maybe we choose to support only the last two Python releases, 3.14 and 3.13. If I own my own custom builder, I can centralize all of that build logic into the builder. So my application teams don't have to worry about that kind of

stuff. So that when we eventually move on to Python 3.15 and I want to deprecate Python 3.13, I do it in one place and not in hundreds of repositories all over the place. Um, how do I patch a security vulnerability? These are good questions to ask as platform operators. And how do I build bundle a proprietary layer? We have a you may have a proprietary middleware that

you need to bundle into your uh application images. How do we do that? Click. Yay. So, one of the answers is, you know, what is our corporate base image? Well, in this example, because I did it on my own laptop, what I did was I pulled down Nvidia's base image, which provided me a CPU version of CUDA libraries. Um, and all I did was add a CNB

user to it. That's all I needed to do. Added a CMBB user, which means that the subsequent builds, application builds, don't require the root user to run at all. So, all of this can happen in user space. your your your uh end user developers do not need access to root on any machine to do this which is a good thing. What language stacks do I support? Well,

in my builder which is defined in this builder.tumbl file, I chose to only support Java. But if I wanted to support Go as well or Python, h it would be a case of adding who here actually reads TML? Yeah, I find it one of those weird languages that I just can never get into my head. This is a list. So, I could add another another uh build

pack onto this list and reference it in an order group. I'm not going to get into exactly what an order group is today, but we can have ask a question at the booth uh later or catch me for a coffee later. But it's a tommo file and it's an image and that defines our builder. What about the answers to the other questions? You know, what runtimes do

we support in production? The Bill Packs booth is now closed, I believe, Joe, isn't it? Yeah, because we've done it. We're doing CubeCon differently this year. But if you want to reach out to us on Slack, if you want to reach out to us um via email or anything, we can explain the answers to this to us. We've got an active Slack channel where lots of users

have solved these problems in the past. cool. Fantastic. So from the build packs perspective, you're not giving your developers access to some kind of container file which they they commit to their repositories and make changes. We centralize all that build logic in a builder. You own the builder and you can pin specific versions of specific build packs. You can create the build pack ecosystem that you support

in your organization. You control the base images. um you know so your security team gets to manage the base image and a lot of organizations you're probably also the security team aren't you um so you know and we can enforce compliance we can make sure that we get high quality sbombs out of these images um we can uh make it uh we can uh we can implement

our organization signing policies and in again integrate these pipelines into vulnerability scanning cool but developers at the end of the They get this simple pack build command if that's what they want to do. A lot of my developers don't even want a pack build command. What they want is Jenkins just to run the build for them or Tectton just to run the build for them or ship

just to run the build for them. And this is cool. That's what I give them. But I control the builder and I control the policy that um builds their The challenge with a IML though is slightly different. Um because the infrastructure requirements are again a level on from just me building my standard fast API application. Um we need GPU drivers. This normally means the NVIDIA CUDA libraries.

Um there's large frameworks, PyTorch, TensorFlow. These things pull in a lot of different dependencies. Um and complex native dependencies as well. You know, some of these dependencies need a C compiler to compile them before they because they don't distribute binary wheels. And you know, as a as an organization, I want reproducible environments across dev and prod. I want to be able to build something in dev and

have quality QA it and make sure that I can get it to prod. And build packs can help tame some of this complexity. And there's kind of two answers to it in the build packs world. we can produce separate builders for different uh for different um target platforms. So your Heroku style pure compute platform where you run web apps might use one builder and your data science

cluster where you deploy data science things on top of GPUs might use a different builder or there's a feature that we've added in the past uh year called image extensions which would allow us to install the CUDA uh libraries from or RPMs or deb files as part of the build. Um, I tend to choose the um the the multiple builder one, particularly in this example, which is

completely fake news, folks. This example is complete fake news. It is pure slideware. And it is slideware because I've got an Intel GPU. Um, and I can't run any of these examples that use CUDA. Um, but if suppose we added Python support to our builder, we might rename the builder then to be polygot builder. So, it's not just building Java anymore. It's building Java and Python. And

I'm going to build a PyTorch Hello World example on my slides which would give us the output the standard build packs output where they're build packs are going on to detect what dependencies are you're using. They're going to figure out what runtime you need install the runtime layer install the dependencies layer and then build your application on top of that and export it to an image called

in this case my PyTorch app. So the workflow is the exact same from a build packs perspective whether it's an AI workflow an ML workflow or we're writing a standard micros service in some kind of language that deploys on a compute platform and similarly you know the kind of third step in all this is to run it on some kind of runtime that has access to GPUs

there's been many talks this week about Kubernetes and access to GPUs so Kubernetes is a great platform to run things that need GPU access but on your desktop you could use Podman or you could use Docker to run and give it access to the GPU and suddenly you've got yourself um a whole data science platform on your desktop and the output might look something like this. But

this is a lie because I do not have that GPU in my laptop. So I do not have an Avidia A100 in my laptop. I'm afraid I would like to have an Nvidia A100 in my laptop but I don't. But from that perspective, from AI and ML build packs give us that nice reproducibility that we're used to. We get kind of consistent CUDA and Python environments in

across our dev and We get this kind of separation of concerns where again as the infrastructure team, you control the builder and we let these expensive brilliant data scientists do the expensive brilliant data science and not have them worry about building images. Um, in terms of security, there's a really cool feature in build packs that you don't get in in many other frameworks, but we can do

automatic base image updates without rebuilding the application. So, if you have a bunch of data scientists churning out thousands of images and if you find that there's a security vulnerability on that CUDA base image, well, updating all those thousand images is a registry only operation for us. It takes a couple of seconds and it doesn't require application rebuilds. Um, and this is particularly because the CUDA libraries

are AI compatible across minor versions. Um, soon as you cross major versions, well, you have to rebuild. That's just life, right? Um, cool. Oh, yeah. Which brings us towards to to our kind of a section is this is a maintainer track talk, folks. So, I am going to talk to some of you who are maybe a bit more familiar with the spec um and what we're trying

to deliver uh before we get to 1.0. If you're just if you're a pure user, hopefully the demo that I showed you at the start motivated why build packs are useful. And if I didn't, then you need to ask me about it at the end. But going towards 1.0, what we want to do is produce um a build packs API and platform API that are very very

stable. They have been stable over the past year or two. arguably even three. Um, we're just mostly trying to get rid of some stuff that we have deprecated and add in one or two new features. But if you have new features that you want to add in, please talk to us soon because at some stage we are going to freeze for 1.0 and have no breaking changes

be uh in the 1.0 release. There'll be a 2.0 at some stage in the future, but it's taken us eight years to get to 1.0. And it is a promise to the ecosystem that we're safe to build upon and safe to depend upon. And a lot of organizations have found this over the past eight years that if you adopted billpack six years ago, even your bill packs

still work today without very much um uh without much maintenance at all, which is good. So what bacon changes do we have to deliver before 1.0? Well, they're mostly done, which is great. Um we did have something called stacks and mixins. those have been deprecated. Um, we no longer use them, which is why it's straightforward to build a a new custom builder. You just add a uh

CNB user to that image. Um, we haven't removed those code paths yet. So, if you're still using stacks and mixins, great. They'll work for the moment, but closer to 1.0, we will remove those code paths. Uh, we're changing how shell processes are handled in the launcher. Um, this has been an RFC. We have an RFC process as an organization. um you propose an RFC, we debate it

and then we implement it. This has been debated a while um and it improves security and signal handling in particular particularly for Python applications. There's there's a lot of interesting things around shells and signals and Python applications. And then the Docker file image extensions that you've seen uh me demo in the case of CUDA where you might want to install those CUDA RPMs or deb files at

build stage that's supported by us supporting a subset of the container file syntax at build time and unreadable table time folks. Yeah, there's a lot of activity going on. Um and all this is communicating is that there is a lot of activity going on. These are RFC numbers for features that folks want. Execution environments has just been done, which means that we can we we can now

distinguish between your test image, your dev image, and your production image at build time and enable different features in each one, which is quite cool. Um, hotel, of course, your application has support, but the build at build time, you want to get just some metrics out of the build system telling you that we're building your images. Again, if anyone wants to work on that with us, we're

always open to help, folks. um image multi-arch support. It's there. We spoke about this about two years ago at CubeCom. It does work on ARM and it does work on AMD 64 and it does work on other architectures. What we want is just a nicer experience at this stage such that when I build on ARM and AMD 64 that we get one image at the far end

rather than the two that we currently produce. Add parallel cache faster builds. But what does it actually really mean for application developers? stable CLI. So the stuff that they learn today will still be able to use in three or four years time. Um build build experience with no surprises. Multiarchchitecture images out of the box. They are there. They're just not out of the box. Better error messages

and observability. Why did my build fail? Well, it's good to know when for build packs authors. So if you're internally in a company particularly and if you need to write a build pack to provide some extra functionality well we have a frozen build pack API when we get to 1.0 currently it's at 0.15 and it has been quite stable but 1.0 will be stable. Um, you know,

image extensions for advanced customization. And as always, since the start of the project, we've had first class sbomb and signing support on for images. Um, if anyone wants to ask me about sbombs later, feel free because they're kind of uh very cool in the build packs world. And then for platform operators, for folks who actually run the Kubernetes clusters, stable platform API to integrate with um and

a builder spec for governance. you can read the spec and see exactly what's going on. And as always, we are OCI native. We have been OCI native since the start. Um, we build things that are ready to deploy onto Kubernetes. They just kind of accidentally run with Docker and Podman as well, but that's kind of cool. Fantastic. How to get involved? Uh, GitHub. All our source code

is on GitHub. um our RFC's in particular. If you want to propose a new feature or if you want to see which features have been proposed, have a look at our RFC documents or better yet just ask Claude to summarize them for you. Yeah. Um tell Claude you need something and tell it to find which RFC is most important. Um we are on the CNCF Slack which

is good. And we do have weekly working group calls that um that that flip-flop between kind of a um an EMIA kind of friendly time zone and a US friendly time zone. So if you're in India or Europe, then we've got uh every second week we've got a call that probably suits you a bit better than the later evening call that our uh US friends join uh

on the alternative week. We are a CNCF project. You know, contributions from everyone make us stronger. We are open source. We've always been an open source project, a multi- vendor project. So please jump in and um help. And other than that, thank you for being a great audience today. Do folks have any questions? This is where I get you calling to ask me about rebates. Well, so

the the kind of feature that I alluded to earlier about being able to replace the base image uh without having an image rebuild is what we call rebase and it is a registry only operation. Um you can see on the left hand side here I kind of draw a picture of of an original image with a base image in it and then some layers on the top

and all we're doing is is rewrite well adding a new manifest on the registry which references the sha of the new base image. So it becomes really really quick. If you've got a CVE in that base image, you can fix that CVE, you know, in less than a minute across thousands of images. Um, some real world numbers, I've heard people say 10 seconds for 10,000 images. Um,

so you know, it's it's pretty good. So if your base operating system has something like lib Z in it, which is something that that has CVs every so often, or if it had, for example, log 4j in it back in the day, um you could rebase the entire fleet of images in in 10 seconds rather than the months that it took a lot of organizations to deal

with with those kind of issues. Thank you for the question, Colin. >> Oh, yeah. Okay. Um, cool. Fantastic. Any other I got a question over here. >> What about signing then? >> What about signing? So, the question is about how do we do image signing? Um, the build packs project does image signing if you use our KPAC uh Kubernetes um operator for building images. But again, we're

using cosign for that. And cosign is a again it we think of these things in the using the Unix philosophy like build packs builds the image cosign signs the image and by separating out the concerns it uh we can collaborate with them. So if you have an existing signing um workflow that uses some other tool to build the image and then cosign to sign them. Well pack

is a dropin replacement for whatever other tool you're using right now. Um, but it does centralize that control of policy for the builds with an infrastructure team rather than distributing around multiple development teams. Does that answer your question? >> Partially. Well, maybe we can get into more detail afterwards. Any other questions? Cool. Thank you folks for your attention.