Harbor Project - The Maintainers Session - Yan Wang, Broadcom & Vadim Bauer, 8gears

This talk introduces advancements in the Harbor container registry, focusing on new features and updates. The speaker, Vadim Bauer, highlights the Harbor CLI, designed for CI/CD workflows, emphasizing its recent popularity and utility for image management. Additionally, he discusses the development of Harbor Satellite, a stateless, secretless registry aimed at edge computing, featuring integrations like SPIFFE and Spire for enhanced security. The presentation also covers improvements to the proxy cache, including connection limit controls and customizable artifact serving behaviors. Updates on performance enhancements and the support for AI modules are shared, indicating Harbor's evolution as a key tool for OCI artifacts management.

Hello everyone. I'm really excited. So many people here and I'm more excited that you're still here in a Thursday afternoon. Until almost everything is over, right? So you're happy that you joined Harbor. So it seems to be important and I'm happy for that. Um my name is Vadim Bauer. I'm one of the maintainers of Harbor alongside Jan. So hi. Um nice to meet you. So I'm a

year from VMware. So I'm the maintainer of Harbor. Thank you. Thank you. >> Exactly. So we're here today to tell you something about Harbor and also what is new in Harbor and what we are working on. Okay, let's get started. you're probably fairly aware what Harbor is, right? So who is new to this Harbor ecosystem? Very brave of you to raise hand. Not many hands, but it's

good to know. So I'm I'm just giving a really short recap. Harbor is a container registry, policy driven. You can use policies to control your image management and and signing artifacts and that things. So you're probably all aware about that Good. Let's start about something that is related to Harbor, but it's not directly Harbor. And this is a project we've been working on for quite some time

and it's the Harbor CLI. So if you don't know what a Harbor CLI is, it's basically an alternative to the web UI and the design why we initially designed the CLI was primarily for CICD workloads and also for people who like the command lines. So you can automate and workflows in your CLI, sorry, in your CI or CD. And nowadays CLI is uh popular uh popular again,

right? For agentic workloads. So, you can hook this into agentic workloads and get information about your images in your registry. This helps sometimes as well. Um And if you didn't try it yet, go ahead, download the Harbor CLI. Um and then you can use it, I think, on all the all the Yeah, Linux, Windows, what whatever you prefer. I'm just giving a a quick demo because there

is some some typical use cases that I like. So, I'm just trying to make it work now. Um Um So, that's basically the CLI. We can have a Harbor command, and I have prepared a list. So, we can just list the images. You see the versions. This is a tabled output. You can output it in JSON, YAML, so that you can process it, you know, with JQ

uh already done um agentic, so that you don't have to deal with tables, which is not nice. Uh and well, you can create projects, tag projects. Basically, nowadays, uh the CLI is covering, I would say, 95% of the use cases, maybe 90% and um Yeah, so we are happy that uh Yeah. And the interesting part about the CLI, it was mainly created with uh uh mentees from

the Alithis mentorship program. So, over the last, I think, 2 years, every couple of months, new mentees um contributed to the project and evolved the project. And I would like to highlight three per- three people here, Patrick, Lakshit, and Prasanth, who is also present here, who contributed significantly to the project. And they are already all maintaining and sticking around and and and and and maintain the CLI

now. So, it's not just a short-term engagement that they did. Well, there is some announcements we want to make today and we're celebrating 130,000 downloads of the CLI already, which is a nice number. And another big announcement is uh we are now kind of a unboxing or unsandboxing the CLI. And uh I wanted to do this empty Yeah, I wanted to do this actually uh straight away

and go to the repo and remove the annotation And then we just We just make it a little soft launch here and just Remove did work. Let me just try it again. All right. And you're now participating and Right. The Yeah, give it a try. It's really nice. It makes life easier. Um The next thing excited to share with you is we made some progress on the

CL on the Harbor satellite part. So, if you know don't know what satellite is, our goal with the satellite is to create um a stateless, secretless, zero-trust registry for the edge that you can um to distribute your container images to edge locations. And it's Is it not It's not a proxy. It's really an active replication, so you actively uh replicate images into the satellite, so you can

manage hundreds of registries with hundreds of different configurations, right? So, that's the goal to every satellite has different images, different setups, different configuration, and this with ground control and Harbor paired, this makes easy to manage all this complexity with satellite. Um recently, there were two additions. Um one um one addition was the um capability of uh using SPIFFE and Spire integration, so that you have a identity

back with SPIFFE and Spire, and now satellite can also act as a proxy, because sometimes, if you don't have control about images that are on the edge, you need to need you need to proxy things, and um although that's not the whole concept of the satellite, because we think it makes sense to be really deterministic and know what should be on the edge, uh we still see

the possibility that a proxy might be a good intermediate solution until people are getting there to really control what should what image should be on which edge location. Uh you can learn more about uh this on the on the on the site. And yeah. Um for the future, we are working and now on the air-gap support of satellite, because initially satellite was not designed for air-gap, because

there is always an outbound connection, but we see that this is important part, and we're working on ways how we can make this work. And our core requirement that we want to be fulfilled is that we want to have the same process, same tools, same workflows that are working on the connected edge, right? Or on the air-gapped edge, and we want to have this fluidity of all

the different types of air-gaps that exist so that one tool can support all of it. That's that's our goal and we're trying to make this work. So it is more of a concept phase at the moment. But this is our let's say um primary goal that we want to achieve. Same tool, same processes does in in different types of edge or air gap uh environments. All right.

Then I hand over to Yan to talk about what's uh what we have in in Harbor. Yeah. Thank you, Adi. So hi. Um let me give you some updates regarding the uh latest uh enhancement on new features in the latest uh several Harbor and patch releases. And um the first one is the um enhancement regarding the proxy cache. And so um we know that the proxy cache

is one of most of popular feature that um of Harbor. It's used user using the proxy cache to to serve the the upstream and artifacts locally. So we resolved several hot issues from the community. And the first one is that in we um introduced a new parameter to um control the limit connection that Harbor sent to the uh upstream. So um because that a lot of user

from community reports they hate the rate limit from the upstream when try to proxy images with Harbor because there are a lot of users are using Harbor internally. So they are using pipeline or anything. So So up to avoid to uh hit the rate limits from the upstream. So, we add a new parameter so you can up uh set it to uh you can fail earlier. I

mean, that you can fail at the Harbor side rather than fail at the upstream side because the once you hit the rate limit rate limits, you cannot able to pull any other images from upstream. So, this is uh um um useful feature for you to try. And uh another one is the A lot of user ask us to to modify the behavior. Um so, the current Harbor

proxy cache just align with the status from the upstreams uh repository. Um I mean, when you try to update your image in the upstream registry like Docker. So, Harbor will immediately save the latest one for you and for your pipeline. And uh if you remove it from the upstream, so Harbor will not serve it anymore and return a 442 your CLI client. But, but a lot of

users uh came to us to ask that I still want to serve the removed artifacts locally. So, uh we're going to um update the behavior so that you can decide whether to serve the removed locally. So, by default, we change the here from not to no to yes. That means we cancel the removed ones. So, um if you don't want if you don't want do not want

to serve it anymore, you can remove it from the proxy cache um um projects. So, the third one is that um we fix one um problem that is pull time. So, um previously uh Harbor has one issue that we do not update the pull time for index. So, when you try to pull a uh reference from index, like you you just pull a um um image from

any other any index, Harbor just update the pull time for last specific reference, not update the pull time of the index. That means for some certain uh situation, Harbor will uh the the tag retention will remove the whole index, include the pulled reference. So, we um in the latest patch release, we will resolve that one. And uh the last one is the uh we uh a lot

uh update that um um proxy adapter list, uh because some of the proxy adapter are auto maintain. So, the maintainer are um uh lack of the API knowledge for those uh adapters. You You mean You mean a lot Harbor supports a bunch of um adapter list, like GCR, ECR, Docker Hub, but some of them are auto maintained. We just uh hide them from the UI. So, if

anyone from uh community have knowledge to help us to maintain them, we can add them back. So, next, what we will uh do for the proxy cache. The first one is that we want to enable the referral API support. So, um the referral API is introduced in distribution supply 1.1. So, that means user can attach anything to your subject manifest, like signature, SBOM, anything else. So, if

you do this in the upstream um uh registry, Harbor right now do not help you to proxy that. So, when you try to pull a image from upstream, Harbor do not help you to pull the S-Bom or signature alive with it. So, for the future, we want to support this. so, the last is the another hot topic from the community. So, user want to ask to limit

So, what can be proxy cached? So, right now, Harbor can help you to proxy everything from upstream. So, but came to say, "Hi, I just want to limit the proxy scope. Like, I just want to use the whole or my pipeline to proxy several specific repository or save several specific versions of of my um artifact. So, and we we plan to do enhancement on this part. yeah,

um another enhancement is that we we do some performance enhancement on Harbor garbage collection. So, you may know that a lot of people report the issue that we hit the performance when to use Harbor garbage before I I I tell the the details, let me firstly to break down a push operation from the Harbor perspective. So, when a OCI client, when you try to use the OCI

client to push a artifact with tag, so, the first Harbor will Harbor core will receive the request and create a tag in the Harbor database and then forward the request to the distribution. So, distribution receive the request and land the tag file into storage and as well as the layers and the manifest. So, you may notice that we learn the tag both in the Harbor database and

in the back end storage. because Harbor leverage the database to manage the tag, so the tag in the storage is useless. We do not use it, the garbage collection has to deal with it in the deletion in the artifact deletion. So, this is the um um performance bottleneck because the the tag deletion in the storage layer is the file system layer operation. It takes time. So, it's

especially when the distribution called the storage API to remove a tag file, it has to uh travel all the files under a certain um repository. So, uh especially for some object storage, it takes some time to finish this operation. So, this is the performance bottleneck. In the in the latest Harbor uh minor release, we update the behavior. That means when Harbor core receive a request and push

by tag request, we just learn tag in the redirect the request to the Um from push by tag and to push by digest. So, um you can see that in this page, you will see a tag file. When the be- the previous behavior that the you learn a tag file in the back end storage. So, after we modify the logic, so when after you push a image,

there's tag file generated. So, that means the garbage collection will not need to care about the tag deletion. So, garbage collection um becomes quite greater and the tighter. So, um and the there's no API and AI break change because Harbor's you can still pull by tag. because Harbor um manage the tag in the database. And we also provide a a checkbox for you because the the cur-

the existing image still have tag landed in the that storage. But, if you want to try the performance enhancement immediately, you can just enable this checkbox to for the existing uh artifacts. So, okay. So, um So, nice to is the regarding the module distribution. So, we did some uh um explore around the AI modules. So, we did some collaboration with the um Alibaba from another company from

uh China. And they contribute a module spike module pack um um project into uh CNC CNCF. It is the sandbox project right now. And uh So, regarding the AI modules, we we know that there are several issues um currently because the um compare with the traditional OCI artifacts. So, the first one is the size. So, you may know that the the the AI modules are much bigger

than the OCI artifacts or the container images. So, user may want to to distribute them quickly. So, they want to set up a private module registry locally to serve the those large size uh module. And uh another one is the immutability. So, so similar ways the OCI content. So, as for the AI modules, they just want to uh share the same uh um principle like the build

ones and the distribute everywhere. so if we um so Harbor can handle this for the OCI artifacts, so we want to try to uh upgrade Harbor to handle this for the AI module artifacts. And uh we also find that there's no standard uh industry standard for AI modules rather than the OCI artifacts. And even user uh deploy their AI workload bit on top of Kubernetes, they and

use the uh custom solution around object store. So, they may use gate or uh up as to for them module management. So, so based on that, we collaborate with Alibaba to uh set up a standard for the AI modules and uh we try to leverage the the the project modules back to build a AI module artifacts and then use Harbor to manage them. So, um here is

the flow to about the integration. So, uh regarding the um modules pack uh repository, you can go to the link I paste it here. You can go to link to know more details. It It is the sandbox um project for our CNC. So, here's the build process. So, so the first thing if you hold the a module in a hugging face, you can clone the entire repository

locally and then use module CLI to generate a module file that is similar format for a docker file and then you can log into the harbor repository and then push the generated module artifact into harbor because the module artifact is OCI compatible. So, then after you push your AI module artifact into harbor, so then you can leverage all the capabilities of harbor to manage your uh module

uh artifacts like audits, versioning, uh any uh replication, access control, etc. And then I want to um a demo here. let me try to uh navigate to the uh hugging face here. let's try to clone the module repository. So, locally. So, after you after we uh clone it, we will use the uh mo- module CLI to generate a module file. So, after we run the clone clone,

so there is a module file generated. And we can see the module file format is very similar. It's quite easy to understand. So, the name, the arc, and the with the configuration files, the module here. So, after that, we can use the uh to log into the harbor because the module CLI is also a OCI compatible after we log in, we will build from the module file.

So, you can see that we build the module file into the OCI layers and as well as the OCI manifest. So, we can also see the the builded So, then we can try to use the CLI to push the AI module artifact into harbor. So, you will see those layers, blobs, and config manifest are pushed successfully. And so, after I let's try to um log into the

harbor website. So, we could see the updated the uploaded um AI artifact here. And the we have a uh specific icon for the AI module OCI spike. And then as for the details page, you can see all the details of um like the labels. So, they are the same ways the hugging face. as well as any configurations in the license license here. You can also view all

the summary uh summary and the license. Also, view the files. This is the file list. The same ways uh you can see the the lock layers, the file layers. So, after that, you can leverage all the existing capability to manage your AI artifact. So, then your workload can deploy the AI module to into your AI workload. Okay. So, when you try to deploy a AI modules, you

can just uh specify specify the reference. It's very It's same as the container image. So, last um I want to share one survey from uh CNCF. CNCF did a survey last uh last December. Um to uh regarding the graduated to ask the CNCF users a- about the uh adoption rate of the graduated projects. So, I highlight the Harbor. The Harbor is in the middle of this chart.

So, from the result, they are um more than um user are using Harbor in their production environment in a more about five are investigating Harbor into their uh production environment. Uh That means Harbor is becoming a de facto OCI registry within the CNCF community. So, So, yeah. Uh I will end with uh something I heard from community. Uh one user come to say to me that it's

worth to uh maintain this project because use and love it. Thank you. Thank you. All right, thank you. Um I think we have some time for questions, no? There's two two minutes for questions, okay? So, best would be if someone come here if you have questions, otherwise I can repeat the question so we have it on recording. Are there any questions? One, two, three questions, no? Okay.

Thank you very much for coming. Thank you for everyone for being here. Thank