Project Lightning Talk: Argo Workflows 4, What's New And What's Next - Alan Clucas, Lead

This talk focuses on the updates and new features in Argo Workflows, a Kubernetes-native batch processing engine. The speaker discusses the recent release of version 4.0, highlighting significant enhancements such as artifact drivers that enable custom plugins for data transport, and improved full cell validation in Custom Resource Definitions (CRDs). The session also touches on the new controller capabilities, including pod restart strategies, and a brief overview of planned features for future versions like workflow tracing and a reworked Directed Acyclic Graph (DAG) engine. The speaker aims to simplify expressions within Argo Workflows to improve usability and integration with tooling like Helm. Overall, this session provides insights into both the current state and future developments of Argo Workflows.

Hi. Um, you've probably heard of Argo in general, Argo Workflows is one of the sub projects. It's actually the oldest. Um, and if you don't know what Argo Workflows does, it's a batch processing engine for uh, Kubernetes. It's very Kubernetes native. It runs all of your steps in pods. Um, so you can define a series of pods as steps or a DAG and it orchestrates them and

provides facilities for passing data between those pods. Um, that's the detail I'm going to be able to give you in five minutes. Um, because what I'm going to talk about is what we've done recently. Um, 4.0 came out earlier this year. It's got a number of new features on the screen. Um, artifact drivers are probably the biggest bit which is so you can write your own plugins

to uh transport data in and out of those pods. Um, I did a whole talk on this about two hours ago. So, um, that will be on the Argoon YouTube site, um, in a couple of weeks, I imagine. Um, we've got, uh, full cell validation in our full CRDs. Up until about 3.7, our CRDs were very, very shrunk. Um, and they are now very full. So hopefully,

if you submit a workflow that really, really won't run, it'll tell you you've spelled stuff wrong and all of that kind of thing. Um, we've got, uh, in 3.6, Six, we introduced mult uh plural versions of a number of uh primitives and they have been removed in 4.0. The singular versions have and there's a new tool argo convert to help you um get updated to version 4.

Um the uh controller will also restart pods that have never started. It's got a restart strategy. If what you're doing is your pod might be not um item potent that would be bad if we restarted it. But under these cases we do know we can restart it. So we we do that. Um already merge for 4.1 is workflows tracing. There's a QR code and a link to

the YouTube talk on that done a couple of years ago. It's not that long. Um but this is tracing of the full workflow. Um working with CI/CD uh SEMCOM in open telemetry to help define what uh traces of this sort of size mean. They're they're working on GitHub. You can get the same sort of thing out of GitHub now. Um but this is allow allows you to

see how your workflow is using its time what bits are taking too long and you can you can also imp trace inside your workloads as well and that will become all part of one big trace. some future plans uh we've started working on reworking the DAG engine uh to separate it from the part that creates the pods so that they are we can do clever things with

the result of that. At the moment, steps are implemented completely separately to DAGs, uh, which leads to some interesting bugs that are on one side and not the other. So, um, this will also mean that steps are implemented by transforming them into a DAG and then running them through the DAG engine may lead to DAG plugins in the future. Um, and on top of that work, we're

hoping to be able to basically reuse pods because some use cases, uh, the pod spin up time is significant and means that workflows isn't really the right tool for the job all the time. Um, we'd like to allow you to use workflows in that way. You'll just be able to annotate existing workflows saying this is a reusable type of thing. Um the trade-off is you lose your

pod isolation, but um that is the only way of fixing that problem. If you don't want to start a pod, you can't start not start a pod. Um and the other thing that we're working on is uh expressions. Expressions are difficult in cargo workflows. They're quite legacy. They've got uh a library called complate that does some of them and then other libraries that do other bits of

them. And this is all a bit uh of a headache to keep maintained. Uh we would like to make expressions much easier to work with so that when you create a workflow, you know it's going to work. You can understand what the expressions really mean rather than having to guess. you can test them maybe before you run them in workflows and they become a bit more readable

and they don't conflict hopefully with Helm's templating system which is fun if you want to template a a workflow in Helm as well as using Argo Workflows' templating engine um I will be here all week um I work for Pipkit our booth is 494 we'll be in the project pavilion on the Argo stand if you'd like to ask any questions about this thank you very Awesome. Thank

you, Alan.