Virtual Power Plants (VPP): How They Work and What They Are - LeRenzo Malcom & Mario Flores, Enpal

This talk focuses on the concept of virtual power plants and their role in stabilizing energy grids, particularly through the lens of a company called Enpal from Germany. The speakers explain the workings of traditional power grids and the crucial synchronization of generators to maintain energy supply and demand. They cite the Texas power grid failure as a cautionary tale of how extreme weather can disrupt energy supply, leading to catastrophic results. The talk introduces the virtual power plant model, which leverages batteries and renewable energy sources to provide grid stability and supports energy trading systems like day-ahead and intraday trading. The speakers detail how their system can respond to energy demand fluctuations and grid events more rapidly than traditional energy sources, reducing reliance on fossil fuels. They also discuss the integration of IoT technology and platforms, including Kubernetes, to manage the virtual power plant infrastructure efficiently.

Hello. Thank you all for coming. Uh and we are about to start our KubeCon presentation on what is a virtual power plant. Who's excited? Woo! Woo! Wait, no, no. Come on, come on. Wait, one more time. Who's excited? Woo! >> [screaming] >> Woo! Okay, cool. Let's start. All right. Oh, no, go back. Okay, there. So, I am Lorenzo. And I'm Mario. And we are both at Enpal,

which is a solar panel company in Germany, and we are trying to build Europe's largest virtual power plant. And we're going to go through some very difficult concepts. Uh and we're going to start with first like talking about the star of the show, which is actually the grid. After we talk about that, we'll talk a little bit more about renewables and how they actually fit in. And

then we're actually going to talk about the concept of a virtual power plant and how it unifies both of these things together. So, we want to start with the grid. And to understand the grid, we actually want to understand a little bit about how power generators work. So, I think this is one of the hardest concepts to actually understand. And if you think of a power generator,

think of it a lot like an airplane engine. It actually goes around in circles at a certain frequency. It has a certain rotations per second. In Europe, that frequency is 50 hertz. In the United States, that frequency is about 60 hertz. When we put multiple generators together, we make sure that we sync them to the same frequency. And this gives us a stronger signal. Uh but when

things go wrong and the generators get out of sync, then that's when stuff starts to break down, right? So, as it gets worse and worse, we get a weaker signal. And when things start to deteriorate, the actual machinery can break. Right? So, at that point, we have to turn off uh cut off some of the generators in order to make sure that all of the other don't

get damaged in the process. So, let's look at this in a little bit of an example. So, I am from Texas. I don't know if I I didn't actually say that. I'm actually from Dallas, which is right there in the northeast. And one thing I guess a lot of people know about Texas is we don't really do cold weather. So, in 2021, we had one of the

coldest winters on in history uh in Texas. And that caused a lot of issues with our grid. It caused one of the biggest grid failures in US history, actually. And first, if we look at this map, what you want to notice is So, we have all of the generators in Texas put together, right? So, look at the different colors. So, the blue ones in North Texas, those

are wind generators. The orange ones around are gas generators. We have two nuclear generators and a few extra coal generators around, right? Again, all of these are connected together, and they're all rotating at exactly the same frequency. So, when we have our giant winter blast, what happens? So, the first thing that happens is in North Texas, the wind generators actually freeze, like actually freeze, which means they

stop rotating, right? So, obviously, when that starts to happen, things go wrong. Uh next thing that happens is our gas wells actually froze, like literally freeze. So, then we have to start to cut off power to those in order to make sure that the other generators don't fail. Then pretty soon, Texas power starts to go out Dallas power starts to go out. This is where I'm from.

Soon after that, the actual coal started to freeze, which I was very surprised that this was possible. Um and in Houston, this is where Beyoncé's from, by the way, uh power starts to trip, and the actual nuclear plant in Houston, uh the sensor lines actually froze. Right? So, all of this happened in the span of about 70 hours, and we soon started to get a cascading failure

across the entire state. So, when that happens, basically, if everything starts to go offline, you have very little time to fix the issue uh or else everything becomes a blackout. And when that happens, you have you have basically no power for several weeks. And in Texas, we actually avoided this by 4 minutes and 37 seconds. Right? So, like it's a very very small window in which to

fix these things. Um and in our case, one of the other problems that happens is when all of these go off, people continue to use their energy, right? Actually, it's really cold, people want more energy. So, there's a lot of demand, but there's not very much supply. So, then in that case, variable energy prices kick in. And so, obviously, energy prices become very very expensive. Some people

had prices of up to 10,000 uh dollars for that month. My parents actually had a bill of about 3,000 for that month. And in the end, no one actually wanted to pay it, so that company went out of business. And this type of thing causes, you know, 200 billion dollars in damages within the first 5 days alone. Sorry, the first Yeah, 5 days alone, we lost about

50 billion dollars in damages. So, we want to prevent these things from happening. All right. So, let's try to understand this problem a little bit better with a little bit more context. Let's talk about the grid. >> [clears throat] >> The grid is actually uh the European grid is actually the largest machine ever. Um it's kind of hard to conceptualize it when you're thinking at human scale.

So, let's think about let's look at the uh one of the largest factories in the world, the Wolfsburg car factory. It's 60,000 workers over 6 and 1/2 kilometers. But that really pales in context of the grid. The European grid has 2.3 million workers working on it. Uh it it It's over 305,000 lines of transmission lines uh connecting 400 million consumers generating just an ungodly amount of energy

between all the generators in between. Uh and here's the thing. This machine has never been down. This is the hardest [clears throat] SLO of all time. If you thought 5 9's was difficult, try that. uh so, this this power grid is actually, you know, it's designed for a different world. It's a few large central generators transmission transmitting over high voltage um energy backbone with one-way power flow

to passive consumers to achieve that electrification. Uh the breadth is, you know, all across Europe starting from a few power plants around Munich re- reaching out to the regional network, national scale, into a unified grid. Uh all synchronized on that 50 hertz that uh we spoke about before. Uh and here here's the thing about this. energy consumption and demand uh is almost instantaneous. It's about 2/3 the

speed of light. When supply is less than demand, that frequency starts to dip. When supply is greater than demand, the frequency starts to spin up. So, the grid has no buffer. Every watt it must be consumed almost instantly. The difference between everything being fine and total collapse is 2.5 hertz. That's actually less than you can hear. That is less than the difference between two piano keys. Fortunately,

we have had a few traditional tools to help us balance that grid. Uh so, let's assume, you know, we're on a pleasant day, everything's operating normally, we have spinning reserves on standby. These are actually gas usually gas plants that are kept running at a low capacity levels that if something goes wrong, we can spin them up when the when the frequency starts to drop. Uh if things

as that starts to happen, our peaker plants start to fire. These are gas plants that are specially engineered for cold starts to respond quickly to grid events. They'll max out as they try to keep that frequency at a stable level. And you can see poor coal here in the corner can barely move. It's a little small to see, but he can bump his production only by about

3%, which is not very much within not fast enough to respond to grid events. If things go too fast and the frequency dips further, load shedding uh will start to happen. That's when um consumers are literally disconnected from the grid so that we can try to uh preserve that frequency. >> [snorts] >> Uh eventually, if things continue to go poorly, the generators will literally disconnect from the

grid so that they don't damage their equipment. And the the a black complete system failure could occur. Let's look at this a little bit in action. Uh this is sped up by quite a bit, but you um the grid, you know, it's it's it's designed for recovery. A small dip, no problem. Three megawatt of power knocked offline, you barely notice. Let's say demand drops and uh we

have too much electricity. Also, we can curtail production and keep going. But occasionally, there are catastrophic >> Oh, no! Actual fishing attacks that >> This actually happened. This actually Oh my god, IT'S A BLACKOUT! WE'VE BEEN HACKED! >> [laughter] Uh so, this this is not theoretical. This really has happened before. >> has happened. This is just for context. All right. So, all this effort into keep the

grid balanced is incredibly expensive, right? There's peaker plants uh that are kept idle most of the year. Uh we talked about those spinning reserves. Also, these plants are just running, burning fuel that isn't uh delivering usable electricity. Uh when we had do have too much wind and solar, we have sometimes have to turn it off because uh it has nowhere to go at certain intervals. Uh there's

also as the renewable energy and uh revolution and we get variable power costs, we have grid bottlenecks that actually prevent that energy from going going where it needs to go, and it's trapped in a localized environment. Okay, so let's come back to Texas and then look at that again and try to understand what what, you know, we had from earlier, right? So, to start off, uh we

had extreme cold. Texas doesn't do well in the cold, right? And then many of the assets actually froze. This caused a cascading effect, in which case we had to turn off other generators, and actually start to do load shedding. Doing load shedding means that there's also less supply as as we start to turn off more and more generators, and this creates a cycle, right? So, this cycle

is actually what we want to avoid, and this cycle is one of the main problems that we're going to be looking at. How do we fix this when it comes to a virtual power plant? Now, this isn't an isolated incident. This type of thing happens all the time. You might remember last year in Spain and Portugal there was a gigantic blackout that happened. Lots and lots of

people lost power. In Berlin recently there was also like several events that caused power to go out. We're actually based in Berlin, but this is not uncommon across the world. This happens pretty often. And so, one of the biggest issues we have to deal with going forward is limited flexibility. The old grid actually just doesn't have really good ways of responding to events. It doesn't have a

really good way of getting energy into the grid quickly when you need to. And so, this is what we want to fix by adding in other energy sources like renewables. So, it'll probably surprise nobody here that renewables are exploding, right? This is Germany alone. Growth is not slowing down. But what this does, it introduces a new problem. And this is called a duck curve curve problem. So,

every year new solar comes online, and as that solar comes online, you can see that in the yellow area here, we get more and more energy production from solar. And what happens to the grid? The we start to have so much energy in the middle of the day that we don't know what to do with it. Prices turn negative. Producers are literally paying money to put money

into the grid, and consumers are getting paid to consume it. As the sun goes down towards the end of the day, the solar production starts to decrease, and all these other fossil fuel plants have to step in to shift the load. Now, clean energy has outgrown the grid. This chart shows the cumulative amount of curtailment that we've done. This is free energy that we basically have to

throw away. Since 2015, we've had about 66 terawatts of energy that we just could not use. We paid producers 6 and 1/2 billion to manage that, and we generated almost 28 megatons of CO2 that didn't need to be there. I mean, just just in 2024 alone, the amount of energy that was curtailed is enough to power 2.7 million homes in a year. So, can actually can you

go back a slide? I think it really one more, cuz I think it's really bears repeating. And that, you know, we generate so much energy with solar and renewables that we don't have a place to put it. Like, that's actually the problem is we're generating so much energy, but there's nowhere for it to go. So, when we generate energy during the day, most people are at work,

they go to school, they do things. It requires less energy from the population to just do that, right? So, when we're generating solar, we actually can't use most of it, and this is one of the biggest problem with renewables, right? So, one of the ways that we can fix this Oh, sorry. One of the ways we can fix this, right, is in a nutshell, we just add

batteries, right? Like, that's that's actually really really easy to do. So, in this way, we can allow solar producers and also wind producers, a lot of other people, to become a part of the infrastructure, right? So, people who have solar installations at home, they have solar installations that are connected to an inverter and a battery. A lot of those people also have an electric electric car. And

what is an electric car if nothing other than a giant moving battery, right? So, we can take all of this extra energy, especially for the car when we're generating we have cheap energy prices during the day, you go to work, you charge your car, you get paid to charge your car, and now you can use that extra energy and sell it back on the energy market later.

We'll talk a little bit more about this, but what this means is that we can help control the energy flow on the grid. We can help do stabilization on the grid. So, that's what we're going to talk about with the virtual power plant. Okay. All right, so now we're finally here. Okay, it's KubeCon. I'm going to talk about Kubernetes. Woo! Yeah. All right. what is a virtual

power plant? Finally, we got here. Okay, so what we're going to be looking at in a basic sense, on the right-hand side we have a whole bunch of homes, right? And those homes have solar panels. And if you also look, I mean, it's a little hard to see, there's little batteries, right? So, the sun comes out, we charge the batteries, right? And then when the sun goes

down, when people need energy, we can sell it back and put it on the energy market. So, we're going to look at the energy market in a little bit, and then also specifically how Kubernetes fits into But for reference, the number of homes that we work with that Impel, we have about 100,000 homes in our network. That gives us access to about 1 gigawatt of energy when

we do a virtual power plant. So, with all of those homes combined, we have effectively enough energy at our disposal in the batteries to be equivalent to about one nuclear reactor, right? 1 gigawatt is one nuclear reactor. So, if you remember earlier today in the keynote when they said, "We're going to take 93 gigawatts of energy for AI." That's 93 gig That's 93 nuclear reactors around the

world. That's a lot of energy, right? Or in this setup, it'd be about 90 million people. Right? So, that's just for reference, right? Some numbers. Cool. So, the energy market on the left is very much like the stock market. There's actually a group of people that do, you know, buy, sell, buy, sell, buy, sell all day long. And there's, you know, a market trader that acts like

a stock market trader. They act on your behalf to buy, sell, buy, sell for you. And then we have a virtual power plant controller. Their job is basically to do say smart AI stuff and say, "We know about these 100,000 homes. The energy market wants to, you know, in there's two different types of trading. There's day-ahead trading, and there's intraday trading. Day-ahead trading, you predict how much

you can sell in the next day, and that's what your responsibility is. Intraday trading, every 15 minutes the market will say, "We want 1 megawatt of energy." And then they talk to the controller, and the controller says, "Okay, out of these 100,000 homes, we know that 1,000 of them have this extra energy. So, each one of those can give up 1 kilowatt of energy into the grid

in the next 15 minutes." And that's what the different controllers do. And then we as Impel Cloud, we actually act kind of like a broker. You know, we're a platform effectively to help, you know, make sure that telemetry data goes back and forth, commands go back and forth, and that all of this works seamlessly. So, now let's zoom in a little bit on the architecture. Okay, so

on the left-hand side, you'll see what is supposed to be a home. And on the right-hand side, you'll see what is supposed to be the cloud. So, over here on the left-hand side, we have the IoT HEMS. This is the home energy management system. That's the IoT box that we make. And this device connects to the inverter, to the battery, to the heat pump, to the, you

know, solar panel, wall box, you know, that's your EV charger if you have one, and also to the smart meter gateway. And that's the thing that does the logic for if we tell it to charge or discharge, it's going to tell the proper components to do those things, right? And then over here, we have our connection to the cloud. So, we'll talk a little bit more about

this in the next slide, but we do, you know, MQTT. If you want to understand more about these sort of like specific components, MQTT is the pub/sub that we use in order to work with IoT devices. It's got a lot of features that work well for things that can randomly turn off or have really bad internet, stuff like that, but it's very much like RabbitMQ. we also

have our control loop. So, when we work with the controller, we do about telemetry and commands back and forth, so there's a loop that happens. And we also do real-time streaming in order to get sort of real-time analytics, and telemetry data that we can also put into something like OpenTelemetry or DataDog or Grafana, one of these other tools to be able to see what's going on. So,

one aspect we'll talk about is the control plane. As you said, we have EMQX, or our our broker's EMQX, but there's other ones like, you know, Mosquito, things like that. And it passes messages back and forth between our cloud and the device. And we The typical way to map out an IoT device is to have a digital twin, right? So, on the right-hand side, we actually use

Dapr in order to have a really simplified actor system. And we have one actor per home, and it's meant to copy the state and give us an understanding of what's going on in the home without ha you know, without having to always do a request model with the home in order to understand what's going on, because IoT devices can just turn off, right? Or they could just

have bad internet, so if you're trying to contact them, and, you know, they have LTE, and there's like a road cloud, so they can't see anything, so then you can't find out what's going on. So, we keep, you know, we keep some state in the cloud where we can in order to make sure that we know what's going So, this is primarily choreography. If you're interested in

like, you know, consistency models, we don't do orchestration, because we only care about this one-to-one mapping. And we use Argo as well in order to help us do our deployments. And we're also looking into stuff like KubeEdge, which is a really really fantastic technology for doing these deployments as well for IoT devices. And another fun slide here is our progressive aggregation. So, we do real-time aggregation that

allows us to get freshness, you know, fresh analytics from our devices using Spark and going into our data lake. We use Databricks. But from here, we can actually get real-time metrics. So, we can actually put on our dashboards like Data Dog dashboards, Grafana dashboards, what is the, you know, power production across Germany right now or within the last 10 minutes, right? What is the power consumption within

this zone of Germany within the last 15 minutes, right? So we can get And I just want to emphasize this has been a real game changer for our organization being able to use Apache Spark and being able to get the data freshness that we've never had before. This is a a bigger project that we did over the past year. And being able to see this on like

Grafana and Data Dog is really, really amazing. Being able to see real time on a map like where is power production happening in Germany is actually pretty cool. So, Mario is going to talk a little bit about how we actually respond to events with a VPP. So now that you know a little bit about, yeah, of the internals, let's talk about how we actually respond to some

of these grid events that we talked about earlier. There's four key ways and I'm going to simplify a little bit here, but um uh two are frequency containment and frequency res- restoration. Basically, this is just responding to these dips in frequency that we talked about previously um with our virtual power plant and I'll show you an example in a couple of minutes. The other uh two cases

are what are called peak shaving and energy arbitrage. This is effectively just load shifting from that uh uh day the peak of when solar energy is being generated to times when it's actually needed later in the evening um as the sun has set. And and there's one super important thing about the way we're able to respond to grid grid events that sets uh virtual power plant and

especially batteries apart from other energy sources and that is how fast it is, right? Within milliseconds a battery can respond where hydro will take quite a bit longer, gas turbines will even peaker plants will take up to 10 minutes where uh our coal plants will take a couple of hours. There's a a slight um typo in this slide. I don't it's supposed to be 10 minutes not

10 hours in case that's ambiguous. Uh so let's look at a real life example. All right, let's pretend it's a few years in the future. This is Berlin. It's a sunny day morning in July. We have 53,000 homes connected and generating solar energy. Midday, all that solar leads to the price collapse that we talked about in our duck curve. Fortunately, our controlling partners at Flexa uh prepared.

We're holding those batteries empty ready to soak up the negative energy. We tell our homes to charge up that battery, charge up the EV, run our heat pump on air conditioning hard. Let's really consume as much as we can cuz we're literally getting Every home is literally getting paid to use that The whole fleet is storing up batteries. We're ready to go when the evening peak comes.

When the sun set, prices spike, our batteries discharge at peak prices um uh both reducing CO2 emissions, reducing the need for fossil fuel plants, and generating revenue for us and our consumers. So you can kind of visualize this here. On the bottom uh you can see this green spot where we stored up all the energy and you can see the yellow line is the sun going up.

You can see the peak of solar generation. And then at the tip on the right is when we release that energy at the end of the day. Here's another example of a frequency response incident and this incident is not theoretical. This happened in real life. This was Tesla. They were just getting started. They had about 1,100 homes in their virtual power plant. Uh one of the largest

energy producing plants in that area tripped dropping 750 MW of energy. There were 600,000 homes at risk of maybe having a blackout. But those home batteries stepped in autonomously responding before any human individual even realize what was happening. Within 6 seconds the frequency was restored and any risk of a cascading event was completely absolved. The grid was stable. coming back to Texas again. Do you we narrowly

avoided disaster by 4 And one of the ways that we could have prevented this problem is by having other smaller units around that could help supply energy when we needed it. So in the future, what we want to do for the grid is help by having a virtual Thank you. Yeah, thank you very much. >> [applause] >> You can you can find our slides online. We set

up a nice accompanying website with a lot of educational content at whatisavpp.com. Check it out. All of the simulations are there as well including other stuff. Yeah. Any questions, comments? All right, well, drop us Oh, you have a question? Do you want to give him your mic? Um I was wondering um the example you used was specifically over winter when um in Texas when uh solar panel

like energy production is usually very low in northern Europe. how are you handling this whole uh managing the frequency in in winter period when >> Sure. Yeah, so Our batteries don't only charge on um on solar, right? Like we can also even even in winter months you can consume energy when uh prices are lower during the day. Um and you have that energy available at other times.

So even though solar is the cheapest source of of renewable energy for those batteries, alternatives exist, right? We can consume then charge the batteries directly from the grid. Another energy source that's really useful in the winter is wind. Right? So there's still lots of wind energy that's available in northern Germany for example and that can also be one of the sources for our doubles. Yeah, of course.

I mean there are limits to battery storage, right? There's uh what is it called? Uh where did it go? The the Dunkelflaute. It's a period that occasionally happens where there's no renewable energy for, you know, 100 hours and that actually can be quite difficult. You really do have to spin up your your your gas plants and your fossil fuels to handle this dip. Oh, where did it

go? I got another question um regarding um the small assets that you have there that uh everyone makes money out of it, but bringing grid stability is not always about money. It's about stability. How How would you support basically make it for the small consumers also accessible that they contribute to the grid stability even if it doesn't make the Uh so you know, we are There's there's

two things to this. One is that grid operators care a lot about stability so they do actually compensate uh uh folks like us for um uh making our fleet available uh to for frequency response. Um and right on a from a regulatory perspective uh there are programs that we are starting to build into that allow us to participate in this market space. It's My understanding is that

it's actually quite beneficial for us to do so because we actually do make revenue being a participant in this space. And we can offer the service cheaper than a traditional power plants. Another great point that I would make is that you can participate in this you don't have to have, let's say, solar panels. You can actually do this with just batteries by doing the grid stabilization effect,

which is also something that we're working on. So if you want to participate in a VPP, the actual main requirement is having a battery. Right? And then ideally something to put the energy, you know, energy into the battery, but that can come from the grid when prices are cheaper negative, right? So uh that's also something that we're working on, but that's another avenue for participating in VPP.

Next question. Oh. So when you saying you're going to sell in the next 15 minutes um electricity from the house you're managing, do you have to somehow prove that you reinjected this electricity into grid? How does this work? Yeah, we do have to prove it. There's smart metering and other metering systems that come from grid providers that um ensure that we're delivering on what we say we're

going to deliver on. We actually get charged if we don't uh respond appropriately. It's called an imbalance fee um and if we don't deliver what we say we are going to deliver, it's actually extremely expensive. Okay. So there is a requirement to do VPP. You have to have a smart meter as well basically. >> Okay. Thank you. Wait, wait, hold on. Hold on. Take a picture. Wait,

hold Ah. Yes. I have two questions. First one, do I need to change my inverter to be able to participate or you can integrate it or what what happens if I already have um like all the hardware? Uh we So right now we're actually presently building capacities that we can integrate existing um systems. Right now we're focused on individuals who have existing PV panels cuz that's often

one of the most expensive parts and integrating those and we are expanding the offerings we have to folks like yourself with existing inverters. And the second one is typical from Germany. Uh what happens if there is no internet and like my my infrastructure is connected and basically I don't want to give all the control to some cloud service. I'm running home assistant and EVCC like on prem

to be able to to control my environment by myself. Do I have this possibility like to say, "Okay, now I don't want uh to participate on the grid stabilization, but I want to give have control on my own infrastructure. So, our model is grid operators come in terms of how we behave, then consumers, and then the VPP fleet. So, Yeah, I don't think we've explicitly explored quite

the diversity of interest your home assist setup probably has, uh but in theory, as we continue to mature our system for folks like you with third-party um I I think that's a use case we could probably manage. But, we already integrate this kind of behavior with our our EV chargers and our heat pumps so that consumers rule and can locally manage their systems independent of our cloud.

We also have redundant connections on most of our systems, including all new installs, so that also helps in case of infrastructure issues or connectivity issues. So, I I would see in the future because there's a law in Germany that you can give control to the grid operators to control your infrastructure optionally. I would see the whole grid as a federated service where you have a possibility, like

I don't know, Matrix, for example, the messaging service. You can like be autonomous, but you can also participate on a federated grid. That would be from my point of view. >> That's absolutely the direction we want to go. And Paul is really behind it. We really are looking forward to contributing. Cool. Thanks. Yes. Uh yes, maybe I have a question a little bit about your infrastructure side

of things, which first of all, yes, super super what you built here. Uh but, I was wondering what are the numbers, let's say, for what it actually takes to host this, you know, how many how many machines you have in your fleet backing this, how many machines are there in Databricks, what's your kind of data data ingestion volume on a on a busy day? Yeah, I can

speak a little bit to that. Um so, we have about 100,000 systems. Not all of them are participating in VPP. At the moment, we have 20,000. We're going up. We're trying to grow up to 100,000. Uh but, in terms of like, say, events and stuff, so for our regular fleet, we do about I I want to say a million messages per 5 minutes. Okay. Right. So, it's

not like overwhelming yet, but that we haven't yet gotten to the part where we have everyone doing very fast data. For our regular fleet, we're much slower on our intake. For VPP, right now, we do, say, every 20 seconds, everyone sends data. As we move to automated frequency response, that requirement goes down to every 500 milliseconds, right? And that has to also go to other third parties

who do, like, you know, sort of governmental stuff to sort of stabilize things, right? So, But, to your question, yeah, how many machines are we actually running and what scale? In the cloud. Yeah. >> Ah, okay. That that is the follow-up question is yeah, where are you running in the cloud? >> I'd say so like in Databricks, we're actually running things pretty pretty small when we have

our streaming aggregations. That was a very much a benefit. So, our cluster sizes at the moment are like, you know, um a few, like three or four, that can manage 20, you know, events from 20,000 systems in real time and do the aggregations as well, right? So, very small clusters. I think the streaming aggregations in particular allow us to make really small clusters, which is very different

from the batch sort of batching paradigm >> Yeah. that you normally do. And so, what the what was normally very large clusters with like 64 CPUs or something, uh sorry, yeah, something like that, we were able to reduce it down to like a cluster of four or five with like two CPUs and like eight or 16 gigs of RAM to do the same thing, but it happens

in a streaming fashion instead of once every 24 hours, which is the the big problem that we had. >> exactly. So, so you started with this as a a conscious design decision or is this something you evolved to after It I mean, it was something we evolved to, but we also had requirements where like everyone wants the freshness to be very fresh, right? The only way we

could do that is by having this sort of streaming paradigm instead of waiting 24 hours. And it just so happened to also be cheaper and faster and better. So, it's like a win-win-win. It was a really pleasant discovery that shifting from like really large volume batch jobs to streaming aggregates was cheaper. Yeah, it's really great design you did with the whole aggregations, so Credit to this man.

>> I read a lot of documentation. It's great. But, you picked Spark Streaming specifically because it was you were already using Databricks. >> that, yes, exactly. Yeah. We also We also explored other platforms besides Databricks, and Databricks had the best integration with Spark for our use case. But, uh mostly you're on the cloud then. It's not apart from your edge, obviously, where Yeah, yeah, so almost everything

is in the cloud. What happens on the IoT device is really the the basics of stuff cuz it has to be autonomous, right? So, none of the aggregations or anything really happens there. We just send the telemetry data to cloud. Yeah, it's it's like any almost any other MQTT client. Okay. And which which cloud? We have Azure. Azure, okay. Yeah. Okay. Okay. Thank you very much. >>

Wait, wait, wait, wait, wait, wait. I'm I'm really bad at this, so Hello. Um So, you mentioned the the somewhat scary example of of someone hacking into the grid, right, and causing a blackout. >> Yeah. Uh can you tell us a bit about how the IoT devices establish their identity, like, and authenticate with the MQTT broker? So, [clears throat] I think at the moment for the IoT

devices, uh they use They're very very simple networking with EMQX. So, EMQX allows you to have sort of like username password, and each one each user can have their own sort of encrypted key that we can pass along to them in a very manual sense. It's very hard to update Um but, over time, we'll be able to have it more integrated within our cloud and have things

like sort of like SSO and sort of uh auth, you know, better authentication. That type of thing. For now, it's still very basic, but that's the ideal goal is what we're working towards, right? Yeah. So, the the authentication does require like a human It does. It does. And that's what We also have a manual installation process, so like people actually go out and install the devices and

connect them to the solar panels. So, during that process is when a lot of this happens, and then we can update and rotate things over time. Yeah. Yeah, wait, wait, wait, wait. Okay. And then, one more question over Sorry. So, I have a couple of questions. One of them is a follow-up for from previous question. So, you're on Azure right now, right? Do you have any plans

of having a backup in AWS or Google Cloud or something in case Azure falls down or something? >> No, we're moving multi-region with Azure, but other than that, no. Uh and the other question is related to the actual power grid. So, um you're saying that the the small consumers are Like, the the batteries, right? But, how important are still the huge batteries that are built by, you

know, by by countries and stuff like that. Like, for example, water batteries or salt batteries or what Still totally important. I think they're complementary our our team has done like sort of business analysis what happened if all of a sudden there's massive investment in grid-scale batteries. And we still feel there's a really compelling case for consumers to participate in the energy market still. And one one more

question, what is the I mean, if there is an estimated timeline when we will have sufficient amount of batteries to sustain the renewable energy we have cuz like the big problem >> time horizon. I think that I think there's a lot of technological advancement that can happen over that time horizon, and I think that um it'll be years before we hit hit a limit, right? Like, the

especially when you compare it to um the costs of incidents, right? Like, the cost the current anticipated estimated investment batteries in Europe through 2030 is about, I want to say, 25 to 30 billion. The cost of ERCOT was three times that, Yeah, of course. so it it really pales in comparison to the grand scheme of things. If anything, it's it's a really cheap way to ensure the

stability of our grid. Yeah, but also the the the the amount of renewables is still growing, right? Mhm. So, will we It's growing at an accelerating rate. Yeah, so will the batteries ever catch up? That's the question mostly. Uh I I'm not sure. I I I think we're going to move to a point where it's all so cheap that uh if we can load shift just enough,

we'll be doing really well. You also want to take into account that there's a lot of people that have EVs, and EVs are also a really really great source for batteries, and they're move they move, right? So, they actually help a lot with sort of like grid response in localized areas. So, that's another source of like very big big batteries that are growing. In that context, yeah,

most home batteries are typically 10 to 20 kilowatts, whereas you can pull in your EV, which is like 80-ish, that's a really big jump in in in inability. All right. I have two easy questions. The first one, do you operate beyond Germany? I'm sorry, what? Do you operate beyond We do, but not with our virtual power plant yet. All right. Uh the second one, it's very gorgeous

presentation. It's beautiful and visually striking. What did you use to make the presentation? We We made it custom. We got a lot of help from Claude. Wow. Yeah. >> Wow, that's incredible. I guess simple one on the >> probably 60 hours of of coaching Claude. But, it was really fun. I I'm actually thinking maybe I should automate this and try to turn it into a little business.

We'll see. Oh, yeah, that's a really good idea. Yeah, good-looking good-looking presentation. I guess on the consumers on not the consumers side, but rather on the endpoint side, how do you I mean, because the present presentation you said that when the price is going to negative, so basically, you are buying the or rather get paid to use the energy, right? You have have shown that you you'd

like to consume the energy by everything on your on the end points on the homes basically. So, how do you plan to connect? I mean, do you have a plan to get the to have a gateway to interconnect everything directly like a smart gateways or We already have that. You already have that? >> Yeah, that's our IoT device. >> basically I guess you get the gateway from

you and then everything is connected and distributed from there. Any plans like for example, you know, open source home assistant integration if people already have that? Yeah, we actually are I I don't want to share too much. That's internal information, but it is something we're actively thinking about. Okay, that's so cool. Basically, if you if you Yeah, yeah, that's so cool. Yeah, thanks. The last one. Oh,

no. Sorry. >> Hey, so I'm sorry. My question is a bit of a duplicate, but how do you guys plan on exporting this VPP model to other countries because I I think TSO operates differently in let's say France. >> You're absolutely right, and it's really difficult. So, we um we worked really hard to get it working. We had to understand the regulatory environment very well. Some of

our partners even in heavily involved in um advocacy for the kinds of policies that would enable would enable a VPP to be effective in Germany. Um one of the key things is um grid fees. Uh the regulatory model treated every home as a traditional energy producer and charged them fees appropriate for a you know, like a nuclear power plant or something. And so, uh our team advocated

for new paradigm for how to approach individual homes. And that's now legislation. And different TSOs and DSOs have adopted um newer standards where it makes it much easier for these uh individual homes to participate in the energy market. Um that model doesn't replicate easy easily, but we have uh the early steps to exploring other um regions and regulatory environments here in Europe. And my guess is in

a certain period of time we'll we'll start moving in those directions. Like we have a huge demand and huge interest from customers and in other partners. Um so, it's just really a matter of time to to figure out the regulatory hurdles. All right, thanks. what's your um stance on the merit order? And um in the future is there a a world without this principle if you have

enough um battery power? I'm sorry. Can you repeat the question? What is your stance on the merit order principle? if there is a future where we don't need gas turbines, but we just use um battery storage? I think it's hard because I think lithium ion is not the only type of battery storage. There's a lot of new technologies that are in pilot stages that could be incredibly

useful for long-term storage, and I think that's really where the gap still is. Um you know, I talked about the the uh period where that's dark days, no wind, no renewable energy being generated. In that period, you would run out of battery energy, right? You really need much larger scale storage that we just don't have right now. And and there are limits to I think how we

could get there. Um I don't actually I'm not sorry. I'm not I'm not being an energy expert. I'm not familiar with the other principle that you mentioned. Um if I'm not mistaken, it's the principle that the most expensive um energy source um dictates the price for energy. So, if you if you need gas turbines at I don't know, 50 cents a kilowatt hour. Um I think that

that's becoming less I mean, I'm I'm being fair theoretical now. These are just my personal opinions at this point, but if you can store a lot of energy uh and if that storage becomes um uh more generalized uh you I think that we untether ourselves from being beholden to that. Another thing I would another thing I would add is like if you remember on one of the

slides, we talked about how long it takes the energy to actually get into the grid with different sources. And actually like nuclear, oil, gas, coal, they take hours to get into the grid. Whereas batteries take milliseconds, right? So like, you know, I think also from a practical standpoint we would be pushing more and more to have those types of things because it would be cheaper to prevent

issues on the grid. Like I I see that would be a we basically just have to get to a critical mass point. But like it will we don't want to wait, you know, our latency metrics for websites aren't 2 to 6 hours. Like that's ridiculous, We want the same thing for our grid. And so, we'll probably push more and more for something like that, I think. >>

Yeah. And I think I but I have to be a little bit speculative, but I I struggle to see how that can maintain in in a VPP enabled world. Uh of course, politics aside, right? >> Hi. Uh so, you are mentioning Azure. Are you using IoT Hub or are you using Cloud Native? >> We moved away from IoT Hub. Why? Do you know? Most No, mostly because

it was uh yeah, that was the main reason for that. Um yeah, also overall it's been very good having EMQX as a as a broker. Um so, that was the that was the primary reason was we needed the higher throughput and some MQTT 5 features. Yeah, and EMQX is the vendor name. MQTT is the protocol. Mhm. Yeah, I know. And um the other question is do you

use any physical models to prognose the behavior of your systems or are you only based on sensor data? It is entirely based on sensor data right now. Yeah. also thank you a lot for the presentation. I also really like the visuals. Thank you. And what's your opinion on the new regulation from our I would say Ministry of Fossil Fuels to make it harder [laughter] for smaller producers

to join the market? So, is it good for you know, it's really funny because it makes the case for a VPP even more interesting, right? Because now the sudden uh the subsidies for uh green energy are reduced. And so, you're not if the feed-in tariff they're talking about eliminating that. So, now what options do you have other than self-consumption? It really like reduces the benefit of having

your own solar panel and batteries, but if you had a VPP in now the difference between just having a home system and having a home system with a VPP becomes much they're much further away, right? So, uh the benefits of being in our VPP really just go through the roof. Mhm. And we see this I think in in the people who we talk to in the marketplace

like, oh, can I get your VPP into uh my installations? Um and that's why we're working so hard on trying to integrate third-party hardware that we currently can't work with. Mhm. Nice, thanks. And second question, um the concept of the virtual power plant, is it like a like a general concept because I guess there are like a few competitors. I think also one of them having a

presentation at the same time from here from Hamburg. So, 1.5 Yeah, yeah. And since you mentioned you want to be the biggest grid, of course that's the goal. Do you have like uh current market share where maybe you 1.5 Octopus >> I don't I think I'd like to think that it's a friendly uh relationship. I think that there the market is the energy market is big. And

I think you know, from my personal perspective, I care a lot about um the well-being of our planet for my children and all of our children. And I think any momentum that we can achieve together in the space is wonderful. So, I think it's great that we have more people actively pushing green energy and helping create uh virtual power plants. >> The more the on that. >>

Okay, thanks. Oh, one more. Yes, so yeah, one more. Uh I have a question about the infrastructure part like uh you told us that you are using uh KubeEdge and uh Argo CD to handle your fleet, right? We're investigating KubeEdge. At the moment, we don't use KubeEdge, but that's one of the deployment models we're investigating. Currently, we use Balena uh for our deployment. You are using you

are using Balena? >> Balena isn't I'm sorry. Balena? Balena? >> Yeah. That's a it's a different alternative. KubeEdge is much more modern, right? So, that's one that we're investigating looking maybe we'll switch to. But yes. So, what was your question actually? Uh it was uh But technically speaking I want to have a more details of the architecture patterns also about how are you deploying your Yeah, that

would be the ideal case. Yeah, that would be the ideal case cuz you can use Argo CD with KubeEdge. And then from that perspective from Argo CD, a node a home is effectively a node in Kubernetes, and it would have its own little page in Argo CD. So, then you'd be able to like deploy individually and and see what's running on each device and things like that.

Yeah. Okay, thank you. Interesting. And just to say, Balena has been great. We really are happy with them. But I think our scale is is exceeded what what what what they were designed for. And so, we're thinking about KubeEdge. Okay. Thank you very much. Great. Okay, thank you so much.