Lightning Talk: Why Your Forecasting Transformer Isn’t Working (And How To Fix It... Rosheen Naeem

In this talk, Roshni Naeem discusses challenges in forecasting solar energy generation using a transformer model in Python. The speaker emphasizes the importance of solar forecasting in stabilizing electricity grids and reducing reliance on fossil fuels. She shares her experience with the Open Climate Fix project, highlighting the temporal fusion transformer model selected for its ability to handle complex time series data. Roshni explains her data collection process, combining PV generation data, site metadata, and numerical weather prediction data while ensuring no null values. She showcases the implementation of the model using PyTorch and specifically mentions techniques like learning rate optimization and early stopping to boost performance. The session concludes with key insights about the critical role of data quality in achieving accurate predictions.

Hello everyone, and good morning. Um I'm going to talk today on the topic of why your forecasting transformer model is not working and how I dealt with it in Python. Um as this is the first talk of the day, I hope it's not very heavy. Um I'll start with a little bit background about myself. Uh my name is Roshni Naeem. Uh I'm working as a software engineer

at Miro. I recently graduated from a master's program, which is called Erasmus Mundus Masters in software engineering for green deal. So, the little bit context about the program is that they teaches you that how as a software engineers, you can play your part in promoting the green environment and how we can make this planet sustainable for our future generations. So, during this program, I did uh 3

months open source internship, which is called Google Summer of Code. So, this uh internship is an open source program, which is organized by Google, um which promotes uh non-profit open source organizations to participate in it. And uh it also promotes students and open source beginners to participate in it and work on on the projects of those um open source organizations. So, when I had to apply for

it, I wanted to select an organization which was working on renewable energy and I came across Open Climate Fix. So, Open Climate Fix is working for sustainability by IT. Their goal is to promote uh green environment and also to reduce carbon emissions. And they had variety of projects, which varied from solar forecasting, cloud casting, wind forecasting, um to making the energy data publicly available, which we can

use for our own applications as well. So, from these topics, uh solar forecasting was the one which caught my eye and I quickly want to go through why solar forecasting is important. So solar forecasting is the source of uncertainty in energy industry. PV generations and predictions are really volatile because it can change drastically within minutes due to cloud coverage. So the thing is that we cannot control

the weather but we can predict it. So the electricity in order to keep the electricity grids stable, there are backup fossil fuel generators which are running and they cost money and also waste energy and reduce carbon emissions. So if we have accurate predictions, we can control the supply and demand of these generators and save The third and important point is smart homes and trading. So if we

have accurate predictions, we can optimize the usage of our appliances at homes and furthermore, we can also trade energy. So the project which was related to solar forecasting was open quad solar forecast project and the project had an regular machine learning forecasting which was using public data of UK and it was originally only a single model architecture. So what I proposed was that adding a multi-horizon multi-source

forecasting model on top of it which would be fusing the three different type of data sources and I will also create a data pipeline which will also result into a 36-hour continuous multi-horizon prediction. So you can predict for the next 36 hours about the PV generation. So the model that I selected was temporal fusion transformer model because of multiple reasons. First is that it is provided by

PyTorch Forecasting library so it is easy to implement. The other reasons were related to architectural reason. The first one is that it combines LSTM encoders plus that tension layer, which provides both long-term and short-term dependency, and it has the ability to handle and model complex time series data. Second, the attention mechanism helps dynamically weigh the importance of different variables. For example, if we consider weather variables, so

it checks different weather variables can have different importance at different time of the day. Third is that it handles three feature type. Static, which is being handled by static covariates and counters, and also time waiting known and time waiting unknown. The last but not least point is interpretability of this model. So, interpretability is that once I'm done training, after training the model, it shows that which which

were the variables or the features which influence my model the most, and which were contributing towards the results more. So, for example, if I'm doing solar forecasting and I have solar radiation and UV radiation or maybe the cloud coverage, which are the most important variables which are influencing my model, so it means that yeah, this was the this was the plan. Like these were the variables which

I was expecting to influence the model more, and they are working. But for example, if I see that the most variable which influence the results or which feature influence the results most was wind factor, so maybe there is something wrong with my data or something is off and I need to double-check it. So, the second part was having dealing with the data, and this was the part

where I spent the most time. So, I was presented with first PV data, which was available on Hugging Face data set, so I had hourly prediction, hourly generation values. So, I chose the random sites, I got the data and then I built a horizon of 36 hours. So, every data point I had 36 hours ahead of generation with it. Very interesting thing about TFT is that it

does not accept null values. So, I had to like carefully choose all the points which like had no null value at all for the next 36 hours. After that, I had the static data which was the site metadata. So, I incorporated for all the random sites that I selected their metadata which was like their capacity, orientation, and etc. different things. Then came the NWP weather data which

is numerical weather prediction data. I got it from the integrated forecasting system model of ECMWF and I got 14 weather variables. Um so, now I had data, PV data, and site metadata combined and I combined this data depending upon the timestamps. So, for every data point that I had, I combined the numerical weather prediction data very uh carefully with it because if anything goes mismatch or if

the timings goes mismatch, then it would not be giving me the accurate results which I am predicting. Um and I think that this step was really important and like I needed to be very careful here that I accurately match all the time all the data depending upon the And in the result, I had a merged data set which was 36 hours continuous batch data with no null

values. Uh the next step was model implementation. So, the model implementation the TFT model implementation it used by torch forecasting. Um on top of that, I used like by torch lightning. And there's an interesting method LR find method which I used to optimize the learning rate. Um it takes a few minutes and it gives you a learning rate and it's always good to use this method. Um

I also did early stopping to prevent the over fitting so model does not learn the training data and does not like work good for the unseen data. And gradient clipping was set to 0.1. So, to test it, I used the unseen data for different PV sites, and I predicted for the next 18 hours. And as the result shows, the generation and the prediction was quite close. And

I also checked that which variables were the ones which were influencing them more in my model. And as I got to know that, okay, solar radiation was the top one, and then UV radiation followed by low cloud cover. Um so, it showed me that it it is what I was aiming for, and these are the variables which were showing me the most. And that was also the

point of using TFT model. So, the key takeaways that I got from this project. So, when I started working on this project, I thought I would be spending most of my time on the model. I dealing with the complexity of the model maybe or doing the analysis. But, the most time that I spent on this project on the data set. And I also ended up creating the

data pipeline, which will take different so different types of data sets, and also give me a final data set, which I can feed to So, the data quality, in my opinion, for this project mattered more over the model quality. And I built the continuous window with zero null values, and I added horizon column with it as well. Um I also classified every feature. So, classifying is also

very important because wrong classification can silently kill your performance. So, part is also very important. Um also, I would say that it's always a good idea to run the LR find method to find the learning rate for it, and setting the gradient clip to 0.1 or your value. So, thank you for listening to my project. Um I have given the project code here available, and also the

organization link and the project code which they're working on. Um I'm a bit short of time, but if you have any questions, please like you can come get me after it. Thank you so much for listening to this presentation.