Lightning Talk: Backpropagation-Free Optimization in PyTorch - Andrii Krutsylo

This talk discusses gradient-free optimization methods applied in deep learning, particularly implemented in PyTorch. The speaker explores various strategies such as random search and evolutionary algorithms for weight optimization in neural networks. They introduce Hebbian learning, a method that predates backpropagation, illustrating its potential for efficient weight updates without requiring gradient information. The speaker highlights the importance of local updates, memory efficiency, and the ability to integrate non-differentiable components in models. Additionally, they present practical techniques like feedback alignment and the Bio Torch library, which offers tools for biologically plausible optimization of PyTorch models. The session concludes with a brief overview of the 'pepita' method, which transforms loss into input for further optimization.

My name is Andre. I'm from the Polish Academy of Sciences. This is not even topic of my doctoral dissertation, so you can imagine how how much I want to discuss it with someone finally. And yeah, we are talking about gradient free optimization methods today. It is implemented in PyTorch, not specific to PyTorch at all. It is general approaches, general methods to backpropagation free optimization. And again, first

of all, I'm pretty sure that you familiar with random search. It is the most basic methods to to train a network. Not not much of a trend to guess the right weights or guess the weights of your network that would yield a proper output that align with what we expect, but this search would be very very long and we probably shouldn't rely on it. Alternative is evolutionary

algorithms. Now, this closer to optimization, the actual optimization, when we can create a population of weights to our model and select the weights that perform slightly better than the rest of the population, do some hybrids, do some mutations, and receive pretty solid results. Actually, reinforcement learning uh pretty much reliably can use uh genetic algorithms or evolutionary algorithms to solve problems. But again, it's um it's a matter

of size of the model and dimensionality of the input data. So, yeah, don't do that. And uh yeah, pretty much any optimization that you can use in maths you can apply to search uh of your weights in the models. For example, you can do some small perturbation of uh of losses and find out which direction we should change our weights, do some hill climbing, and solve the

problem. So, before backpropagation, we had advanced stuff called Hebbian learning introduced in 1949. Uh it is famous statement, famous phrase that neurons that uh fire together wire together. It could be explained as we want to increase the weight between two neurons that are activating a lot as as a response to some input. And anti-Hebbian rule would be we want to decrease the connection between neurons that doesn't

activate together during some some input. Uh Yeah, of course, it will introduce a lot of problems like uh uh blow up blowing up of these uh weight values, but again, you can solve it. And on this illustration, you can see Hebbian rule applied to solve continual learning problem quite recently by uh by me and my team. We What you can see here a graphical representation of weights

connected to individual neuron. So, this square every of these squares Okay, doesn't work. Okay. is actually an image representations of 32 by 32 weights connected to one neuron, and you see these nice airplanes from the CIFAR-10 data set. So, how this Hebbian learning works, how the uh very specific modification of Hebbian learning works in this case, it it accumulate features from similar samples to store their averaged

representations directly in weights. Pretty good idea, very biologically inspired. And you don't need gradients here at all. You don't need backpropagation here at all. All you need is presynaptic activity and postsynaptic activity. You find the difference between them, and this is your uh delta scaled on how you want to change the weight between these two activations. What does it mean? It means that you don't have to

store everything in your memory. You don't care about the back propagation chain and you don't care about gradients. It's very memory efficient, but not really accurate for reasons. So, yeah, we have this local updates. Uh it is of course unsupervised because in this presynaptic postsynaptic activity we don't really use uh labels. yeah, it save memory for us. What it doesn't do, it of course don't create hierarchy

of features or rather we don't have proofs that hierarchy of features would be created that way. And of course, this signal will not propagate too deep. this images were generated was generated from the one layer right after the And we still have a lot of semantic information about um locations of pixels, so we can create this nice airplanes because a lot of images are looked one to

another like look similarly and we have some features of airplanes stored in the weights. Now, when we move to the next layer, the output of the next layer would not have this rich semantic information about shapes of airplanes. It would be simple activations of uh of neurons. and you wouldn't have this nice ability to accumulate similar samples in one neuron or in uh several neurons. this is

problem common to all back propagation free methods. And you can always see when some back propagation methods are in is invented as someone will post uh uh next paper telling that okay, we improve dramatically results of the baseline method by adding residual connections to the inputs. Basically, by feeding the inputs into deep uh deep layers and avoiding this um uh signal propagation problem. Like, it is a

solution, but it is a reduce a lot of additional parameters, a lot of additional connections, and you can't really tell that it is deep if every of your deep layers is also connected to the directly to the input. So, uh yeah, don't do that if you if you would have such idea. Or else I don't post it on NeurIPS or something. Just do it in blog. yeah,

basically, what we want to take from this Hebbian learning, which is predates back propagation, we want to take this very nice ability to use unlimited information uh about two neurons, about one connection, or about one layer, or about a group of layers because if we're talking about um ResNets, for example, when we have blocks of uh of layers that connected with residual connections, of course, that they

have to be updated simultaneously. So, yeah, we also use scale from one neuron to the whole block to to to to several uh layers. But, we still can train the model without storing whole gradients with the only local rules. And yeah, I hope it would be fun for you to to to learn how to do that. So, one why we're talking about uh back propagation free methods

at all? Uh first of all, yeah, local updates, as I told, that we we'd care about only uh connections between neurons. Uh it saves memory. It allow us to parallelize very very efficiently and out of the box because all we need to know is a very limited information about some chunk of of the model so you can spread this model to your infrastructure and train it as

well as much and any way you you want. And the best thing, I know that a few physics are currently in in audience, you can just plug in some non-differentiable parts to your model. For example, if you want to have physics-informed layers, you can just put it inside your model prevent some distribution shifts in in deep layers, hopefully would be useful for your research. Very simply, very

fast about a few methods that are actually used right now much closer to our days, much closer to the performance of backpropagation. First one is a feedback alignment. It is not backpropagation-free method. It's it's a method that was that created the family of the methods. It is a basically baseline for all backpropagation-free methods. We we compete with feedback alignment. Very simple idea. We took backpropagation and we

remove weights of the model from the training signal. What we're doing is uh we replace weights of the model with randomly initialized matrices and during backpropagation, we multiply our error to this random matrices and we train model because model forward forward connections, forward weights of the model would align themselves with this new signal based on during this back propagation. Well, we will move forward. You can approach

to me after the presentation. I I'll explain how this matters work because we have only 10 minutes. I want to introduce you to the bio torch library. It is not my library. I'm just very huge fan of it and you can just uh install it. It is the best thing about it it already provides some benchmark on common data sets and with convolutional Unfortunately, none of this

gradient free or back propagation free methods was tested on other lands because well, we don't have resources to do that and this is a research gap. You can hire me to to work in this direction. I'm available. Yes, and the best part is that this code is actually a proper library. It is not some dot here fast research code. So, you can use it. You can take

your uh pytorch model and use a simple wrapper to convert it to biologically plausible uh optimized model. Very cool. Okay, we're running out of time. Probably this last method for today is a pepita. Very cool idea. So, instead of doing back propagation, instead of moving backward, what you do is you put input to the model, propagate through all the and then you receive your loss, which you

transform into the new input. You you use some random matrix to to transform the shape of your loss into the shape of your input and you add this new loss to your input to create new input to the model. You make second pass, second forward pass, and you find the difference between these two inputs to the model and this would be your training signal. We are running

out out of time and thank you very much. Please approach to me. I'm really happy to talk about it.