About this talk
This talk explores the limitations of large language models (LLMs) in solving complex planning problems, such as scheduling for nurses or delivery routes. The speaker emphasizes that while generative AI has gained significant attention for its capabilities, it cannot effectively tackle specific tasks that require detailed context and optimization. He reviews the reasons behind this, including issues such as context window size and the sheer complexity of problem spaces. The speaker introduces Timefold, an open-source mathematical optimization tool that can solve these planning challenges more effectively, arguing for a broader understanding of AI that includes older methodologies. By integrating LLMs with traditional AI techniques, new solutions can be developed to enhance scheduling efficiency across various industries.
Full transcript
It's for the It's for the shock effect, right? Wow. Okay. Mic's already on. >> I have three more minutes. No, before I start or do I start now? >> You You guys are just like, "Yeah, whatever. Whatever. Belgian guy. Just do whatever you want, right?" Uh, fine. I guess we'll start. I guess So, good afternoon everyone. How are you doing? >> Good. Still excitement on a Friday
afternoon. I love to see that. See, even yelling. Good, good, good. So, I hope you enjoyed the conference. I I'm not quite sure what I will tell my wife once I get back to Belgium because I learned a couple of things today. I learned about llama. I learned about goose. I learned about duck. So, I'm probably going to have to tell my wife that I went to
some sort of petting zoo today. but there's one animal that I absolutely love that was missing today and that's a cat. Any of you are cat lovers? Cat lovers. Great, great, great. So, I actually tried to bring my cat here but uh he was not cooperating this morning. Uh, he did not want to join me to this conference and I I understand. I mean, he's he likes
his comfy little space at home. And I love cats. And what I love about cats is that they have become the de facto kings and queens of feel-good internet content. Whenever you want to feel good, you just oh, they look at some cat pictures and cat videos and you feel better. And what I like about current times is we have genai so we can make more cat
content. For example, a cat sitting on a crocodile while playing a buno. Boom. There you have it. A cat in a tiny top hat hosting a stand-up show for a crowd of impressed goldfish. Boom. Although to to be fair, this one does not look impressed. And you know, I wasn't able to bring my cat, but I did uh try out Sora today. So, if you don't know,
uh Sora is a way by open AI to generate not just images because these are all ChatGpt Dolly images, but video and it looks like this. So this this could have been today if my cat would have been cooperative, but it was not. just just to say Genai is awesome and I think it's very cool technology that we should use and abuse to create more cat content,
but also to help us with other things in our lives that make it be easier. You have this mail and you want to send it politically correct, but you're actually very angry. You type a very angry mail. you ask, please make it politically correct and GI will take care of that for you. So, there's a bunch of good things I can say about Genai. Whenever I read
about Genai online, it is spouted as this is going to be the solution for absolutely everything. We are done as human beings in life. It's going to solve everything. And then I think, yeah, maybe it isn't the solution for The problem is when I try to uh you have these people that are super hyped up about Genai and when I say or is it when I when
I doubt our new overlords I get a lot of butts uh single t uh so a lot of butts and it is yeah but it will get better with rag but it will get better with fine tuning but it will only get better but and then the next idea that comes like in a couple of weeks Well, what I want to talk about in this talk today
is talk about something that LLMs can't do because I think too many other speakers, well, it's a good thing that a lot of other speakers are speaking about this, but I think it's important to still realize that these tools can't solve everything. And that's what I want to talk to you about today. One thing LLMs can't, and that's something I'm very passionate about, is solving planning problems.
And these planning problems are actually everywhere. For example, when you have curriers that actually need to deliver packages to different locations, someone needs to plan that. If you have 2,000 packages and 20 vehicles, what is the best combination of packages, vehicles, and routes? Someone has to calculate Airports are full of scheduling and planning problems. Airlines need to be scheduled. when you uh when you go to the
desk to to check in, did you notice that that's the same people that also check you in at the airplane, that also needs to be scheduled. Maintenance of these things needs to be scheduled. The world is really full of these problems. And recently I was talking to my neighbor who actually does the scheduling for nurses in a hospital, you know, uh 50 nurses and they each need
to be assigned and everything needs to be taken into account like holidays and skill levels and so on. And my neighbor actually asked like, "Yeah, but these geni things, can it solve my planning problem?" And I thought this is the thing we see, right? People when whatever problem we're talking about. People think that with LLM it's going to be a walk in the park every problem on
the planet. So we put it to the test. We tried it like how does it solve these problems that are everywhere around us all these planning problems and uh the results were very disappointing. Uh I had a few examples where uh one person got assigned all the shifts. I remember in my example data his name was Carl. I hope there's no corals here. I hope not everyone
is assigned to one or just just random weird stuff, you know. But instead of just going, "Ah, yeah, LM are bad at this," it's why are they bad at this specific problem? Why are they bad at solving these planning problems? And we actually identified two potential reasons. Challenge one, context window size. So, in case you don't know, the context window is basically how much things the uh
the uh LLMs can process in one go. Like it's like a random access memory, like a RAM memory, but then for an LLM, just simply put, I know I'm taking some shortcuts here, but these things, these context winners are limited by an amount of tokens. You've probably heard this if you've been involved with LMO there, it cost this many tokens. So the data you need to schedule
10 employees is about 3,400 tokens. So that's all the data being the name, last name, uh when they are available, what what skills they have, and so on. If we scale that out to 100 employees, we're already at 34,000 And 100 employees is like an average use case that I'm seeing out in the wild. If you go to a thousand employees, which sometimes happens, you're at 340K
tokens, and this is just scheduling those employees for one week. Now, I don't know about you, but if you uh are living by a schedule, you don't want it to be scheduled one week. You want to be scheduled a month so you can plan like personal life and other stuff, right? So if we take the middle one and we extrapolate it to time street. So let's say
we need about 100k tokens to to schedule something. So what's the context window for currents? Well, past generation it was 8k. I'm asking for 100k for like a basic case. So that wasn't working out. But more and more recently you can see that context windows are being expanded. And currently we're at 120k tokens to like 2 million tokens. So probably this context window problem that we're feeling
with these scheduling things is temporary because all these context windows are just increasing and increasing and that's what you hear a lot of marketing people say as well. We're racing to the biggest context window. But there's a big there's a big butt again. Still with a single T. Still with No, no, no. exception. So there is this this GitHub repo and I'll show you if my Yes.
So does anyone know this repository Nvidia ruler? It's actually a benchmarking tool that uh the guys from Nvidia made. So there might be a bit of bias because it's a company that makes a lens you know. Uh, but what they actually did was they compared the context length and they they checked like if if something reports I'm going to make it a little bit bigger for you.
So, uh, if an LLM says that it has a context window of uh, 1 million, can it actually do sensible things if you fill it with a million tokens? Because a lot of these things are saying, I have a context window of 500K, but they become useless as soon as you put like 64K in there. And that's what this benchmark says. And I actually like this one
a lot. Uh where where's the one? Uh here. GLM4. They say that they have a context window of 1 million, but they kind of become useless after 64K. So I'm not fully convinced that the context window problem will resolve itself uh in the end. So context windows might be a problem. Problem will probably get better. but some nuance, right? It's not just about the biggest context window.
But that's not the real big problem that I think these planning problems have when it comes to LMS. The second one is the problem size. Just the sheer amount of possibilities. And let me uh let me do a short pop quiz. Uh imagine you have 10 employees, 10 employees with different skills, preferences, and so on. You're going to schedule them for seven days. There are three shifts
per day. So, morning, afternoon, evening. And on average, you need three employees per shift. How many different combinations can we make for 10 employees? And anyone? I see people on their phone. So, you're disqualified. Well, this might surprise you, but even a simple problem, 10 employees for a has 10 to the 63 power combinations. Now, that that's a format we don't see often. So, I also did
uh did this so it gives a bit of a better idea how big the problem spaces. So, these are all the different combinations you can make with 10 employees. So this is very difficult problem to solve to solve efficiently and uh uh in case you don't know but I think most of you do by now LLM basically take the statistically most likely next token when you have
a problem space that that's that's that large statistically most probable next token isn't going to do much for you. So now there's this new thing, right? There's these reasoning models that can reason and and can iterate on themselves. So how are they doing then? To be fair, sometimes they succeed on very small data sets, but they think like humans. You can actually see it in the prompt
output. It thinks like, oh, I have Ann and Anna and Carl and I need to schedule them this and so. But you don't want to think like humans necessarily when you're talking about this larger scale. Human logic doesn't really work that well like a textual uh doesn't work that I actually have an example here. So this is part of the reasoning prompt. So you ask it to
check GPT and it will start reasoning for three and a half minutes and then it will come with somewhat of a decent solution. But you can see it's actually thinking like a human and that makes it really complex because sometimes the best solutions are actually something completely different or like a different approach. So this just doesn't work for this kind of So in short, why do LLMs
failure? It's the context window and the problem size. Those are the two big problems here. So, LLMs can't make these schedules even though the scheduling problems are everywhere as I showed you like uh even school timetabling. The schedule of this conference is a problem to combine correctly. You know, you need to take into account uh uh speakers when they are available and so on. So, it these
problems are everywhere but and LLMs can't do it. But surprisingly AI can and we're re at least in recent times we've gotten so stuck and made our definition of AI very narrow where if it's not Gen AI people don't say it's AI anymore right if it can't generate cat pictures are you really AI but I think the solution for these planning problems and for a bunch of
different problems where LLMs suffer and genis suffer are actually found in the outer circles of AI, the more older forms of AI, for example, in machine learning. And machine learning is uh it was a branch of AI where you give it enough data and it will learn from the data to to process things. One of the things I love about machine learning is that it has enabled
so many use cases. if you have enough data like this is sign language, you can train a machine learning model to interpret sign language just by giving it examples. I once had I should have included the video for this, but I once trained on stage a machine learning model where I had a big plush animal and I trained it to to act as a controller in a
video game. You can do those things with machine learning. So machine learning is very but can you use it to create these schedules right these planning problems and actually no because there's a few problems because to do machine learning it learns from patterns but every schedule is kind of kind of different they don't have that much in common even in the same company a schedule during the
holidays will look very different than a schedule during other days or even a schedule made during this week at your companies will be different because you spend a bit of a few days here. So learn from data. That's where these problems really become a big problem because often scheduling nurses is still done in Excel. There is no large amount of highquality data available to use machine learning
to train a model so that it can actually solve these And machine learning is actually really good at uh analyzing things and making uh uh predictions and making single decisions. But in the case of a schedule, you need to make many decisions. It's like what what shift are you going to be on? What shift are these people going to be on? Like it's a ton of different
decisions that impact each other. So you can't use machine learning just for that. So we need to zoom out even more. And now we come into the circle of artificial intelligence in the broadest sense. In here are a lot of techniques that people will now say this is not artificial intelligence anymore because we've gotten so used to it. And you know artificial intelligence is always a thing
you don't understand because once you understand it you go h that's not intelligent. You can see it now with LLMs as well. What are we saying? We have we probably have the most advanced AI at our hands and we say that's not No, it can generate cat pictures, but it just creates the next token. So, we can ruin everything like that. You know, we make it uh
I uh small tangent. I think we as humans, we don't want artificial because once it's intelligent, then what are we as humans? So, we keep moving the goalpost like no this is not AI. No, this is not AI. We've been doing that for about 25 years. So anyway, but in the artificial intelligence, that's where a lot of interesting things are. There is an area called mathematical optimization,
and that's basically using a function, a math function, and optimizing it to to make the best result. Now, there's a problem with mathematical optimization. There's not really a problem, but it's math. We some people really don't are a bit afraid of math. You know, math is scary and and and big and difficult and it feels ah I can't do this stuff, you So, luckily I uh I
I ran into a project a few years back and I've been a fan since and uh disclaimer uh I actually got hired by the company between sending this uh talk in and talking here today. So, uh I'm very hype about this. I was hyped before I started working there, but now I accidentally work there because it's an open source project and they hire me. So disclaimer. So
timefold is a way that makes planning optimization and mathematical optimization simple by taking out the It allows you to do operation uh uh optimization algorithms in domain language. It's open source. The solver part is open source. We do have some enterprise offerings around it, but you can use the solver. the really calculation part for free. It's in Java, Cotlin, and Python at the moment, and we might
expand in the future. And it's actually a framework that's existed for more than 10 years that can solve all these complex planning problems that are still everywhere in the world. So, it's actually quite incredible that all of the attention now goes to generating things with Genai. And these kind of technologies that have been around for 10 years are not used that much. So it actually behind the
scenes uses something called metauristics. So if you want to make a big schedule, you can't brute force it because there's too many combinations. It will take forever. So what you do is you make a quick first initial solution and then you start moving things around. So you just basically you take a box of a puzzle box with all the puzzle pieces. You throw it on the floor
and then you start moving things around until the the puzzle makes sense. So let me give you an example of what that looks like. Uh let me start the backend server. I'll make it bigger in just a second. So when I said you can define it in domain code. I'm going to show the Java example. Okay. So if you look at it, we basically have two classes.
I have an employee. Is this large enough to read for everyone? By the way, we have an employee which is just a simple Java class with some extra annotations on it. And you can see it has a name. It has skills, unavailable dates, desired dates, and undesired dates. So that that's things you can configure for this model. What else do we have? We have we want to
assign these employees to shifts. So in the shift class, we have some data as well. And then we have a planning variable, which is a an annotation that comes from timefold. And by combining these two classes together, it will actually try to fill all the employees in the shifts, but it's going to do it in a uh metal uristic style. So it's first going to create an
initial solution and it will move things around. This will become clear when I show you a demo. So uh I started the demo. So it should actually work. Uh no, that's wrong. Yes. So this is a scheduling problem. Uh it's nurses and you can see it's uh every nurse. Oh sorry that's location. Let me show you. So you have a few nurses and they have undesired dates.
They have unavailable dates and as a planner you need to take all this into account. Right? So this is a really difficult problem that many people suffer from every week because this is done manually in a lot of companies. And with timefold with the two classes I showed you and a bit more code that I won't go into too much detail with you can actually press solve
and then it starts solving it and it's creating this this difficult schedule So there's one more piece of the timefold code that I need to show you and that's actually here. These are constraints and it's basically uh the rules that you want the schedule to obey. So you see uh a required skill constraint. That means that if you have a shift that requires someone that's an emergency
nurse, that the nurse you assign to that shift has the skill to do emergency procedures. You can see that there's two groups, hard constraints and soft constraints. Now, uh just by the terminology hard constraints means if you break this constraints, you can't do the schedule. For example, the uh one shift per day, if the scheduler accidentally schedules two shifts for one person on one day, it's infeasible.
So that's a hard constraint. The soft constraints are like the nice to haves. For example, if an employee uh has a desired day that they want to work, if you take it into account, it improves the schedule, but if you can't do it, it's not the end of These hard and soft constraints, they each have a score. And let me just show you. It's all Java lambdas
where the the API is a bit confusing but once you get used to it it actually is is quite okay. And for example this one just checks if uh has a required skill then the employee that you assign to it needs to have that required skill and that's a hard constraint. So if you go back to schedule you can actually see on the top left you see
a score and you can actually ask it to explain the score. So you can exactly see why the scheduler or the time fold solver has decided to make the schedule the way it is. So for example, I have no hard constraints. I have a positive score of 3,300 because I have seven employees that got their desired day. So this is kind of a form of explainable AI.
It's generating something, but it also explains why certain things were assigned this way. So this is very powerful tech. Now this was a very simple one. It was only 130 shifts. How about 960 shifts at the same time? 50 employees and we just get going. So now what's happening is first it makes an initial schedule and then it starts moving things around as I said like right
now if I go here you can actually see in the logs you should be able to see in the logs here. Uh I for some reason I can't zoom in, but this says 77,000 moves per second. So it's actually moving things around at a speed of 77,000 per second. So it can try a lot of different combinations this way. Then when we get back, it has actually
created a feasible schedule for all these shifts. And uh in case you are not impressed, it's these are all the shifts, right? and it goes down quite a bit. Imagine doing that by hand like as a manual planner. It would drive you insane. So that was a very short demo. So what you do with timefold is basically you model your domain in Java classes. So if you're
a Java programmer or Cotlin or Python, you can just use your own programming language. You define some constraints. You say I want it to react like this and then you run it. And that's how you can create these very complex schedules that are honestly everywhere and is so many people have not automated or optimized for this. Uh we have a few example use cases. So we have
vehicle routing, order picking, bet allocation, meeting scheduling. I think there's there's a cool thing you could build there. Uh employee scheduling, uh school time tableing, even conference scheduling because this is used behind the scenes for the DevOps conferences to make this this big schedule. So there are so many use cases that can be built with this. So LLMs can't do this stuff, but there's older forms of
AI, old school AI like the things that Timefold is doing to do uh to do these things. So where do we think the future of AI is So I really believe that now we are too focused on the Geni thing. A lot of people are focusing on Geni Geni and what can we uh what can GI do for us and I was talking before and actually have
a funny twist on that. Maybe we should think what we can do for Genai. how can we make it better and expand upon it and as time folds we are in the outer uh artificial intelligence circle but even beside that circle there's a bunch of cool things like workflow integrations that you could do and I think combining things will be awesome so for the old school AI
that's timeful but other forms of of programming as well we can use LLM to explain these constraints in a human language we can support the domain expert when they're picking these Java classes and such to help them build it. And we can write code, of course, because that's like the thing that most people are doing now. We can write code. Well, you can write timefold code that
then does this. And there's so much more things that we could be doing. Uh we actually uh created our own GPT. It was very simple. Go to chat GPT. You just upload your documentation and you have your own GPT. It's not fancy, but it really helps users of Timefold or maybe your own products to to get on board. So that's how LLMs can support current technologies. But
what about LLM? What can we do for LLMs? Well, that's we can integrate them. So who was here for the goose talk? A few people. So in case you don't know, model context protocol is a way that allows uh tools like claude or goose to connect LLMs with servers that you built yourself and for example for timefold we did this uh this is just clawed and if
you click on a certain button it shows which tools are available and we actually made available two tools automate schedule and optimize demo data. This is a a a proof of concept. It's not live. It's just on my PC. But the LLM is actually capable of calling these tools. So in this way, you can expand LLMs to use all the power of previous forms of AI, previous
technologies. And I think this expansion is probably the biggest thing happening now in AI. It's not the LLMs itself. That's like a separate category. But finding cool ways, cool tools to integrate with LLMs and make uh things possible for the LLM that they normally couldn't do is probably even a bigger interesting field to work in. So to conclude, because I think I'm almost out of time, we
had fun, right? We started with cats and now we're all the way down to LLM. So it's all good. Uh so to conclude, Genai is awesome. It is. There's so much stuff we can do with it. I think it's here to stay and we can do just amazing things with it, but it can't do everything and we need to keep this sharp because so many people are
now looking at chat GPT or cloud or LLMs and they think I'm just going to let it do everything and then they get frustrated because it can't do certain things. So whenever that happens, try to investigate why can't an LLM do this? And then maybe with MCP and such, you can find a way to to make it possible for the LM to give it tools so it
can do these complex things. So especially uh since we're now so super focused on GI, it might be a good idea to look at other forms of AI and machine learning because there's a lot of value in there. And I uh I was very excited that I could join Timefold after being a fanboy for the product for for quite a while. Uh because I believe that that's
an amazing technology that will enable a lot of people to to do their job a lot better. Like if you don't have to spend an hour or two every day to create a schedule for your employees, if you can make a timet for for the school in the neighborhood, if you can uh insert other use case, uh there's so much more time left. So as part of
uh timefold, our mission is kind of to free the world from wasteful scheduling because we think a lot of time and money and effort is being wasted. And I would like to invite you to also give it a try. If you don't uh think this is for you, all good. I think it's the most awesome thing I've ever worked on. Uh it will help you create schedules.
We actually have a quick start page that has a few quick starts for different use cases that you can just pull from git run. Uh actually the employee uh shift scheduling use case I showed you is just a quick start. So it's available uh to use and uh that's kind of where we're going towards. So if you take anything away from today, you learned a lot about
AI if you stayed in this track. Use it. But don't just look at what AI can do for you, but also what you can do for the LLM. And that's the end of the talk. I hope you had a nice conference. See you next time.
