Walid Tazout: Simulating the Chaos of Real-World Networks for Resilient Applications

This talk focuses on the importance of addressing network instability while testing applications, particularly for platforms like Spotify. The speaker, a QA engineer at Spotify, discusses the limitations of testing under ideal network conditions and emphasizes the necessity of replicating real-world scenarios where users encounter poor connections. He shares a specific case involving Spotify's offline mode failure in cars, illustrating how even a slight network quality can lead to significant user experience issues. To address these challenges, he introduces an in-house tool that simulates various network impairments, allowing teams to test apps under different conditions, such as latency and packet loss. The session highlights the need for consistent app reliability, advocating for early and frequent testing in realistic environments to preempt user-facing problems and improve overall app performance.

Thank you. Thank you. And thank you all for being here. I know there's a lot of talks and thank you for choosing to be in this one. And I appreciate how hard it is to focus after lunch. So I hope you're hopped up on coffee and sugar and you're ready to focus. I'm going to go a little bit beyond that and I'm going to poke at the

audience a little bit because I like to get a sense of who I'm talking to. So let's start with this question. Uh, who among you in their current line of work is working on an app that's completely offline, does not need internet to operate? All right, two people. Great. I love that. That's not the trend today. I hope you get as much value out of this as

everyone else. Let's move on. Who everyone else who they have they have apps that are connected? Who among you have management for like an offline mode or for when internet drops? So either uh for example like an offline mode with like a subset of features or just a page that says oops you're offline try again. Basically failing gra gracefully. Show of hands. All right hands. I like

that. Now I'm going to challenge you a little bit. Uh as you know internet connection is not an onoff switch. It's usually a slider. More realistically it's about like a 100 sliders that determine how good or bad your connection is. So for people who have offline mode, the business logic to detect offline mode is it as simple I'm not getting anything anything from the back end or

is it more nuanced than that and there's more business logic to detect for example I'm not getting enough packets to load my media who among you has a more complex uh kind of business logic all right fewer hands a lot fewer hands great that tells me that everybody here is in the right room and you're going to get as much value out of this. So, I'll introduce

myself a little bit. My name is Wed. I'm a QA uh engineer at uh Spotify uh particularly in the PPX studio. Uh that's platform and product experiences. And that's a studio that focuses on the Spotify experience that's outside of just the core uh experience like your phone, your headsets. So, think Spotify in your speaker, Spotify uh on your car. Uh, believe it or not, I worked on

Spotify in your coffee machine, Spotify in your TV, and I still can't believe I'm saying this. Spotify and your connected toothbrush. I am not lying. Uh, but yeah, but most importantly, I'm going to tell you about Spotify and car because that's what inspired this whole thing. I was talking to one of RBD representatives, that's business development, and he was telling me about an issue he had with

Spotify where the offline mode just does not trigger. What happens is that he tells me that uh it just no matter how much I try I download the the the data the music and when I try to use it in the car using Android Auto or uh yeah was it was Android Auto it just does not la it keeps trying to reload and and and I I

was like I took it personally. I was a little bit offended. I was like what do you mean? We tested this times and times over. Uh, and he was like a bigger person, so he wasn't offended that I was calling him a liar. But basically, we kind of was like, "Show me what you do every morning." And he was like, "Fine. I wake up. I know internet

is not great on my way to work. So, I download a playlist. When I get in the car, I hope that that playlist will be triggered when the offline mode is triggered, but it never does. It keeps trying to load it." I was like, "That's a real problem. Why does it not trigger?" And then when we like I'll save you a lot of the investigation, but basically

the Spotify app it has just enough internet to think that it's online and load the metadata but not enough internet to fetch the media from the CDN and it kept it keeps uh getting like trying to stuck trying to load that data or loading it super slowly without realizing that it can't and switch into the offline mode. And that's our first problem, the fact that our users

got this issue before we do. So I took it back to the dev team and I was like, we have this issue. Uh let's work on it. And they're like, great, let's try to reproduce it so we can get some logs uh from our back end. And I was like, hey, hit me. Second problem. I can't reproduce this. I can't I don't have the tools. Either I'm

working on online or offline. I don't have like the slider we just talked about. And that's issue number two. So we thought they they they kind of guessed what the issue would be. They implemented solutions and we had like three or four candidates that we needed to test. I couldn't test them. So what I did is I grabbed our test car and I went around Stockholm trying

to find a spot with bad uh network. And I feel like I was the first person in history who said, "Shit, Stockholm has really good coverage. I can't find a spot to test this." But I ended up finding it after like I think a day and a half. Uh but my manager was not amused that I was driving around for a day and a half. So which

leads us to our third problem. It's really expensive. Um so yeah, that's how this all started and that's how I started thinking about a solution for this. So let's formulate our uh problem statement. Uh the problem is that uh bad network is the norm. It's not the edge case and we shouldn't treat it as such. And when I say bad network, I don't mean slow network. I

mean unstable network. A network that changes over time, which if that's all networks. Why? Because we don't control as developers. We don't control the whole pipeline. If we work car manufacturers, we don't control the road. So you don't control the coffee Wi-Fi. You don't control the what the elevator that the user is going to get into is made of. You don't control how fast the train that

user is going to be in is made of. or like how fast is it going? So there are like a lot of mobility there. There's like cell tower handoffs and basically you can't control that. So you need to plan for it. If you assume the chaos and it works there, it's going to work everywhere. So we created a tool. The solution is that we create a tool

that simulates that chaos to our devices. And before we move into the how, let's talk about why we need to simulate. Uh one is because or like the main reason is that because production is hostile as we just saw and we needed to fix the solutions that we found in our car story. We need to catch issues before our users do and we need to make bugs

reproducible and usually it's a whole lot less expensive to fix this beforehand then try to do some firefighting afterwards. So, in summary, if you're only testing in your office Wi-Fi, you're basically running a very, very exclusive beta test, and you're probably not going to catch your app in its worst behavior, which is basically what it's going to do in production, and give you headaches. So, the test

philosophies behind all of this is don't wait for the field or production to teach you what your app behaves like. And the goal that I want you to take home with you is that your app needs to be boringly reliable even on bad days. So this is our solution that we created. It's an in-house solution that kind of helps us deliver on what we do. What is

it? It's a routerbased network impairment controller that you kind of drive from your computer, a simple UI from your computer. And it allows you to impose uh network limitations or impairments or like latency, jitter, packet loss, and complete clean connection drops. uh allows you to kind of formulate those into a scenario or a test case and then you can like uh you can uh run those on

your device under test. Uh a lot of predefined scenarios are available for our testers but you can also use AI because that's everywhere now to generate those scenarios. Uh so now we saw the what is it let's move into how it's made and the goal for this is that to kind of inspire you if you are in the same use case make something like this that works

for you. So how does it work? So basically it's uh it's let's start from the client UI. It's just a flask uh Python app that's uh a Flask Python app that kind of controls that has the UI and that lets you uh connect to the router and it lets you target one specific device and uh uh it lets you kind of impose uh individual impairments on that

device. uh that uh that Flask app container talks to the router which uh just for info any open WRT router could work. What we chose is the smallest and the cheapest and the most portable so they can we can carry it around and we can exchange it over with labs but any uh router that runs uh or can run OpenWRT uh for info. OpenWRT is a Linux

distribution that works for embedded systems most specifically routers and it's highly customizable and that's why we went with this. Uh so in the router uh we created like kind of a router API service so we can access and control it. It has uh four endpoints uh connect target uh so connect so the computer can connect to the router uh target so you can specify which IP address

is the one that belongs to the device under test apply network impairments and uh resets and because we're running scenarios we needed some sort of session management to kind of know where we are where we're going and when to stop and finally uh it forced us to put in some authentication for sense of security because uh the moment we plugged an external router they came running they're

like what is this what are you doing uh yeah and that API service controls uh uh a network simulation engine and think of this as uh the kind of thing that actually does the impairments it's a shell script uh with uh basically uses the command uh TC or uh traffic control it's a very simple but powerful command that allows you to do four things to the traffic

that goes through that system. One, it allows you to shape it, schedule it, police it, and drop it. And in combination with all of this, you can basically do any everything we just talked about like, you know, uh introducing jitter, introducing packet loss in any uniformity and any shape you want. And uh yep what it does is basically it targets one specific device under tests without uh

letting all the other traffic pass through so you don't form a bottleneck. As you can see here speaking of bottlenecks uh that the control path and the data path are completely separate and that is by design. So we don't introduce any artificial bottlenecks that would kind of uh create issues that we would not see in production. moving on next, I'm going to walk you through just the

UI and to give you a idea about how it actually works. Uh so starting with our uh setup and logs. So uh here you input your uh your router's uh your router's IP address and then you input your uh API port that you designed in the in the router and then you put the target configuration which is the DUT's uh uh IP address and then the router

interface that we're using in our router. there was only one interface because we only needed to test one device at a time. But for multi-device testing or like IoT devices, you can buy a fancier router with multiple interfaces and you can run those tests as well. And finally, or like underneath it, there's a log section that uh you can see what's happening not only on your client

but also on the router as well because it sends the logs and the errors upstream uh to your client. Uh after that we have individual impairments which is basically impairments that you can put in uh that are isolated. So we can see here the first one is a download speed limiter which is a very common one a lot what a lot of our developers were interested in

is what my app would look like if I gave it for example uh I don't know 100 kilobits per second. Uh after that we uh added in network emulation uh options. The most common ones that people wanted to use or that we thought were going to be more useful at the latency, jitter and packet loss. And uh these were super useful in kind of a scenarios where

like internet is like download speed is good but uh the the the internet quality or network quality is bad. So for example, this is what would happen in a conference like this where or like in a concert a lot of people are connected to the same cell tower. You're going to see five bars but you're not going to be able to do much with it because while

the download speed might be high there's a lot of like uh congestion in the network and uh we have some uh Linux fiends that were like this is not enough we need more. uh the TC command as I said it's very powerful and offers you the chance to add a lot more um um what do you call it uh arguments or options and for that uh we

added an option there where like they can just write it and it would be passed through the command line and basically what this allows you do is not only to control how much latency how much jitter or how much racket loss but it's also like its distribution so for example if we say 5% packet loss that's 5% of your packets not getting back. But what this allows

you to do is basically how to distribute the if you don't put any uh option, it's going to be uniform. But again, that doesn't reflect the real world scenarios. So if you want to get to the real nitty-gritty situations, you're going to be able to do that. Also, if you like the nuclear option, there's also activate black hole that completely drops all network and uh the command

Z or control Z that kind of uh resets all the the impairments that you do. But as we talked about before, uh real life network is not uniform. You don't have like a strict uh a strict network limitation or just like network or packet loss or jitter that's like constant. It's usually moving. So what you need to actually simulate what happens in the real world is uh

scenarios. So if you think about individual impairments as Lego blocks, this is your uh scenarios are when you put them together and you get like your nice Lego piece at the end. So you can uh we have uh we we we kind of partnered with our like network engineers to kind of guess what for example uh driving through a congested area would look like, going down the

elevator would look like. And then we put some predefined scenarios that people can uh export, they can modify, they can import back for sake of consistency of tests between the teams and uh within the team itself. And as I said, uh I am not a person that's going to shove AI down people's throat, but we found it really useful here because you can't expect all developer engineers

to have like deep knowledge of network engineering. So what we did is we partnered with the same kind of network engineers to draft uh uh system prompt that kind of helped people to just say for example the example that we have here is uh underground parking resulting in gradual loss of connectivity and it would generate the scenario for you. You can take it tweak it and upload

it back and it would work. So uh I just want to reiterate or like flag that this is the tool that worked for us. It doesn't mean that you need to go build your in-house tool as well. There are multiple tools out there that we're going to see in a second. This is just the one that fits our needs because we work with individual devices that are

external like a speaker, uh a head unit for a car, a simulation device, and we needed to fit all of those. So, let's look at some of the alternatives. First one that comes to mind, and I think it comes to everyone's mind when it comes to network manipulation is HTTPS proxies. And here we're talking about for example uh Charles proxy uh or like uh HTTP fiddler I

think it's called. Uh those are excellent for API debugging. You can see the headers, the bodies, uh the status codes, retries, and you can easily kind of uh rewrite the request response, you can record and replay, you can script it. It's perfect, but it's weak at simulating real world uh real world transport chaos. So it can't introduce jitter loss because it works if you know the OC

model for the IP it works more closer to the application layer but what you need is something a little bit up higher to the hardware layer. Uh moving on to the OS link conditioner that's a little bit closer to the hardware layer but uh so here I'm talking about like the Mac OS link conditioner or third party solutions uh like uh I forgot the Windows uh uh

it'll come back I think it's called Windows dummy or something. You can find it on GitHub and download it. And it's really cool to kind of limit the network of your own computer, but that's the catch. It only works on your computer and it only works on the system you're in. Uh, and it doesn't allow you to hit other systems on your network. Uh, which leads us

to the bespoke solution, which is the one emulator or wide area network emulator. This this is enterprise grade. A few companies that do this. You get a router, you get their own software and it allows you to kind of do exactly what we wanted to do. It's expensive to buy and it's expensive to maintain because you need the license for the software, you need support. Uh, but

this is where the situation we were in. This is what we were faced with. And our solution was to take what worked for us in all of these and create our own tool. And I'm not going to lie to you, it was easier than expected. It cost I think the router was like around hundred euros and it took a week weeks worth of work to get a

working prototype and a couple of weeks to refine it and like kind of share it with the team. So I really I highly encourage you to create a in-house solution that fits your exact need. It's not going to be that big of an uh of an investment. So moving on, this is the the the feature that pushed us to finally create this uh because we were pushing

kind of this boulder down the hill until we reached uh the when we wanted to launch lossless and we were like yeah we need to test on uh diverse uh network conditions but we had no tool to do so and that's when we actually pulled the trigger on So why is lossless very very tricky? Uh is because instead of uh if you know a little bit about

codecs it instead of streaming vorbis uh uh files which kind of consume at most 320 kilobits per second we're working here with flack lossless files. So the same sd quality that needs to stream without any glitches. And for you to do that, you need upwards of 200 uh 2.4 2.5 megabits per second just so you can play without an issue and keep your buffer and we were

sure that most of our users do not have that consistently throughout their day. So we needed to test what would happen when the eventual case of like you know few fluctuations in the system dropped and that's when we created the tool and what we watched out for is uh kind of non-functional criterias like for example startup and seek latency when you start your music how fast does

it play or how slow does it play depending on your network condition when you seek through the bar how slow does it play and uh also How much time does it stutter when the buffer runs out? Does it figure out that it's actually not playing? Does it figure out that the speaker is actually stuttering? We would not have been able to figure this out without actual real

life network simulation. And and those are like all blackbox kind of attributes. And we also looked at white box attributes like how our internal systems uh rebuffer and uh when do they trigger the rebuffer, how many times do they trigger the rebuffer before they announce that they can't actually uh finish the streaming or continue the stream. And our goal was to have a consistent UX on variable

UI or cellular network without any regressions for the normal uh for the normal streaming. And uh most importantly, something that we haven't thought about before because we never had streaming that was this costly is quick and interruptionfree recovery. When an issue happens, we needed to recover fast. Issues like network did not happen as much before, but that's what we were going to introduce with lossless. And that's

exactly what we caught and we fixed before the actual launch. So, uh, to kind of bring this all together, if I need to leave you with three things, it would be this one. Consistency always beats raw speed or raw performance. Why? Because users usually forget or forgive a fast enough app if it's consistent. But they will not forgive a super fast app if every day, every time

they have to use it, it's like a roulette of how good, how bad is it going to be. The second thing is that you need to simulate early and often. Do not wait until it hits production for you to figure out the non-functional issues that you're going to have. Especially when it comes to network, I feel like it's a blind area for a lot of people. And

third thing, uh, scenarios always beat ad hoc poking. So, even in exploratory testing, make sure to create a scenario rather than just keep poking at your app because you're going to be able to repeat that scenario you and your teammate as uh development goes along. And uh yeah, that's it everyone. Thank you so much for paying attention and I hope you got something out of Okay, we

have enough time uh to go on with the Q&A. Okay, there is one question now. We start with this and I hope to see more of course. Um, okay. Our app runs smoothly on our network but slows down for clients overseas. How would you test and troubleshoot this kind of network biased performance issue? That's a tricky one. >> Okay. So, I'm I'm I'm not an expert on

this particular topic, but it seems like this is something that's uh server related. Your servers are if it works uh when you say on our network, you mean in the office uh Okay. So that's exactly a perfect illustration of what we need to test. We need to test uh outside of the office. A very cheap alternative. Wait, like you think it's cheap, but it's going to turn

out to be expensive is to go out there and try to test in the field, but then you you really hardly can replicate exactly what the user who replic who told you there's an issue, you can't replicate what they actually went through. And they don't usually like users when they find an issue, they're not going to give you a descriptive uh fault report. they're going to tell

you, "Oh, the app breaks." So, uh I would advise you to move a little bit more towards uh network simulation to see why your uh users are having an issue. Unless it's a back or like a server distribution issue, then that's a completely separate topic. >> Okay, next one. Can Spotify tool be open sourced? Uh that was my first instinct when this was created and but then

you have to go through like the the opensource people and once it's open source they have to maintain it. So you have to convince them to actually own and develop this tool. Uh you can make a case for it uh but it requires investment from their end and uh I can try uh but I'm not sure that that's the direction they would want to go with. >>

Okay. Well hopefully it works. Hopefully Okay. How did you fix the flag uh lossless issue after you simulated it? >> Uh so uh to be like exact when I tested I tested the flag issue kind of on speakers. So it was a collaborative uh efforts between us and our partners. It was a lot of like kind of back and forth. We realized that the buffers that the

partners are using are not big enough for the actual flag files. So, as I said, a song that's a few uh megabits, uh if you have like kind of a buffer of like I don't know uh and if if you know a little bit about the embedded world, uh hardware is costly. You can't have just four gig gigabits of RAM. So, they try to reduce that as

much as possible. But what they realized is that their buffers are too small to run Flack. So, they had kind of we pushed them to have an adaptive uh memory allocation for the buffer whenever Flack is running. So, it allowed them to have a little bit more overhead and uh while they're fetching the the music, they can tell when there is an issue and they can kind

of uh bring it down or bring it up depending on network conditions. >> Okay. Uh the next one, did you run scenarios using VPNs and does it matter? >> Actually, it matters and you need to disable the VPN because it routes your traffic. What it basically what it did for us is that it disrupted the control path of our uh because we needed to talk to the

router directly and with the authentication bit that we added we enforced that rule is that whoever talks to the API needs to be local uh and the API breaks that. So we had to disable the API whenever you needed to test uh this because it needed to happen on the local area network and not outside of Okay, then next one. Okay, they're just jumping in between them.

Let's take this one. Why not to use out of the box solutions? Uh there was something like microtic. Did you see it? Why not to use out of the box solutions? I think we we can work with that. >> Yeah. Uh I mean the the one we use is an out of-box sol. It comes with double open uh WRT connected. the ones that we found that has

like for example very friendly UIs, it did not offer the level of flexibility uh that we wanted uh especially for our test cases. So uh the I what we would have happened is that we got those and we were going to need to install our own shell script nonetheless and they're way more expensive than your run-of-the-mill uh OpenWRT router. So we decided to go with a cheap

solution that has the potential uh because while microte is like really good, I use it at home. Uh it wasn't the right fit for uh for this particular use case. Uh especially that we needed to buy a bunch of them to distribute them over people uh across Stockholm and Goththingberg and they needed to be small and movable. This one that we chose actually works with USBC power.

So you can hook it to a power bank and uh put a 4G chip in it and you can move anywhere you want. It's very portable. Okay, then next one. Do you also have automated pipelines? If yes, where they're running and how to simulate network luck? >> Uh, for this for testing for like the the actual like phones, yes, we do. But for what I was working

on, which is like kind of speakers is hard to automate the pipelines there because you're running a kind of a hardware and the loop for a hardware that you do not own. While we make the the the SDKs that go in the speaker, we don't own the speaker and we don't own the tech. So, it's a hardware in the loop and it's a real blackbox situation that

we have there. Uh, that being said, we managed to kind of hack our way into kind of a uh sending uh the the commands through the back end to the speaker uh and read through the back end. So, it's not a true blackbox. Uh but uh I left that door open by uh creating the API on the uh on the router. So when or if we can

actually do it any curl command would kind of trigger these scenarios in the router in case we want to explore that route. >> Okay, great. The next one uh how did you get the rout router to tell you the list of devices for which it chooses one that will basically that that will have the traffic throttled. >> Uh we actually don't ask the router for that. they

have to figure out uh your your your devices IP address either if you're using your your phone you can find it or you can find like uh use the the the router's UI that just basically it's any router has the list of devices that are connected to it and you can find the IP address of your device. Most devices that we use, you can actually know which

IP address it has uh either from like the device itself or from the companion app. Like for example, for speakers, there's a companion app that can tell you uh in the dev mode which uh what's the IP and MAC address of that >> Okay, great. Uh so is this tool more for mobile or is it also for the websites? why we created this tool the way it

is now, why voted uh we avoided uh proxies is for it to work with everything anything that can be connected to the internet via uh uh in uh Wi-Fi or Ethernet you can use this tool with it. I'm guessing for a site for a website it would be a little bit of a over spending, but it didn't cost that much. So, it would work perfectly and it

would simulate perfectly how the user would be uh actually using it because the user would be on a laptop and it's their internet that's going to be uh fluctuating as opposed to testing on your browser for example with dev tools that limit the bandwidth just for your tool. That's too synthetic to kind of uh replicate what's actually happening out Okay, great. Uh, yeah, that's a nice one.

Okay. Do you currently have open QA engineers positions? >> No comments. >> In our team, actually, I don't have a view over the whole company, but in our team, we just filled one. Uh, but I mean, we we we the the the career websites is pretty up to date. So, there are no ghost postings. There are if there is an open position, it's going to be there.

Okay, perfect. So now the next one, what was the solution for app stabilization? Was it to increase the buffer size? Uh, as I said, I mean, I think I answered this before. I wasn't working on the mobile app itself. My focus was more on embedded devices. And yes, the solution was either uh increase the buffer or improve how it works. >> Okay, then the next one. Uh

so the okay what was again the US conditions for Windows? >> Uh I keep forgetting its name but wait I I'll come back for it. >> Okay maybe >> it's called clumsy uh Windows clumsy. You can find it on uh GitHub. It's a third party. It's not produced by Windows like the Apple one. So uh do your research about how it works before you actually put it

on your computer especially your work computer if it allows for open source uh software. So be careful with that. >> Perfect. Yeah. So you said also it was easy to build in house and we are wondering here how many devs were needed and how many resources were needed to build the system. >> So it's uh I don't know if you know in Spotify we have like something

called hack week. It's a week out of the year where you could just play around, build whatever feature you want. Uh, not everything, almost not all of it gets shipped to production, but basically you have a cart blanch to build whatever you want. For me, as a tester, I'm passionate about test tooling. So, this is what I went with. Uh, as I said, it took the five

working weeks of working just on this to actually get a working uh version. uh and a couple of weeks after that with me and like the as I said the network engineers to actually craft the the scenarios. So uh not much with Genai. Now you can basically uh gap the gap or like bridge the gaps that we have for example like in uh coding to actually like

build this and I mean the code for all of it is not that long because as I said simple UI you can make it fancier, you can make it more complex but we just needed it to do that one thing and we were not going to expose it or give it to clients. So we kind of uh jumped a lot of uh security measures because it's just

going to run in house. Uh but if you need just something that works and works internally, I don't think it's going to take that much. >> Okay. Then can we use blaze matter to basically do performance testing for different locations? >> Uh I'm sorry I am not familiar with blaze meter. >> Anyone wants to elaborate or not really? Then we skip this one. uh in iOS um

app development they suggest best practice to try all requests until timeout even if the device is offline. Is this good or bad in your opinion? >> My personal opinion is no but that depends on your app and your product. uh how would your clients uh react if they had to wait uh for a long time before the app actually times out. And as we saw in the

examples I gave with the with the car is that sometimes it doesn't time out. Sometimes you have just a smidgen of internet that you actually get a response from the the back end without actually like you know the functionality being able to run. So you need to define what is a workable solution and you need to define when you don't get that solution that's a broken experience

and you need to let the user know that that's a broken experience instead of just like uh showing them the infinite uh the infinite loading screen or like uh as I saw like uh somebody joked that they removed or they changed the loading screen with thinking to give the illusion that it has geni. So you can do that and it would be fine. >> Okay. Then the

last question uh could an automated Charles proxy script that fluctuates bandwidths and delay be enough for most apps and test scenarios? >> Um well uh [sighs] yes but as I said uh again it for the app it's going to be too synthetic because it works on the very highest level of the OC. It works on the application layer. So the actual scenarios where it's going to actually

uh hit the base layer of your OC of your IP traffic or even if it's not IP if it's like UDP traffic if you use it in your app you're not really going to be able to replicate that. While this covers what a lot of people actually miss, if you want to have a complete and simple solution, you need to simulate the whole traffic.