From Bytes to Brilliance : Optimizing Large File Delivery in REST APIs | Nupur Agarwal (EN)

In this talk, Nupur, a technical lead at ING, discusses the challenges faced while building an API to deliver large zip files using Spring Boot. The speaker reveals how the initial design led to performance issues due to loading entire files into memory, resulting in out-of-memory errors and corrupted downloads. By transitioning to a streaming approach, they drastically improved memory usage and eliminated crashes. Nupur emphasizes the importance of managing the stream lifecycle and aligning HTTP headers and timeout settings properly to ensure successful file delivery. The session concludes with key takeaways to avoid common pitfalls when developing large file delivery systems.

[music] >> Thank you all for joining on the last day of the conference. I'm really glad you guys can make it. Let me start with a confession. This talk is not about AI. But AI has a major role to play in it. Yay! >> [laughter] [gasps] >> Last year our team built an API using AI that worked perfectly. It passed all our test cases. It ran well

in development, in staging environment. But as soon as we promoted it to production with broad like data, it started crashing our servers. And not because of any bug, not because of Kafka, not because of database, but because of a single perfectly reasonable line of code that everyone of us in the room has written at least once. Today I'm going to show you that mistake, why it nearly

took down our system, and more importantly, what it taught us to send large files with Spring Boot. If you have ever built a simple download file API or going to build one soon, I'm thinking that you will love this presentation. Before we dive into the damage, a quick intro. I'm Nupur, technical lead in ING. I mostly work on Java, Kafka, Spring Boot, and data heavy streaming applications.

If you are wondering what do I mean by heavy streaming, I'm talking about data more than 4 to 5 millions. I care deeply about clean architectures, and more than that, architectures that are scalable and can survive production. Originally from India and currently working in the beautiful Amsterdam. If you want to connect, here is the LinkedIn QR code. And if this talk triggers some painful memories, let's connect

afterwards. Sorry for bombarding with one more QR code. This is the code which you can scan for putting down any questions if in between the talk or now if you have any. Please feel free to scan it. Let's talk about what we were trying to do. Our API needed to deliver large zip files containing reconciliation data. It's kind of a whole data dump from the system of

record to system of copy. On paper, this problem looks promising and kind of boring. Fetch files, zip files, and send them via REST API. That's doable, right? REST APIs do this all the time. What we underestimated is that REST was never designed for gigabyte scale binary Keeping that in mind, we started designing our application with the REST This was not a convenience feature for us, a very

important financially critical data that we needed to deliver and we could not afford any failure or any delivery time out. That meant zero tolerance for corrupted files, guaranteed end-to-end data accuracy, observable because it's a long-running download, so of course visibility into the long-running downloads. Um reasonable performance and resilience against network failure. At this point, we were still confident we could do that. This is just a file

delivery, how hard it could be. Here is what we built first. The API fetched inventory file and checksum file from S3 bucket, zipped it, and returned it back as a byte array response to the client. It was elegant, simple, readable solution, and that's exactly why it was dangerous. Because what it really meant was before sending file, we loaded entire zip into our memory. And as I mentioned

that our data was not not in megabytes, but it was 4 to 5 millions, mostly in GBs. This slide shows the kind of code that caused all of this. Fetch files, zip files, convert it to byte array, and send it a response. If you have a 500 MB file, you are sending response to MB file, and you all hit at once. So, for 500 MB request, you

have 500 MB in heap per request. Multiply that by concurrency, and your back-end application is already a memory bomb. It was kind of a journey where we encountered different timelines, different issues at different point of And first one was out of memory error. Not occasionally, not during peak loads, just reliably. Every time a few downloads happen, the JVM filled up, and server went down. Suddenly, the file

downloads were started restarting our >> [snorts] >> And even when memory survived, time did not. Large zip files take time to make and send. Default HTTP timeouts kicked in. Spring framework timeouts kicked in. Requests were killed midway. Clients got corrupted downloads. And we got confused because sometimes the solution was working and sometime it was not. At this point of time, we have to admit something. The system

was not buggy, but there was a design problem. Design was not right. Something was wrong fundamentally. And after fixing these solutions, these problems, we went to production and we are doing well in production. It was a challenging journey for us, but everything is stable. We have not encountered any issue yet. Now, every month data grows, but still the solution that I'm going to share is working fine

for us in the live So, let's see what is the problem with byte array. So, byte array load entire file into the memory, which is the biggest problem. And then it converts it to byte array. And then it sends entire array in single response, which takes up your heap. Let's put number into it. 300 for 300 MB file, it was fine. For 500 MB, it was still

okay. For 1 GB, our system got crashed. So, why I'm putting these numbers here? Because it's not compulsory that what solution worked for me also worked for you. So, it really depends what's best is how much data you are taking care of. I will pause here for a moment. If this were your service in production, how would you approach this? So, there is no right or wrong

answer. I every solution, every design can have different trade-offs. But anyone in the room like what how would you approach this problem? Thank you. Bravo. Make hands for you. And that's what we did. Streaming. This was breakthrough was good. Large files should never be fully loaded into the memory and sent. They must always be streamed. Instead of load all and send, it should be read a little,

send a little, read a little and send a little. That's what streaming is. Instantly, memory usage stopped growing with the file size. And this single decision eliminated all our out of memory errors instantly. Let's see how streaming works. Streaming is not a magic. It's just a loop. Read a small chunk. Write to the network. And then, repeat. No buffering of the full file. No memory spike. Just

do it in a flow till the file ends. In Spring Boot, we eliminated this every implemented this using streaming response body. It gave us direct control over the response output stream. We zipped our files and we streamed them. This was the moment where our production graph started finally went flat. It felt quite for some some time at least after this solution. And quiet is always good in

production, right? Our production graphs were fine after that. But streaming introduced a new mic nightmare. Corrupted zip files. So, when you when you buffer whole file into the memory and you send it, so then you know, either you load the file or send it or you don't have any response. But there are less chances of corrupted files. But with streaming, because you are reading a little, sending

a little, so there are more chances of files getting corrupted, your zip getting corrupted. So, because of that, you need to understand and own your stream life cycle as well. For us, streams were not always getting closed correctly. And because of that, our zip metadata getting updated on the stream correctly. And closing the stream helped us. Closing the stream finalizes, flushes, and writes your zip's metadata correctly

on your output. And our data integrity came back. As I mentioned before, that was very important for us, and it worked fine for Then came one of my favorite moments, compression. So, we added one line, setting the zip compression level. Technically, this line changes the zip compression level from default to best compression. Same streaming logic, smaller compressed It simply asked the JVM to spend more CPU per

chunk of stream to deflate the size of the stream. For reconciliation data, where it's mostly text and repeating kind of structure, it really helped us a lot. This paid off massively. The key point is that optimization is sometimes only about one line of code, and it changes a lot in your code and in your performance. So, let's see how does the whole flow look like after streaming

and compression. On the left, we read small small chunks from S3 at a time. These chunks uh were converted to byte. Then, we had uh we are doing using Deflator best compression on the zip And then, we simply send this compressed bytes to the socket as the response. The important part is that nothing waits for the full file. So, for the small chunk, we are reading it,

converting to byte, compressing it, and then sending to the socket. Up to this point in our journey, everything we fixed lived in the Java But, here is a thing that we need to remember always, that perfect streaming code can still completely completely fail at HTTP layer as well, not only at the Java layer. Streaming does not only depend on how you write how you write your byte

code, but it also depends on what you promise your client with your HTTP headers. So, for example, for content type, you have to set binary response when you are sending files. So, it could be either either application octet stream or application zip. Then, setting content disposition as attachment is not a only cosmetic thing. By this, you are telling your client that this is a stream. If you

don't send it, then your client will try to open this attachment as a response inline. As as a result of which, client will see either corrupted downloads or broken downloads or broken files. So, it's very important to use the correct content disposition and content type headers HTTP headers in your Avoid content length with streaming. Yeah, we are not buffering anymore. We hate buffering. We are streaming. So,

of course, don't use content length at all with streaming. Because with content length, you are mentioning to the client, "Okay, this could be the size of the output that you can expect." So, client will start buffering and calculate the content length of your So, avoid that. This slide is my favorite matrix slide. After streaming with some header changes, with some compression, not only we fixed our out

of memory errors, but also we uh improved our resource usage. So, if you see here, our CPU usage was reduced, our memory usage was reduced drastically from 5 GB to it uh came down to 3 GB. So, it was a big win for us. Not only we fixed the errors, but we reduced our And uh in our case, the solution worked. Just when we thought we are

done, timeouts came back to haunt us. The story is not finished yet. Spring was terminating our request because the asynchronous processing exceeded configured time limits, resulting in HTTP 503 errors. We have a Tomcat connection timeout So, it uh uh server Tomcat connection timeout tells it controls basically how long the server keeps your connection without any network activity. So, if there is no network activity, it will close

the connection. By default, it's configured as 20 seconds. Then second is second important one is spring MVC async request. It comes into the picture especially for asynchronous processing of spring. By default, uh it's not configured, but generally asynchronous request gets timed out um before other timeouts are achieved, timeout values are achieved. Um when these timeouts are misaligned, the client may disconnect before streaming the whole response, and

they may get corrupted downloads. And Tomcat may close the socket while streaming your streaming code is still writing the data to the stream. So, these two parameters need to be very much aligned. Otherwise, you will have corrupted downloads and intermittently failures and HTTP 503 errors as we encountered. We had We had to align timeout across layers, Spring async timeout and Tomcat socket timeout. One thing to remember

here is that Spring async timeout should always be lesser than your Tomcat socket timeout. Why? Because if you have not always, but yeah, of course, when you are using asynchronous processing. Because for your long-running asynchronous processes, like here we have streaming, so there it could happen that your Tomcat connection is killed, but still you're you're writing to the stream. So, you to avoid that, you should always

have your Tomcat idle timeout more than the Spring So, that when you're you're you are processing asynchronous request, you you are ensured that your socket is open to write content to the client. So, final setup So, we played around with few of the values. So, our final setup looked like this. So, we set time Tomcat socket idle timeout as 3 minutes and Spring async timeout as 2

minutes. So, for us this worked This This was the worst-case scenario value, although it was taking for us almost 1 minute to register it. But, for worst cases, when there is a network issue, when there is a huge load, we set it for this timeout and it worked fine for us. So, you need to play around with these parameters, what works for you. Always remember that Spring

async timeout should be lesser than your Tomcat After that, our performance improved naturally. Faster downloads, fewer retries, predictable behavior, our resource usage improved. And we started trusting our system again, which was working something and break sometimes and breaking sometimes. None of these issues were encountered in development. Every single of them only showed up with the real data, real network, real file sizes. And that's why uh we

have collected some pitfalls that we hit so that you don't hit that. First is loading entire file into memory. You should never do that. Instead, use Streaming without owning its life cycle. When you are using streaming, close your streams. Assuming correct Java streaming was not was enough. Streaming can success at your Java level, but it can fail at your HTTP So, always keep these combinations in mind.

Promising file size while streaming. If you are streaming, don't promise any file size. Never use content length in your headers. Misaligned timeouts. Your time allowance timeouts should be aligned. Spring casing timeout and Tomcat timeout should already always be aligned for asynchronous processes. No visibility long running running downloads. If you are using streaming, sometimes it gets difficult to uh track your download that where system goes wrong. These

are five key takeaways for us. If you remember these five things, you can easily build your large files delivery system using Spring Boot API. Simplicity can be dangerous. Streaming is not optional. HTTP headers matter more than you think. Monitoring is survival. Timeouts must agree. Large file delivery system is not a feature. It's a system. That's it for my today's talk and thank you for staying with me

this afternoon. I'm happy to take questions now or continue with this conversation afterwards. Just one request, please scan this barcode and share your feedback if any, positive, negative, anything. It will help me to improve my presentation for future and my content for future. Thank you so much. Have a lovely day ahead. >> [music]