Skip to content

Instantly share code, notes, and snippets.

@JAndritsch
Last active February 3, 2023 07:46
Show Gist options
  • Save JAndritsch/3920385 to your computer and use it in GitHub Desktop.
Save JAndritsch/3920385 to your computer and use it in GitHub Desktop.
Nginx settings and jQuery File Uploader chunk sizes
jQuery File Uploader (blueimp) and nginx benchmark notes.
Hypothesis:
Assigning the chunk size to be greater than the client_body_buffer_size will result in slower uploads and corrupt files.
Table 1: Stability
------------------
The table below shows the number of successful file uploads after a random number of pause and resume operations.
Note that these tests include nginx tweaking of the "client_body_buffer_size".
File size: 43 MB to 730 MB videos
Chunk size nginx client_body_buffer_size files successfully uploaded
-----------------------------------------------------------------------------------------------
1024 * 128 (128 KB) 128 KB 7 / 7
1024 * 128 (128 KB) 256 KB 7 / 7
1024 * 256 (256 KB) 256 KB 7 / 7
1024 * 128 (128 KB) 512 KB 7 / 7
1024 * 256 (256 KB) 512 KB 7 / 7
1024 * 512 (512 KB) 512 KB 7 / 7
1024 * 128 (128 KB) 1024 KB 7 / 7
1024 * 256 (256 KB) 1024 KB 7 / 7
1024 * 512 (512 KB) 1024 KB *
1024 * 1024 (1 MB) 1024 KB *
* = skipped
Note: In most cases where the chunk size was higher than the client_body_buffer_size, several of
the uploads ended up corrupt. We concluded that the tests were not valuable in cases where the chunk
size was greater than the buffer size.
Table 2: Performance
--------------------
The table below shows the performance of uninterrupted chunk uploading at various chunk sizes
against different nginx client_body_buffer_size.
Number of simultaneous uploads: 5
File size: ~730 MB
Chunk size nginx client_body_buffer_size time to complete
------------------------------------------------------------------------------------------------
1024 * 64 (64 KB) 128 KB 6m 28s
1024 * 128 (128 KB) 128 KB 3m 48s
1024 * 128 (128 KB) 256 KB 3m 58s
1024 * 256 (256 KB) 256 KB 3m 10s, 3m 15s
1024 * 128 (128 KB) 512 KB 4m 52s
1024 * 256 (256 KB) 512 KB 3m 34s
1024 * 512 (512 KB) 512 KB 31m 20s
1024 * 128 (128 KB) 1024 KB 4m 12s
1024 * 256 (256 KB) 1024 KB 3m 20s
1024 * 512 (512 KB) 1024 KB *
1024 * 1024 (1 MB) 1024 KB *
* = skipped
Note: Specifying a chunk size of 512 KB or higher resulted in a drastically slower upload.
Conclusion:
Hypothesis was correct. Based on the data collected, it seems our most optimal configuration is
to use a maxChunkSize of 256 KB and a client_body_buffer_size of 256 KB because it performed the
best out of all the other configurations.
It is possible that the other configurations did not work as well due to how the test environment
was configured. These settings did work a bit differently in a much more realistic setting, but that
could be related to the difference between hardware. The important takeaway from this documentation
is to realize that your chunk size must be less than or equal to your client_body_buffer_size.
@kerstner
Copy link

+1

@nvolungis
Copy link

👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment