Skip to content

Instantly share code, notes, and snippets.

@courtneymyers
Last active January 10, 2019 20:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save courtneymyers/f5eeae1da04eaf973c589657d122f321 to your computer and use it in GitHub Desktop.
Save courtneymyers/f5eeae1da04eaf973c589657d122f321 to your computer and use it in GitHub Desktop.
Utility function for spitting an array into chunks.
// Helpful if you're working with a very large array, that you need to send to a web server for processing.
// For example, after chunking the array, you can send a number of request to a web server in in parallel,
// and handle the response after all the chunks have been processed (see example below).
function chunkArray(array, chunkLength) {
const length = array.length;
const chunks = [];
let index = 0;
while (index < length) {
chunks.push(array.slice(index, (index += chunkLength)));
}
return chunks;
}
// --- Example ---
// unitIds is an array containing the numbers 1 through 1M.
// this represents a large dataset you want to process against some web server,
// that can't handle a set of data that large in a single api call
const unitIds = Array.from(Array(1000000), (d, i) => ++i);
// unitIds, in 100 item chunks, to not overload web service call
const unitIdsChunks = chunkArray(unitIds, 100);
// request data with each chunk of unitIds
const $deferreds = [];
unitIdsChunks.forEach(function(chunk) {
const = `https://www.example-web-service.com?data=${chunk.join(',')}`
$deferreds.push($.get(url));
});
$.when($deferreds)
.done(function(requests) {
// chunkedData will hold the results of each web service request
const chunkedData = [];
requests.forEach(function(request) {
request.done(function(data, textStatus, jqXHR) {
chunkedData.push(data);
// after the last request has been resolved
if (chunkedData.length === requests.length) {
console.log(chunkedData); // <----- handle chunked data!
}
});
});
})
.fail(function() {
console.error('Error with web service');
});
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment