Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Async/await function to fetch data from multiple URLs in parallel
/* Client side, works in Chrome 55 and Firefox 52 without transpilation */
//https://blogs.msdn.microsoft.com/typescript/2016/11/08/typescript-2-1-rc-better-inference-async-functions-and-more/
async function fetchURLs() {
try {
// Promise.all() lets us coalesce multiple promises into a single super-promise
var data = await Promise.all([
/* Alternatively store each in an array */
// var [x, y, z] = await Promise.all([
// parse results as json; fetch data response has several reader methods available:
//.arrayBuffer()
//.blob()
//.formData()
//.json()
//.text()
fetch('https://jsonplaceholder.typicode.com/posts').then((response) => response.json()),// parse each response as json
fetch('https://jsonplaceholder.typicode.com/albums').then((response) => response.json()),
fetch('https://jsonplaceholder.typicode.com/users').then((response) => response.json())
]);
for (var i of data) {
console.log(`RESPONSE ITEM \n`);
for (var obj of i) {
console.log(obj);
//logger utility method, logs output to screen
console.log(obj);
}
}
} catch (error) {
console.log(error);
}
}
/* NodeJS version */
//uses the `request` package which makes working with Node's native http methods easier
const request = require('request');
var requestAsync = function(url) {
return new Promise((resolve, reject) => {
var req = request(url, (err, response, body) => {
if (err) return reject(err, response, body);
resolve(JSON.parse(body));
});
});
};
const urls = [
'https://jsonplaceholder.typicode.com/posts',
'https://jsonplaceholder.typicode.com/albums',
'https://jsonplaceholder.typicode.com/users'
];
/* Works as of Node 7.6 */
var getParallel = async function() {
//transform requests into Promises, await all
try {
var data = await Promise.all(urls.map(requestAsync));
} catch (err) {
console.error(err);
}
console.log(data);
}
getParallel();
@mrwillis

This comment has been minimized.

Copy link

@mrwillis mrwillis commented Sep 16, 2017

Great this is awesome. Thank-you.

@GulfamAnsari

This comment has been minimized.

Copy link

@GulfamAnsari GulfamAnsari commented Feb 26, 2018

Lovely bro. Thank you great help

@evanmcd

This comment has been minimized.

Copy link

@evanmcd evanmcd commented Mar 1, 2018

This helped me out a ton. Thanks.

@danilogila

This comment has been minimized.

Copy link

@danilogila danilogila commented Mar 18, 2018

Awesome

@Bosco2k6

This comment has been minimized.

Copy link

@Bosco2k6 Bosco2k6 commented Apr 4, 2018

I wasted 1+ day trying to figure out how to do this on my own. Very awesome utility/example - thanks so much!

@bschwartz757

This comment has been minimized.

Copy link
Owner Author

@bschwartz757 bschwartz757 commented Oct 15, 2018

Glad it helped!

@jorgea1986

This comment has been minimized.

Copy link

@jorgea1986 jorgea1986 commented Nov 22, 2018

Thanks!! works great, question: how can i add a custom header to one of the urls? im getting this error from one of the requests (coinbase pro)

"message: User-Agent header is required."

sorry if this is a dumb question im new to node.js

thanks again

@matthewcummings

This comment has been minimized.

Copy link

@matthewcummings matthewcummings commented Jan 10, 2019

Amen for this gist! I thought it would be easy to find a parallelized http download example in node. . . boy was I wrong. This is the most useful link I've come across, thank you!

@subba9301

This comment has been minimized.

Copy link

@subba9301 subba9301 commented May 19, 2019

Very useful, thank you
How do i continue even if any of the url failed.. the above code running fine if all url's are accessible.. I still want to continue display even if any one of the url stuck or failed with other promised urls. Please help..

@openmindgap

This comment has been minimized.

Copy link

@openmindgap openmindgap commented Jul 4, 2019

This is great, really helped me get a grip with "super" promises - thanks a lot!
Combining with the information here https://developers.google.com/web/fundamentals/primers/promises (another good resource) I've modified code above to do:
.fetch(...).then(status).then(json).then(myFunction).catch(error) (client-side code)

status, json, myFunction and error are all methods. The error() is so I can catch the error from a specific fetch() rather than wrapping the whole lot in a try-catch.

I'm very new to promises (and fetch for that matter) so may be doing it wrong but seems to work well

@imfunniee

This comment has been minimized.

Copy link

@imfunniee imfunniee commented Nov 29, 2019

oh shit, thank you dude

@quoctrung163

This comment has been minimized.

Copy link

@quoctrung163 quoctrung163 commented May 24, 2020

thank you

@Not-your-average-coder

This comment has been minimized.

Copy link

@Not-your-average-coder Not-your-average-coder commented Jul 18, 2020

Thanks a lot man!
You made my day :)

@bschwartz757

This comment has been minimized.

Copy link
Owner Author

@bschwartz757 bschwartz757 commented Jul 18, 2020

Glad it helps!! 👏🏼

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.