Skip to content

Instantly share code, notes, and snippets.

Embed
What would you like to do?
Async/await function to fetch data from multiple URLs in parallel
/* Client side, works in Chrome 55 and Firefox 52 without transpilation */
//https://blogs.msdn.microsoft.com/typescript/2016/11/08/typescript-2-1-rc-better-inference-async-functions-and-more/
async function fetchURLs() {
try {
// Promise.all() lets us coalesce multiple promises into a single super-promise
var data = await Promise.all([
/* Alternatively store each in an array */
// var [x, y, z] = await Promise.all([
// parse results as json; fetch data response has several reader methods available:
//.arrayBuffer()
//.blob()
//.formData()
//.json()
//.text()
fetch('https://jsonplaceholder.typicode.com/posts').then((response) => response.json()),// parse each response as json
fetch('https://jsonplaceholder.typicode.com/albums').then((response) => response.json()),
fetch('https://jsonplaceholder.typicode.com/users').then((response) => response.json())
]);
for (var i of data) {
console.log(`RESPONSE ITEM \n`);
for (var obj of i) {
console.log(obj);
//logger utility method, logs output to screen
console.log(obj);
}
}
} catch (error) {
console.log(error);
}
}
/* NodeJS version */
//uses the `request` package which makes working with Node's native http methods easier
const request = require('request');
var requestAsync = function(url) {
return new Promise((resolve, reject) => {
var req = request(url, (err, response, body) => {
if (err) return reject(err, response, body);
resolve(JSON.parse(body));
});
});
};
const urls = [
'https://jsonplaceholder.typicode.com/posts',
'https://jsonplaceholder.typicode.com/albums',
'https://jsonplaceholder.typicode.com/users'
];
/* Works as of Node 7.6 */
var getParallel = async function() {
//transform requests into Promises, await all
try {
var data = await Promise.all(urls.map(requestAsync));
} catch (err) {
console.error(err);
}
console.log(data);
}
getParallel();
@mrwillis

This comment has been minimized.

Copy link

commented Sep 16, 2017

Great this is awesome. Thank-you.

@GulfamAnsari

This comment has been minimized.

Copy link

commented Feb 26, 2018

Lovely bro. Thank you great help

@evanmcd

This comment has been minimized.

Copy link

commented Mar 1, 2018

This helped me out a ton. Thanks.

@danilogila

This comment has been minimized.

Copy link

commented Mar 18, 2018

Awesome

@Bosco2k6

This comment has been minimized.

Copy link

commented Apr 4, 2018

I wasted 1+ day trying to figure out how to do this on my own. Very awesome utility/example - thanks so much!

@bschwartz757

This comment has been minimized.

Copy link
Owner Author

commented Oct 15, 2018

Glad it helped!

@jorgea1986

This comment has been minimized.

Copy link

commented Nov 22, 2018

Thanks!! works great, question: how can i add a custom header to one of the urls? im getting this error from one of the requests (coinbase pro)

"message: User-Agent header is required."

sorry if this is a dumb question im new to node.js

thanks again

@matthewcummings

This comment has been minimized.

Copy link

commented Jan 10, 2019

Amen for this gist! I thought it would be easy to find a parallelized http download example in node. . . boy was I wrong. This is the most useful link I've come across, thank you!

@subba9301

This comment has been minimized.

Copy link

commented May 19, 2019

Very useful, thank you
How do i continue even if any of the url failed.. the above code running fine if all url's are accessible.. I still want to continue display even if any one of the url stuck or failed with other promised urls. Please help..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.