Skip to content

Instantly share code, notes, and snippets.

@ktheory
Last active May 11, 2024 10:59
Show Gist options
  • Save ktheory/df3440b01d4b9d3197180d5254d7fb65 to your computer and use it in GitHub Desktop.
Save ktheory/df3440b01d4b9d3197180d5254d7fb65 to your computer and use it in GitHub Desktop.
Easily make HTTPS requests that return promises w/ nodejs

Javascript request yak shave

I wanted to easily make HTTP requests (both GET & POST) with a simple interface that returns promises.

The popular request & request-promises package are good, but I wanted to figure out how to do it w/out using external dependencies.

The key features are:

  • the ability to set a timeout
  • non-200 responses are considered errors that reject the promise.
  • any errors at the TCP socker/DNS level reject the promise.

Hat tip to webivore and the Node HTTP docs, whose learning curve I climbed.

It's great for AWS Lambda.

Examples

const url = require('url');
const req = require('httpPromise');
// Simple get request
req("https://httpbin.org/get")
  .then((res) => { console.log(res)})
  .catch((err) => { console.log("oh no: " + err)});

// Get req w/ timeout and extra headers:
req(Object.assign({}, url.parse("https://httpbin.org/get"), {timeout: 2000, headers: {'X-MyHeader': 'MyValue'}}))

// POST data:
req(Object.assign({}, url.parse("https://httpbin.org/post"), { method: 'POST', headers: {'Content-Type': 'application/x-www-form-urlencoded'}}), "foo=bar")
  .then((res) => { console.log(res)})
  .catch((err) => { console.log("oh no: " + err)});
import https from 'https'
export default (urlOptions, data = '') => new Promise((resolve, reject) => {
// Inspired from https://gist.github.com/ktheory/df3440b01d4b9d3197180d5254d7fb65
const req = https.request(urlOptions, res => {
// I believe chunks can simply be joined into a string
const chunks = []
res.on('data', chunk => chunks.push(chunk))
res.on('error', reject)
res.on('end', () => {
const { statusCode, headers } = res
const validResponse = statusCode >= 200 && statusCode <= 299
const body = chunks.join('')
if (validResponse) resolve({ statusCode, headers, body })
else reject(new Error(`Request failed. status: ${statusCode}, body: ${body}`))
})
})
req.on('error', reject)
req.write(data, 'binary')
req.end()
})
@natachaS
Copy link

natachaS commented Sep 19, 2016

Nice ! I'll test it next time i need to make a request ! 🎉

@zwacky
Copy link

zwacky commented Jun 7, 2018

hey there, thanks for this snippet! let data have a default value of ''. then it won't spit errors when using super simple stuff like requests like req(simpleUrlWithoutData).

module.exports = ((urlOptions, data = '') => {
…

cheers from 2018

@melroyvandenberg
Copy link

Be aware that a string (or object) is not an error. Only new Error() is.

@santanaG
Copy link

santanaG commented May 8, 2022

since its 2022...

import https from 'https'

export default (urlOptions, data = '') => new Promise((resolve, reject) => {
  const req = https.request(urlOptions, res => {
    // I believe chunks can simply be joined into a string
    const chunks = []

    res.on('data', chunk => chunks.push(chunk))
    res.on('error', reject)
    res.on('end', () => {
      const { statusCode, headers } = res
      const validResponse = statusCode >= 200 && statusCode <= 299
      const body = chunks.join('')

      if (validResponse) resolve({ statusCode, headers, body })
      else reject(new Error(`Request failed. status: ${statusCode}, body: ${body}`))
    })
  })

  req.on('error', reject)
  req.write(data, 'binary')
  req.end()
})

@latobibor
Copy link

latobibor commented Jul 26, 2022

@santanaG
No, no, no, please never do this for the sake of your colleagues:

export default (urlOptions, data = '') => {}

Default exporting an anonymous function breaks all IDEs. There is nothing you can type and this is recommended when you code, you won't be able look for usages, you won't be able to find the function. There is no gain, only loss here.

// this plays well with everything
export function req(urlOptions, data = '') {}

@santanaG
Copy link

@latobibor

I do not know which is your IDE of choice but mine is vscode. As of right now, I have done the following:

  1. Found a usage of a default exported anonymous function (as I have used it above), highlighted the text, and pressed F12. Took me right to the declaration.

  2. Proceeded to highlight the default keyword within that declaration then pressed SHIFT + F12 keys. Gave me a list of all function usages throughout the project.

  3. Held my mouse over a randomly chosen usage from the list just mentioned and could see the type data. As a bonus, went back to the declaration, held my mouse over the default keyword, and saw the type data there as well. I even played a bit with the types to see them changing on the usages.

For the sake of random strangers (maybe even future colleagues) on the internet, please consider that when you make statements such as "No, no, no, please never do this for the sake of your colleagues" and "There is no gain, only loss here." you might be coming on as a bit too aggressive and unwelcoming. These are the sorts of interactions that people attempting to join this field mention when they say that the field is unwelcoming to newcomers. It is just plain bad manners to state something of this nature in a less than compassionate way (we are all learning here). I would note this point in particular because your statement: "Default exporting an anonymous function breaks all IDEs." is just flat out wrong, as I have just confirmed with my testing.

Here is another point I've decided to make. The sentence "Default exporting an anonymous function breaks all IDEs." is not just wrong because vscode indeed is not broken but because you are working from the mistaken belief that the language syntax can somehow break a tool that is meant to understand and properly interpret said syntax. If vscode could not provide any of the data or features it usually provides in this particular instance of a function declaration, that is not the language's fault but a bug or a lack of the tool.

My final point is more controversial: Types are not the be-all and end-all (and that is a good thing). Code style decisions should not be made to accommodate typing but for clarity. You should consider that there might be enough people that feel this way that your above statements may seem elitist and misguided.

@ktheory
Copy link
Author

ktheory commented Jul 30, 2022

@santanaG Thank you! I updated the gist w/ the code in your comment.

@hertzg
Copy link

hertzg commented Aug 1, 2022

for people who just want to deal with buffers and nothing more (also with listener cleanup)

import https from "https";

export const request = (urlOptions, body) =>
  new Promise((resolve, reject) => {
    const handleRequestResponse = (res) => {
      removeRequestListeners();

      const chunks = [];
      const handleResponseData = (chunk) => {
        chunks.push(chunk);
      };

      const handleResponseError = (err) => {
        removeResponseListeners();
        reject(err);
      };

      const handleResponseEnd = () => {
        removeResponseListeners();
        resolve({ req, res, body: Buffer.concat(chunks) });
      };

      const removeResponseListeners = () => {
        res.removeListener("data", handleResponseData);
        res.removeListener("error", handleResponseError);
        res.removeListener("end", handleResponseEnd);
      };

      res.on("data", handleResponseData);
      res.once("error", handleResponseError);
      res.once("end", handleResponseEnd);
    };

    const handleRequestError = (err) => {
      removeRequestListeners();
      reject(err);
    };

    const removeRequestListeners = () => {
      req.removeListener("response", handleRequestResponse);
      req.removeListener("error", handleRequestError);
    };

    const req = https.request(urlOptions);
    req.once("response", handleRequestResponse);
    req.once("error", handleRequestError);

    if (body) {
      req.end(body);
    } else {
      req.end();
    }
  });

@santanaG
Copy link

@santanaG Thank you! I updated the gist w/ the code in your comment.

That was unexpected but I am honored! Thanks!

@Et7f3
Copy link

Et7f3 commented Jan 20, 2024

@hertzg Why do you removeRequestListeners if you only add once ?

Using once might not be needed because:

The optional callback parameter will be added as a one-time listener for the 'response' event.

From documentation 'abort' event can be emitted

'abort' ('aborted' on the res object)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment