Skip to content

Instantly share code, notes, and snippets.

@mecampbellsoup
Last active December 1, 2020 22:32
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save mecampbellsoup/f4410002b6ed60ef6ca6beea62ade548 to your computer and use it in GitHub Desktop.
Save mecampbellsoup/f4410002b6ed60ef6ca6beea62ade548 to your computer and use it in GitHub Desktop.
Efficiently condense an unordered list of integers to an HTTP querystring param

Condense a List

When requesting resources from a JSONAPI-compliant API, you can request a filtered subset of the resource by including a request parameter filter, e.g.:

GET /api/v1/jobs?filter[id]=1,2

In this example you are requesting, in effect, the following SQL query:

SELECT * FROM jobs WHERE id IN (1, 2);

This pattern is very useful for preventing unnecessary data transfer over the wire if e.g. your client is polling the API for data.

Gotcha'

You may naively write some JS code that does something like:

const myIds = [1, 2, ..., n]; // Imagine `n` is a really large number like 10,000
const url = `http://api/jobs?filter[id]=${myIds}`;
fetch(url);

You may then be surprised to see a status code 431 from your API!

What is a 431?

The HTTP 431 Request Header Fields Too Large response status code indicates that the server refuses to process the request because the request’s HTTP headers are too long. The request may be resubmitted after reducing the size of the request headers.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/431

As the code states, this request failure is a result of putting too many bytes into the filter request header.

(Note that querystring parameters are sent as request headers; you can see this by inspecting a network request with querystring params.)

image

Workaround

Most JSONAPIs can handle range literals specified by clients. So the following should return identical sets of data:

GET /jobs?filter[id]=1,2,3,4,5
GET /jobs?filter[id]=1..5

Note that .. is typically an inclusive range, while 1...5 would be exclusive and yield values 1,2,3,4.

We need a JS function that can take as input an unordered list of integers, and output a condensed querystring representation e.g.:

console.log(condense([1,2,3,4,5])); // "1..5"
console.log(condense([1,2,3,5,6,7])); // "1..3,5..7"
console.log(condense([1,2,3,5,8,9,10,11,13,15,16,17,40])); // "1..3,5,8..11,13,15..17,40" 

Remember that the input list may or may not be ordered!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment