Skip to content

Instantly share code, notes, and snippets.

@bradennapier
Last active May 27, 2019 21:40
Show Gist options
  • Save bradennapier/3d2b84dcb3fcf2429ee49b5c6e796c91 to your computer and use it in GitHub Desktop.
Save bradennapier/3d2b84dcb3fcf2429ee49b5c6e796c91 to your computer and use it in GitHub Desktop.

redis-rate-limiter

This document specifies a concept for a multi-level rate limiting procedure that can be used to implement rate limiting of many kinds. It should be considered a work in progress!

This document describes the schema that should be utilized.

Limiting Keys

Each user shall be assigned a multi-level set of limiting that must pass for any request to continue.

Configuring Rate Limits

We utilize a generic cell algorithm (GCRA) to implement intelligent backoff and burst handling using the Redis Module redis-cell

This is configured using limits.yaml. We use a recursive data format for building the rate limits and when a request is made against a given key, we call each step in the tree.

Internally this is handled efficiently using a built-in lua script.

schema:
  # user:
  user:
    children:
      # if no others match
      "*":
        # limits against user:${username}
        limits:
          - 15
          - 30
          - 60
        children:
          # limits against user:${username}:trade
          trade:
            limits:
              - 5
              - 10
              - 15

Based on the above configuration, we can walk through an example situation. Lets say we have a user, alex. Whenever alex makes requests, we will run a check based on his action(s) to determine if he/she should be allowed to make the given request:

import { authorizeActionForUser } from "abstracted-redis-library-here";

// convenience function against the above request.  "alex" will be
// substituted for the "*" value
authorizeActionForUser("alex", "trade").then(limited => {
  if (limited) {
    // user is rate limited!
  }
});

In this case, limited would be undefined since it is our first request. However, with each request that is made, the system will parse the tree and add a "token" against each limits key it encounters along the way.

In practice this looks like this:

# limits found at path schema.user.children["*"]
LIMITER user:alex 15 30 60 1
# limits found at path schema.user.children["*"].children.trade
LIMITER user:alex:trade 5 10 15 1

We aggregate the results of all the requests (after they have been made) and return a summary of the results. This summary becomes the final descriptor of the limit against the given parameters. Limits may be nested as deeply as needed, remebering that every limits key encountered along the way will always be checked against and will cause a rejection if crosses the threshold.

It is important to realize that this means that user:alex:trade and user:alex:withdrawal means that both requests go against the user:alex quota whereas they each also maintain their own quotas on top. This is critical to consider when building limits as each root level must have enough of a quota to

In each step of the check, the replies we receive will each be an array:

/* @flow */
/**
 * 0 = User is not being limited
 * 1 = User is being limited
 */
type Limiter$IsLimited = 0 | 1;
/**
 * Burst allowed requests in any interval
 */
type Limiter$Burst = number;
/**
 * Remaining requests allowed before
 * limited unless timer expires
 */
type Limiter$Remaining = number;
/**
 * If the user is limited, how many seconds must they
 * wait until they will be allowed to make the requested
 * value again? This is -1 if the user is not being limited.
 */
type Limiter$RetryAfter = number;
/**
 * How many seconds remain until the limiter resets to
 * its maximum value?
 */
type Limiter$ResetsAfter = number;

type Limiter$Response = [
  Limiter$IsLimited,
  Limiter$Burst,
  Limiter$Remaining,
  Limiter$RetryAfter,
  Limiter$ResetAfter
];

This means that in our existing example, we will end up with a aggregated result looking something like:

type ExampleResponse = [Limiter$Response, Limiter$Response];

In order to provide a valid final result, we then need to parse the response to provide a summary which matches Limiter$Response but takes into account every set of limits along the way.

type FinalSummary = Limiter$Response;

If mapped to the standard rate limiting headers, this would mean the following for each item in the array response:

The meaning of each array item is:

  • Whether the action was limited:
    • 0 indicates the action is allowed.
    • 1 indicates that the action was limited/blocked.
  • The total limit of the key (max_burst + 1). This is equivalent to the common X-RateLimit-Limit HTTP header.
  • The remaining limit of the key. Equivalent to X-RateLimit-Remaining.
  • The number of seconds until the user should retry, and always -1 if the action was allowed. Equivalent to Retry-After.
  • The number of seconds until the limit will reset to its maximum capacity. Equivalent to X-RateLimit-Reset.
@bradennapier
Copy link
Author

bradennapier commented Aug 9, 2018

Update

An example using the setup script to implement a workin test command and benchmark

const { performance } = require("perf_hooks");
const redis = require("./cell-setup");

function checkPath(...path) {
  return redis.limiter(path.length, ...path);
}

async function start() {
  await checkPath("user", "alex", "trade");
}

async function request(...args) {
  const startTime = performance.now();
  await checkPath(...args);
  return performance.now() - startTime;
}

module.exports = {
  start,
  request
};

@bradennapier
Copy link
Author

bradennapier commented Aug 9, 2018

Benchmark

Quick benchmarks against RateLimit.js in redback. Note that results got as high as a 5,000% increase in performance - however, this is likely due to Javascript Engine / Heap & Memory (a valid concern). RateLimit.js could not run when making more than 60,000 requests simultaneously whereas I was able to make nearly 1,000,000 requests simultaneously with the cell implementation before running out of heap memory (default node settings)

Important: Note that these implementations are not apples-to-apples. redis-cell implements GCRA which includes burst handling and backoff whereas the RateLimit.js implementation is pretty much just saying "how many requests has the client made in the last n seconds". This makes the performance difference much more impressive.

All tests run against v10.8.0 on Redis version 4.0.11

With Hardware Specs:

Hardware Overview:

  Model Name:	iMac Pro
  Model Identifier:	iMacPro1,1
  Processor Name:	Intel Xeon W
  Processor Speed:	3.2 GHz
  Number of Processors:	1
  Total Number of Cores:	8
  L2 Cache (per Core):	1 MB
  L3 Cache:	11 MB
  Memory:	32 GB
Calls Setup, Starting Requests
Running Tests for:  cell
[ 'user', '1', 'trade' ]
[ 'user', '2', 'trade' ]
[ 'user', '3', 'trade' ]
[ 'user', '4', 'trade' ]
[ 'user', '5', 'trade' ]
[ 'user', '6', 'trade' ]
[ 'user', '7', 'trade' ]
Running Tests for:  rl
[ '1', 60, 'user:1:trade' ]
[ '2', 60, 'user:2:trade' ]
[ '3', 60, 'user:3:trade' ]
[ '4', 60, 'user:4:trade' ]
[ '5', 60, 'user:5:trade' ]
[ '6', 60, 'user:6:trade' ]
[ '7', 60, 'user:7:trade' ]

    --- Test Results ---

    Total Iterations: 5000 * 7 Tests (35,000 iterations each)

    rl:
      Total Duration: 343693971.74181193
      Average: 9819.82776405177
      Max: 14198.909738004208
      Min: 5541.352702006698

    cell:
      Total Duration: 26785993.573875546
      Average: 765.3141021107299
      Max: 1454.7679300010204
      Min: 322.44201999902725

    Diff: cell is 1183% faster
// RUN FOR CELL ONLY (Dont Require rl or include in scope at all)
Calls Setup, Starting Requests
Running Tests for:  cell
[ 'user', '1', 'trade' ]
[ 'user', '2', 'trade' ]
[ 'user', '3', 'trade' ]
[ 'user', '4', 'trade' ]
[ 'user', '5', 'trade' ]
[ 'user', '6', 'trade' ]
[ 'user', '7', 'trade' ]

    --- Test Results ---

    Total Iterations: 10 * 7 Tests (70 iterations each)

    cell:
      Total Duration: 231.97511593997478
      Average: 3.3139302277139255
      Max: 4.6971050053834915
      Min: 2.4761980026960373
// RUN FOR RL ONLY (Dont Require cell or include in scope at all)
Calls Setup, Starting Requests
Running Tests for:  rl
[ '1', 60, 'user:1:trade' ]
[ '2', 60, 'user:2:trade' ]
[ '3', 60, 'user:3:trade' ]
[ '4', 60, 'user:4:trade' ]
[ '5', 60, 'user:5:trade' ]
[ '6', 60, 'user:6:trade' ]
[ '7', 60, 'user:7:trade' ]

    --- Test Results ---

    Total Iterations: 10 * 7 Tests (70 iterations each)

    rl:
      Total Duration: 2249.6415529847145
      Average: 32.13773647121021
      Max: 36.03293499350548
      Min: 28.242240011692047
Calls Setup, Starting Requests
Running Tests for:  cell
[ 'user', '1', 'trade' ]
[ 'user', '2', 'trade' ]
[ 'user', '3', 'trade' ]
[ 'user', '4', 'trade' ]
[ 'user', '5', 'trade' ]
[ 'user', '6', 'trade' ]
[ 'user', '7', 'trade' ]
Running Tests for:  rl
[ '1', 60, 'user:1:trade' ]
[ '2', 60, 'user:2:trade' ]
[ '3', 60, 'user:3:trade' ]
[ '4', 60, 'user:4:trade' ]
[ '5', 60, 'user:5:trade' ]
[ '6', 60, 'user:6:trade' ]
[ '7', 60, 'user:7:trade' ]

    --- Test Results ---

    Total Iterations: 10 * 7 Tests (70 iterations each)

    RateLimit:
      Total Duration: 2168.367660999298
      Average: 30.976680871418544
      Max: 34.8785649985075
      Min: 26.95397099852562

    Cell:
      Total Duration: 331.32161206007004
      Average: 4.733165886572429
      Max: 5.786581993103027
      Min: 3.023066997528076

    Diff: Cell is 554% faster
Calls Setup, Starting Requests
Running Tests for:  rl
[ '1', 60, 'user:1:trade' ]
[ '2', 60, 'user:2:trade' ]
[ '3', 60, 'user:3:trade' ]
[ '4', 60, 'user:4:trade' ]
[ '5', 60, 'user:5:trade' ]
[ '6', 60, 'user:6:trade' ]
[ '7', 60, 'user:7:trade' ]
Running Tests for:  cell
[ 'user', '1', 'trade' ]
[ 'user', '2', 'trade' ]
[ 'user', '3', 'trade' ]
[ 'user', '4', 'trade' ]
[ 'user', '5', 'trade' ]
[ 'user', '6', 'trade' ]
[ 'user', '7', 'trade' ]

    --- Test Results ---

    Total Iterations: 2 * 7 Tests (14 iterations each)

    RateLimit:
      Total Duration: 106.0299790352583
      Average: 7.573569931089878
      Max: 8.384077996015549
      Min: 6.889536008238792

    Cell:
      Total Duration: 17.149763986468315
      Average: 1.2249831418905939
      Max: 1.5108399987220764
      Min: 0.9881349951028824

    Diff: Cell is 518% faster

@bradennapier
Copy link
Author

bradennapier commented Aug 9, 2018

Update

Without completely thinking it through, the aggregation would likely end up looking something like this

function aggregateResults(results) {
  const response = results[0].slice();

  results.forEach(
    ([isLimited, burst, remaining, retryAfter, resetAfter]) => {
      if (isLimited === 1) response[0] = 1;
      if (response[1] > burst) response[1] = burst;
      if (response[2] > remaining) response[2] = remaining;
      if (response[3] < retryAfter) response[3] = retryAfter;
      if (response[4] < resetAfter) response[4] = resetAfter;
    }
  );

  return response;
}

Note: This could be made significantly more performant if the aggregation was implemented in the lua script as well.

Which returns the following against the 2 example tiers (TODO: Study to see if any flaws in logic currently exist here). First line shows the result of the request (from the lua script) and the second line shows the aggregated response to be given.

Note: The imposed limits in the example are obviously far too low and easily adjusted using the configuration yaml, this is just an example to test the logic of the concept and proof of concept code.

[ [ 0, 16, 14, -1, 3 ], [ 0, 6, 4, -1, 2 ] ]
[ 0, 6, 4, -1, 3 ]
[ [ 0, 16, 13, -1, 5 ], [ 0, 6, 3, -1, 4 ] ]
[ 0, 6, 3, -1, 5 ]
[ [ 0, 16, 12, -1, 7 ], [ 0, 6, 2, -1, 5 ] ]
[ 0, 6, 2, -1, 7 ]
[ [ 0, 16, 11, -1, 9 ], [ 0, 6, 1, -1, 7 ] ]
[ 0, 6, 1, -1, 9 ]
[ [ 0, 16, 10, -1, 11 ], [ 0, 6, 0, -1, 8 ] ]
[ 0, 6, 0, -1, 11 ]
[ [ 0, 16, 9, -1, 13 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 13 ]
[ [ 0, 16, 8, -1, 15 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 15 ]
[ [ 0, 16, 7, -1, 17 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 17 ]
[ [ 0, 16, 6, -1, 19 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 19 ]
[ [ 0, 16, 5, -1, 21 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 21 ]
[ [ 0, 16, 4, -1, 23 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 23 ]
[ [ 0, 16, 3, -1, 25 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 25 ]
[ [ 0, 16, 2, -1, 27 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 27 ]
[ [ 0, 16, 1, -1, 29 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 29 ]
[ [ 0, 16, 0, -1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]
[ [ 1, 16, 0, 1, 31 ], [ 1, 6, 0, 1, 8 ] ]
[ 1, 6, 0, 1, 31 ]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment