Skip to content

Instantly share code, notes, and snippets.

@texodus
Last active August 17, 2020 03:45
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save texodus/483a42e7b877043714e18bea6872b039 to your computer and use it in GitHub Desktop.
Save texodus/483a42e7b877043714e18bea6872b039 to your computer and use it in GitHub Desktop.
regular-table / Two Billion Rows
license: apache-2.0

Two Billion Rows

An example of a regular-table data model which generates data on-the-fly to simulate a 2,000,000,000 row <table>.

HTML

You'll need a <regular-table> for this example, which you can create in HTML:

<regular-table></regular-table>

Virtual Data Model

To simulate a really big data set, this model will take advantage of the regular-table Virtual Data Model to generate data only for the window currently visible on screen. This is how really big:

const NUM_ROWS = 2000000000;
const NUM_COLUMNS = 1000;

The dataListener function for this virtual data set is simple, and returns the static dimensions directly:

function dataListener(x0, y0, x1, y1) {
    return {
        num_rows: NUM_ROWS,
        num_columns: NUM_COLUMNS,
        row_headers: range(y0, y1, group_header.bind(null, "Row")),
        column_headers: range(x0, x1, group_header.bind(null, "Column")),
        data: range(x0, x1, (x) => range(y0, y1, (y) => formatter.format(x + y))),
    };
}

It makes copious use of the range() function, which generates a sequence from [x0 .. x1], mapped by the function argument f().

function range(x0, x1, f) {
    return Array.from(Array(x1 - x0).keys()).map((x) => f(x + x0));
}

Generated row and column headers, as well as header groups for every group of 10, are also done on demand via group_header(), this time using the clamp() function.

function group_header(name, i) {
    const group = clamp(i, 10);
    return [`Group ${group}`, `${name} ${formatter.format(i)}`];
}

clamp() formats a number x to it's nearest y

const clamp = (x, y) => formatter.format(Math.floor(x / y) * y);

regular-table Initialization

With these, all that's left is to register the dataListener and draw the <table>.

function init() {
    const table = document.getElementsByTagName("regular-table")[0];
    table.setDataListener(dataListener);
    table.draw();
}

We'll initialize this table within a <script> tag, so that this ".js" output from literally of this Markdown file will not initialize, allowing dataListener() to be re-used as a data model in other examples.

<script>window.addEventListener("load", () => init())</script>

Styling

We want to distinguish the cells from the headers:

td {
    color: #1078d1;
}

And some special styling to separate the row_headers from the column_headers:

tbody th:last-of-type,
thead tr:nth-child(2) th:nth-child(2),
thead tr:first-child th:first-child {
    border-right: 1px solid #ddd;
}

Appendix (Utilities)

A formatter for Numbers:

const formatter = new Intl.NumberFormat("en-us");

Appendix (Dependencies)

<script src="https://cdn.jsdelivr.net/npm/regular-table@0.0.7/dist/umd/regular-table.js"></script>
<link rel='stylesheet' href="https://cdn.jsdelivr.net/npm/regular-table@0.0.7/dist/css/material.css">
license: apache-2.0
td {
color: #1078d1;
}
tbody th:last-of-type,
thead tr:nth-child(2) th:nth-child(2),
thead tr:first-child th:first-child {
border-right: 1px solid #ddd;
}
<!DOCTYPE html>
<html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, minimum-scale=1, user-scalable=no">
</head>
<body>
<regular-table></regular-table>
<script>window.addEventListener("load", () => init())</script>
<script src="https://cdn.jsdelivr.net/npm/regular-table@0.0.7/dist/umd/regular-table.js"></script>
<link rel='stylesheet' href="https://cdn.jsdelivr.net/npm/regular-table@0.0.7/dist/css/material.css">
<link rel="stylesheet" href="index.css">
<script src="index.js"></script>
</body>
</html>
const NUM_ROWS = 2000000000;
const NUM_COLUMNS = 1000;
function dataListener(x0, y0, x1, y1) {
return {
num_rows: NUM_ROWS,
num_columns: NUM_COLUMNS,
row_headers: range(y0, y1, group_header.bind(null, "Row")),
column_headers: range(x0, x1, group_header.bind(null, "Column")),
data: range(x0, x1, (x) => range(y0, y1, (y) => formatter.format(x + y))),
};
}
function range(x0, x1, f) {
return Array.from(Array(x1 - x0).keys()).map((x) => f(x + x0));
}
function group_header(name, i) {
const group = clamp(i, 10);
return [`Group ${group}`, `${name} ${formatter.format(i)}`];
}
const clamp = (x, y) => formatter.format(Math.floor(x / y) * y);
function init() {
const table = document.getElementsByTagName("regular-table")[0];
table.setDataListener(dataListener);
table.draw();
}
const formatter = new Intl.NumberFormat("en-us");
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment