Skip to content

Instantly share code, notes, and snippets.

@kainino0x
Last active August 29, 2015 14:25
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Save kainino0x/c082067f61ac9356ac04 to your computer and use it in GitHub Desktop.
mesh_compression_open3dgc proposal & mock-up

Overview

This extension inserts an optional layer in between bufferViews and buffers. This layer, called decompressedView, performs Open3DGC decompression on part of a buffer before it is accessed by the bufferView.

Example

In this example (a ball), we have:

  • 960 indices (2 bytes/index): 1920 bytes
  • 162 vertices (12 bytes/position, 12 bytes/normal): 3888 bytes
  • 5808 bytes total uncompressed data
  • 1194 bytes total compressed data (as compressed by collada2gltf -c Open3DGC -m binary, default compression parameters)

Uncompressed dae and various binary formats here.

In an engine, this can be loaded bottom-up (creating buffers first, and so on): buffer [<- decompressedView] <- bufferView <- accessor <- mesh (This is, roughly, the approach Cesium uses.)

or top-down (lazily loading objects as necessary): mesh -> accessor -> bufferView -> [decompressedView ->] buffer (This seems to be the approach MontageJS uses.)

As proposed, all bufferViews, regardless of whether they come from compressed data, will represent data that can be passed directly to gl.bufferData; decompression happens before this. No information needs to travel up or down the chain for decompression to be performed.

For discussions on alternative ideas, see Open alternative ideas.

For pseudocode for functionality added by this proposal, see Pseudocode.

Example glTF File Mock-up

{
  "allExtensions": [
    "mesh_compression_open3dgc"
  ],

Since an extra layer is inserted between bufferView and buffer, that layer is stored here, in "extensions".

  "extensions": {
    "mesh_compression_open3dgc": {
      "decompressedViews": {
        "decompressedView_ball": {

A decompressedView behaves much like a buffer or bufferView, but must be processed before it is used. It points to some data in a buffer, which would be decompressed, then referenced by a bufferView. In practice, decompressedViews will map one-to-one with glTF meshes. However, they are not extensions to the mesh object, because it breaks the "bottom-up" method described above and breaks separation of concern in the "top-down" method.

          "buffer": "buffer_ball_compressed",  // buffer containing compressed data

          "byteOffset": 0,     // buffer offset pointing to compressed data
          "byteLength": 1194,  // compressed size
          // no type; compressed data is always read as a straight byte array.

          // TODO: useful but not necessary; will probably keep
          "decompressedByteLength": 5808,

          // Since Open3DGC distinguishes between floating-point and integral types
          // but not between different sizes of those types, this defines the types
          // of the arrays to decompress into.
          "floatAttributeSizes": [ 5126, 5126 ],
          "intAttributeSizes": [],

          // Compression metadata
          "mode": "binary"
        }
      }
    }
  },

accessors behave just as they normally do without the extension. There can be multiple accessors for mesh indices into the same bufferView into a decompressedView - each one is one primitive.

  "accessors": {
    "accessor_ball_idx": {
      "bufferView": "bufferView_ball_element",
      "byteOffset": 0,
      "byteStride": 0,

      "count": 960,
      "componentType": 5123,  // uint16
      "type": "SCALAR"
    },

    "accessor_ball_pos": {
      "bufferView": "bufferView_ball_vertexdata",
      "byteOffset": 0,
      "byteStride": 0,

      "count": 162,
      "componentType": 5126,  // float32
      "type": "VEC3",

      "max": [ 1, 1, 1 ],
      "min": [ -1, -1, -1 ]
    },

    "accessor_ball_nor": {
      "bufferView": "bufferView_ball_vertexdata",
      "byteOffset": 1944,
      "byteStride": 0,

      "count": 162,
      "componentType": 5126,  // float32
      "type": "VEC3",

      "max": [ 1, 1, 1 ],
      "min": [ -1, -1, -1 ]
    }
  },

bufferViews for compressed data point into a decompressedView instead of a normal buffer. If there are any uncompressed meshes, they work exactly as usual.

Since the "buffer" property isn't used, it could be omitted when using the extension.

  "bufferViews": {
    "bufferView_ball_element": {
      // No buffer
      "byteOffset": 0,     // Offset into decompressedView
      "byteLength": 1920,  // Length in decompressedView

      "target": 34963,  // ELEMENT_ARRAY_BUFFER

      "extensions": {
        "mesh_compression_open3dgc": {
          // The buffer property can go here, instead of outside.
          "decompressedView": "decompressedView_ball"
        }
      }
    },

    "bufferView_ball_vertexdata": {
      // No buffer
      "byteOffset": 1920,  // Offset into decompressedView
      "byteLength": 3888,  // Length in decompressedView

      "target": 34962,  // ARRAY_BUFFER

      "extensions": {
        "mesh_compression_open3dgc": {
          // The buffer property can go here, instead of outside.
          "decompressedView": "decompressedView_ball"
        }
      }
    }
  },

Buffers behave as usual.

  "buffers": {
    "buffer_ball_compressed": {
      "byteLength": 1194,  // compressed size
      "type": "arraybuffer",
      "uri": "ball.o3dgc.bin"
    }
  },

Meshes behave as usual.

  "meshes": {
    "geom-ball": {
      "name": "ball",
      // TODO: in core, consider moving "attributes" from below to here
      "primitives": [  // TODO: in core, consider renaming "primitives"
        {
          "material": "whatever",
          "mode": 4,  // TRIANGLES. (See #341.) Implementation currently uses "primitive" as in glTF 0.8.
          "indices":    "accessor_ball_idx",
          "attributes": {
            "POSITION": "accessor_ball_pos",
            "NORMAL":   "accessor_ball_nor"
          }
        }
      ]
    }
  }
}

Internal formats

In the Open3DGC library, decompressing a single model results in individual homogeneous arrays for indices, positions, normals, and other attributes (which maps to a glTF mesh primitive). In order to define an output format which is efficient to upload and access from attributes, a format must be predefined for accessors to point into.

When decompressed, the resulting blob will look like this:

[--------model (primitive) 1---------]
[----indices----][positions][attrs...]

Those individual arrays will be be arranged into bufferViews as follows:

[----------model (primitive) 1------------]
[------indices-------][positions][attrs...]
 ↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓  ↓↓↓↓↓↓↓↓↓  ↓↓↓↓↓↓↓↓
[element array buffer][---array  buffer---]

This allows an engine to simply construct a buffer from each of the decompressed arrays, then call gl.bufferData exactly once for each bufferView.

It is also possible to upload each part of an ARRAY_BUFFER bufferView using gl.bufferSubData, avoiding the CPU/memory overhead of constructing the complete array for upload using gl.bufferData.

This decompressed format will be defined by the extension specification. In particular, this defines the order in which arrays are stored into the GL buffer, and it defines the alignment of each of these arrays. For WebGL in particular, due to typed array semantics, this is an important consideration. See the pseudocode for constructing the decompressed blob.

  • Another, very different option is to connect model attributes directly with the homogeneous arrays that are returned by the decompression. Naively, this can be implemented with a single GL buffer for each attribute (positions, normals, etc.)

    • Efficient implementation would require in-engine amalgamation of various arrays into one GL buffer. This can be done with gl.bufferSubData if sufficient metadata is provided, but prevents glTF from mapping closely to the GL API calls.
  • To allow for full expressiveness of layout (allowing multiple primitives to be stored in one bufferView, or attributes to be interleaved), accessors into bufferViews could be used as if they are writable memory by the decompressor. Thus, some memory would be allocated by a bufferView, then the decompression step would write out index and attribute arrays into the accessors, allowing the format to be completely defined by information already stored in core glTF.

    • This may have significant conflicts with how a loader would be implemented without the extension (especially with regard to uploading to GL).
    • There are also potential undefined behavior issues caused by this idea: since the loader must write into accessors, problems will arise if two accessors point into the same part of a bufferView - depending on which model or attribute is loaded first, the results might be completely different. The extension (or the glTF spec) would have to specify that either (a) accessors cannot overlap or (b) decompressing must always write the same data to the same place in memory.
  • I have considered structural compatibility with hypothetical hardware/API extensions for mesh compression. However, since many possible formats or structures could be used (including those described above plus considerations for interleaving), I deemed it not worth accounting for.

glTF loading, with decompression branch

function loadGLTF() {
  for (b in buffers) {
    b.data = b.load();
  }
  for (d in decompressedViews) {
    var decomp = Open3DGC.decompress(d.buffer.data[d.offset .. d.offset + d.length]);
    d.data = produceDecompressedBlob(decomp.models);  // (defined below)
  }
  for (bv in bufferViews) {
    var data;
    if (bv has extension open3dgc) {
      data = bv.open3dgc.decompressedView.data[bv.offset .. bv.offset + bv.length];
    } else {
      data = bv.buffer.data[bv.offset .. bv.offset + bv.length];
    }
    gl.bufferData(bv.target, data, gl.STATIC_DRAW);
  }
  for (m in meshes) {
    for (p in m.primitives) {
      create VAO;
      bind element array buffer from p.accessor;
      for (a in p.attributes) {
        bind array buffer from a;
      }
    }
  }

  // ...
}

This enforces an alignment scheme which is supported by Typed Arrays (e.g. Float32Array), whose offsets must be aligned with the type in the array.

function produceDecompressedBlob(decompressedModelObjects) {
  var blob = [];
  for (m in decompressedModelObjects) {
    blob.padToMultipleOf(m.indices.BYTES_PER_ELEMENT);
    blob.appendBytes(m.indices as bytes);
    for (a in m.attributes) {  // including positions and normals
      blob.padToMultipleOf(a.BYTES_PER_ELEMENT);
      blob.appendBytes(a as bytes);
    }
  }
}

Changelog

  • decompressedBuffer now can only contain one primitive, because multiple isn't possible with the schema I had. This is actually the same as how Fabrice had done the implementation. It may change again, but the schema will also have to change.
  • Removed [indices1][indices2][vertices1][vertices2] layout idea as that's no longer valid.
  • Removed unnecessary decompressedBuffer fields.
  • Rename decompressedBuffer to decompressedView.
  • Remove bufferView.buffer for bufferViews using compression.
  • decompressedBuffer can now hawe multiple primitives, but they must all use the same vertex data.

Please comment in KhronosGroup/glTF#398. Notifications don't work on Gists.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment