Skip to content

Instantly share code, notes, and snippets.

@javagl

javagl/glTF-loading.md Secret

Created January 27, 2019 16:54
Show Gist options
  • Save javagl/bfde5cfab4240843120ed6eb38f4af87 to your computer and use it in GitHub Desktop.
Save javagl/bfde5cfab4240843120ed6eb38f4af87 to your computer and use it in GitHub Desktop.
Download ZIP
A summary of approaches for loading glTF
Raw
glTF-loading.md

(Mainly summarized from KhronosGroup/glTF#755 )

Loading glTF assets

The glTF format is primarily intended as a delivery format. As such, it encapsulates single assets that usually can be loaded atomically. There are dedicated solutions for really large assets that would take a long time to download or could not even fit into the memory at once. For example, the 3D Tiles standard from Analytical Graphics Inc. uses glTF assets as the payload for the tiles, and manages the loading and assembly of large scenes that consist of these individual glTF assets.

But although glTF is not supposed to be a streaming format, there are different possible strategies for loading the asset and its associated resources. A rough categorization of these approaches could be:

  • Block until the whole glTF asset (including external data) is loaded, and then send it to the renderer.
  • Asynchronously load the whole glTF asset, and then send it to the renderer.
  • Asynchronously load the external data (buffers and images) and update the rendered scene incrementally.

Some details of these approaches will largely depend on the environment. For example, in a desktop application, the term "synchronous" may indeed imply that the UI is blocked and a progress dialog is shown, whereas in a web-based application, "synchronous" rather means that the full asset with all associated resources is loaded atomically and resolves a single promise. This may be the case even though the tasks to load associated resources (like buffers and images) are internally still handled asynchronously via the responses to multiple HTTP requests.

Application-specific loading strategies

One could imagine different strategies for determining the order in which the resources of a glTF asset should be loaded. These strategies might be

  • Sorting the meshes (and thus, the tasks to load their resources) based on their distance to the viewer
  • Do occlusion tests to see which meshes have to be loaded and rendered at all
  • Prioritize the meshes in a queue based on the approximate screen space occupation (computed from the bounding box of the accessor.min/max and the view configuration),

These approaches could be referred to as forms of "Visibility-guided rendering" (VGR). Regardless of the exact approach, they all have in common that they require a mechanism for asynchronous loading dedicated meshes and their associated resources. So the following will not focus on one particular strategy, but rather on the loading and rendering infrastructure in general.

Implications for implementations

When a glTF asset is loaded from a given URI into a viewer application, it may be desirable to load certain resources lazily or asynchronously.

  1. When a mesh is loaded, show it with a default material. Later, when the material textures are loaded, update the rendered mesh to have the actual material.
  2. When a mesh and its associated material textures are loaded, show the final rendered mesh.

Which of these approaches for asynchronous loading are applicable depends on the desired user experience. For example, whether it is acceptable for the user to see a mesh with a default texture. In such a case, the geometry of an asset may already be shown with a default material, while the actual PBR material textures are still downloaded in the background. Similarly, if the asset contains animations, one could download the geometry and textures and show the initial, rendered model, and load animation data in the background.

Synchronous loading

For the case of synchronously loading the whole glTF asset and its associated resources, the infrastructure for the renderer may be simple. In pseudocode, it may be sufficient to connect the loader and renderer as follows:

loadWithAllResources(gltfUri).andWhenReady(renderIt());

The glTF and its resources are loaded, and once all the resources are loaded, the whole asset is passed to the renderer, which does the initialization of the rendering data structures, uploads the data to the GPU, and renders the result.

Asynchronous loading

In a sophisticated viewer application, people might expect that it will render as much as it can, at any point in time. This means that the meshes of one glTF asset should appear one by one, as their required data is loaded, and maybe even be displayed with a default texture until their texture is loaded completely.

In order to achieve this, fine-grained asynchronous loading may be necessary. Here, "fine grained" refers to the fact that each buffer and image may be loaded asynchronously and individually. (For embedded- or binary glTF, this might not be possible - at least not beyond decoding and uploading the data asynchronously).

This means that the renderer has to collect information from various sources. For example, when rendering a mesh, there may be two elements that may involve a lookup into the lazy-loading-infrastructure:

mesh->meshPrimitive->material->texture->image->uri
mesh->meshPrimitive->accessors[x]->bufferView->buffer->uri

Each of them may cause the actual rendering to be deferred to the next pass if the required data still has to be loaded. Additionally, for an OpenGL or WebGL based renderer, the initialization of GL data structures has to happen on the rendering thread (or in a "rendering call"). The initialization of a GL texture has to assume that the image data is already loaded - otherwise, it will have to be deferred to a later rendering pass.

Scheduling of loading tasks

There are different conceivable approaches for the scheduling of the tasks that load the external resources. This scheduling might happen immediately after the asset was loaded, or when the resource is required for the first time. The following pseudocode illustrates the difference:

  1. Create tasks for loading and initializing all resources at the beginning, and render only the elements whose resources are already been loaded:

    function init(gltf) {
    // Create tasks to load and initialize ALL textures here  
    scheduleTasksForLoadingTextures(gltf.textures); 
}

function render(gltf) {
    ...
    // For each mesh, render it only when its associated resources
    // have already been loaded 
    if (texturesHaveBeenLoadedFor(someMesh)) render(someMesh);
}

  2. Try to render each mesh, and schedule the tasks to load the associated resources only when they are required:

    function render(gltf) {
    ...
    // For each mesh, render it only when its associated resources
    // have already been loaded 
    if (texturesHaveBeenLoadedFor(someMesh)) render(someMesh);
    
    // Otherwise, schedule the tasks to load the textures of 
    // the mesh primitive that is currently rendered
    else scheduleTasksForLoadingTexturesOf(someMesh); 
}

Further reading

The following is a small collection of links to resources, issues and discussions that are related to progressively loading and streaming glTF assets:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment