Magnum and ECS - Part I

README.md

Magnum ECS Example

How to data-orient data and processing with Magnum.

Table of contents

• Goal
• Build
• Overview
• Problem
• Solution
• Discussion

Goal

To ease comprehension and management of data within a render loop for a content authoring application, akin to Blender, Maya and Houdini.

Specifically, this data:

• Simple geometry, e.g. box, sphere, capsule
• Complex geometry, e.g. convex and concave meshes
• Curves, e.g. straight and curved lines
• Lights and shadows, for presentation (1-3 lights sufficient)
• Picking, for interactive manipulators
• Integration of physics simulation, via PhysX
• Playback of baked simulation
• Threaded simulation, rendering and GUI
• Data exchange with off-the-shelf DCC software via Pixar's USD
• Visualisation of e.g. trajectories, velocities, contacts, limits, constraints, tension, stress etc.
• Interoperability with a Qt-based GUI; e.g. visualise data in tables and tree-views, respond to clicks etc.

Implementation

Separate data from function.

Where data used to be coupled with function, and calls to OpenGL was made from within a deep call stack. Instead, I'd like for both data and function to lie flat (or as flat as possible), with "systems" gaining overall access to multiple instances and components at a time.

Disclaimer

This idea is not yet fully formed, and my experience in GPU and games programming is 2 months old (!). However, I have over a decade of experience working on the other side of the fence, as a user and developer in Maya, Houdini and other DCCs

Build

I've built on-top of the PrimitivesExample, and involved EnTT into the mix; but the particular framework used isn't relevant or limited to this particular implementation or idea.

• Build the PrimitivesExample
• Copy entt.hpp into externals\entt.hpp
• Copy ECSExample.cpp into PrimivitesExample.cpp

Overview

There are 5 types of components.

struct Position;
struct Orientation
struct Scale;
struct Identity;
struct Drawable;

And 5 systems; both being a quantity of 5 is coincidence.

static void MouseMoveSystem;
static void MouseReleaseSystem;
static void AnimationSystem;
static void PhysicsSystem;
static void RenderSystem;

Whereby "systems" (function) iterate over "components" (data).

Problem

Most components work OK.

• MouseMoveSystem correctly interacts with orientation components, except it should really affect only the (non-existent) camera entity
• MouseReleaseSystem correctly interacts with the color components
• Animation and Physics are mock systems that do nothing
• RenderSystem is where things start to break down, namely the Drawable component.

struct Drawable {
    GL::Mesh        mesh;
    Shaders::Phong  shader;
    Color4          color;
};

...

drawable.mesh(drawable.shader);

Here, color is plain-old-data, but both Mesh and Phong are complex classes that combine data and functionality. In this case, the Mesh::draw method handles rendering from inside-out. That is, each instance of a Mesh calls out to OpenGL independently. Furthermore, Mesh::draw takes a shader as argument, when shaders are the programs responsible for transforming vertices and related data into pixels. If either takes the other as argument, it should really be the shader taking the Mesh as argument (?).

Solution

Here's how I'd like to break things down, based on my current (minimal) understanding of data-oriented design. Note that this isn't meant to compile, but to convey the idea.

auto box = registry.create();
auto sphere = registry.create();
auto phong = registry.create();

// Box
registry.assign<PendingUpload>(sphere);
registry.assign<Position>(box);
registry.assign<VertexBuffer>({3, 2}, 3, { // assume float
     0.0f, 0.5f, 0.0f, 1.0f, 0.0f, // pos, uv
    -0.5f, 0.0f, 0.0f, 0.0f, 0.0f,
     0.0f, 0.0f, 0.5f, 0.0f, 1.0f
}, {0, 1, 2});

// Sphere
registry.assign<PendingUpload>(sphere);
registry.assign<Position>(sphere);
registry.assign<VertexBuffer>(sphere, ...);

// Phong
registry.assign<PendingUpload>(phong);
registry.assign<ShaderProgram>(phong, {
    { ShaderProgram::Type::Vertex, "shader.vert" },
    { ShaderProgram::Type::Fragment, "shader.frag" }
});

// Connect vertex buffers to shader program
// Called on changes to assignment, e.g. a new torus is assigned this shader
registry.assign_or_replace<ShaderAssignment>(phong, {box, sphere});

Where components would look something like..

struct PendingUpload {};

struct VertexBuffer {
    std::vector<unsigned int> layout;
    unsigned int count;
    std::vector<float> vertices;
    std::vector<unsigned int> indexes;
};

struct ShaderProgram {
    enum Type { Vertex, Fragment };
    std::map<Type, std::string> stages;
};

// Connection between drawable entities and a shader entity
struct ShaderAssignment {
    std::vector<entt::registry::entity_type> entities;
};

// OpenGL handles to uploaded resources
struct UploadedVertexBuffer {
    unsigned int id;
};

struct UploadedShaderProgram {
    unsigned int id;
};

And their systems something along the lines of..

/**
@brief Upload new data to the GPU

Whenever a new item spawns, it'll carry data pending an upload
to the GPU.

*/
static void SpawnSystem(ent::registry registry) {
    registry.view<PendingUpload, ShaderProgram, Assignment>().each([](auto entity,
                                                                      auto& pending,
                                                                      auto& program) {
        // Create vertex shader
        // Create fragment shader
        // ...
        // Link
        // Release

        // Signal that this program is ready to go
        registry.remove<PendingUpload>(entity);
        registry.assign<UploadedShaderProgram>(entity, programId);
    });

    registry.view<PendingUpload, VertexBuffer>().each([](auto entity,
                                                         auto& pending,
                                                         auto& buffer) {

        // Create VAO;
        // Create VBO;
        // Create IBO;
        // Set attribute layout
        // Release;

        // Signal that this buffer is ready to go
        registry.remove<PendingUpload>(entity);
        registry.assign<UploadedVertexBuffer>(entity, vaoId);
    });
}

/**
@brief Produce pixels by calling on the uploaded shader

Meshes are drawn per-shader. That is, a shader is associated to multiple renderables

*/
static void RenderSystem(ent::registry registry) {
    GL::defaultFramebuffer.clear(GL::FramebufferClear::Color | GL::FramebufferClear::Depth);
    GL::defaultFramebuffer.setViewport({{}, windowSize()});
    GL::Renderer::enable(GL::Renderer::Feature::DepthTest);

    registry.view<ShaderAssignment, UploadedShaderProgram>().each([](auto& assignment, auto& programId) {
        // Use programId

        for (auto& entity : assignment.entities) {
            auto& [pos, ori, buffer] = registry.get<Position, Orientation, UploadedVertexBuffer>(entity);
            // Bind buffer
            // Build transform
            // Draw elements
        }

        // Release programId
    });
}

Discussion

All of this is hypothetical. I'm having trouble finding examples somewhere inbetween basic and complex. Something to illustrate high-level concepts such as these, in particular one that implement a render loop.

Looming questions

1. Is this reasonable?
2. Is there a better way?
3. Is this possible with Magnum, or am I contorting it to do my will?
4. Does is make sense for a VertexBuffer to be associated with a given ShaderProgram, and to then iterate over each assigned VertexBuffer given a ShaderProgram? In Magnum, the default is the inverse; i.e. Mesh::draw(shader).
5. How does shadow maps fit into this? I.e. do I implement a ShadowRenderSystem an run this prior to a ColorRenderSystem, feeding the former into the latter?
6. Speaking of which; does systems take arguments? In the "complex" example above, systems take no arguments, and is part of a World object. Which to me has "globals" stamped in its forehead.
7. As new entities are spawned, there's an "initialisation" or "upload" step that must happen prior to being drawn. Currently, this initialisation happens as part of one SpawnSystem which "signals" to another system when it's ready by assigning a new component, e.g. the UploadedVertexBuffer; carrying only it's OpenGL identifier.
   • This (probably) works, but what's the alternative?
   • Magnum currently combines upload and use into the constructor of Mesh, with an optional NoCreate argument to prohibit upload on construction.
   • That's convenient, because it means you can synchronise the creation and destruction of the data in both CPU and GPU memory, but creates an implicit dependency and reference to a "global" or "current" OpenGL context; how does that work across contexts or across threads?

References

A selection of some of the resources I found that are either too basic or too complex.

• https://github.com/miguelmartin75/anax
• https://github.com/vinova/Artemis-Cpp
• https://github.com/seanfisk/ecs
• https://ibob.github.io/dynamix/index.html#features
• http://gameprogrammingpatterns.com/component.html
• https://blog.molecular-matters.com/2013/07/24/adventures-in-data-oriented-design-part-3c-external-references/
• http://bitsquid.blogspot.com/2011/09/managing-decoupling-part-4-id-lookup.html
• https://ourmachinery.com/post/ecs-and-rendering/
• https://gist.github.com/mottosso/9981d33f96bc0c810bd3fc04f8063999
• https://skypjack.github.io/2019-02-14-ecs-baf-part-1/
• skypjack/entt#169 (comment)
• https://github.com/skypjack/entt/wiki/Crash-Course:-entity-component-system
• YT: A Data Oriented Approach to Using Component Systems
• YT: OOP is dead, long live Data-Oriented Design
• YT: bennybox ECS series
• YT: Implementation of a component-based entity system in modern C++
• YT: Rez Bot ECS series (incomplete)

ECSExample1.cpp

#include <Magnum/GL/DefaultFramebuffer.h>
#include <Magnum/GL/Mesh.h>
#include <Magnum/GL/Renderer.h>
#include <Magnum/MeshTools/Compile.h>
#include <Magnum/Platform/Sdl2Application.h>
#include <Magnum/Primitives/Cube.h>
#include <Magnum/Shaders/Phong.h>
#include <Magnum/Trade/MeshData3D.h>
#include <Magnum/Math/Quaternion.h>

#include "externals/entt.hpp"

namespace Magnum { namespace Examples {

// --------------------------------------------------------------
//
// Components
//
// --------------------------------------------------------------

using Position = Math::Vector3<float>;
using Orientation = Math::Quaternion<float>;

struct Scale : public Vector3 {
    using Vector3::Vector3;
};

struct Identity {
    std::string name;
};

struct Drawable {
    GL::Mesh mesh { NoCreate };
    Shaders::Phong shader{ NoCreate };
    Color4 color;
};

// ---------------------------------------------------------
//
// Systems
//
// ---------------------------------------------------------

static void MouseMoveSystem(entt::registry& registry, Vector2 distance) {
    registry.view<Orientation>().each([distance](auto& ori) {
        ori = (
            Quaternion::rotation(Rad{ distance.y() }, Vector3(1.0f, 0, 0)) *
            ori *
            Quaternion::rotation(Rad{ distance.x() }, Vector3(0, 1.0f, 0))
        ).normalized();
    });
}

static void MouseReleaseSystem(entt::registry& registry) {
    registry.view<Drawable>().each([](auto& drawable) {
        drawable.color = Color3::fromHsv({ drawable.color.hue() + 50.0_degf, 1.0f, 1.0f });
    });
}

static void AnimationSystem(entt::registry& registry) {
    Debug() << "Animating..";
}

static void PhysicsSystem(entt::registry& registry) {
    Debug() << "Simulating..";
}

static void RenderSystem(entt::registry& registry, Matrix4 projection) {
    Debug() << "Rendering..";

    registry.view<Identity, Position, Orientation, Scale, Drawable>().each(
        [projection](auto& id, auto& pos, auto& ori, auto& scale, auto& drawable)
    {
        auto transform = (
            Matrix4::scaling(scale) *
            Matrix4::rotation(ori.angle(), ori.axis().normalized()) *
            Matrix4::translation(pos)
        );

        // Problem area 1: Shader program with function and data combined
        // Ideal solution: Uniforms a separate component
        drawable.shader.setLightPosition({7.0f, 7.0f, 2.5f})
                       .setLightColor(Color3{1.0f})
                       .setDiffuseColor(drawable.color)
                       .setAmbientColor(Color3::fromHsv({drawable.color.hue(), 1.0f, 0.3f}))
                       .setTransformationMatrix(transform)
                       .setNormalMatrix(transform.rotationScaling())
                       .setProjectionMatrix(projection);

        // Problem area 2: Vertex data and rendering function combined
        // Ideal solution: Vertex data a separate component, shader takes mesh as component
        drawable.mesh.draw(drawable.shader);
    });
}

// ---------------------------------------------------------
//
// Application
//
// ---------------------------------------------------------

using namespace Magnum::Math::Literals;

class ECSExample : public Platform::Application {
public:
    explicit ECSExample(const Arguments& arguments);

private:
    void drawEvent() override;
    void mousePressEvent(MouseEvent& event) override;
    void mouseReleaseEvent(MouseEvent& event) override;
    void mouseMoveEvent(MouseMoveEvent& event) override;

    entt::registry _registry;

    Matrix4 _projection;
    Vector2i _previousMousePosition;
};

ECSExample::ECSExample(const Arguments& arguments) :
    Platform::Application{ arguments, Configuration{}
        .setTitle("Magnum ECS Example") }
{
    GL::Renderer::enable(GL::Renderer::Feature::DepthTest);
    GL::Renderer::enable(GL::Renderer::Feature::FaceCulling);

    _projection =
        Matrix4::perspectiveProjection(
            35.0_degf, Vector2{ windowSize() }.aspectRatio(), 0.01f, 100.0f) *
        Matrix4::translation(Vector3::zAxis(-10.0f));

    // Create entities
    auto box = _registry.create();

    // Assign components
    _registry.assign<Identity>(box, "Box");
    _registry.assign<Position>(box, 0.0f, 0.0f, 0.0f);
    _registry.assign<Orientation>(box, Quaternion::rotation(30.0_degf, Vector3(0, 1.0f, 0)));
    _registry.assign<Scale>(box, 1.0f);
    _registry.assign<Drawable>(box,
        MeshTools::compile(Primitives::cubeSolid()),
        Shaders::Phong{},
        Color4(.4f, .2f, .9f)
    );
}

void ECSExample::drawEvent() {
    GL::defaultFramebuffer.clear(
        GL::FramebufferClear::Color | GL::FramebufferClear::Depth
    );

    // Should the system take _projection as argument?
    RenderSystem(_registry, _projection);

    swapBuffers();
}

void ECSExample::mousePressEvent(MouseEvent& event) {
    if (event.button() != MouseEvent::Button::Left) return;
    _previousMousePosition = event.position();
    event.setAccepted();
}

void ECSExample::mouseReleaseEvent(MouseEvent& event) {
    if (event.button() != MouseEvent::Button::Left) return;

    // Should the system handle all mouse events, instead of individual ones?
    MouseReleaseSystem(_registry);

    event.setAccepted();
    redraw();
}

void ECSExample::mouseMoveEvent(MouseMoveEvent& event) {
    if (!(event.buttons() & MouseMoveEvent::Button::Left)) return;

    const float sensitivity = 3.0f;
    const Vector2 distance = (
        Vector2{ event.position() - _previousMousePosition } /
        Vector2{ GL::defaultFramebuffer.viewport().size() }
    ) * sensitivity;

    // Should the system compute delta?
    // If so, where does state go, i.e. _previousMousePosition?
    MouseMoveSystem(_registry, distance);

    _previousMousePosition = event.position();
    event.setAccepted();

    redraw();
}

}}

MAGNUM_APPLICATION_MAIN(Magnum::Examples::ECSExample)

Continuing from mosra/magnum#314

Most of the issues you describe stem from how GL is designed

This makes sense with my findings so far; have been briefly looking at how Vulkan does it and get the feeling it's exposing more of the things I'm learning happens implicitly with OpenGL, like the swap chain and command buffer.

the mid-term roadmap for Magnum is about creating a Vulkan-based renderer (and going very DoD with it)

Neat! The only hurdle for me with Vulkan specifically is that I'd really like for my application to also run in the browser, and from what I gather OpenGL is the only option there, via Emscripten. Still, I figure it has at least documented these implicit things, and that some of that may be applicable at a higher level even in OpenGL.

Made a few changes, and have new questions. I figure I'd leave this gist as-is to maintain some form of progression/history of both code and conversation.

• Part II