gistfile1.txt

[00:00.000 -> 00:04.080]  First of all, thank you very much, TJ and
[00:02.000 -> 00:06.240]  François, to give this opportunity.
[00:04.080 -> 00:08.960]  My name is Kristof. I am the principal
[00:06.240 -> 00:10.480]  technical artist at Remedy Entertainment,
[00:08.960 -> 00:12.560]  and today we're going to talk about how
[00:10.480 -> 00:14.320]  we're using USD in our next generation
[00:12.560 -> 00:15.520]  game development pipelines.
[00:14.320 -> 00:17.040]  I will be presenting today's
[00:15.520 -> 00:18.960]  presentation, but our director of
[00:17.040 -> 00:20.480]  technology, Mika Vähkala, is also in this
[00:18.960 -> 00:21.600]  call, so if you want to introduce yourself,
[00:20.480 -> 00:24.640]  Mika.
[00:21.600 -> 00:25.760]  Oh, hello. Yeah, this is Mika. Oh hello yeah
[00:23.000 -> 00:30.680]  this is Mika and as Christoph said
[00:25.760 -> 00:33.320]  I'm the TD and kind of also PO for
[00:30.680 -> 00:35.720]  this like product owner for the
[00:33.320 -> 00:37.200]  application that's using all this this use
[00:35.720 -> 00:39.600]  the technology and you'll get a sneak peek
[00:37.200 -> 00:41.920]  of something we call OmniTool which
[00:39.600 -> 00:48.000]  is not to be confused with omniverse, but yeah, so I'm also happy to be able to share this stuff
[00:48.000 -> 00:49.000]  with you guys.
[00:49.000 -> 00:56.440]  All right, first a bit of obligatory history. If you are unfamiliar with our studio, we
[00:56.440 -> 01:02.880]  hail from the northern part of Europe in Espo Finland, and we were established in 1995.
[01:02.880 -> 01:05.760]  We are the original creators of the Max Payne franchise,
[01:05.760 -> 01:08.400]  after which we created Alan Wake, American Nightmare,
[01:08.400 -> 01:10.480]  Quantum Break, Control, and most recently,
[01:10.480 -> 01:13.640]  the single player campaign for Crossfire X.
[01:13.640 -> 01:15.840]  I personally joined the studio in 2013,
[01:15.840 -> 01:19.160]  so that's like early stages of Quantum Break's production.
[01:19.160 -> 01:20.960]  And historically speaking, we have worked
[01:20.960 -> 01:25.000]  in a single project and single engine setting.
[01:25.000 -> 01:28.000]  The past couple of years, this has changed.
[01:28.000 -> 01:34.000]  Remedy today is a multi-project and to an extent, multi-engine studio.
[01:34.000 -> 01:38.000]  So we are actively working on five different projects with four different publishers,
[01:38.000 -> 01:43.000]  two of which with 505 Game being Condor and an unannounced AAA project.
[01:43.000 -> 01:46.000]  With Epic Games, we are building the next installment
[01:46.000 -> 01:47.760]  in the Alan Wake franchise.
[01:47.760 -> 01:50.680]  And together with Tencent, we are creating Vanguard,
[01:50.680 -> 01:52.320]  which is our Unreal project.
[01:52.320 -> 01:53.840]  And most recently, we have announced
[01:53.840 -> 01:55.920]  that we are working on the Max Payne 1 and 2 remix
[01:55.920 -> 01:57.360]  together with Rockstar Games.
[01:58.560 -> 02:01.280]  Now with this out of the way, let's talk about our technology.
[02:01.280 -> 02:04.800]  So Remedy's technology stack is known as Northlight.
[02:08.120 -> 02:08.160]  And Northlight consists consists of a proprietary engine
[02:12.200 -> 02:15.640]  and proprietary tooling. So this means our world editor, our runtime engine, our data conversion pipelines, our DCC
[02:15.640 -> 02:19.160]  pipelines, etc. So most things that essentially touch a
[02:19.160 -> 02:22.160]  NorFlight game in our production is found in the NorFlight
[02:22.160 -> 02:26.920]  moniker. NorFlight historically is made out of many proprietary data
[02:26.920 -> 02:27.520]  formats.
[02:27.520 -> 02:31.120]  So level and world data, animation markup,
[02:31.120 -> 02:34.320]  animation graphs, behavior trees, material definitions,
[02:34.320 -> 02:37.400]  and so on have always been proprietary.
[02:37.400 -> 02:39.040]  And in the end, any interrupt that we
[02:39.040 -> 02:41.920]  wish to do with DCCs with these formats,
[02:41.920 -> 02:43.680]  we'd either have to do import export
[02:43.680 -> 02:46.400]  or do a bunch of plug- plugin work or some custom Python stuff.
[02:46.400 -> 02:48.840]  And it's, yeah, it's all very custom.
[02:49.720 -> 02:52.360]  But a few years ago, we wanted to build a new editor.
[02:53.560 -> 02:56.760]  This new editor has been mulling in the Northlight team
[02:56.760 -> 02:59.360]  for quite some time, but we figured a couple of years ago,
[02:59.360 -> 03:01.200]  now's the right time to do so.
[03:01.200 -> 03:02.640]  And we had a few decisions to make,
[03:02.640 -> 03:04.600]  whether we wanted to build it from the ground up
[03:04.600 -> 03:08.440]  and what technology to base it on if we do so. And we had a few decisions to make, whether we wanted to build it from the ground up and what technology to base it on if we do so.
[03:08.440 -> 03:10.720]  So we decided to both build it from the ground up
[03:10.720 -> 03:13.240]  and base it entirely on USD.
[03:13.240 -> 03:15.600]  So for lack of a better description,
[03:15.600 -> 03:19.640]  what we've ended up with is a remedy-specific USD assembler.
[03:19.640 -> 03:22.600]  This is a screenshot of our editor.
[03:22.600 -> 03:25.320]  It has all the things that you would expect from a USD assembler.
[03:25.320 -> 03:29.400]  So we have a stage view, composition stack,
[03:29.400 -> 03:32.160]  our edit targets that you can choose from,
[03:32.160 -> 03:34.520]  print properties that you can set.
[03:34.520 -> 03:39.120]  We have something specific called an opinion inspector.
[03:39.120 -> 03:41.560]  We also have variant editing capabilities and so on.
[03:41.560 -> 03:43.400]  And we, of course, have a viewport.
[03:43.400 -> 03:45.280]  The main thing that I want to point out here with the viewport
[03:45.280 -> 03:48.520]  that this is in fact our engine that is being attached to our editor.
[03:48.520 -> 03:50.480]  So this is not a Hydra renderer.
[03:50.480 -> 03:53.920]  So the data that you're seeing here is essentially our internal
[03:53.920 -> 03:56.840]  runtime data formats.
[03:56.840 -> 04:00.160]  If people are interested in learning more about the editor itself,
[04:00.160 -> 04:04.000]  feel free to hit myself or Mika up after this talk.
[04:04.000 -> 04:06.280]  And maybe we can arrange for yet another presentation
[04:06.280 -> 04:07.120]  in the future.
[04:07.120 -> 04:09.840]  But there's a lot of stuff that we can talk about
[04:09.840 -> 04:12.360]  with the editor alone.
[04:12.360 -> 04:14.880]  Now, why USD?
[04:14.880 -> 04:18.440]  As we have moved on from single project into multi-project,
[04:18.440 -> 04:21.120]  we wanted to be, first and foremost, scalable.
[04:21.120 -> 04:23.680]  And we felt that USD offered this
[04:23.680 -> 04:25.720]  through its many different composition arcs
[04:25.720 -> 04:28.620]  and the resulting collaborative workflows that it offers.
[04:29.580 -> 04:32.480]  On top of that, we were able to get some very flexible
[04:32.480 -> 04:35.920]  version control software workflows with USD
[04:35.920 -> 04:38.100]  by kind of utilizing its plugin mechanism.
[04:39.320 -> 04:41.480]  Additionally, we have kind of out of the box
[04:41.480 -> 04:45.840]  DCC interoperability, which for people on my side of the spectrum
[04:45.840 -> 04:47.840]  is very, very interesting.
[04:47.840 -> 04:51.360]  So we can essentially use Maya USD or Solaris and Houdini
[04:51.360 -> 04:55.160]  or even the one in Blender, just kind of out of the box
[04:55.160 -> 04:56.760]  with what we are doing.
[04:56.760 -> 04:58.920]  And of course, the premise of having long-term data
[04:58.920 -> 05:03.360]  portability through USDs kind of standardized schemas.
[05:03.360 -> 05:06.200]  Next up is that we wanted to consolidate things.
[05:06.200 -> 05:09.840]  So we've had a lot of similar concepts internally,
[05:09.840 -> 05:12.920]  things that we describe as levels, prefabs, archetypes,
[05:12.920 -> 05:15.120]  presets, whatever.
[05:15.120 -> 05:16.440]  All of these could essentially be
[05:16.440 -> 05:19.040]  expressed as standard composition arcs, or layers,
[05:19.040 -> 05:20.600]  or overrides, et cetera.
[05:20.600 -> 05:22.680]  And this was, of course, a big plus point
[05:22.680 -> 05:25.040]  for us to go with USD.
[05:30.720 -> 05:35.280]  And last and definitely not least is performance. Being able to load vast amounts of data really fast in a multi-threaded manner is incredibly important to us. And USD offers this out of the
[05:35.280 -> 05:41.680]  box, which was, of course, very important. And then the long-term goal of being able to
[05:41.680 -> 05:46.760]  offer larger worlds quite much easier through utilizing payload composition
[05:46.760 -> 05:49.880]  arcs or hiding, sorry, or muting sub layers
[05:49.880 -> 05:51.920]  or just selecting the layers that you
[05:51.920 -> 05:54.840]  wish to work on individually or collaboratively
[05:54.840 -> 05:57.800]  is extremely powerful.
[05:57.800 -> 06:00.440]  So how does USD fit in the Norfied architecture?
[06:00.440 -> 06:02.400]  This consists out of two components.
[06:02.400 -> 06:04.920]  So the first component is OmniTool.
[06:04.920 -> 06:07.040]  So again, not to be confused with OmniVerse.
[06:07.780 -> 06:09.780]  And this is what, this is our tools framework.
[06:09.840 -> 06:14.660]  Um, this is built, uh, on top of .NET and written in C sharp and
[06:14.660 -> 06:16.460]  consider it a plugin framework.
[06:16.540 -> 06:18.480]  We have many different types of plugins.
[06:18.720 -> 06:21.540]  Uh, for example, for offering animation graphs, behavior,
[06:21.540 -> 06:22.980]  trees, materials, and so on.
[06:23.560 -> 06:27.560]  And the editor that I showed earlier is essentially just a new plug-in to the system.
[06:27.560 -> 06:31.440]  And it's important to note that only the editor knows
[06:31.440 -> 06:33.600]  about USD.
[06:33.600 -> 06:36.720]  Next to that, we have our game, which is a runtime engine
[06:36.720 -> 06:38.080]  written in C++.
[06:38.080 -> 06:40.760]  Game itself, like Omnitool, doesn't know a single thing
[06:40.760 -> 06:42.120]  about USD.
[06:42.120 -> 06:45.680]  But we have a optional module that
[06:45.680 -> 06:49.160]  is loaded on top of that when we are working with the editor.
[06:49.160 -> 06:50.740]  That is called the editor layer, and it
[06:50.740 -> 06:52.440]  has a dependent called Scene API.
[06:52.440 -> 06:54.680]  And they know about USD.
[06:54.680 -> 06:57.000]  So when we are opening a layer in our editor,
[06:57.000 -> 06:59.200]  we kind of tell game, or more specifically,
[06:59.200 -> 07:01.720]  editor layer in the Scene API to open the very same layer
[07:01.720 -> 07:03.920]  in their USD runtimes.
[07:03.920 -> 07:08.160]  And we kind of keep them in sync through gRPC.
[07:08.160 -> 07:11.120]  More specifically, we're kind of utilizing SDF layer state
[07:11.120 -> 07:12.760]  delegates for this.
[07:12.760 -> 07:18.040]  So when layer changes are made on any of these layers
[07:18.040 -> 07:21.320]  in either OmniTool or the game's runtime,
[07:21.320 -> 07:24.240]  we capture these differences as tiny binary blobs,
[07:24.240 -> 07:29.320]  and we send them over the process boundaries via gRPC.
[07:29.320 -> 07:32.640]  For OmniTool, this is quite simplistic.
[07:32.640 -> 07:34.800]  We just receive it and apply it to the root layer.
[07:34.800 -> 07:36.680]  But for game, we also do the extra step
[07:36.680 -> 07:39.040]  of essentially also applying these changes
[07:39.040 -> 07:40.680]  to our runtime components.
[07:40.680 -> 07:44.520]  We do know how to link these together,
[07:44.520 -> 07:48.520]  but we'll go into a bit more detail with that in a moment.
[07:48.520 -> 07:50.080]  What is very important to note is
[07:50.080 -> 07:52.480]  that we do not include USD in our final builds.
[07:52.480 -> 07:56.800]  We only ever use it at edit time.
[07:56.800 -> 08:00.080]  And we have a few reasons for that.
[08:00.080 -> 08:02.360]  So first and foremost, this sounds maybe a little bit
[08:02.360 -> 08:05.480]  silly, but for us, if there is no official support
[08:05.480 -> 08:10.920]  for non-desktop platforms, it's kind of a thumbs down for us.
[08:10.920 -> 08:13.480]  Sorry, to this date, at least to my knowledge,
[08:13.480 -> 08:15.840]  even Windows is still marked as an experimental platform
[08:15.840 -> 08:16.920]  from USD.
[08:16.920 -> 08:19.360]  We need to be able to compile it for PlayStation 5,
[08:19.360 -> 08:22.160]  Xbox Series X, S, Switch, and any other platforms
[08:22.160 -> 08:25.480]  that we can think of that we are targeting with our games.
[08:25.480 -> 08:27.240]  And if USD does not do this out of the box,
[08:27.240 -> 08:31.920]  then we cannot spend time on doing this ourselves.
[08:31.920 -> 08:33.920]  Additionally, as a studio, we strongly
[08:33.920 -> 08:36.280]  feel that USD should be optimized for content creation
[08:36.280 -> 08:39.960]  on desktop and content creation only in the sense
[08:39.960 -> 08:43.720]  that we only care about composition at edit time.
[08:43.720 -> 08:46.120]  We believe that if you want to have your composition
[08:46.120 -> 08:49.280]  represented at the runtime, that you should transform it,
[08:49.280 -> 08:51.680]  bake it, or what we do, or what we call it internally,
[08:51.680 -> 08:54.720]  convert it.
[08:54.720 -> 08:58.000]  Additionally, at the time that we started this work,
[08:58.000 -> 08:59.720]  we had just come out of a big refactor
[08:59.720 -> 09:01.160]  to support another system.
[09:01.160 -> 09:05.760]  And for us, kind of rewriting the runtime yet again just to see
[09:05.760 -> 09:11.760]  if USD would work for us at runtime was not something we could justify. We are not a large,
[09:11.760 -> 09:18.280]  a very large studio and we have limited resourcing so we had to draw the line somewhere. And
[09:18.280 -> 09:25.960]  having worked with the gRPC workflow now we can quite confidently say that we vastly prefer live editing workflows over importing, exporting,
[09:25.960 -> 09:27.960]  or state-reloading workflows altogether.
[09:29.560 -> 09:33.160]  And lastly, we have a flattened scene at runtime.
[09:33.160 -> 09:37.120]  So we are using a highly optimized and custom version
[09:37.120 -> 09:39.800]  of an entity component system runtime.
[09:39.800 -> 09:43.120]  And we do not necessarily need a hierarchy
[09:43.120 -> 09:46.280]  when we're dealing with, when we're looking in a runtime.
[09:46.280 -> 09:49.240]  ECS and scene graphs are kind of diametrically opposed
[09:49.240 -> 09:50.520]  to one another.
[09:50.520 -> 09:52.280]  And for us kind of shoehorning one into the other,
[09:52.280 -> 09:54.020]  it also seemed fairly illogical.
[09:56.280 -> 09:57.960]  So let's take a bit of a closer look
[09:57.960 -> 10:01.520]  at how we're doing these kind of communication flows
[10:01.520 -> 10:02.720]  with gRPC.
[10:02.720 -> 10:26.760]  There's generally two ways of approaching this. One is where the layer changes are done on the editor side, and the other side is where the layer changes. This can be done by manipulating some property,
[10:26.760 -> 10:29.280]  i.e. they're expressing an opinion.
[10:29.280 -> 10:31.320]  Maybe they're chasing the scene graph somewhere.
[10:31.320 -> 10:32.600]  Doesn't necessarily matter.
[10:32.600 -> 10:35.000]  It is a layer change that we can capture
[10:35.000 -> 10:37.520]  in this diffify SDF layer state delegates.
[10:38.440 -> 10:39.720]  And then we package this up
[10:39.720 -> 10:42.520]  into an apply diff remote procedure call.
[10:42.520 -> 10:46.000]  This RPC gets sent over the process boundaries into our game's
[10:46.000 -> 10:50.480]  runtime. And again, just to reiterate, our game doesn't know anything about USD itself,
[10:51.200 -> 10:57.360]  but our editor layer knows what to look out for. It listens to these RPC events and it receives
[10:57.360 -> 11:04.240]  the diff. It then offloads the binary blob that we've sent with it into the scene API where it
[11:04.240 -> 11:08.120]  gets applied. And it applies it by modifying its version of the
[11:08.120 -> 11:09.600]  layer that we have loaded in the editor.
[11:11.240 -> 11:14.120]  This in turn triggers an internal entity changed callback.
[11:14.880 -> 11:19.040]  And we are using this callback to kind of create an in process, um,
[11:19.600 -> 11:21.320]  life edit, if you want to call it that.
[11:21.720 -> 11:25.560]  Uh, and this life edit is used to update one of our entities, or at least the
[11:25.560 -> 11:27.800]  entity that we have been editing so far.
[11:29.160 -> 11:32.540]  So if you are adding new prims, then we would create new entities.
[11:32.540 -> 11:36.360]  If you're changing components through changing properties in the editor, then
[11:36.400 -> 11:39.680]  all of these components are also now updated for those entities themselves.
[11:40.920 -> 11:44.680]  This is kind of, you have to keep in mind, this is also quite simplified, but I
[11:44.680 -> 11:46.960]  believe this brings the idea across quite well.
[11:48.080 -> 11:52.300]  The next version that we have of this is working,
[11:52.300 -> 11:54.180]  for example, with commands.
[11:54.180 -> 11:56.840]  And commands are, for example, copy pasting,
[11:56.840 -> 12:01.400]  scene graph bits, or maybe it could be spawning a gizmo
[12:01.400 -> 12:03.680]  by clicking a button in the editor.
[12:03.680 -> 12:06.000]  Regardless, we do not make any edit,
[12:06.000 -> 12:09.000]  we do not make any layer changes in the editor side itself.
[12:09.000 -> 12:12.000]  As per usual, the user of course is the
[12:12.000 -> 12:15.000]  interface to this and they want to execute a command.
[12:15.000 -> 12:18.000]  A command, as I said, could be a copy-paste or clicking a button
[12:18.000 -> 12:21.000]  or something. And this just creates an invoke command
[12:21.000 -> 12:24.000]  from a procedure call and sends it again
[12:24.000 -> 12:26.200]  into our game's runtime for our GRPC.
[12:26.200 -> 12:28.720]  And now the editor layer either immediately executes
[12:28.720 -> 12:32.000]  the commands, if it's, for example, a copy-pasting,
[12:32.000 -> 12:33.760]  or it spawns a gizmo.
[12:33.760 -> 12:37.160]  And with a gizmo, these only live in our game.
[12:37.160 -> 12:39.760]  We don't really have gizmos in an editor, per se,
[12:39.760 -> 12:43.200]  but the users can interact with them within our viewport.
[12:43.200 -> 12:44.960]  When a user does interact with a gizmo,
[12:44.960 -> 12:45.160]  for example, a translation, or they execute the command but the users can interact with them within our viewport. When a user does interact with a gizmo,
[12:45.160 -> 12:47.880]  for example, a translation, or they execute the command
[12:47.880 -> 12:50.240]  straight away, the net result is the same
[12:50.240 -> 12:53.560]  that the scene API itself edits the scene.
[12:53.560 -> 12:55.440]  And in this case, we are creating the diff
[12:55.440 -> 12:58.880]  from the game's version of the same layer.
[12:58.880 -> 13:00.360]  And then the process is essentially
[13:00.360 -> 13:03.800]  the same as the original case where
[13:03.800 -> 13:15.000]  we're going from editor into runtime, but now we're doing the opposite direction. So this triggers a diff callback, where we send the very same apply diff RPC over the process boundaries back into the editor where it gets
[13:15.000 -> 13:17.000]  applied.
[13:17.000 -> 13:25.400]  When it gets applied here. This is also the point where user are able to save these layers to disk. They can also keep on working in memory if they so desire,
[13:25.400 -> 13:26.660]  or they can have auto save on.
[13:26.660 -> 13:29.000]  That's entirely up to them.
[13:29.000 -> 13:32.760]  But generally, this is how we are working together
[13:32.760 -> 13:34.400]  with the editor and our game, and how
[13:34.400 -> 13:38.720]  USD is used, broadly speaking, inside North Flight
[13:38.720 -> 13:41.680]  and its technology stack.
[13:41.680 -> 13:43.640]  We can talk about this for much longer,
[13:43.640 -> 13:45.840]  but we have to condense this in a 30 minute
[13:45.840 -> 13:48.960]  presentation. So if you do want to know more, of course, you're free to hit us up.
[13:50.640 -> 13:59.120]  Now let's talk about content. Content historically has been kind of, we have source data files on
[13:59.120 -> 14:05.680]  disk, these get converted and they are then used in our editor. The same holds true now for our current editor.
[14:06.760 -> 14:10.880]  Meaning that for example, mesh or other source data, if you want to call it that,
[14:11.000 -> 14:13.360]  are asset paths and applied schema attributes.
[14:14.240 -> 14:17.320]  What you're seeing on the right is a screenshot of our mesh components
[14:17.360 -> 14:19.080]  properties for a entity node.
[14:19.920 -> 14:23.760]  And this mesh component properties, they're a single applied API schema.
[14:24.200 -> 14:26.640]  And they define a mesh source attributes,
[14:26.640 -> 14:29.120]  which is essentially an asset path.
[14:29.120 -> 14:31.140]  And this has an opinion saying that,
[14:31.140 -> 14:33.260]  use shared mail published.fbx.
[14:33.260 -> 14:35.860]  That's the only thing we can do with this.
[14:35.860 -> 14:38.860]  Meaning that we can only ever compose
[14:38.860 -> 14:42.560]  world or entity concepts at this point in time.
[14:42.560 -> 14:44.500]  So we cannot express opinions on meshes.
[14:44.500 -> 14:47.080]  We cannot express opinions on material attributes
[14:47.080 -> 14:48.840]  within material files,
[14:48.840 -> 14:52.360]  anything that is an extraneous source data file,
[14:52.360 -> 14:55.560]  we cannot really compose at this point in time.
[14:55.560 -> 14:58.360]  So what you're seeing on the right is our opinion inspector.
[14:58.360 -> 15:01.120]  It's kind of like a stack of author opinions
[15:01.120 -> 15:03.560]  based on the prim that you have selected.
[15:03.560 -> 15:06.640]  And you're seeing that for the basic player prim, we have the mesh component
[15:06.640 -> 15:09.600]  and the mesh source defined, and there's an authored opinion saying
[15:09.600 -> 15:11.120]  use shared mail products FBX.
[15:11.760 -> 15:15.360]  So we can wholesale say use a different FBX, but that's kind of it.
[15:16.560 -> 15:20.160]  Additionally, we have the historical problem, if you want to call it that,
[15:20.160 -> 15:21.920]  that we have very monolithic data.
[15:22.720 -> 15:27.760]  This may be a remedy only problem, but this is what we're dealing with now,
[15:27.760 -> 15:30.280]  is that we have a lot of duplicated content.
[15:30.280 -> 15:31.640]  When looking at characters,
[15:31.640 -> 15:34.440]  if you take, for example, Jessie from Control,
[15:34.440 -> 15:37.720]  and let's assume she has 50 different outfits,
[15:37.720 -> 15:40.760]  her skeleton information, her face geometry,
[15:40.760 -> 15:43.000]  sometimes hair or other pieces of garment,
[15:43.000 -> 15:44.600]  hands, feet, whatever,
[15:44.600 -> 15:47.040]  they all get copied and saved
[15:47.040 -> 15:49.200]  into these 50 different FBX files.
[15:49.200 -> 15:51.360]  And even those can have permutations over them
[15:51.360 -> 15:55.120]  for enabling things like cloth simulation physics.
[15:55.120 -> 15:57.720]  This holds true, but to a lesser extent for environment assets
[15:57.720 -> 15:59.120]  as well.
[15:59.120 -> 16:01.840]  And this is not great from a compositional point of view,
[16:01.840 -> 16:04.360]  and we kind of want to move away from this.
[16:04.360 -> 16:07.800]  So we have to start enabling source data composition
[16:07.800 -> 16:09.480]  into our level layers.
[16:11.000 -> 16:13.760]  But as mentioned before, we are not a large studio,
[16:13.760 -> 16:16.400]  so we have to be quite pragmatic in our approach for this.
[16:16.400 -> 16:18.440]  And the first thing that we wanted to do
[16:18.440 -> 16:21.600]  is essentially support existing data via custom plugins.
[16:21.600 -> 16:25.920]  So meaning for FBX, we would have an SDF file format plugin for our custom
[16:25.920 -> 16:29.760]  animation for our custom market files, we would have that plus some additional schemas that we
[16:29.760 -> 16:36.480]  have altered in-house, etc. This first step is incredibly important to us because it allows us
[16:36.480 -> 16:45.000]  to start writing a data transformation layer for USD, meaning that we can take USD, either via raw USD,
[16:45.240 -> 16:46.880]  or in this case via an FBX file
[16:46.880 -> 16:48.680]  and convert it to our runtimes.
[16:51.020 -> 16:54.780]  The reason why this is important is that it creates a slow,
[16:54.780 -> 16:59.620]  but steady migratory path for the DCC pipeline teams.
[16:59.620 -> 17:01.900]  Their pipelines wouldn't need to change overnight.
[17:01.900 -> 17:04.360]  They can still be outputting FBX.
[17:04.360 -> 17:07.960]  And over time they can switch this over into just outputting USD.
[17:07.960 -> 17:12.320]  But they have to decide how they want to structure their assets.
[17:12.320 -> 17:14.800]  And until they're ready to do so,
[17:14.800 -> 17:16.040]  we can have this layer running.
[17:16.040 -> 17:18.320]  So we can have this conversion layer
[17:18.320 -> 17:22.480]  running in the background for as many games as we want, really.
[17:22.480 -> 17:24.840]  The other thing with having a singular entry point
[17:24.840 -> 17:28.720]  with a USD converter is that it's very easy to extend.
[17:28.720 -> 17:31.680]  You can just essentially add support for animation
[17:31.680 -> 17:34.760]  if you want to, same for USD physics,
[17:34.760 -> 17:36.720]  for skeletons and so on.
[17:36.720 -> 17:39.400]  In my R&D phase, I have written a prototype converter
[17:39.400 -> 17:41.160]  that does all of these things, and it was actually
[17:41.160 -> 17:42.560]  quite easy to get going with this.
[17:48.460 -> 17:52.360]  So even with these two components, uh, we can essentially start composing a source assets sort of as they are, uh, this would still
[17:52.360 -> 17:56.400]  require a assets, uh, sorry, a pipeline change from the DCC pipelines.
[17:57.300 -> 18:01.640]  To, for example, output modular chunks as FBX files, not necessarily, uh, the
[18:01.640 -> 18:06.320]  wholesale, uh, collapsed, uh, character outfits or something.
[18:06.320 -> 18:09.800]  But the idea that we can now compose it as VXs
[18:09.800 -> 18:14.280]  is we opened the floodgates quite a bit already.
[18:14.280 -> 18:16.520]  So on the right, you're seeing a mock-up layer.
[18:16.520 -> 18:19.160]  This is not at all an indication of where
[18:19.160 -> 18:21.040]  we want to go to for the future.
[18:21.040 -> 18:24.400]  But it's just to exemplify what we're trying to achieve here.
[18:24.400 -> 18:26.440]  So at the top level, we have an entity nodes.
[18:26.480 -> 18:31.640]  Um, this is our custom is a schema to define entities in our levels.
[18:32.460 -> 18:37.400]  And in order to apply components to them, you just apply component API student.
[18:38.000 -> 18:39.920]  So in this case, we have that very same mesh component
[18:39.920 -> 18:41.320]  API, as I mentioned earlier.
[18:42.060 -> 18:47.480]  But rather than having a mesh source asset path attributes, it could be a relationship to a different prim.
[18:48.040 -> 18:49.720]  Now this prim could be a child.
[18:49.760 -> 18:50.660]  It could be a sibling.
[18:50.660 -> 18:52.140]  It could be part of a different hierarchy.
[18:52.660 -> 18:56.960]  It doesn't necessarily matter, but this data transformation
[18:56.960 -> 19:00.760]  layer mentioned before could essentially be looking for these relationships
[19:00.800 -> 19:05.220]  and it could take the composed state of the, sorry, of the
[19:05.900 -> 19:09.680]  prims that, uh, that of the prims that they are pointing to and essentially
[19:09.680 -> 19:11.280]  convert these into the runtime.
[19:12.300 -> 19:17.460]  This would allow us to a compose in a modular fashion, express opinions on
[19:17.460 -> 19:20.820]  source data, and also de-duplicate a lot of content.
[19:20.900 -> 19:24.180]  So you essentially only start using, you only pay for what you're
[19:24.180 -> 19:26.240]  using at that point in time.
[19:26.240 -> 19:27.560]  There is still duplication,
[19:27.560 -> 19:30.020]  but the duplication is on the runtime side.
[19:32.080 -> 19:33.440]  And this takes, as I mentioned before,
[19:33.440 -> 19:36.400]  a very small change on our pipelines.
[19:36.400 -> 19:38.400]  This is essentially where we are trying to move to
[19:38.400 -> 19:39.580]  at this point in time.
[19:40.600 -> 19:42.000]  We're kind of halfway there.
[19:43.040 -> 19:46.000]  And so far the file format plugin works extremely well for us.
[19:46.000 -> 19:49.000]  And I can showcase this in one of the later slides as well.
[19:51.000 -> 19:53.000]  Now, where do we really want to move to?
[19:53.000 -> 19:55.000]  Is of course to have most of our data as USD.
[19:55.000 -> 19:58.000]  So we're talking animations, skin and static geometry,
[19:58.000 -> 20:01.000]  material definitions, our custom markup files,
[20:01.000 -> 20:03.000]  physics definitions.
[20:03.000 -> 20:05.560]  Most of our data that we currently have,
[20:05.560 -> 20:07.600]  we can and very likely should be expressing
[20:07.600 -> 20:09.800]  as USD in the long-term.
[20:09.800 -> 20:11.680]  During my R&D phase, I have done tests
[20:11.680 -> 20:14.000]  with most of these, except for a markup
[20:14.000 -> 20:16.360]  and materials to a smaller extent.
[20:16.360 -> 20:19.200]  And I was able to get all of this into our engine,
[20:19.200 -> 20:21.280]  surprisingly fast, actually.
[20:21.280 -> 20:23.000]  And of course, the bonus with this is,
[20:23.000 -> 20:26.080]  is that you still have to, sorry, that you still,
[20:26.080 -> 20:31.280]  that you have the benefit of using pure USD. So the multi-threaded file loading works proper and
[20:31.280 -> 20:36.480]  the caching works proper as well, which is something that you don't really have if you're
[20:36.480 -> 20:39.440]  using FBX references, at least I couldn't get it to work properly.
[20:42.560 -> 20:47.000]  And this would enable long-term reusable assets across projects and engines as well.
[20:47.000 -> 20:51.000]  So the cool thing with USD of course is that you can layer on it what you would need.
[20:51.000 -> 20:58.000]  And there is no theoretical limitation as to why we couldn't have something called asset.usd that is bare bones.
[20:58.000 -> 21:05.640]  It could just be describing geometry, animation, any kind of data that is standardized.
[21:05.640 -> 21:08.040]  So if project A is a version of Northlight,
[21:08.040 -> 21:10.200]  and project B is yet another version of Northlight,
[21:10.200 -> 21:12.840]  and project C is an Unreal project,
[21:12.840 -> 21:15.200]  we should be able to take in Asset.USD
[21:15.200 -> 21:18.000]  and essentially layer on it what we need for the engine
[21:18.000 -> 21:21.400]  or even the project-specific needs.
[21:21.400 -> 21:23.120]  So if we do need to do some restructuring,
[21:23.120 -> 21:30.920]  we can essentially do that via these new layers that we throw over on top of it. Long term, we also
[21:30.920 -> 21:34.640]  want to try to get rid of our file system dependencies. This is of course
[21:34.680 -> 21:37.600]  something we wish to experiment with. This is not a guarantee that will
[21:37.600 -> 21:40.960]  happen, but we want to look into it. So at the moment, all of our layers are
[21:40.960 -> 21:47.960]  stored essentially on a Perforce server. And we kind of want to see if we can, we can get rid of that dependency altogether.
[21:48.520 -> 21:54.080]  Be it via internal cloud delivery, sorry, content delivery network or a database
[21:54.080 -> 21:56.080]  somewhere or another system altogether.
[21:56.320 -> 22:00.520]  We're not sure yet, but this is a long-term future thinking that we're doing here.
[22:02.400 -> 22:04.520]  And another thing that we are actively working on right now
[22:04.560 -> 22:07.960]  is the USD version problem.
[22:07.960 -> 22:09.440]  So we have many different versions
[22:09.440 -> 22:11.880]  of our engine, which potentially correlate
[22:11.880 -> 22:15.240]  to many different versions of USD, which then, in turn,
[22:15.240 -> 22:16.740]  correlate to many different versions
[22:16.740 -> 22:19.240]  of the DCC integrations, and all of our plugins,
[22:19.240 -> 22:21.280]  and schemas, et cetera.
[22:21.280 -> 22:23.600]  But we want to move into a system
[22:23.600 -> 22:28.240]  where USD, DCC integrations or schemas etc.
[22:28.240 -> 22:34.240]  are essentially all installed on demand and per project. So the projects define what they need
[22:35.040 -> 22:42.320]  and this configuration system pulls it from a centralized repository. You can use whatever
[22:42.320 -> 22:49.360]  you want for this. We are currently looking to using Sonotype Nexus. We have something working over it. We have already something working
[22:49.360 -> 22:53.360]  for this very small scale, but we're hoping to push this out studio wide fairly soon,
[22:53.360 -> 22:59.040]  actually. But if you combine this with good CI, CD, you can essentially put out as many
[22:59.040 -> 23:05.920]  permutations that you want, and they're just artifacts that you can pull in through this repository manager.
[23:05.920 -> 23:07.800]  And last, and definitely not least,
[23:07.800 -> 23:10.960]  is the live editing with DCCs.
[23:10.960 -> 23:13.400]  So this whole gRPC workflow, where
[23:13.400 -> 23:18.280]  we're communicating with the editor and the engine,
[23:18.280 -> 23:19.640]  it's basic RPC.
[23:19.640 -> 23:22.560]  So you should be able to do this with DCCs as well.
[23:22.560 -> 23:24.440]  And that's what this prototype is.
[23:24.440 -> 23:25.600]  So on the left, you have Maya.
[23:25.600 -> 23:29.880]  And on the right, you have our editor and our game running.
[23:29.880 -> 23:32.360]  And essentially, what you're seeing
[23:32.360 -> 23:37.280]  is lifetime changes in Maya being reflected
[23:37.280 -> 23:39.320]  on the editor and the game side.
[23:39.320 -> 23:41.760]  The way this works is that in Maya,
[23:41.760 -> 23:43.400]  I have an anonymous layer in which
[23:43.400 -> 23:48.520]  I reference this level layer. So this scene is authored in our editor.
[23:48.520 -> 23:50.880]  So it has no notion of any pieces of geometry.
[23:50.880 -> 23:53.080]  It only knows which FBX files to point to
[23:53.080 -> 23:55.040]  as asset attributes.
[23:55.040 -> 23:57.880]  So what we're doing here is that we're referencing this layer
[23:57.880 -> 24:00.120]  through our custom asset resolver.
[24:00.120 -> 24:04.480]  I find any print that has a mesh component API applied to it.
[24:04.480 -> 24:08.720]  If I do find one, I can see if I can resolve its mesh source attributes.
[24:08.720 -> 24:12.640]  And if I do, I essentially create a metric corrective underneath,
[24:12.640 -> 24:16.320]  because as a side note, we are authoring our worlds in one meters per unit,
[24:16.320 -> 24:21.160]  but our plugin, our FBX plugin, and generally data as a whole,
[24:21.160 -> 24:25.440]  we tend to output in one meter, sorry, in 0.01 meters per unit.
[24:28.960 -> 24:33.760]  So you have to correct for that. But underneath that, I essentially just created a reference to the FBX file. So what you're seeing in Maya as well is the result of our file format plugin for FBX.
[24:34.720 -> 24:41.040]  This has a bit of a downside in the sense that if you are doing this on the fly, every time you add
[24:41.040 -> 24:47.240]  a reference to it, it does it serially. So we will take the hit on importing the FBX scene
[24:47.240 -> 24:50.200]  even before it makes its way into USD land.
[24:50.200 -> 24:52.400]  And this is something that we hope to change in the future,
[24:52.400 -> 24:57.200]  of course, by moving away from that into actual USD.
[24:57.200 -> 25:00.480]  The more interesting part about this is that in Maya itself,
[25:00.480 -> 25:03.680]  I've created another plugin, a transformation handler
[25:03.680 -> 25:06.440]  in the UFE system.
[25:06.440 -> 25:10.520]  And we're doing this because it's easy to hook into these kind of pre and post transformation
[25:10.520 -> 25:11.980]  events.
[25:11.980 -> 25:15.720]  And doing so allows us to also start the diffing operation and closing it and then sending
[25:15.720 -> 25:20.020]  the RPC of that binary diff back into our editor in a game.
[25:20.020 -> 25:22.120]  So that's what you're seeing on the right as well.
[25:22.120 -> 25:25.480]  So all of these, like the Z component being changed
[25:25.480 -> 25:29.720]  is essentially a bunch of RPCs being applied continuously
[25:29.720 -> 25:32.000]  and quite fast.
[25:32.000 -> 25:34.080]  And we want to take this quite far.
[25:34.080 -> 25:36.080]  There's no reason why we wouldn't be able to,
[25:36.080 -> 25:39.600]  for example, allow animators to animate parts of our levels
[25:39.600 -> 25:43.480]  within their preferred DCC, be it Maya, Houdini, or God
[25:43.480 -> 25:47.920]  forbid, MotionBuilder. Maybe we want to bring
[25:47.920 -> 25:52.240]  in white boxing levels into Houdini to do generative modeling workflows on top of that.
[25:52.240 -> 25:57.600]  Like the sky is essentially the limit. As long as you have an USD layer to track to,
[25:57.600 -> 26:01.440]  you can kind of create a diff from it. And if it can be diffed, we can apply it as a
[26:01.440 -> 26:05.040]  live edit. Of course, there are caveats, there always will be,
[26:05.040 -> 26:06.940]  but that's the general gist of things.
[26:09.480 -> 26:12.400]  Yeah, and that's essentially what we are working on,
[26:12.400 -> 26:16.020]  what we currently have planned for the future.
[26:16.020 -> 26:17.680]  So we do have some working group topics
[26:17.680 -> 26:21.000]  that we want to raise as well.
[26:21.000 -> 26:31.000]  So we have a lot of the same problems, if you will, that have been discussed in the past. So lots are something that are, sorry, are unanswered for us.
[26:31.000 -> 26:36.000]  We have things like missing schemas that we also need, but then we have some more
[26:36.000 -> 26:45.760]  kind of specific problems that we've come across. So first and foremost, hierarchy iterative restructuring. By that, I mean that when you're working on video games,
[26:45.760 -> 26:47.400]  it's very iterative.
[26:47.400 -> 26:50.080]  And if you're looking at the whole world
[26:50.080 -> 26:53.520]  from a USD point of view, it is an enormous scene graph.
[26:53.520 -> 26:56.760]  So having to do any deletions or moving operations
[26:56.760 -> 26:59.520]  or changing relationships on these massive scene graphs,
[26:59.520 -> 27:03.440]  and especially if they affect prims hidden behind references
[27:03.440 -> 27:06.720]  that are hidden behind variants that are inactive.
[27:06.720 -> 27:09.280]  I'm sure you can imagine the type of situation
[27:09.280 -> 27:10.680]  you can get into quite fast.
[27:11.840 -> 27:14.320]  In addition to that, this is more of a nuisance really,
[27:14.320 -> 27:17.960]  but we do have a lot of schema changes all the time.
[27:17.960 -> 27:21.760]  This correlates to the schema versioning issue
[27:21.760 -> 27:24.480]  that has been brought up in other groups.
[27:24.480 -> 27:25.760]  We have a custom solution for this. So we can do schema versioning and we can do schema patching issue that has been brought up in other groups. We have a custom solution for this.
[27:25.760 -> 27:29.800]  So we can do schema versioning and we can do schema patching, but we do
[27:29.800 -> 27:31.380]  also wish to hot load these schemas.
[27:31.920 -> 27:35.580]  Maybe this has now changed with a more recent version of USD, but the version
[27:35.580 -> 27:37.880]  that we are using, this doesn't work very well.
[27:37.880 -> 27:39.200]  So we would have to shut down.
[27:39.720 -> 27:44.240]  Essentially our editor or engine, all the USD runtimes, and then kind of restart
[27:44.240 -> 27:45.440]  them to see these schema changes
[27:45.440 -> 27:47.280]  live.
[27:47.280 -> 27:49.040]  The next one is variant introspection.
[27:49.040 -> 27:52.040]  We kind of do not know what is in a variant unless it is active.
[27:52.040 -> 27:56.520]  And for a assembler that also allows authoring variants,
[27:56.520 -> 27:58.680]  this can be quite problematic because sometimes you
[27:58.680 -> 28:03.040]  can also just leave the wrong variant active, for example.
[28:03.040 -> 28:06.800]  It's not a huge deal, but it is quite a pain point
[28:06.800 -> 28:09.560]  for the person who had been working with this.
[28:09.560 -> 28:12.160]  In a similar fashion, figuring out
[28:12.160 -> 28:14.600]  when changing edit targets, which
[28:14.600 -> 28:17.600]  prims or metadata or whatever in your stage
[28:17.600 -> 28:19.520]  are actually part of that edit target.
[28:19.520 -> 28:25.000]  So figuring out what content relates to an edit target
[28:25.200 -> 28:27.760]  from your stage has been quite challenging as well.
[28:28.720 -> 28:30.760]  And another thing that we have been thinking about
[28:30.760 -> 28:34.120]  mulling over more than anything really is seeing
[28:34.120 -> 28:37.400]  is it possible to list edit uniform properties?
[28:37.400 -> 28:40.320]  So I know list editing normal attributes is a no-go
[28:40.320 -> 28:42.280]  because of the time series dependencies,
[28:42.280 -> 28:45.040]  but uniforms are not that.
[28:45.040 -> 28:48.480]  And we were thinking about skeleton composition.
[28:48.480 -> 28:51.400]  What we have internally is an animation skeleton,
[28:51.400 -> 28:54.320]  deformation skeletons, and runtime deformers.
[28:54.320 -> 28:56.000]  And from a compositional point of view,
[28:56.000 -> 28:58.600]  you would like to layer them above each other.
[28:58.600 -> 29:00.600]  At the moment, this is very difficult
[29:00.600 -> 29:02.800]  because you'd have to redefine the skeleton prim.
[29:02.800 -> 29:04.360]  You'd have to essentially redefine
[29:04.360 -> 29:07.040]  the entire joint influences attributes on the geometries
[29:07.040 -> 29:08.160]  that you're targeting.
[29:08.160 -> 29:12.280]  And this goes for anything that touches USD-SKAL.
[29:12.280 -> 29:15.680]  Being able to maybe prepend or append
[29:15.680 -> 29:20.200]  joints, matrices, joint influences, and so on,
[29:20.200 -> 29:22.840]  in composition stacks would be actually quite fun.
[29:22.840 -> 29:24.200]  I don't know if this is possible,
[29:24.200 -> 29:26.280]  but we wanted to bring it up.
[29:26.280 -> 29:27.660]  And then, of course, the last one
[29:27.660 -> 29:30.800]  is missing schemas such as cloth simulation.
[29:30.800 -> 29:33.200]  I now notice now that in the GTC keynote
[29:33.200 -> 29:34.960]  that NVIDIA does have something for cloth,
[29:34.960 -> 29:37.760]  so it will be kind of interesting to look at that.
[29:37.760 -> 29:39.800]  But also destruction is missing for us,
[29:39.800 -> 29:42.960]  and the more generic animation curves.
[29:42.960 -> 29:45.320]  We are more interested in keeping animation curves
[29:45.320 -> 29:49.540]  as they are from the DCC, and also to a much higher extent,
[29:50.800 -> 29:54.000]  authoring our own data curves for random things.
[29:54.000 -> 29:55.640]  This is something that we've actively working on
[29:55.640 -> 29:56.740]  at this point in time.
[29:58.320 -> 30:01.320]  But yeah, these are some topics that we could bring up
[30:01.320 -> 30:03.120]  or if people are willing to discuss this right now,
[30:03.120 -> 30:04.460]  feel free, of course.
[30:04.460 -> 30:05.960]  But essentially, this was it.
[30:05.960 -> 30:11.960]  I'm sorry if I went a bit fast or rambled a little bit, but the gist of it is that Remedy loves USD.
[30:11.960 -> 30:13.960]  We are heavily invested into USD.
[30:13.960 -> 30:15.960]  We want to see it succeed, of course.
[30:15.960 -> 30:20.960]  And yeah, we want to, of course, contribute as much as we can to the working groups as well.
[30:20.960 -> 30:22.960]  So that was it. Thank you.
[30:19.950 -> 30:22.070]  working groups as well.
[30:22.070 -> 30:23.370]  So that was it, thank you.

https://m.youtube.com/watch?v=FI2pyzTOvaQ&pp=ygUTcmVtZWR5IHVzZCBpbiBnYW1lcw%3D%3D