Asset Map 0.5 - Packages

README.md

Asset Map

• v0.5

The conceptual overview of content in Magenta along with it's manifestation as a directory tree.

Version 0.5 is about "packages".

• Proof of concept video
• Go to Map
• Go to Tree
• Go to Dropbox
• Download Hierarchy

Previous

• v0.2
• v0.3

Motivation

Speaking with artists at MPC and Framestore, to solve the problem of knowing which versions of render layers or animated characters belong together they both implement the concept of "packages". A package is simply a collection of versions. Artists then work directly with packages, as opposed to "subsets" (see below).

For example, it is common for a lighter to pick up an "animation package" for him to start lighting, just as it is common for a compositor to pick up a "lighting package".

To the artist, the package is versioned and each version contains all files necessary for him to light or animate. To us, the developers, the package is merely a .json file with name/version pairs. The package is also the place where captures are made for review, as they encompass the whole shot.

Workflow wise an artist imports a package as he would any regular file. The file extension for the package is registered in Maya and shows up in the Import dialog. From there, our tool, the importer, handles finding each file and corresponding version and references them appropriately.

This way, artists rarely import e.g. "ben01" or "backgroundLayer" individually, but instead work with packages. Packages are created implicitly and without the influence of the artist.

The first package is created either during Layout or Animation. Layout imports the individual items into the scene, positions them, and outputs a "layout package" that the Animator then picks up and animates.

CHANGES.md

Changes from 0.3

• Removed "family"
  The hierarchy no longer reflects family and instance separately; instead, an Instance results in a file with it's name at the end. The family is in its file-extension. Think of it like ben_GRP or body_GEO as opposed to /geo/ben and /grp/ben. A final name can look like this.

• thedeal_ben_modelDefault_v001_myModel.ma

  • .ma is generated from model family

• thedeal_ben_modelDefault_v001_myCapture.mov

  • .mov is generated from review family

• thedeal_seq01_1000_ben01_v001_myPointcache.abc

  • .abc is generated from pointcache family

• Removed "task"
  Tasks were removed from both shots and assets, meaning that artists now work directly on 1000, animRig and modelLow etc.

• Added "subset"
  A "subset" is part of an Item. Every publish is made to a subset. For example, in rendering it's the render layer, from animation it's ben01. From any asset, it's just default. Only subsets are versioned!

  • Subset

• Added "package"
  A package is a group of Subset versions. For example, ben01 -> version010 and table01 -> version002.

  • Package

LEGEND.md

Node	Description
	A transformation; i.e. input A turns into output B
	Transformation from -> to
	Instance, the inverse of a file. May have one or more representations
	A representation, such as a .jpeg file or .exr sequence
	Updated individually
	A comment
	Artist
	Item
	Subset
	Family
	Package
	Versioned package

MAP.md

thedeal_seq01_1000_animPkg_v006_animPkg.pkg

{
  "ben01": "v004", 
  "ben02": "v004", 
  "camera01": "v001", 
  "sofa01": "v001"
}

TREE_ANNOTATED.yaml

# Top level directory. Everything within this directory is
# self-contained and makes no outside references.
thedeal:

  # Re-usable items of a project are stored together here
  assets:
    ben:
      modelLow:
        
        # The public directory contains shared content
        public:
          
          # Default subset. Assets typically aren't divided further than the whole
          default:
          
            # Each version may contain one or more files/series of files
            v001:
            v002:
            v003:
            
              # Each file contains the full topic, along with version and name
              # of instance. The family is represented by it's extension.
              thedeal_ben_modelDefault_v001_myCapture.mov
              thedeal_ben_modelDefault_v001_myModel.ma
              thedeal_ben_modelDefault_v001_myMetadata.json
        
        # The private directory contains personal content
        private:
          marcus:
          roy:
            maya:
            zbrush:
            fusion:
      modelProxy:
      modelDefault:
    table:
    chair:

  # Items are combined and outputted as packages
  film:
    seq01:
      1000:
        public:

          # Characters and render layers mix together
          backgroundLayer:
            v001:
            v002:
            v003:
              ao.exr:
                thedeal_seq01_1000_backgroundLayer_v001_ao.0001.exr
                thedeal_seq01_1000_backgroundLayer_v001_ao.0002.exr
                thedeal_seq01_1000_backgroundLayer_v001_ao.0003.exr
              diffuse.exr:
              beauty.exr:
              beauty.mov
          dust:
          flames1:

          ben01:
            v001:
            v002:
            v003:
              thedeal_seq01_1000_ben01_v001_myAnimation.atom
              thedeal_seq01_1000_ben01_v001_myPointcache.abc
          ben02:
          table01:
          chair01:
        
        # A group of animated components
        animPkg:
          v001:
          v002:
          v003:
            thedeal_seq01_1000_animPkg_v001_versions.json
            thedeal_seq01_1000_animPkg_v001_capture.mov
        
        # A group of rendered elements
        lightPkg:
          v001:
          v002:
          v003:
            thedeal_seq01_1000_animPkg_v001_versions.json
            thedeal_seq01_1000_animPkg_v001_capture.mov

        private:
          marcus:
          roy:
            maya:
            fusion:

TYPE_INSTANCE.md

Instance

The inverse of a file.

When a file has been opened/imported into an application, the result is the "instance" of that file.

TYPE_ITEM.md

Item

A directory with /private and /public directories.

Private and Public represents development files and shared files respectively. The private files are worked on in isolation by an individual artist who then "publishes" to the equivalent /public directory where other artist then may fetch it.

TYPE_PACKAGE.md

Package

The package is a group of subset versions.

• Artists use packages directly
• Artists publish packages in-directly
• A new version of a package is created for each published Subset, such as ben01
• Packages are only relevant within the /film directory

For example the animator imports a layoutPkg to start animating. He produces a animPkg to be used by lighters who produce a lightPkg to be used by compositors who produce compPkg used by grading and editing.

Motivation

The package compensates for the lack of association between subsets, such as ben01 and table01, where each subset is versioned independently. A package wraps one or more of these versions.

ben01                              animPkg
 v001                     /------->  v001
 v002                    |
 v003 -------------------|
table01                  |
 v001 -------------------/
 v002

TYPE_SUBSET.md

Subset

A "subset" is part of an Item.

A subset consists of one or more Pyblish "instances".

Motivation

Sometimes, part of an Item needs to update independently of one another.

For example, in a shot with two characters only one has changed and needs publishing. Another example, a shot where only one render layer has changed and needs publishing.

Instead of publishing the entire Item, needlessly consuming time, space and cpu cycles, a subset may be published and versioned individually.

The name "subset" reflect that it is not the whole Item, but only part of it.

x.py

"""Source code to represent a directory tree as JSON"""

import os
import json


def hierarchy_to_dict(path):
    if os.path.isfile(path):
        return ""

    d = {n: hierarchy_to_dict(os.path.join(path, n)) for n in os.listdir(path)}

    return d


d = hierarchy_to_dict(r'C:\Users\marcus\Dropbox\projects\MagentaHierarchy0.5')
print(json.dumps(d, indent=2, sort_keys=True))

Make asset more like shots in published contents

The graph states that a lookdev artist operates on a single look at a time which I think is too limiting. (Similarly the current structure does the same with "modeling" where it only allows one "model" among its outputs). This reintroduces that it's too limiting what is valid output from any scene.

Instead it could be more like shots where it allows multiple outputs, like subsets.

Also I think it makes much more sense to put the packages under film/seq01/1000/publish into their own folder and doing the same for the subsets:

film/seq01/1000/publish/packages/
film/seq01/1000/publish/subsets/

Or if subsets are ever only present in a single package then even add the subsets within the package.

film/seq01/1000/publish/animPkg/
film/seq01/1000/publish/animPkg/subsets

Having them separate might mean more conflict between what "ben01" is in its contents. One might have that name in there for an "instance" in the scene for a rig, whereas the other one has it for a model. This also would mean there's a big chance that one "subset" would be present in multiple packages. And... do we want that "subset" in each "package" to be the same "subset", or does it belong specifically to the package?

---

As stated during our chat on Hangouts I don't really know what to think about a "free-for-all" within the shot folder separating solely by artist.