Workaround for Maya renderSetup layer overriding multiple same type/name AOVs

README.md

Workaround for Maya renderSetup layer overriding multiple same type/name AOVs

This is a note to self, to know why Maya renderSetup fails to apply renderLayer override on AOVs that are same type/name. (e.g. not able to disable specific AOV in renderLayer)

In Arnold, this happens when creating custom AOV with existing AOV name.

In Redshift, this happends when creating more than one AOV node that are same AOV type.

Hopefully, this will be fixed officially someday.

What I have found

RenderLayer overriding in RenderSetup uses collection to operate value override, and the collection needs a selector to filter/select nodes that needs to be overridden.

In maya.app.renderSetup.model.renderLayer.RenderLayer class, there's a method getCorrespondingCollection which seems to be the one who is resposible to create the layer override collection for specific renderSetting related node.

When dealing with AOV attribute override, the method getCorrespondingCollection reach down to the renderer's own subclass of maya.app.renderSetup.model.rendererCallbacks.AOVCallbacks class (if the renderer is Redshift, the subclass is rsmaya.rendererCallbacks.RedshiftAOVCallbacks), calling it's getChildCollectionSelector method to create renderer specific selector node and register the name of AOV node which needs to be overridden, into that selector node.

But somehow, AOVCallbacks class (superclass of, e.g. RedshiftAOVCallbacks) doesn't get to know the actuall name of AOV node that is being operated and need to assemble that name by itself from AOV type name with custom prefix, take Redshift for example it's rsAov_{aovType}, generated from getAOVName method.

Hence, if there's another AOV node that is using existing AOV name or same AOV Type, the node name cannot be assembled correctly and breaks the layer override.

RenderLayer.py

# In maya/app/renderSetup/model/renderLayer.py

class RenderLayer(RenderLayerBase, nodeList.ListBase, childNode.ChildNode):

    def getCorrespondingCollection(self, nodeName, selectedCollectionName):
        ...
        # around L1031
        if collection.AOVCollection.containsNodeName(nodeName):
            ...
            with undo.CtxMgr(kCreateAOVChildCollection % self.name()):
                callbacks = rendererCallbacks.getCallbacks(rendererCallbacks.CALLBACKS_TYPE_AOVS)
                aovName = callbacks.getAOVName(nodeName)
                coll = collection.create(aovName, collection.AOVChildCollection.kTypeId, aovName=aovName)
                # +++ The workaround
                _selector = coll.getSelector()
                if hasattr(_selector, "setAOVNodeName"):
                    # We need to check attr exists because it's not defined by the RenderSetup superclass
                    # `maya.app.renderSetup.model.selector.Selector`
                    _selector.setAOVNodeName(nodeName)  # works for Arnold and Redshift.
                # +++
                aovCol.appendChild(coll)
                return coll
        ...