22nd July 2019 - Visual Patching

README.md

In the midst of converting existing packages to Rez and discovering new requirements for Allzpark as we go.

Development Packages

Previously, versions listed in the Packages tab adhered to the "Use Development Packages" and "Use Localised Packages" checkboxes, but the override-dropdown did not. The same applied to the project version select.

For the overrides, there's a more significant workflow issue at hand, which is that the versions present in that list are versions discarded by Rez. By using any of those versions, you are effectively overruling Rez's decision to pick a particular version over another. In most cases, it's probably fine. The difference between a 1.0.0 and a 1.0.1 is unlikely to cause any issues. However, because the overruling happens post-solve, it means that if 1.0.1 introduced a new requirement, or changed the version of any of its requirements, then those new requirements would not become part of the launched application. Likewise, if any other package dependend on 1.0.0, than that package may break as well.

As a first step, I've made the dropdown versions adhere to the checkboxes.

For projects, the issue was also a little bit complicated.

1. Versions available did not respect the checkboxes
2. However the resolve did

So if 2.1 was selected but 2.1 was a development package and the "Use Development Packages" was unticked, then Allzpark would not be able to find 2.1 even though the user had explicitly selected it.

I've made this dropdown take the checkboxes into account as well.

Full Command

Last week, I added a "full command" representation to the bottom of the GUI. This didn't take checkboxes into account either, but now it does. With one caveat.

1. --no-local is appended if "Use Development Packages" is unticked
2. --paths is appended if "Use Localized Packages" is unticked

The latter, --paths, can make the command quite a bit longer. But since there is no official flag for localized packages, as there is with --no-local, then the only other option is to explicitly include every package path in the command itself. Relative the number of packages already in the command, it shouldn't make much difference. However, there is a limit to how long a command is able to get. The exact limit however I'm a little unsure of, but I don't think we're there quite yet. Just something to keep in mind!

Cache Refresh

I noticed new package versions weren't showing up in the GUI, until after a restart. Turns out this had to do with the 2 layers of caching going on.

1. Memcached, where requests go and resolves are returned. But also calls to os.listdir
2. lru, which is more or less a replicat of what goes to memcached, but stored in local memory for even faster lookup.

Normally, when using Rez from the command-line, the local memory is wiped on each invokation. Which would explain why the problem wasn't present anywhere but the GUI.

Luckily, there was a convenient way of clearing these in-memory cached.

from rez.package_repository import package_repository_manager

# From the controller
for path in self._package_paths():
    repo = package_repository_manager.get_repository(path)
    repo.clear_caches()

Where each "repository" carries each own cache that we need to clear. These are effectively each sub-directory of packages we've got on our REZ_PACKAGES_PATH.

But the question then is, when should we clear the cache?

Considering the cache doesn't survive invokations on the command-line, I figure it's safe to almost completely ignore it. However it does help reduce the number of calls to memcached, so I set up an automatic timer to clear the cache every 10 seconds.

That means that at most an artist or developer would have to wait 10 seconds prior to seeing their new package appear in the GUI.

Visual Patching

So far, we've got two methods of overriding Rez's decision to choose one version over another.

1. Patching
2. Post-resolve overruling

Patching is done by typing into the patch text field. It'll re-resolve the context given the updated requirements and enable alternative versions than the latest ones, however it does not permit choice of version that breaks the constraints of any other package. So it's safe to do, and Rez will tell you about when it isn't.

Overrides on the other hand is in our hands. Rez resolves your request, and then you overrule it. Entirely ignoring the constraints. Sometimes, this is desirable.

For example, if your project determined that Maya 2017 was the latest permitted version, you could not select 2019.

Other times however it may break things.

For example:

1. Rez resolves core_pipeline-1.0
2. You overrule core_pipeline-2.0

Technically, core_pipeline-2.0 has different requirements than 1.0, but those are not taken into account. And any package that depend on 1.0 would likely break too.

So what I've done is removed (2) - post-resolve overruling - and turned the mechanism of overruling into a visual means of typing into that Patch text field.

Dropdown = Patch

Manual Typing Works Too

"Set to default" translates to removing a patched package

"Set to latest" translates accordingly

Removing all patches restores the original resolve

It's not perfect, but it's in the right direction.

So what if you did want to pick Maya 2019 when 2018 was the latest permitted version? You would need to update the package making this requirement, to permit 2019 to be selected. A little more tedious, at the gain of a safe resolved context.

Tomorrow

I'll start having a look at error handling. As I've mentioned previously, errors are mostly unhandled at the moment, most errors coming from Rez and the packages it's trying to resolve.

Some of the most common errors are:

1. A project does not resolve This prevents any further action from happening, as every subsequent action depends on a valid project.
2. An application does not resolve Every project has one or more applications that must resolve correctly.
3. A package has an error There are a few variations here.
   • Missing Variable Reference For example, a package may be calling {env.SOME_VARIABLE} whereby it's expecting this variable to be declared by another package, or come from the parent environment.
   • SyntaxError Not as common, but can happen in any @late variable or in the command() function, since those aren't actually checked on release.
   • Unresolvable For example, one of its requirements cannot be met. This would typically bubble up to the project itself not resolving, but given a @late requires variable this can happen at run-time too.

At the moment, these all trigger a similar-looking error.

Which I'd like to convert into something more useful.

1. What happened, specifically?
2. Which package(s) was involved?
3. What can you do about it?