The starting point to using and understanding Allzpark.
Estimated reading time: 20 mins
By the time you're done with this chapter, you'll be able to call the below command, and understand what it does.
rez env allzpark bleeding_rez-2.31+ pyside2 python-3 -- allzpark
Allzpark isn't just a pretty face, it's the backbone of any competent production studio working in visual effects, feature animation, commercials or games.
That backbone is made up of packages.
What is a package?
A package is a group of files with some metadata attached, declaring a name, version and its relationship to other packages. When one package requires another, a requirement hierarchy is formed.
For example, consider this requirement.
requires = ["maya-2019", "arnold", "cmuscle"]
From looking at this, you'd expect a version of
cmuscle compatible with
maya-2019 (note that we didn't request a particular version of these). Because only a subset of versions of
arnold are compatible with
maya-2019 what happens is a resolve.
Resolving a request means solving the equation of a requirements hierarchy until exact versions of each package in a request is found, and goes something like this.
iteration 01 # maya-2019 arnold cmuscle iteration 02 # maya-2019.0.3 arnold-4.12 cmuscle-1.4 iteration 03 # maya-2019.0.3 arnold-4.12 cmuscle-1.4 libpng-12 libtiff-1 qt-5.12 iteration 04 # maya-2019.0.3 arnold-4.9 cmuscle-1.4 libpng-12 libtiff-1 qt-5.12 qtbase-5.12 qtgui-5-12 openiio-3.41 complete
- In this example, the first iteration is your original request.
- The second iteration expands on this request to include specific versions of
cmuscle; both of which are deemed compatible with
- Now things start to get interesting, where did
libpng-12come from?! Well, that's a requirement of
arnold-4.12, so if we want
arnoldwe're going to have to get its other requirements too.
- But see, now things get even more interesting.
arnold-4.12was just downgraded to
arnold-4.9! That's because
openiiowas a requirement of
qtand conflicts with
arnold-4.12. As a result, an older version of
arnoldwas picked, one that is compatible with
That last step is one of the thing that separates Rez from package managers like
conda; the retroactive downgrading of a version to conform to a given constraint. This is one of the things that makes Rez more capable and safer than your typical resolver.
As you can see, the number of iterations and complexity therein can grow significantly. It is not uncommon for the number of packages involved to grow into the hundreds and run for dozens to hundreds of iterations per solve, with both off-the-shelf software like above and internally developed projects intermingled. Think about what you would have to go through to solve such a hierarchy yourself - which many do.
To resolve requirements, we'll utilise bleeding-rez.
pip install bleeding-rez rez bind --quickstart rez --version # bleeding-rez 2.33.1
pip not found
It's possible you have
pip installed, just not on your
PATH. Try this.
python -m pip install bleeding-rez
If this doesn't work, let's install pip.
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py python get-pip.py
The above command assumes admin/sudo access to your machine which isn't always the case. If so, you can install Rez using a virtual environment.
$ python -m pip install virtualenv $ python -m virtualenv rez-install $ rez-install\Scripts\activate (rez-install) $ pip install bleeding-rez
rez not found
If installation went successfully, but you aren't able to call
rez then odds are the Python executable path isn't on your
PATH. On Windows, this directory is typically at
c:\python37\scripts but may vary depending on how Python was installed, and varies across platforms.
Following the installation of
rez, you should have gotten a message about which path was missing from your
PATH, you can either add this yourself, or use the
virtualenv method from the above
Permission denied box.
This will make
rez available via the command-line and establish a few default "packages" in your
~/packages directory, which we'll talk about later.
Your first project
In order launch an application in the context of a project using Allzpark, we must create one.
Your project will be a directory with a file inside called
mkdir spiderman cd spiderman touch package.py
This will create a new file called
package.py in your newly created
spiderman directory. Edit this file with the following.
name = "spiderman" version = "1.0" build_command = False
Now we can "build" and make use of it.
rez build --install $ rez env spiderman > $
>character denotes that you are in a Rez "context", which is its virtual environment.
Let's keep going
name = "spiderman" version = "1.1" build_command = False def commands(): global env env["PROJECT_NAME"] = "Spiderman"
Now what happens?
> $ exit $ rez build --install $ rez env spiderman > $ Write-Host $env:PROJECT_NAME Spiderman
Great, we're now in control over the environment of this package. What about requirements?
name = "spiderman" version = "1.2" build_command = False requires = ["texteditor"] def commands(): global env env["PROJECT_NAME"] = "Spiderman"
texteditor in order to run. Let's build it.
> $ exit $ rez build --install 09:15:46 ERROR PackageFamilyNotFoundError: package family not found: texteditor
Your first application
Woops! We haven't got a package for
texteditor, let's make one.
> $ exit $ cd .. $ mkdir texteditor $ cd texteditor $ touch package.py
name = "texteditor" version = "1.0" build_command = False def commands(): import os global alias if os.name == "nt": alias("texteditor", "notepad") else: alias("texteditor", "nano")
Now let's build and use this package.
$ rez build --install $ rez env spiderman > $ texteditor
Viola, a platform-dependent text editor, tied to a given project. This is one way of tying applications to a project, but we'll look at some more as we go along. In general, you'll want to keep packages self-contained and portable, such that you can deploy them elsewhere. In this case, we utilised a widely accessible application we can expect to exist on almost any workstation. But we aren't always so lucky.
Let's make another one to illustrate this point.
> $ exit $ cd .. $ mkdir maya $ cd maya $ touch package.py
name = "maya" version = "2018.0" build_command = False def commands(): import os global alias if os.name == "nt": alias("maya", r"c:\program files\autodesk\maya2018\bin\maya.exe") else: alias("maya", "/usr/autodesk/maya2018/bin/maya.bin")
In this example, we're making some assumptions that may or may not be appropriate for your environment. If you are in control over workstations and installation paths, then this is fine. But if you can't make that guarantee, you'll care about portability which we'll cover a little later.
Before we move on, make another package for
maya-2019 as well in a similar fashion. We'll need this for later.
Let's now update our project to require
maya and see what we end up with.
name = "spiderman" version = "1.3" build_command = False requires = ["texteditor-1", "maya-2018"] def commands(): global env env["PROJECT_NAME"] = "Spiderman"
To run it..
$ rez env spiderman > $ maya > $ texteditor
As you can see, you now have both
texteditor available, at the same time. This typically is not what you want, and comes with a few gotchas. Consider for example if
texteditor had a requirement for another project, such as
msvcrt<=2011, and that
maya has a similar but conflicting requirement, such as
msvcrt>=2013. In isolation, this isn't a problem, because you can happily run
texteditor without requiring
maya and vice versa. But because these are both requirements of
spiderman, you've now made
spiderman impossible to use.
To account for this, we need to use "weak" references for both
name = "spiderman" version = "1.4" build_command = False requires = ["~texteditor-1", "~maya-2018"] def commands(): global env env["PROJECT_NAME"] = "Spiderman"
spiderman is requested, neither
texteditor is included.
~/spiderman $ rez build --install ~/spiderman $ rez env spiderman > ~/spiderman $ maya # Unrecognised command
Instead, you ask for them as part of your request.
> spiderman/ $ exit spiderman/ $ rez env spiderman maya # resolved packages: # maya-2018.0 ~\packages\maya\2018.0
But then, what was the point of making these requirements of
spiderman if they weren't going to become part of the resolve unless? Surely you can just leave them out, and include
maya in the request?
If you notice above, the resolved package was
maya-2018. If it wasn't for this weak reference, Rez would have picked the latest version,
This is how you can tie applications to your project, without including each of them into the same context. Think about how chaotic this would be if your project involved dozens of applications!
So you've made a project and given it a unique environment and applications. What's stopping you from launching these applications directly from the command-line? Why do you need Allzpark?
Every command we've typed so far has been entirely in the hands of Rez and you can safely run productions in this way. What Allzpark does is put a face on this system, something for the less technical-minded artists to wrap their heads around, and establish a few ground-rules about how to make the most out of Rez. We'll get into these rules a little later, but for now, let's see what Allzpark looks like on your machine.
For this next part, we'll need
git --version # git version 2.16.1
git not found
Git is required in later chapters, so you may as well get it up and running right away.
Allzpark is a Python package, and whilst we could install it like any other Python package, what we're going to do instead is install it as another Rez package. For that, we'll need
git clone https://github.com/mottosso/rez-pipz.git cd rez-pipz rez build --install
pipz is a wrapper around
pip for use with Rez. It can take any request
pip can, and turn it into a Rez package. This saves from having to create a Rez package ourselves, when it's already a Python package. Neat!
To test out the installation, let's install
six as Rez package.
rez env pipz -- install six -y
This is the equivalent of
pip install six. Now let's try it with Allzpark.
git clone https://github.com/mottosso/allzpark.git rez env pipz -- install ./allzpark
In this case, we'll install Allzpark from the cloned repository directly (as it isn't yet on PyPI). We'll also need a Qt binding. Any binding will do, in this example we'll use PySide2.
rez env pipz -- install pyside2 -y
And there you have it. We are now ready to launch Allzpark.
rez env allzpark python pyside2 -- allzpark --root ~/packages
One of the thing that separates Res from other package managers like
conda is that packages you install are shared. Not only can they be shared across multiple machines, but also across multiple operating systems. Once a package has been installed, you'll never have to install it again. It is permanent, immutable in fact. This is how you can build up a personal- or studio-repository of packages that you can build a pipeline upon, making strong and controlled assumptions about what packages are available, which version they are at, and that they are compatible with each other.
So far, we've installed all packages into their default location, which is
ls $env:USERPROFILE/packages # With PowerShell ls $HOME/packages # With Bash
If your graphical application depends on Qt.py, then Qt.py needs to be loaded before your application. This is where loading order comes in. By establishing a requirements hierarchy, the order in which libraries load is included for free.
The recommended layout for Rez packages are as follows.
int/Internal projects, such as
core_pipeline. You develop and release new versions internally.
ext/External projects, such as
Qt.py, you typically install these with
rez env pipz -- install
td/Packages developed by TDs themselves, such as small utility scripts
proj/Project such as
app/Applications, such as
converted/Automatically converted packages from the old Template-based system
There are two additional paths.
~/packagesYour local development packages, from your home directory
~/.packagesYour localised packages
Any package from PyPI can be installed using a utility package called
$ rez env pipz -- install pyblish-base --release
- See rez-pipz for details.
Any package from Scoop can be installed using another utility package called
$ rez env scoopz -- install python python27 git
- See rez-scoopz for details.
For greater performance, any package may be localised to your local disk.
- See rez-localz for details.
$ rez env pyside2 allzpark bleeding_rez -- python -m allzpark ============================== allzpark (1.1.79) ============================== - Loading Rez.. ok - 0.75s - Loading Qt.. ok - 6.14s - Loading allzpark.. ok - 0.53s - Loading preferences.. ok - 0.00s ------------------------------
Notice how long it took to load
Qt, let's localise this.
$ rez env localz -- localise PySide2
Now try launching again.
$ rez env pyside2 allzpark bleeding_rez -- python -m allzpark rez env pyside2 allzpark bleeding_rez -- python -m allzpark ============================== allzpark (1.1.79) ============================== - Loading Rez.. ok - 0.91s - Loading Qt.. ok - 0.36s - Loading allzpark.. ok - 0.70s - Loading preferences.. ok - 0.00s ------------------------------
That's much better.
To save disk space, you can delete any or all localised packages from your
Packages, like Python modules, are discovered from a list of paths. If there are identical packages on two or more paths, the first one is picked.
We can leverage this behavior to override one package with another.
Overriding requirements enable you to test new packages, or packages of different versions, in an existing project and works like this.
- Copy project onto local development directory
If the version remains the same or higher then your edited project is now picked up in place of the original, providing final control over which packages are used in a given project.
Overriding environment variables can be achieved in a similar fashion to requirements, but is even more flexible.
Packages with regards to environment variables act akin to CSS, or Cascading Style Sheets, from the world wide web in that every change augments - or cascades - a previous change.
def commands(): global env env["PYTHONPATH"].append("/a")
requires = ["a"] def commands(): global env env["PYTHONPATH"].append("/b")
In this example, the package
b augments an existing
PYTHONPATH created by package
a. It does so by appending the value
"/b". You can also
prepend and overwrite by assigning it directly.
env = ""
Example - Legacy Viewport
We can leverage this behavior to override the behavior of a program using dedicated "override packages".
name = "maya_legacy_viewport" version = "1.0" requires = ["maya"] def commands(): global env env["MAYA_ENABLE_LEGACY_VIEWPORT"] = "1"
Including this package in your resolve results in Maya exposing the Legacy Viewport option, to circumvent that pesky Viewport 2.0.
Rez is heavily dependent on a server-side application called
memcached, which as the name suggests is a cache for queries involving the filesystem. Without it, resolving new environments can take seconds to minutes compared to milliseconds, so it's recommended that you set it up.
If you've got access to Docker, then this is the simplest option for both a local and distributed install.
docker run -ti --rm -p 11211:11211 memcached
This will expose
memcached to its default port 11211.
Memcached isn't particularly memory intensive (>=32 mb of RAM) and can run on low-end hardware like a Pi, and should keep you covered up to 500 versions or so.
apt-get install memcached
This will expose
memcached to its default port 11211.
You can also install onto your local machine, however it requires a Linux or MacOS operating system for that.