Computing in multiple conflicting environments with Parsl

conflicting_envs.md

Working with conflicting environments

Let's assume that you are using a environment management system such as virtualenv, conda or modules(on clusters), it is possible that some of your workflow applications might have conflicting library requirements. Eg. one application requires the latest tensorflow library, whereas the other requires an older version.

Parsl can launch workers into completely different environments by using different executors for each app. The executor defines the resources (nodes/gpus) to request from say a batch scheduler like slurm and the commands to initialize the environment on the compute nodes such that parsl workers can be launched.

Note: For this to work all the target environments must be using the same version of Python and Parsl.

Here's a short demo:

1. we first create two conda environments named after two versions of tensorflow.

$> conda env list
tf_1.5.0_py3.7           /home/yadu/anaconda3/envs/tf_1.5.0_py3.7 </br>
tf_2.2.0_py3.7        *  /home/yadu/anaconda3/envs/tf_2.2.0_py3.7

2. Next we create a parsl script test.py that uses these two envs for two different applications::

import parsl
from parsl import python_app
from parsl.providers import LocalProvider
from parsl.channels import LocalChannel
from parsl.config import Config
from parsl.executors import HighThroughputExecutor

config = Config(
    executors=[
        HighThroughputExecutor(
            label="TF_1.5.0",
            max_workers=1,
            provider=LocalProvider(
                channel=LocalChannel(),
                init_blocks=1,
                max_blocks=1,
                worker_init="""#!/bin/bash
source /home/yadu/anaconda3/bin/activate
source activate /home/yadu/anaconda3/envs/tf_1.5.0_py3.7
"""
            ),
        ),
        HighThroughputExecutor(
            label="TF_2.2.0",
            max_workers=1,
            provider=LocalProvider(
                channel=LocalChannel(),
                init_blocks=1,
                max_blocks=1,
                worker_init="""#!/bin/bash
source /home/yadu/anaconda3/bin/activate
source activate /home/yadu/anaconda3/envs/tf_2.2.0_py3.7
"""
            ),
        )
    ],
)
parsl.load(config)

@python_app(executors=['TF_1.5.0'])
def App1():
    import os
    return os.getenv('CONDA_PREFIX')

@python_app(executors=['TF_2.2.0'])
def App2():
    import os
    return os.getenv('CONDA_PREFIX')

print("Conda env that App1 executed in :", App1().result())
print("Conda env that App2 executed in :", App2().result())

3. Execute the test.py in a compatible env.:

$> conda activate parsl_1.0.0_py3.7
$> python3 test.py
Conda env that App1 executed in : /home/yadu/anaconda3/envs/tf_1.5.0_py3.7
Conda env that App2 executed in : /home/yadu/anaconda3/envs/tf_2.2.0_py3.7