yadudoc

(wrench) /tmp/simgrid:>cmake -DCMAKE_INSTALL_PREFIX=/opt/simgrid .
-- Configuring SimGrid v3.32.1
-- Cmake version 3.22.1
-- The C compiler identification is Clang 14.0.6
-- The CXX compiler identification is Clang 14.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Check for working C compiler: /Users/yadu/opt/anaconda3/envs/wrench/bin/x86_64-apple-darwin13.4.0-clang
-- Check for working C compiler: /Users/yadu/opt/anaconda3/envs/wrench/bin/x86_64-apple-darwin13.4.0-clang - broken
CMake Error at /Users/yadu/opt/anaconda3/envs/wrench/share/cmake-3.22/Modules/CMakeTestCCompiler.cmake:69 (message):

yadudoc

Working with conflicting environments

Let's assume that you are using a environment management system such as virtualenv, conda or modules(on clusters), it is possible that some of your workflow applications might have conflicting library requirements. Eg. one application requires the latest tensorflow library, whereas the other requires an older version.

Parsl can launch workers into completely different environments by using different executors for each app. The executor defines the resources (nodes/gpus) to request from say a batch scheduler like slurm and the commands to initialize the environment on the compute nodes such that parsl workers can be launched.

yadudoc

"""
Specification of user-specific configuration options.

The fields must be configured separately for each user. To disable any associated configurations, commen\
t
out the entry.

User specific overrides that should not go in version control can be set by creating a
file called local_user_opts.py, which declares a dictionary local_user_opts. Top level
keys in that dictionary will replace entries in the below user opts file, so it should

yadudoc

def generate_batch(filename, start=0, batchsize=10, max_batches=10):

    counter = 0
    if max_batches == 0:
        max_batches = 999999999

    with open(filename) as current:        
        while current and counter < max_batches:
            yield current.tell()
            counter += 1

yadudoc

def compute_descript(smile, walltime=1):
    """                                                                                                                                                                                                                                        
    import random                                                                                                                                                                                                                              
    import time                                                                                                                                                                                                                                
    if random.randint(0,8) == 0:                                                                                                                                                                                                               
        time.sleep(1)  

yadudoc

Problem

Executors may own one provider+launcher combo which is used to provision a set of resources for some duration from the native cluster scheduler (eg. slurm). Since each provider's provider.submit(..) method only takes a single command which is launched across the job any dynamic partitioning of the job is not well supported by the current model.

However, what we need is a mechanism by which provider.submit(..) takes a proxy command that may be used to partition the job in a dynamic fashion that tracks and reports the available capacity on the job.

yadudoc

import parsl
from parsl.providers import LocalProvider
from parsl.channels import LocalChannel

from parsl.config import Config
from parsl.executors import HighThroughputExecutor

config = Config(
    executors=[
        HighThroughputExecutor(

yadudoc

@python_app
def square(x):
    return x**2
@python_app
def average(x):
    return sum(x)/len(x)

output_list = []
for i in range(1,5):
    output_list.append(square(i))

yadudoc

# We'd ideally want to import our modules from a separate module :
# import methods



class MpiMethodServer:

    def __init__(self, input_queue, output_queue, methods_list=None, load_default=True):
        """ If load_default = True, we load the default methods list from a separate methods file
        else the user must pass a list of methods via methods_list kwarg.

yadudoc

Release Process

This is a first attempt at documenting the release process and the various testing/dev processes that go into making master release ready. Following is a checklist of items that are to be cleared for the 0.9.0 release

Checklist

Documentation updates :