pietromarchesi

pietromarchesi

pietromarchesi

import os

import neo
import numpy as np
from quantities import s
import matplotlib.pyplot as plt

import p_utils

### EXTRACTING AND PREPARING THE DATA ###

pietromarchesi

import luigi
import sciluigi
import os

class MyMetaWorkflow(sciluigi.WorkflowTask):

    runmode = luigi.Parameter()
    n_tasks     = luigi.IntParameter()

    def workflow(self):

pietromarchesi

Packaging data following the Neo model

General considerations

• A block corresponds to a recording session. Consequently, a full dataset will be composed of several blocks. A block should have annotations regarding the animal and all the dummy codes used in the different fields (e.g. if different trial types have specific numeric labels)
• Segments are used for trials. They should contain annotations regarding trial-specific variables, and event/epoch objects.
• Since this is a multi-area recording, channel indices correspond to a single area/bundle, and tetrode and channel information are ralegated to the annotations.

pietromarchesi

In this example we show how to adapt the example workflow so that we can run it on a cluster using the Slurm Workload Manager (Slurm).

We will write a workflow composed of two tasks, one which creates a file called foo.txt and writes foo in it, and one which reads foo.txt and swaps every occurrence of foo with the name of the cluster node on which the workflow is running.

To do this, we have to slightly change the workflow definition. In particular, we set up a runmode parameter which will allow us to specify from the command line whether we want the workflow to be run locally or on the cluster.

When we define the tasks, we have to pass an additional SlurmInfo object, which contains the specification of the resources we want to allocate to our tasks and other Slurm parameters.

pietromarchesi

import os
import luigi
import sciluigi

class MyWorkflow(sciluigi.WorkflowTask):
    runmode = luigi.Parameter()

    def workflow(self):

        if self.runmode == 'local':

pietromarchesi

Python environments on the JULIA system

There are different tools for managing virtual Python environments (see this answer on Stack Overflow for an overview). Here we give some instructions on how to get started using virtual environments on the JULIA system using venv (Python 3 support only) and virtualenvwrapper (an extension of the popular virtualenv, which can be used with both Python 2 and 3).

Recommended: managing environments using venv (Python 3 only)

venv is a package automatically shipped with Python 3. This method is quite straightforward, but it does not support Python 2 (for that, you can look at the second method using virtualenvwrapper which works for both Python 2 and 3).

To begin, we find the Python installation that we want to use by doing

$ module avail python

pietromarchesi

Python environments on the JULIA system

There are different tools for managing virtual Python environments (see this answer on Stack Overflow for an overview). Here we give some instructions on how to get started using virtual environments on the JULIA system using venv (Python 3 support only) and virtualenvwrapper (an extension of the popular virtualenv, which can be used with both Python 2 and 3).

Recommended: managing environments using venv (Python 3 only)

venv is a package automatically shipped with Python 3. This method is quite straightforward, but it does not support Python 2 (for that, you can look at the second method using virtualenvwrapper which works for both Python 2 and 3).

To begin, we find the Python installation that we want to use by doing

$ module avail python

pietromarchesi

Setting up a Python environment on JULIA with virtualenvwrapper

I find the system Python I want to use by doing

$ module avail python

Then we can load Python 3.6 with

$ module load python/3.6.1