gistfile1.txt

filo@filo-dtc:~/workspace/nipype$ git checkout 8c388d3262b1b3eedf79
Note: checking out '8c388d3262b1b3eedf79'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 8c388d3... changing the number of residual saved in defaults


filo@filo-dtc:/media/sdb2/nipype_examples$ python ~/workspace/nipype/examples/bootstrapping.py 
/usr/lib/pymodules/python2.6/IPython/kernel/clientconnector.py:43: DeprecationWarning: Importing class Tub directly from 'foolscap' is deprecated since Foolscap 0.4.3. Please import foolscap.api.Tub instead
  self.tub = Tub()
DEBUG:workflow:(preproc.realign, preproc.coregister): No edge data
DEBUG:workflow:(preproc.realign, preproc.coregister): new edge data: {'connect': [('mean_image', 'source'), ('realigned_files', 'apply_to_files')]}
DEBUG:workflow:(preproc.coregister, preproc.smooth): No edge data
DEBUG:workflow:(preproc.coregister, preproc.smooth): new edge data: {'connect': [('coregistered_files', 'in_files')]}
DEBUG:workflow:(preproc.realign, preproc.art): No edge data
DEBUG:workflow:(preproc.realign, preproc.art): new edge data: {'connect': [('realignment_parameters', 'realignment_parameters')]}
DEBUG:workflow:(preproc.coregister, preproc.art): No edge data
DEBUG:workflow:(preproc.coregister, preproc.art): new edge data: {'connect': [('coregistered_files', 'realigned_files')]}
DEBUG:workflow:(detrend.modelspecd, detrend.level1designd): No edge data
DEBUG:workflow:(detrend.modelspecd, detrend.level1designd): new edge data: {'connect': [('session_info', 'session_info')]}
DEBUG:workflow:(detrend.level1designd, detrend.level1estimated): No edge data
DEBUG:workflow:(detrend.level1designd, detrend.level1estimated): new edge data: {'connect': [('spm_mat_file', 'spm_mat_file')]}
DEBUG:workflow:(detrend.level1estimated, detrend.join_residuals): No edge data
DEBUG:workflow:(detrend.level1estimated, detrend.join_residuals): new edge data: {'connect': [('residual_images', 'in_files')]}
DEBUG:workflow:(detrend.join_residuals, detrend.img2float): No edge data
DEBUG:workflow:(detrend.join_residuals, detrend.img2float): new edge data: {'connect': [('merged_file', 'in_file')]}
DEBUG:workflow:(analysis.modelspec, analysis.level1design): No edge data
DEBUG:workflow:(analysis.modelspec, analysis.level1design): new edge data: {'connect': [('session_info', 'session_info')]}
DEBUG:workflow:(analysis.level1design, analysis.level1estimate): No edge data
DEBUG:workflow:(analysis.level1design, analysis.level1estimate): new edge data: {'connect': [('spm_mat_file', 'spm_mat_file')]}
DEBUG:workflow:(analysis.level1estimate, analysis.contrastestimate): No edge data
DEBUG:workflow:(analysis.level1estimate, analysis.contrastestimate): new edge data: {'connect': [('spm_mat_file', 'spm_mat_file'), ('beta_images', 'beta_images'), ('mean_residual_image', 'mean_residual_image')]}
DEBUG:workflow:(analysis.contrastestimate, analysis.threshold): No edge data
DEBUG:workflow:(analysis.contrastestimate, analysis.threshold): new edge data: {'connect': [('spm_mat_file', 'spm_mat_file'), ('spmT_images', 'spmT_images')]}
DEBUG:workflow:(analysis.level1estimate, analysis.threshold): No edge data
DEBUG:workflow:(analysis.level1estimate, analysis.threshold): new edge data: {'connect': [('RPVimage', 'RPVimage'), ('mask_image', 'mask_image'), ('beta_images', 'beta_images'), ('mean_residual_image', 'mean_residual_image')]}
DEBUG:workflow:(bootstrap_wf.detrend, bootstrap_wf.bootstrap): No edge data
DEBUG:workflow:(bootstrap_wf.detrend, bootstrap_wf.bootstrap): new edge data: {'connect': [('img2float.out_file', 'original_volume')]}
DEBUG:workflow:(bootstrap_wf.bootstrap, bootstrap_wf.analysis): No edge data
DEBUG:workflow:(bootstrap_wf.bootstrap, bootstrap_wf.analysis): new edge data: {'connect': [('bootstraped_volume', 'modelspec.functional_runs')]}
DEBUG:workflow:(bootstrap_wf.detrend, bootstrap_wf.analysis): No edge data
DEBUG:workflow:(bootstrap_wf.detrend, bootstrap_wf.analysis): new edge data: {'connect': [('level1estimated.mask_image', 'level1design.mask_image')]}
DEBUG:workflow:(level1.infosource, level1.datasource): No edge data
DEBUG:workflow:(level1.infosource, level1.datasource): new edge data: {'connect': [('subject_id', 'subject_id')]}
DEBUG:workflow:(level1.datasource, level1.preproc): No edge data
DEBUG:workflow:(level1.datasource, level1.preproc): new edge data: {'connect': [('func', 'realign.in_files')]}
DEBUG:workflow:(level1.datasource, level1.preproc): Edge data exists: {'connect': [('func', 'realign.in_files')]}
DEBUG:workflow:(level1.datasource, level1.preproc): new edge data: {'connect': [('func', 'realign.in_files'), ('struct', 'coregister.target')]}
DEBUG:workflow:(level1.preproc, level1.bootstrap_wf): No edge data
DEBUG:workflow:(level1.preproc, level1.bootstrap_wf): new edge data: {'connect': [('realign.realignment_parameters', 'detrend.modelspecd.realignment_parameters'), ('art.outlier_files', 'detrend.modelspecd.outlier_files'), ('smooth.smoothed_files', 'detrend.modelspecd.functional_runs')]}
DEBUG:workflow:setting nodelevel input concatenate_runs = True
DEBUG:workflow:setting nodelevel input input_units = secs
DEBUG:workflow:setting nodelevel input output_units = secs
DEBUG:workflow:setting nodelevel input time_repetition = 3.0
DEBUG:workflow:setting nodelevel input high_pass_filter_cutoff = 120.0
DEBUG:workflow:setting nodelevel input subject_info = [Bunch(amplitudes=None, conditions=[], durations=[], onsets=[], pmod=None, regressor_names=None, regressors=None, tmod=None), Bunch(amplitudes=None, conditions=[], durations=[], onsets=[], pmod=None, regressor_names=None, regressors=None, tmod=None), Bunch(amplitudes=None, conditions=[], durations=[], onsets=[], pmod=None, regressor_names=None, regressors=None, tmod=None), Bunch(amplitudes=None, conditions=[], durations=[], onsets=[], pmod=None, regressor_names=None, regressors=None, tmod=None)]
DEBUG:workflow:setting nodelevel input interscan_interval = 3.0
DEBUG:workflow:setting nodelevel input timing_units = secs
DEBUG:workflow:setting nodelevel input blocks_info = Bunch(duration=[10, 10], onsets=[[5, 25, 45, 65, 90, 110, 130, 150, 175, 195, 215, 235, 260, 280, 300, 320], [15, 35, 55, 75, 100, 120, 140, 160, 185, 205, 225, 245, 270, 290, 310, 330]])
DEBUG:workflow:setting nodelevel input time_repetition = 3.0
DEBUG:workflow:setting nodelevel input subject_info = [Bunch(amplitudes=None, conditions=['Task-Odd', 'Task-Even'], durations=[[5], [5]], onsets=[[5, 25, 45, 65, 90, 110, 130, 150, 175, 195, 215, 235, 260, 280, 300, 320], [15, 35, 55, 75, 100, 120, 140, 160, 185, 205, 225, 245, 270, 290, 310, 330]], pmod=None, regressor_names=None, regressors=None, tmod=None)]
DEBUG:workflow:setting nodelevel input input_units = scans
DEBUG:workflow:setting nodelevel input output_units = scans
DEBUG:workflow:setting nodelevel input interscan_interval = 3.0
DEBUG:workflow:setting nodelevel input timing_units = secs
DEBUG:workflow:setting nodelevel input contrasts = [('Task>Baseline', 'T', ['Task-Odd', 'Task-Even'], [0.5, 0.5]), ('Task-Odd>Task-Even', 'T', ['Task-Odd', 'Task-Even'], [1.0, -1.0])]
DEBUG:workflow:setting nodelevel input contrast_index = 1
DEBUG:workflow:(level2.l2source, level2.frequency_map): No edge data
DEBUG:workflow:(level2.l2source, level2.frequency_map): new edge data: {'connect': [('outfiles', 'binary_images')]}
DEBUG:workflow:(level2.frequency_map, level2.simple_threshold): No edge data
DEBUG:workflow:(level2.frequency_map, level2.simple_threshold): new edge data: {'connect': [('frequency_map', 'volumes')]}
DEBUG:workflow:in: ('realign.realignment_parameters', 'detrend.modelspecd.realignment_parameters')
DEBUG:workflow:(level1.preproc, detrend.modelspecd): No edge data
DEBUG:workflow:(level1.preproc, detrend.modelspecd): new edge data: {'connect': [('realign.realignment_parameters', 'realignment_parameters')]}
DEBUG:workflow:in: ('art.outlier_files', 'detrend.modelspecd.outlier_files')
DEBUG:workflow:(level1.preproc, detrend.modelspecd): Edge data exists: {'connect': [('realign.realignment_parameters', 'realignment_parameters')]}
DEBUG:workflow:(level1.preproc, detrend.modelspecd): new edge data: {'connect': [('realign.realignment_parameters', 'realignment_parameters'), ('art.outlier_files', 'outlier_files')]}
DEBUG:workflow:in: ('smooth.smoothed_files', 'detrend.modelspecd.functional_runs')
DEBUG:workflow:(level1.preproc, detrend.modelspecd): Edge data exists: {'connect': [('realign.realignment_parameters', 'realignment_parameters'), ('art.outlier_files', 'outlier_files')]}
DEBUG:workflow:(level1.preproc, detrend.modelspecd): new edge data: {'connect': [('realign.realignment_parameters', 'realignment_parameters'), ('art.outlier_files', 'outlier_files'), ('smooth.smoothed_files', 'functional_runs')]}
DEBUG:workflow:out: ('img2float.out_file', 'original_volume')
DEBUG:workflow:(detrend.img2float, bootstrap_wf.bootstrap): No edge data
DEBUG:workflow:(detrend.img2float, bootstrap_wf.bootstrap): new edge data: {'connect': [('out_file', 'original_volume')]}
DEBUG:workflow:out: ('level1estimated.mask_image', 'level1design.mask_image')
DEBUG:workflow:(detrend.level1estimated, bootstrap_wf.analysis): No edge data
DEBUG:workflow:(detrend.level1estimated, bootstrap_wf.analysis): new edge data: {'connect': [('mask_image', 'level1design.mask_image')]}
DEBUG:workflow:in: ('bootstraped_volume', 'modelspec.functional_runs')
DEBUG:workflow:(bootstrap_wf.bootstrap, analysis.modelspec): No edge data
DEBUG:workflow:(bootstrap_wf.bootstrap, analysis.modelspec): new edge data: {'connect': [('bootstraped_volume', 'functional_runs')]}
DEBUG:workflow:in: ('level1estimated.mask_image', 'level1design.mask_image')
DEBUG:workflow:(bootstrap_wf.detrend, analysis.level1design): No edge data
DEBUG:workflow:(bootstrap_wf.detrend, analysis.level1design): new edge data: {'connect': [('level1estimated.mask_image', 'mask_image')]}
DEBUG:workflow:in: ('mask_image', 'level1design.mask_image')
DEBUG:workflow:(bootstrap_wf.detrend.level1estimated, analysis.level1design): No edge data
DEBUG:workflow:(bootstrap_wf.detrend.level1estimated, analysis.level1design): new edge data: {'connect': [('mask_image', 'mask_image')]}
DEBUG:workflow:in: ('func', 'realign.in_files')
DEBUG:workflow:(level1.datasource, preproc.realign): No edge data
DEBUG:workflow:(level1.datasource, preproc.realign): new edge data: {'connect': [('func', 'in_files')]}
DEBUG:workflow:in: ('struct', 'coregister.target')
DEBUG:workflow:(level1.datasource, preproc.coregister): No edge data
DEBUG:workflow:(level1.datasource, preproc.coregister): new edge data: {'connect': [('struct', 'target')]}
DEBUG:workflow:out: ('realign.realignment_parameters', 'detrend.modelspecd.realignment_parameters')
/home/filo/workspace/nipype/nipype/pipeline/engine.py:301: UserWarning: Module bootstrap_wf has no input called detrend.modelspecd.realignment_parameters

  info[2]))
Traceback (most recent call last):
  File "/home/filo/workspace/nipype/examples/bootstrapping.py", line 278, in <module>
    l1pipeline.run()
  File "/home/filo/workspace/nipype/nipype/pipeline/engine.py", line 400, in run
    self._create_flat_graph()
  File "/home/filo/workspace/nipype/nipype/pipeline/engine.py", line 511, in _create_flat_graph
    workflowcopy._generate_execgraph()
  File "/home/filo/workspace/nipype/nipype/pipeline/engine.py", line 554, in _generate_execgraph
    self.connect(srcnode, srcout, dstnode, dstin)
  File "/home/filo/workspace/nipype/nipype/pipeline/engine.py", line 303, in connect
    raise Exception('Some connections were not found')
Exception: Some connections were not found

ok the reason i didn't see the error is because it was not in the connection part, but the expansion part

just put
l1pipeline._create_flat_graph()

in if main

don't have the time right now to debug it, but if i can i'll take a look at it later.

yes the problem lies somewhere in _create_flat_graph() it is still undeterministic thou (crashes every few calls).

Satrajit Ghosh 10:50
i know the problem

Chris Filo Gorgolewski 10:50
oh wow!
although I silently hoped for that... 10:51

Satrajit Ghosh 10:51
there is no particular order in which nested workflows are being expand

Chris Filo Gorgolewski 10:51
that's great

Satrajit Ghosh 10:51
so depending on which one gets expanded first and your set of connections, sometimes you see the error and sometimes you don't

Chris Filo Gorgolewski 10:52
still the order should not influence the result

Satrajit Ghosh 10:53
think of A(C)B(D) and now you have connections between C and D
if B(D) gets expanded first then C will connect appropriately 10:53
if A(C) gets expanded first then C will not have an appropriate connection. i thought we had addressed this, but i very much suspect that this is the root cause 10:54

Chris Filo Gorgolewski 10:54
by expanding you mean flattening and renaming?

Satrajit Ghosh 10:55
yes
there is actually a separate problem in expanding 10:55

Chris Filo Gorgolewski 10:55
I see the problem now
so shouldn't we sort them first? 10:56

Satrajit Ghosh 10:56
i don't think that handles all the cases

Satrajit Ghosh 11:02
but doing a topological sort at each workflow level is probably a good idea
nodes = nx.topological_sort(self._graph)
for node in nodes: 11:02
if you put those lines in _generate_execgraph 11:02
but i'm really not sure that's the answer 11:02

Chris Filo Gorgolewski 11:03
I'll try this

Satrajit Ghosh 11:03
it works for your case, but you need to create some test cases
if after the sort line you put nodes.reverse() 11:04
it will crash everytime 11:04

Chris Filo Gorgolewski 11:05
it should be at least deterministic...
even if we are not sure if it solves all of the cases 11:06

Satrajit Ghosh 11:08
i think doing the topological sort will address your case, but the following case will likely break
consider two workflows A->B and C->D 11:08
now consider D getting an input from the output of A and B getting an input from the output of C. i have a feeling the execgraph is going to break 11:09

Chris Filo Gorgolewski 11:12
cant we do a two stage expansion?
first flatten and rename and then reconnect? 11:12

Satrajit Ghosh 11:13
the key is to determine where a connection port (stored in an edge data) connects to.
anyway, i really need to get back to finishing a grant 11:13

Chris Filo Gorgolewski 11:13
sure
thanks for your help 11:14

Satrajit Ghosh 11:14
no problem
but we should fix this bug. 11:14