Example of recipe for iteratively building a template using recursion (view graph with: snakemake -np --dag | dot | display)

snakemake_iterative_template

input_images = ['A','B','C']
shell('touch {input_images}')

max_iters = 4

#need this to make sure iterations don't go below 0!
wildcard_constraints:
    iteration="[0-9]+"

rule all:
    input: 'iter_{iteration}/template.nii.gz'.format(iteration=max_iters)

rule init_template:
    input: input_images
    output: 'iter_0/template.nii.gz'
    shell:
        'init_template {input} {output}'

rule reg_to_template:
    input:
        template = 'iter_{iteration}/template.nii.gz',
        image = 'inputs/{image}.nii.gz'
    output:
        warp = 'iter_{iteration}/{image}_warp_{iteration}.nii.gz'
    group: 'reg'
    shell:
        'reg_to_template {input} {output}'

rule update_template:
    input:
        images = lambda wildcards: expand('iter_{iteration}/{image}_warp_{iteration}.nii.gz',image=input_images,iteration=int(wildcards.iteration)-1),
        template = lambda wildcards: 'iter_{iteration}/template.nii.gz'.format(iteration=int(wildcards.iteration)-1)
    output:
        template = 'iter_{iteration}/template.nii.gz'
    shell:
        'update_template {input} {output}'

DAG is visualized here:

Running snakemake -np gives the expected result:

Building DAG of jobs...
Job counts:
    count    jobs
    1    all
    1    init_template
    12    reg_to_template
    4    update_template
    18


[Fri Oct  2 19:43:30 2020]
rule init_template:
    input: A, B, C
    output: iter_0/template.nii.gz
    jobid: 17


init_template A B C iter_0/template.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_0/template.nii.gz, inputs/B.nii.gz
    output: iter_0/B_warp_0.nii.gz
    jobid: 15
    wildcards: iteration=0, image=B


reg_to_template iter_0/template.nii.gz inputs/B.nii.gz iter_0/B_warp_0.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_0/template.nii.gz, inputs/C.nii.gz
    output: iter_0/C_warp_0.nii.gz
    jobid: 16
    wildcards: iteration=0, image=C


reg_to_template iter_0/template.nii.gz inputs/C.nii.gz iter_0/C_warp_0.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_0/template.nii.gz, inputs/A.nii.gz
    output: iter_0/A_warp_0.nii.gz
    jobid: 14
    wildcards: iteration=0, image=A


reg_to_template iter_0/template.nii.gz inputs/A.nii.gz iter_0/A_warp_0.nii.gz


[Fri Oct  2 19:43:30 2020]
rule update_template:
    input: iter_0/A_warp_0.nii.gz, iter_0/B_warp_0.nii.gz, iter_0/C_warp_0.nii.gz, iter_0/template.nii.gz
    output: iter_1/template.nii.gz
    jobid: 13
    wildcards: iteration=1


update_template iter_0/A_warp_0.nii.gz iter_0/B_warp_0.nii.gz iter_0/C_warp_0.nii.gz iter_0/template.nii.gz iter_1/template.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_1/template.nii.gz, inputs/A.nii.gz
    output: iter_1/A_warp_1.nii.gz
    jobid: 10
    wildcards: iteration=1, image=A


reg_to_template iter_1/template.nii.gz inputs/A.nii.gz iter_1/A_warp_1.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_1/template.nii.gz, inputs/B.nii.gz
    output: iter_1/B_warp_1.nii.gz
    jobid: 11
    wildcards: iteration=1, image=B


reg_to_template iter_1/template.nii.gz inputs/B.nii.gz iter_1/B_warp_1.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_1/template.nii.gz, inputs/C.nii.gz
    output: iter_1/C_warp_1.nii.gz
    jobid: 12
    wildcards: iteration=1, image=C


reg_to_template iter_1/template.nii.gz inputs/C.nii.gz iter_1/C_warp_1.nii.gz


[Fri Oct  2 19:43:30 2020]
rule update_template:
    input: iter_1/A_warp_1.nii.gz, iter_1/B_warp_1.nii.gz, iter_1/C_warp_1.nii.gz, iter_1/template.nii.gz
    output: iter_2/template.nii.gz
    jobid: 9
    wildcards: iteration=2


update_template iter_1/A_warp_1.nii.gz iter_1/B_warp_1.nii.gz iter_1/C_warp_1.nii.gz iter_1/template.nii.gz iter_2/template.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_2/template.nii.gz, inputs/B.nii.gz
    output: iter_2/B_warp_2.nii.gz
    jobid: 7
    wildcards: iteration=2, image=B


reg_to_template iter_2/template.nii.gz inputs/B.nii.gz iter_2/B_warp_2.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_2/template.nii.gz, inputs/C.nii.gz
    output: iter_2/C_warp_2.nii.gz
    jobid: 8
    wildcards: iteration=2, image=C


reg_to_template iter_2/template.nii.gz inputs/C.nii.gz iter_2/C_warp_2.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_2/template.nii.gz, inputs/A.nii.gz
    output: iter_2/A_warp_2.nii.gz
    jobid: 6
    wildcards: iteration=2, image=A


reg_to_template iter_2/template.nii.gz inputs/A.nii.gz iter_2/A_warp_2.nii.gz


[Fri Oct  2 19:43:30 2020]
rule update_template:
    input: iter_2/A_warp_2.nii.gz, iter_2/B_warp_2.nii.gz, iter_2/C_warp_2.nii.gz, iter_2/template.nii.gz
    output: iter_3/template.nii.gz
    jobid: 5
    wildcards: iteration=3


update_template iter_2/A_warp_2.nii.gz iter_2/B_warp_2.nii.gz iter_2/C_warp_2.nii.gz iter_2/template.nii.gz iter_3/template.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_3/template.nii.gz, inputs/A.nii.gz
    output: iter_3/A_warp_3.nii.gz
    jobid: 2
    wildcards: iteration=3, image=A


reg_to_template iter_3/template.nii.gz inputs/A.nii.gz iter_3/A_warp_3.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_3/template.nii.gz, inputs/B.nii.gz
    output: iter_3/B_warp_3.nii.gz
    jobid: 3
    wildcards: iteration=3, image=B


reg_to_template iter_3/template.nii.gz inputs/B.nii.gz iter_3/B_warp_3.nii.gz


[Fri Oct  2 19:43:30 2020]
rule reg_to_template:
    input: iter_3/template.nii.gz, inputs/C.nii.gz
    output: iter_3/C_warp_3.nii.gz
    jobid: 4
    wildcards: iteration=3, image=C


reg_to_template iter_3/template.nii.gz inputs/C.nii.gz iter_3/C_warp_3.nii.gz


[Fri Oct  2 19:43:30 2020]
rule update_template:
    input: iter_3/A_warp_3.nii.gz, iter_3/B_warp_3.nii.gz, iter_3/C_warp_3.nii.gz, iter_3/template.nii.gz
    output: iter_4/template.nii.gz
    jobid: 1
    wildcards: iteration=4


update_template iter_3/A_warp_3.nii.gz iter_3/B_warp_3.nii.gz iter_3/C_warp_3.nii.gz iter_3/template.nii.gz iter_4/template.nii.gz


[Fri Oct  2 19:43:30 2020]
localrule all:
    input: iter_4/template.nii.gz
    jobid: 0


Job counts:
    count    jobs
    1    all
    1    init_template
    12    reg_to_template
    4    update_template
    18
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

But using the --group-components option results in only a single job being submittted:
snakemake -np --group-components reg=3 --cluster "sbatch"

touch A B C
Building DAG of jobs...
Job counts:
	count	jobs
	1	all
	1	init_template
	12	reg_to_template
	4	update_template
	18

[Fri Oct  2 19:49:56 2020]
rule init_template:
    input: A, B, C
    output: iter_0/template.nii.gz
    jobid: 17

init_template A B C iter_0/template.nii.gz
Job counts:
	count	jobs
	1	all
	1	init_template
	12	reg_to_template
	4	update_template
	18
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

However, it works as expected if only a subset of the dag is used, when there are no repeated rules, e.g.:
snakemake -np --group-components reg=3 --cluster "sbatch" iter_1/template.nii.gz

touch A B C
Building DAG of jobs...
Job counts:
	count	jobs
	1	init_template
	3	reg_to_template
	1	update_template
	5

[Fri Oct  2 19:51:24 2020]
rule init_template:
    input: A, B, C
    output: iter_0/template.nii.gz
    jobid: 4

init_template A B C iter_0/template.nii.gz
[Fri Oct  2 19:51:24 2020]

group job reg (jobs in lexicogr. order):

    [Fri Oct  2 19:51:24 2020]
    rule reg_to_template:
        input: iter_0/template.nii.gz, inputs/C.nii.gz
        output: iter_0/C_warp_0.nii.gz
        jobid: 3
        wildcards: iteration=0, image=C

    reg_to_template iter_0/template.nii.gz inputs/C.nii.gz iter_0/C_warp_0.nii.gz

    [Fri Oct  2 19:51:24 2020]
    rule reg_to_template:
        input: iter_0/template.nii.gz, inputs/A.nii.gz
        output: iter_0/A_warp_0.nii.gz
        jobid: 1
        wildcards: iteration=0, image=A

    reg_to_template iter_0/template.nii.gz inputs/A.nii.gz iter_0/A_warp_0.nii.gz

    [Fri Oct  2 19:51:24 2020]
    rule reg_to_template:
        input: iter_0/template.nii.gz, inputs/B.nii.gz
        output: iter_0/B_warp_0.nii.gz
        jobid: 2
        wildcards: iteration=0, image=B

    reg_to_template iter_0/template.nii.gz inputs/B.nii.gz iter_0/B_warp_0.nii.gz

[Fri Oct  2 19:51:24 2020]
rule update_template:
    input: iter_0/A_warp_0.nii.gz, iter_0/B_warp_0.nii.gz, iter_0/C_warp_0.nii.gz, iter_0/template.nii.gz
    output: iter_1/template.nii.gz
    jobid: 0
    wildcards: iteration=1

update_template iter_0/A_warp_0.nii.gz iter_0/B_warp_0.nii.gz iter_0/C_warp_0.nii.gz iter_0/template.nii.gz iter_1/template.nii.gz
Job counts:
	count	jobs
	1	init_template
	3	reg_to_template
	1	update_template
	5
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.