lemur jobdef demoing the wait-on-step difficulties.

gistfile1.txt

(catch-args
 [:step-name
  "Set as the name of the step"
  "lemur-is-awesome"])

(defcluster the-cluster
    :app "AnApp"
    :run-id "Unique"

    :num-instances 2 ; Includes the master.
    :master-instance-type "m1.small"
    :slave-instance-type "m1.small"

    :bucket "metail-s3-scratch"
    :runtime-jar "s3://fake-location"

    :keypair "rar"
    :endpoint "elasticmapreduce.eu-west-1.amazonaws.com")

(defstep the-step
  :main-class "example.main"
  :step-name "${step-name}")

(fire! the-cluster the-step)

(println "---------Raw")
(println the-step)
(println (:step-name the-step))

(println "---------Evaluated 1")
;; Not sure how to ask for the step name from this. Attempting to use it like a
;; Clojure map causes an error.
(println (lemur.evaluating-map/evaluating-map the-step))

(println "---------Evaluated 2")
(println (evaluating-step the-cluster the-step))
(println (:step-name (evaluating-step the-cluster the-step)))

(wait-on-step the-step 864000)
; (wait-on-step (lemur.evaluating-map/evaluating-map the-step) 864000)
; (wait-on-step (evaluating-step the-cluster the-step) 864000)

Run with: lemur run test_jobdef.clj --step-name everything-is-awesome or lemur run test_jobdef.clj.

Problem 1: Running this produces a stack-overflow (an endless recursive call). That's because your defstep has a recursive definition:

{... :step-name "${step-name}"}

Asking the step for :step-name returns "${step-name}", which is then evaluated and says to ask the map for :step-name, which returns "${step-name}"... this continues indefinitely. To resolve, change the name of the command line arg to asomething else, e.g. :run-step.

Problem 2: This is what we discussed in email, defstep returns a regular map, you need an evaluating-map. Also, because you want a value from catch-args (which is part of "the-cluster"), you need to use evaluating-step as you suspected, not just evaluating-map.

I posted a corrected, annotated gist at https://gist.github.com/mlimotte/2693120f98adccbec655
Also, you can run it with lemur dry-run test_jobdef.clj. This won't test the wait-on-step, but will let you verify the step that will be run.