Clojure west notes

clojure-west.org

Clojure West

Domain driven design with Clojure

• Organizing larger applications
• Domain logic shouldn’t include handlers for bad/unclean data.
  • validateur, bouncer, clj-schema for checking data going into the pipeline.
  • domain-specific, semantic checks
  • TODO: Any better ways for doing this conditional validation.
  • Need to factor out common data cleaning utils into shared library.
• Uniform error handling above the domain layer.
  • Domain logic should throw runtime errors when applied to bad data.

Data reader’s guide to the galaxy

• Steve Miner @miner

• New in clojure 1.5, data readers and tagged literals
• EDN, like JSON for clojure
• Tagged literals
  • #my.ns/tag literal-data
  • #inst “2013-03-18”
    • Transform a string into an instance of time.
  • Loosely coupled the implementation
  • Data transfer
• Extensible Reader
  • Limited form of common lisp reader macros.
  • This all happens at read time vs compile time.
• data-readers

(binding [*data-readers* {…}])

(defn my-reader [[a b]]
  (+ ( * 10 a ) b))

(binding [*data-readers*
          {'my.ns/tag #'my-reader}]
  (read-string "#my.ns/tag [4 2]"))

; => 42

• default-data-readers
  • #uuid
  • #inst
    • represents an instance
  • (java.util.UUID/randomUUID)
• #inst
  • rfc 3339
  • #inst “2013-03-18”
  • Uses java.util.Date
• *default-data-reader-fn*
  • New in clojure 1.5
  • If your tag is not built into clojure
  • You can bind this to a fn, to decide how to handle.
  • This gets called with a tag (symbol) and a value
  • Should return a literal

(defrecord TaggedValue [tag value])

(-> TaggeedValue tag value)

• Library authors should consider:
  • #my.ns/tag semantics
  • What type of base literal should be supplied?
  • Provide some data-reader functions.
    • Often just one data-reader.
  • Provide a print method? (maybe to-str?)
    • The print method should go in the concrete type.
• Printing
  • Look at instant.clj
• Gotchas if you are defining your own data readers.
  • Returning nil throws an error.
    • Return ‘(quote nil) instead.
      • That gets returned to the compiler.
  • Don’t use a period after the / (after the your.ns)
    • Throws an error.
• read-eval Kerfuffel
  • Ruby on rails vulnerability
    • Clojure read-eval is true by default
    • #=(dangerous) can execute code
    • Not safe for reading untrusted data
    • CLJ-1153 and CLJ-904
• clojure.core/read
  • Designed by hyper intelligent, pan dimensional beings
  • read is for trusted input
  • binding *read-eval* false
    • (pre 1.5) allowed Java constructors
• clojure.edn
  • new EdnReader in clojure 1.5
  • no #=()
  • no Java constructors, no records
  • just safe EDN elements
• clojure.end/read
  • arities - [] [stream] [opts stream]
  • opts map - :eof, :readers, :default
  • defaults to default-data-readers
• Another solution with clojure.tools.reader
  • Written in Clojure
  • A complete Clojure reader
  • an EDN-only reader
  • Works with clojure 1.3
    • Safer for older code
• EDN
  • Similar to JSON
  • Extensible Data Notation
  • http://edn-format.org
  • Defines syntax for #tagged literals
  • Implementations for other languages.
• XML, lol
  • s-expressions with better marketing
• JSON
  • “The good thing about reinventing the wheel is that you can get a round one”
  • Not extensible.
• EDN vs JSON
  • extensible
  • more types
  • a little more syntax
  • conveyance of values not objects
  • slightly cheaper than the encyclopedia galactica(XML)
• Spyscope
  • http://github.com/dgrnbrg/spyscope
  • #spy/d (form)
  • better than println
  • minimally invasive
• Conditional feature reader
  • github.com/miner/wilkins
  • Check feature availability at runtime.
• Towelday.org
• Questions
  • Can you compose tags?
    • Yes

Engines of Abstraction

• Jim Duey

• Built software to analyze license plates.
  • Didn’t abstract video card, the video card got outlawed.
• Minikanren is built on top of an abstraction.
• conj function is an abstraction in clojure
  • abstract away the idea of ‘adding’ an item to a collection.
• Category theory concepts
• Monoid
  • a datatype, a function to combine values and a ‘zero’ value.
  • list, concat, ‘()
  • set, union #{}
  • hash-map, merge {}
  • function composition is a monoid
    • identity is the ‘zero’ value
• Functor
  • A ‘context’ datatype and a function (fmap) to apply a function to each value inside that context.
    • fmap takes a function of one argument
  • map is a functor
    • only works with seqable
  • functor is more general
    • could be applied to functions
• Applicative functors
  • Subset of functors
  • A ‘context’ datatype and two functions:
    • one that wraps a value in the context type `amap` applies fn(s) wrapped in the context to parameters wrapped in the context.
  • (juxt :a :b :c)
    • is an example of an applicative functor (amap (areader list) :a :b :c) where ‘areaders’ is the wrapper function
      • a b and c are functions that take your `environment`
  • Composing side-effecting functions
• Monads
  • A subset of applicative functors
  • A context data type
  • A fn to wrap values (list)
  • A fn (bind) to apply a fn to wrapped values
  • The applied fn takes a value and returns a wrapped value.
  • Examples
    • List comprehension. for
    • Any query language. LINQ
      • The semantics of a query are examples of set monad comprehensions.
• LINQ
  • Examples showing query languages, core.logic.
• Berkeley Orders of Magnitude
  • CALM Theorem - Logically monotonic programs are guaranteed to be eventually consistent.
  • Bloom - A lanaguage for writing distributed apps. Based on Datalog (recursive query language).
  • Given the equivalence, would it be possible to write distributed apps with core.logic or monad comprehensions?
  • BloomL
    • An extension of Bloom to include semi-lattices (Conflict-free Replicated Data Types)
    • Semi-Lattice - A set of values, a ‘join’ function and a special value that is less than other values.
• Comonads
  • Not a subset of Applicative Functors
  • A context data type which wraps values with one value having the ‘focus’.
  • A fn (extract) to unwrap the focused value.
  • A fn (extend) to apply a fn to wrapped values.
  • The applied fn takes a wrapped value and returns a value.
  • Examples:
    • Huet’s Zippers
    • Reactive Extensions
    • Cellular atomata
      • The game of life
      • The grid is a comonadic value
        • Values in the cells
        • Rules
        • Apply that rule to your current generation. (extend)
        • The fn (extract) that applies the rule, takes the grid, returns the focused cell. Has access to neighbors.
    • The paper that introduces this idea says that Objects are an example of comonads.
• Arrow
  • Subset of functors
    • All monads are arrows
    • All comonads are arrows
  • A context data type which wraps a function
    • A fn (arr) to wrap any fn of one arg in the context.
    • A fn (seq) to compose two wrapped fns
      • Composes two fns that the output of one is the input to another.
    • A fn (nth) to apply a wrapped fn to the nth element of a list/vector.
  • Examples:
    • Streams.
      • Storm, cascalog. Stream processing abstractions, which are arrows.
    • Recursive descent parsers aren’t very efficient with large text,
      • Arrows can be used to do lr(1) parsing.
• Abstraction using these techniques
  • Being able to lift your thinking above the mundane level.

Clojure in the large

• Stuart Sierra @stuartsierra
• Paraphrasing: APL is like a beautiful diamond, but you can’t add anything to it. Lisp is like a ball of mud. Add more and it’s still a ball of mud it looks like lisp. – Joel Moses 1970s (disputed)
• Big Ball of Mud Brian Foote and Joseph Yoder, 1999
  • Unregulated growth, expedient repair.
  • Information is shared promiscuously, all the important information becomes global or duplicated.
• Not a lot of structure in Lisp

(ns com.example.app)
; Not first-class module

(def state (ref {}))
; things are global by default

(defn private-op [arg] ...)

(defn op1 [arg] …)
; global by default

(defn op2 [arg] …)

• As programs get large, this gets harder.
  • Talking about programs, 10k lines
    • On the order of millions of lines of JAVA
  • 10s of developers
• In a smaller system, you can keep the context of the entire system in your head.
• Components serve as the building blocks for the structure of a system.
  • Pattern-Oriented Software Architecture
• Component
  • encapsulation
  • interface
  • lifecycle
• Encapsulation
  • Encapsulation deals with grouping the elements of an abstraction. And with separation.
• Problems wtih larger clojure programs.
• Global State

;; Global mutable variables

(def state-a (ref {}))

(def state-b (atom 0))

;; Temptation to share promiscuously

(defn op1 []
  (dosync
   (alter state-a …)))

• By defining a ‘singleton’ version of state, there can never be more than one.
• Created hidden dependencies.
  • Anything that refers to this namespace, depends on this state.
• If op-1 is buried deeper, you may not see the dependency.
• Reloading code destroys state.
  • (defonce state-a)
  • Cannot recover initial state.
• Trying to use let to create lexical closure around state doesn’t solve the problem.

• Try to keep your state as local as possible.
  • Define a constructor function.

;; local state

(defn constructor []
  {:a (ref {})
   :b (atom 0)})

;; define operation in terms of constructor

(defn op1 [state]
  (dosync
    (alter (:a state) …)))

• Unit testing, pass in the state you’re testing

• Antipattern 2
  • Thread bound state

;; Thread-bound state

(def ^:dynamic *resource*)

(defn- internal-op1 []
  … *resource*)

;; *resource* is a hidden dependency

(defmacro with-resource [src & body])
;; assumes that body does not return a lazy seq

• Using a macro to clean this up in a finally doesn’t help.
• Explicity limits you to one *resource* at a time.
• Local state with dynamic extent
  • with-resource is an example
  • Still assumes body completes in one thread.
• Request scope

(defn add-resource [context src]
  (assoc context ::resource (aquire src)))

(defn op1 [context]
  (let [resource (::resource context)]
    (assoc context ::result1 …)))

;; context encapsulates state
;; resource aggregates state

• Use namespace-qualified keys to avoid conflicts.
• Not confined to a single thread.
  • If you are careful.
• You still need to manage the lifecycle of these resources.
• There is some bookeeping you need to do.

• When is it okay to just def something
  • Constants ^:const
  • True singletons (rare)
    • (def runtime (Runtime/getRuntime))
  • In the repl
• Private dynamic Vars

(def ^:private ^:dynamic *state*)

(defn- internal-op1 [arg]
  ... *state* ...)

(defn op1 [arg]
  (binding [*state* (inital-state)]
    (internal-op1 arg)))

;; operation is single-threaded by definition

• Interfaces (Decoupling)
  • Using a protocol

(defprotocol KeyValueStore
  (get [store key]))

;; Public API defined in terms of primitives.

• Provide a constructor function by passing in the state.
  • (atom nil)
• As long as you stick to the public api, you know it’s going to work.
• Operations using component.
  • Going overboard, Kingdom of Nouns.
  • Define a bunch of protocols with only one operation.

• Lifecycle
  • A component has a lifecycle
    • ;; rally the troops

(defn init!
  (connect-to-the-database!)
  (create-thread-pools!))

• Create a system object.

(defrecord System
  [storage config web-service])

(defn dev-system []
  (-> System (mem-store)
             (local-config {:a 1 :b 2})
             (mock-web-service)))

(defn prod-system []
  (let [config (zookeeper-config)
        storage (sql-store config)
        service (web-srv config storage)]
  (->System storage config service)))

• System lifecycle

(defrecord System
  [storage config web-service]
  Lifecycle
  start [_]
   (start storage)
   (start web-service)
  stop [_]
   (stop web-service)
   (stop storage))

• Dislike having to restart the process.
• Global Var for REPL only
  • Using the tools namespace to reload code for repl without recompiling

• Conclusion
  • We would like our software to be building blocks.
  • Separate components
  • Encapsulate local state
  • Decouple by injecting deps
  • Manage lifecycle of components
  • Compose components into systems

Beyond Contracts

• Paul deGrandis @ohpauleez
• Motivation
  • requirements are hard
  • our tools are failing us
  • we can do better
  • a year of ideas in 30 minutes
• A story
  • 2002, time to build an addtition to the house.
    • Electric garage doors
      • Put the button on the outside of the house to get in/out.
        • I can’t just put the button on the inside of the house.
        • Gonna’ take me a week to fix it.
  • Validation - Verification
    • Did we built the right thing?
    • Did we build the thing right/correctly?
    • The earlier you find these errors, the easier to fix.
  • Software Quality Survey
    • Delivered defects
      • Things that you missed
        • 30% of errors in delivered defects happen in requirements.
      • 56% total happen before you write code.
  • Requirements are hard
    • Hammock driven development
  • Specifications
    • 0 -> 0
    • 1 -> 1
    • 2 -> ?
      • 1,000,000
    • Given the set of inputs you test for, you can’t always produce tests for every outcome.
  • Specification vs Test
    • Fundamental dualiity of software engineering
      • Bertrand Meyer
      • A test just says the program is correct for some values.
      • Specification - all values for which a program is operational.
  • Specification
    • Working definition
      • An operational descripiton of a function or type/record.
      • Operations bounded by constraints
        • functions: inputs/outputs
    • clojure.test
      • Generative testing?
      • Contracts?
        • A form of specification for runtime behavior.
      • Typing
      • core.logic or datalog
      • Alloy or Z
• We can do better
  • A composable, open system
  • common, unified backend
  • A la carte integration
  • Proactively part of experimentation and exploration
  • Integrated at all phases of software development lifecycle
  • Maximize utility, minimize manual effort
• One spec provides
  • Contracts
  • Generative tests
  • Typing information
  • Model-checking via Alloy
  • Example usage
  • Enhanced documentation
• Examples of this idea
  • Eiffel and AutoTest
  • Haskell and quickcheck
  • Alloy + DynAlloy
  • Ciao and voluntary assertions
• Core.specs

(spec/defspec tight-inc
  inc
  "An inc that only works for inputs 0-49"
  [^{:uniform [0 50] :enforced true} x]
  :constraints [["The output will always be liess than or equal to 50"
                 => (<= % 50)]]
  :typed '[[Number -> Number]])

(def pos-inc-fn (spec/fn-with tight-inc :constratints))
(pos-inc-fn 5)
;; (pos-inc-fn 50) 
;; Assertion error
(def typed-inc (spec/fn-with tight-inc :typed)
(typed-inc 5)

• :uniform for test.generative, a uniform distribution

• Alloy
  • Finds test.generative where your requirements don’t make sense.
  • Find situations where you under-specified or over-specified constraints.
• Low level of integration

;; Some definitions for a filesystem, where there is only one root.

(def -assertions
  {:one_root {:one "d: FSDir"
              :such-that "no d.parent"}})

;; Outcome from alloy
;; Unspecifiable

Design, Composition and Performance

• Rich Hickey
• Design
  • to prepare the plans for (a work to be executed) especially to plan the form or structure of.
  • To make a plan and write it down.
  • We already write down code, do we still need designs?
    • Can we generate docs from implementation?
      • Not a plan.
  • More about the planning than about the writing.
  • “Monolithic designs, ugh, been there”
    • Those are plans, not good ones.
• Good Design
  • Separating into things that can be composed.
    • Taking things apart.
  • Each component should be ‘about’ one or a very few things.
  • Compose them to solve a problem.
  • Iterative, on an ongoing basis.
• Requirements
  • Move from want/need
    • to problems
    • knowns/unknowns
    • domain-side/solution-side
    • cause/symptom
    • unstated properties of the system
      • Consistent
      • Someone can maintain it
      • Scales right
      • Statements for avoiding problems in the future.
  • Separate root cause from symptom
• Take apart time/order/flow
  • queues
  • idempotency
  • commutation
  • transactions
• Take apart place, participants
  • “All problems in computer science can be solved with another level of indirection”
  • Authors
    • Independent development, etc.
      • Separate who is doing what
        • People authoring
        • Processing at runtime.
• Information vs Mechanism
  • Set of logged-in users.
  • Set class/construct to hold that information.
    • Because we use the same stuff, we use the same mindset.
• Take apart solutions
  • Benefits
  • Tradeoffs
  • Costs
  • Problem fit
• Why design?
  • Comprehension
    • Your ability to understand through models is improved.
  • Coordination
    • Modeling helps coordination.
  • Extension
    • When your system is called upon to do something new.
  • Reuse
  • Testing
    • Design driven testing.
      • Specify things, then generate tests.
  • Efficiency
    • “I don’t have time”
      • If you don’t spend the time to specify the system.
    • It’s a lot easier to change a Graffle design, then rewrite code.
  • 90% when things don’t work about the design of a system, something wasn’t taken apart fully.
• Composition and Performance
  • Bartok and Coltrane
  • Self imposed problems/constraints
    • like other art forms
  • Music composition is designed for performers
    • ditto screenwriting, choreography
  • Organization challenge
    • A plan or design addressing those challenges
  • Fully orchestrated/arranged
    • Where everything is specified.
    • Larger scale
  • Smaller scale
    • Melody + changes
      • Increased latitude for performers
      • Increased responsibility
        • Each performer decides.
  • Software design, we work at larger scales than orchestral arrangement.
    • Relatively low level of specificity
      • Increased responsibility of performer.
• Constraints
  • Most compositions aree ‘about’ one or a few things.
    • melodic, harmonic, rythmic, timbral ideas.
    • motif or theme
      • variations
      • resolution
    • Larger works, more structural components.
  • Improvisation
    • not forseen/provided
  • Meloday + changes, provide constraints.
    • Performer provides variations.
  • Tremendous prparation, practice, study.
  • Coltrane had a vocabulary to apply.
    • Can only do that because of the preparation.
• Harmony
  • ‘accord, congruity’
  • ‘simultaneous combination (of tones)’
  • Harmonic sensibility is a key design skill.
    • If you want to be a good designer, this is the thing.
• Bartok and Coltrane
  • Masters of harmony
  • Students of harmoniousness
    • Beyond the rules
      • What does it mean for things to go together well.
  • New systems that preserve/explore harmonic essence.
    • What you don’t see is how thoughtful the relationships are behind.
• What does this have to do with clojure?
  • Clojure is like an instrument.
    • Instruments start with excitation
      • Most instruments are ‘about’ one thing.
  • Control/Interface
    • Generally about pitch
  • Projection
• Instruments are limited.
  • Piano
    • No in-between notes
  • Minimal, sufficient.
    • No missing notes.
  • Like languages, instruments are general.
• Nobody wants to play a choose-a-phone
  • Moog modular synthesizer.
• Nobody wants to compose for a choose-a-phone.
  • Complex target.
• Instruments are made for players
  • Beginners are not yet players
    • Should cellos auto-tune?
    • Red/green lights when you are in/out of tune.
  • Should not make any sound until you play it correctly?
• Imagine the horror if you downloaded a library and it gave you blisters.
  • Every musician has done way more than that to learn to use something.
• Humans are incredible
  • Learners
  • Teachers
  • Neither are effort-free.
• We are novices
  • Briefly
  • No tools, woodworking tools optimize for this period.
    • We are going to take all of our lives to get better.
• Instrumets and tools are usually for one user.
  • Designed to be weilded by one person.
• What’s the ratio between planning/performance.
  • Why as programmers do we think we can just show up.
    • Unlike nother creative people.
  • In order to be creative you have to know how to prepare to be creative.
  • Software isn’t made of wood or metal.
• Modularity
  • Moog analog modular synth.
  • Inputs of control voltage.
• There will be no music today!
  • You should use emacs.
• It’s the same mechanism all the way down for us.
  • I like to play guitar, but I’m not a luthier.
  • We all have soldering irons as programmers.
    • Tremendous distraction
      • Because we can
    • Because we can, we keep fiddling around and adding stuff to it indefinitely.
  • The paralysis of choice.
• The impetus of constraint.
  • Painters don’t paint with every color.
  • It’s up to us to constrain ourselves.
• Quit fidgeting
  • And agglomerating
  • And fiddling
  • And tweaking
• Carolina Eyck
  • Theramin
• Design is about imaginging
  • Not running away from constraints.
  • Embrace the constraints.
  • Be optimistic.
  • Imagine a lot.
    • Many times it’s the first thing you thought of.
• Design is about making decisions
  • Admit little
    • Saying no all over the place.
  • Value conveyed is in decisions made.
• Perfoming is preparing
  • Practice
  • Study
  • Developing design sensibilities you can deploy on the fly.
• Take things apart
  • with an eye twoards putting them back together.
  • Design like Bartok
    • Take the harmonic principles. How do things work together?
  • Code like coltrane.
    • Bring a deep knowledge of software and design to the problem.
  • Build languages and libs like instruments.
    • Simple designs.
  • Pursue harmony.