outcome-oriented-programming.md

Outcome-Oriented Programming

Mike McNeil, Aug 2014

Humans are not very good at planning. We have no problem running scenarios, thinking through possibilities, and pondering "what if?" questions. I might plan to not eat my cousin's birthday cake before she gets home, for instance. If I'm very serious, I might write down my commitment; or if I'm unsure about the pros and cons, use some organizational tool like a T-chart.

But when it comes to making a decision in the moment, all bets are off. The cake is a goner.

Predictive Analysis vs. Process Design

Below, I've included a figure containing a decision tree diagram.

source: http://en.wikipedia.org/wiki/Decision_tree#Decision_tree_using_flow_chart_symbols

This is not a program; it's a hypothetical analysis.

For our human thought processes, this works great! Each node in the graph contains a word that has meaning for us, and in combination with the directed edges of the graph, it all makes perfect sense. It is an efficient way to encode an analysis constructed to advise a squishy, neural, inductive decision-making process.

But when it comes to programming, we have different priorities: maintainability, security, reliability, performance, and simplicity. With the at-times conflicted nature of these goals, it's amazing we get anything done at all. So where to start?

Reusability

Above all else, it is my position that the wellspring of most of these architectural virtues is reusability. If a module is truly reusable, it can be used by many different developers; allowing that single code base to be optimized for security and performance. Reusability eliminates the need for endless duplication in programs that do almost exactly the same thing, and makes the individual business logic that must be written simpler and more maintainable; precisely because the reusable parts become familiar to developers across disciplines and corporate walls.

Unfortunately, if we were to print the decision tree diagram above onto construction paper, then snip it into pieces, I would end up with word salad:

Case, Proceed, 40%, 50K, 60%, Loss, Win, Costs, Costs, 80%, 20%, 80%, 20%, Zero, $100k, -$100k, Zero, Damages, 5%, 40%, 55%, $500k, $50k, Zero, Proceed

Even if you maintain the order, this jumble of concepts doesn't make a whole lot of sense. And it's particularly hard to see how the concept would be reusable as an instruction or even a decision.

Chopping Code For The Parts

The same experiment could be performed on the source code for most traditional software applications. Print each file out, snip it up into pieces (each line on its own slip of paper), and see if any of them make sense individually:

if (password.length<6) {, alert('your password is too short');, } else {, $('.signup').submit() }

This isn't as bad because we have more context (more words) in each chunk. But it's still pretty useless.

But as you may have gathered, I'm leading us somewhere- turns out there are some interesting patterns buried in the way you can split up and "understand" imperative "algorithms".

Let's split the code salad above up a bit further.

Instead of dumbly snipping line breaks, this time we'll convert into a context-free dialect of LISP-y pseudo-code, identify lambda expressions and separate variables and constants from the operations which call them.

Importantly, we eliminate so-called "structured programming" constructs like "for" and "if". Only function calls, variables, and constants are allowed. The result is that we end up with something that looks a lot more like assembly code; but instead of "GOTO" statements, we use callbacks.

(alert MSG)

(if CONDITION THEN_CALLBACK ELSE_CALLBACK)

"your password is too short"

($ SELECTOR)

".signup"

($.fn.submit JQUERY_COLLECTION) 

PASSWORD

(length SOME_STRING)

6

(< X Y)

Much better! We still don't have any certainty on how these different things are connected, but as atomic parts, they are much more meaningful.

Before we continue, let's categorize the parts we have:

Instructions
($ SELECTOR)
(< X Y)
(length SOME_STRING)
($.fn.submit JQUERY_COLLECTION)
(alert MSG)
(if CONDITION THEN_CALLBACK ELSE_CALLBACK)

Constants
6
".signup"
"your password is too short"

Free Variables
PASSWORD

We now have an abstract instruction set (or "grammar") of reusable symbols from our program.

But we're still missing a key element.

Documenting Our Grammar

Before we move on, let's document the return value for each of our instructions, and their inputs (we'll treat our special if/then callbacks as ordinary inputs for now)

While we're at it, we'll make a note whether the instruction is "referentially transparent" (or "nullipotent"); meaning it has no side effects.

($ SELECTOR)

No side effects.

Returns: ((jquery_element))

Input	Type
SELECTOR	((string))

(< X Y)

No side effects.

Returns: ((boolean))

Input	Type
X	number
Y	number

(length SOME_STRING)

No side effects.

Returns: ((number))

Input	Type
SOME_STRING	((string))

($.fn.submit JQUERY_COLLECTION)

Returns: void

Input	Type
JQUERY_COLLECTION	jquery_element

(alert MSG)

Returns: void

Input	Type
MSG	((string))

(if CONDITION THEN_CALLBACK ELSE_CALLBACK)

No side effects.

Returns: void

Input	Type
CONDITION	((boolean))
THEN_CALLBACK	((function))
ELSE_CALLBACK	((function))

Challenges and Bugs

So how would we reconstruct a program using our new piecemeal components?

We don't have quite enough information in our grammar to do this with 100% confidence. Even though we know what types of data our instructions accept, there would be some guesswork involved. Perhaps most important is that we would need special code to handle certain sequences of instructions-- they don't all just play nicely together.

Type Variability

In loosely-typed language, there is no guarantee the value returned by a function will be anything close to whatever it may have promised. A property might not exist in an object where it was expected, or an array might be empty, or the type of a variable could just be flat-out wrong.

"Edge Cases"

There are always contingencies we don't immediately consider when designing programs.

This is half-solved in certain statically-typed languages through the use of typed exceptions and soft declarations (e.g. @throws). However, exceptions are synchronous and can be (depending on the platform) computationally expensive.

For instance, what if the jQuery selector function cannot find an element which matches the specified selector? Or what if it finds multiple elements? What if the form fails to submit because the user's browser is offline? Or if the server responds with an error? What kind of error?

The Snowball Effect

More troublesome still, these "edge cases" have a way of snowballing:

• What happens if a matching jQuery element cannot be found and then you try to submit the form?
• What if multiple elements are found- do we submit multiple forms? (if so, that's fine, but we never thought about it until now)
• What happens if multiple form elements were found and the server returned an error for one of them but not another?
• What happens if a form element doesn't have a valid form action? What if the URL in that action violates the same-origin policy? Or if it tries to talk to https, but we're on http?
• and so on...

As you can see, this is undeniably the source of software bugs.

So how can we improve the situation?

Introducing Outcome-Oriented Programming

Let's identify the "what if?"s. For each of our instructions (and making the assumption that valid input data types were passed in) we'll consider if there is ANY possibility at all that some other outcome than the expected default successful "exit" might occur.

When defining outcomes (or "exits"), we must first determine an identifier (e.g. "success" or "notFound"). Next, we'll identify an output type. Instead of a single return value, each exit of an outcome-oriented function has its own distinct data type.

Outcome Induction

The generic heuristics below are useful for identifying hidden exits in any instruction:

Anticipate IO errors (i.e. network/disk/child processes)

A general best-practice is to expose a catch-all error exit for any asynchronous operation that communicates with a different process or thread, or with IO hardware that is capable of failing based on various logical or physical error cases (like the network card and hard disk).

For instance, in node.js, that means any write or read operations in fs, http, net, etc.

In the browser, that means HTTP requests, WebSocket connections, etc.

================================================================== ($.fn.submit JQUERY_COLLECTION)

Input	Type
JQUERY_COLLECTION	jquery_element

Exit	Returns
SUCCESS	void
ERROR	((string))

==================================================================

Databases and Shared Memory usually have error exits with special meaning

Many programs need to look-up and/or modify data in files, shared memory state, database tables, etc.

You can sometimes get away with defining a catch-all "error" exit in these cases, but it is generally better for your exits to be more specific and logically useful. Consider: could a non-programmer use the interface you're designing?

Examples:

1. an instruction which does a findOne operation in a database might define a "notFound" exit
2. an instruction which does a create operation in a database with constraints might define an "invalid" exit
3. an instruction which attempts to perform a custom query in a remote LDAP system might define "forbidden", "unauthorized", "notFound", AND "invalid" exits, in addition to "error" and "success" (however realize that this example instruction is probably too complex-- better to create multiple, more specific instructions. Don't forget the data type of a given exit must be consistent for all input combinations)
4. an instruction which sends an AJAX request to a particular custom API endpoint that does exactly what was described in example #2. In that case, this browser-side wrapper of the back-end instruction would have its own "invalid" exit. And if necessary, it could split its catch-all unexpected network "error" exit into two: "serverToDatabaseNetworkError" and "clientToServerNetworkError". It is rare that you would do this, but important to understand the pattern.

Even the simple example we've been working with in this document has two parts that deserve sensible custom logical error exits:

First, we'll handle the case where no matching form element exists in the DOM.

================================================================== ($ SELECTOR)

No side effects.

Input	Type
SELECTOR	((string))

Exit	Returns
SUCCESS	((jquery_collection[]))
NOTFOUND	void

==================================================================

Note that we could have added an additional exit for when there are multiple form elements, but instead, we opted to change the output type of the success exit to a jquery_collection, and then change the input type of $.fn.submit below to match. Generally, it is a more flexible/reusable approach to accept sets of things instead of just one thing.

Next, we'll add a "invalid" exit to our form submission instruction:

================================================================== ($.fn.submit JQUERY_COLLECTION)

Submit a form to the server.

Input	Type
JQUERY_COLLECTION	((jquery_collection[]))

Exit	Returns
SUCCESS	void
ERROR	((string))
INVALID_DATA	((string))
INVALID_URL	((string))

==================================================================

Watch Out For Math

There is no need to examine this particular heuristic in our example since we're not using any mathematical operators that could return an inconsistent type, but for reference, a good example of a math instruction that requires an additional exit is: (/ 2 0)

Factor out callbacks (i.e. inputs with type: function)

So what about if/then? The callbacks being passed for THEN and ELSE could really anything! How can we describe the exit conditions if we don't even know what those functions will do?

We can't. Instead, those function inputs should be replaced with exits.

This allows us to redefine IF as follows:

==================================================================

(if CONDITION THEN_CALLBACK ELSE_CALLBACK)

No side effects.

Input	Type
CONDITION	((boolean))

Exit	Returns
THEN	void
ELSE	void

==================================================================

Wiring Up a Program With Machines

Now that we have these special instructions which keep track of all their possible outcomes (called machines), we could create many different programs from these symbols, all which accomplish different tasks without any additional information, context, type-checking, or undefined states.

For instance, our jQuery selector instruction will never try to continue to the instruction attached to its "success" exit if a matching DOM element was not found, no matter what we're building! That potential bug is gone, forever.

In Pseudocode

In this javascript-esque pseudocode, we use a pretend VALUE_ON_LINE_X variable to demonstrate how previous result values can be used in the inputs of subsequent instructions (this is rather than defining custom variable names, which confuses the topic) For example: VALUE_ON_LINE_3

Let's look at every possible scenario (there are 6):

A. Password too short

This always happens if PASSWORD is less than 6 characters long, since alert only has a single exit (SUCCESS).

1            | if (PASSWORD.length < 6)
 •---THEN--->| 
2            | alert("your password is too short")
 •-------SUCCESS-----> 

B. No form element found

This happens when PASSWORD is at least 6 characters long, but jQuery can't find any elements in the DOM that match ".signup".

1            | if (PASSWORD.length < 6)
 •---ELSE--->| 
2            | $(".signup")
 •------NOTFOUND------>

C. Success

This happens when PASSWORD is at least 6 characters long, jQuery finds at least one element in the DOM that matches ".signup", and then after submitting the data from that form to the URL it specifies, the server responds with a 2xx status code letting us know that whatever the form was supposed to do must have worked.

1            | if (PASSWORD.length < 6)
 •---ELSE--->| 
2            | $(".signup")
 •--SUCCESS->| 
3            | $.fn.submit(VALUE_ON_LINE_2)
 •------SUCCESS------->

D. Network error

Same as C. Success but final exit is •--ERROR->.

E. Invalid data

Same as C. Success but final exit is •--INVALID_DATA->.

F. Invalid url

Same as C. Success but final exit is •--INVALID_URL->.

Look familiar? That's because each of these scenarios is an exit!

Our program is now a machine itself. And any free variables it has (in this example, PASSWORD) are its inputs. So it has 1 input and 6 exits.

Compiling Code

There are many different ways to visualize this, and the process of compiling the concepts discussed in this document to real code is not trivial (fortunately, Treeline takes care of this for us).

Here is a hand-compiled version of our example program in Javascript:

Note: writing it out this way took about 20 minutes, and the code is not nearly as clean or error-proof as what you'll get back from the Treeline compiler (two big differences are that (1) my hand-compiled code has no code comments and (2) I'm not doing any optimizations or reducing the number of function wrappers based on context). Hopefully, dear reader, the hand-compiled version helps you see how this all works.

function ourExampleProgram (masterInputs, masterExits) {
  if (masterInputs.password.length < 6) {
    alertMachine({MSG: "your password is too short"}, {
      success: function (){
        return masterExits.PASSWORD_TOO_SHORT();
      }
    });
  }
  else {
    jQueryGetElement({selector: '.signup'},{
    	SUCCESS: function ($form){
    	  jQuerySubmitForm({ JQUERY_COLLECTION: $form }, {
            SUCCESS: masterExits.SUCCESS,
            ERROR: masterExits.ERROR,
            INVALID_DATA: masterExits.INVALID_DATA,
            INVALID_URL: masterExits.INVALID_URL
          })
    	},
    	NOTFOUND: masterExits.NO_FORM_ELEMENTS_FOUND
    })
  }
}


// Machine definitions:
////////////////////////////////////////////////////////////////////////////

function alertMachine(inputs, exits) {
  alert(inputs.MSG);
  return exits.SUCCESS();
}

function jQueryGetElement (inputs, exits){
  var $elements = $(inputs.SELECTOR);
  if ($elements && $elements.length) return exits.SUCCESS($elements);
  else return exits.ERROR();
}

function jQuerySubmitForm(inputs, exits){
  // TODO: handle case where JQUERY_COLLECTION is an array of form elements

  var formUrl = inputs.JQUERY_COLLECTION.attr('action'));
  if (typeof formUrl !== 'string' || !formUrl.match(/^http/)) {
    return exits.INVALID_URL;
  }
  inputs.JQUERY_COLLECTION submit(function (body,res) {
    if (res.statusCode === 400) return exits.INVALID_DATA(body);
    else if (res.statusCode > 400 || res.statusCode < 200) return exits.ERROR(body);
    else return exits.SUCCESS();
  }
}

// "if" and "<" machines omitted for brevity

Note that, since the compiler understands primitive operators like < and if in every relevant output language, the resulting code contains normal imperative usage of those machines. Note that at design-time, primitive operators like these work just like any other machine.

Finally, remember that the output code is just a representation of the abstract logic you are programming; and it is a lossy transformation.

Conclusion

Orchestral music from centuries ago can still be performed today (even though it was never recorded on tape) thanks to the declarative encoding of the notes and which instruments should be playing them (the "parts") as sheet music. In fact, some might even say that the "sheet music" is the "music", perhaps more so than the orchestra performing it.

Much in the same way, by designing our programs as circuits of inputs and exits performed by reusable, adaptable machines, we are able to compose software which "works according to plan" because it is the "plan".