The Strange Loop 2014
90 diversity scholarships Keynote - Joe Armstrong - Erlang Friday, he had a bad day. New version of open office. Tried to add images. All images disappeared. Apple’s Keynote was broken for him too. (diff versions not compatible) Decided to do slides in HTML Doesn’t produce nice PDFs for later download though. Found one that did, but needed grunt. Installed grunt, but wasn’t in his path. Very confusing. He’s been programming since 1965. Tom Kilburn; first modern programmer? June 21, 1948 - first program. That’s when he started work on Erlang. Compared 1985 vs 2014 machine. 2014 is much more powerful but boots slower than 1985. Why? His generation of programmers deserve medals for creating so many maintenance programmers (and jobs). What do the laws of physics say about computes? code even you cannot udnerstand a week after you wrote it. no comments code with no specifications code is shipped as soon as it runs before it is beautiful code that is very very fast very very very obscure and incorrect code that is not beautiful code that you wrote without understanding the problem Programmers who wrote the code are dead Written in archaic language which nobady understands Management thinks modifying is easy (hah) atoms in universe (big numbers) So a C program with six 32-bit integers can have more states than the number of atoms on the planet. What about Javascript? Don’t ask. A computer is a finite state machine. State x Event -> New State Failures - They will fail so you have to deal with it. Fault tolerance requires the above. Systems should self repair. In 1985 everybody* knew sh, make, C Now programmers have no common language. 2500+ languages to choose from. The Ultimate Laptop (is a 1kg black hole) Physical limits of computation. funny Entropy reverser (break 2nd law of thermodynamics) It needs DNS which can be spoofed Use hashes instead of names hash://e5fa44f2b31c1fb553b6021e7360d07d5d91ff5e How to find machine; Chord Kademlia sha1(ip) Combining ideas of git and bittorrent Merge all similar files (How?) (He’s been thinking about this for about 20 years.) Least compression difference We need to reverse entropy (make things smaller and smaller) Quantum mechanics set upper limits Let’s clean up the mess we made. React RESTful UI Rendering - Pete Hunt Every new UI at Facebook is done with React. Those who cannot remember the past are condemned to repeat it –George Santayana Evolution of Distributed Systems Message Passing is too low level. Real application has state that evolves over time. Distributed Objects (CORBA, RMI, SOAP, DCOM) too much implicit communication possible When is the computation running? multistep, multidirectional messaging -> unpredictable + slow REST (fixing the problems of distributed object) performance (throughput, user-perceived performance) REST constraints (for your success) Stateless (no client state on server) Layered (clients don’t know if you’re hitting the cache server or real server) Uniform interface (GET, POST) Evolution of UI Development Ad-hoc in the beginning; everything jumbled together State Change Observation and Events Order of events can affect the final state you reach (even if they’re the same events) How do you converge to the same state every time? EXPLICIT state transition shouldComponentUpdate can help a lot Uniform Interface (React components render() whether it’s first call or Nth call) (facebook has a new iOS framework) There are good ideas (CONSTRAINTS) in REST. Try to apply them to your non-web problem. use react-art for canvas/svg/vml how does react deal with race conditions? what’s the next frontier in UI dev? constraint-based layout systems what is the relationship between react and web components? why did he say web components aren’t restful? does facebook use immutable data structures? yes, they use immutable.js Typed Clojure in Practice Converting Clojure to Typed Clojure Ran a buggy function and let it crash Now let’s use type checker to find the bug summarise fn needs to be annotated using ann Typed Clojure is an Optional Type System for Clojure Check out Cursive Clojure, because it supports Typed Clojure TypedClojure.vim - mouse over expression gives you type of expression typed clojure is coming to clojurescript inference is getting better HMaps (type for keyword maps) (HMap :mandatory {:a Num :b Str :c Sym}) Variable-Arity Polymorphism (Is :- part of core.typed?) CircleCI is a commercial company that uses Typed Clojure If you don’t have your own lib, adopt one and annotate it! When are you going to have dependent types? Needs to understand it first. Typed racket guys are working on it. This has all happened before and it will all happen again - J. M. Barrie - Peter Pan (Mark Allen) The history of programming languages 1972 - Jean Sammet - Virtually everyone agrees that today programming is an art, not a science. Many although fewer contend that programming can and should become a science.
Is programming today more of an art or a science? first compiler was written in 1952 people didn’t initially believe something like a compiler could be written the more things change, the more things stay the same. the goal of early languages were to overcome hardware limitations Fortran Input: Reason Output: Pleasure (funny picture of boardgame) Lead Fortran Team (1954-57) Invented BNF (Backus Normal Form) National Medal of Science (1975) He was happy with how efficient Fortran code generation was language design was lacking cost saving and efficiency gains are eaten up by increases in program size and complexity Also original member of Algol committee predicted computing utilities (cloud computing) (1961) enable symbolic reasoning representation of data and expressions as lists in memory functional composition and recursion as programming idioms automatic memory management Steve Russel coded the first eval function in asm for IBM 704. McCarthy intented to have a compiler, but eval kind of changed things Interpreted programs are about 60 times slower M-expressions were supposed to replace S-expressions but that never happened Read “The Humble Programmer” 1972 code blocks with lexical scope call by value, call by name keywords (if/else/for/while) formally specified grammar lower the barrier to entry Retaking Rules for Developers Intelligent systems derive their power from the KNOWLEDGE THEY POSSESS. How do we manage this knowledge? Frankly, there were a lot of intellecutally dishonest promises made. Rule authoring is development, whether we call it that or not. Let’s approach expert systems as code Complex wiring of function inputs and outputs is complicated. Can it be done automatically RULES AS A CONTROL STRUCTURE unit-of-logic
Type[constraints]
Type[constraints]
=> action
Idris Practical Dependent Types with Practical Examples Trying to make Dependent Types practical Consistency Without Consensus - Peter Bourgon Partitions and Partitions Tolerance the system continues to operate despite mesage loss or node failure
Consistency
Availability
Partitions
CP systems - consistency protocols ? – Viewstamped Replication CP systems abstract this away AP systems allow these to happen but how do you do that without corrupting system state? CALM Principle - Consistency As Logical Monotonicity A system should only grow in one direction
Associative
Commutative
Idempotent
Distributed, sure, whatever
reuse of ACID acronymn for distributed systems
Conflict-free
Replicated
Data
Type
Increment only counter (example) Sets of User ids for counting
{ 123, 456 }
{ 123, 456 }
{ 123, 456 }
Sets are nice, because the union operation is associative, commutative, idempotent
Interlude - Bending the problem CRDTs in Production - Event Streams Event timestamp, actor, verb, thing fan out on write (every user gets an inbox) fan in on read (every producer has an outbox) the downside is the read becomes more expensive Roshi Set (new kind of set from soundcloud) S+ { A/1 B/2 } # add set
S- { C/3 D/6 } # delete set
Redis has all the primitives necessary Consistency without consensus = CRDT Maybe bend your problem, not your solution Building Realtime Analytics Systems (Initially you might think…) Load all your data into Hadoop. Query it. Done MapReduce/Hadoop has problems. It can be slow. Not optimized for query latency Need to introduce a query layer to make queries faster. What types of queries to optimize for? Then Hadoop for preprocessing and storage -> Various data stores Existing data stores were not good enough, so they made their own called Druid. Druid is optimized for reads Kafka (Loading?) (It’s a message queue of some sort) Kafka Brokers -> Strom Workers -> Druid Realtime Workers -> Druid Historical Cluster
\ /
Druid Query Broker
Difficult to handle corections of existing data Windows may be too small for fully accurate operations Hadoop was actually good at these things Hadoop might be brought back to rectify this. What this might look like Storm
Kafka Druid
Hadoop
Glue to connect these pieces together storm-kafka
Camus
Tranquility
Druid already knows how to talk to Hadoop
Scheme in the Morning - Programming Should Eat Itself - Nada Amin “It made my head hurt.” –Dan Friedman Controlling Time and Space - Evan Czaplicki Undertanding the many formulations of Functional Reactive Programming
First Order FRP - How it works in Elm Input: Explicit structure for event handling
Transform: Raw input gets transformed with lift:
“`lift : (a -> b) -> Signal a -> Signal b“`
State:
“`foldp : (a -> b -> b) -> b -> Signal a -> Signal b“`
Merge:
“`
merge : Signal a -> Signal a -> Signal a
Signals are connected to the world Signals are infinite - There’s no such thing as deleting a signal Signal graphs are static - There’s a known structure Efficiency - event comes from world, flows through graph, world is changed Architecture - elm apps have 4 parts - same general structure Hot-swapping - can change code and have it reflected right away Time-travel Debugger - go back and forth between states Higher Order FRP (alternative to core design) What if signal graphs were dynamic? “`
join : Signal (Signal a) -> Signal a
“`
creating a new signal may need infinite look back - memory growth llnear with time restrict the definition of join with fancier types Higher-order functional reactive programming in bounded space hot swapping not possible Would he use this feature even if he had it? What are the pros of dynamic signal graphs? The only core design thing is left is that signals are connected to the world Signals are not connected to world Signals graphs are again not static No way to talk about signals; What properties do I need for my application? Does the code come out nice? because javascript is a mess Future Ad Labs implemented captchas as little games (in straight js + jquery) Ideally, the game engine would be: Elm is like a user-friendly Haskell unpredictable environment (ie?) PureScript (currently impleented in Haskell) Looks like Haskell, but generates fairly light JS. has side effects system & ffi
extract the essence of map, filter et al
away from the functions that transform seqs/collections so they can be used elsewhere recasting them as process transformations ones that can be defined in terms of a succession of steps where each step ingests an input building a collection is just one instance seeded left reduce is the generalization ‘lead back’
‘carry into’
‘lead across’
on the way back/in, will carry inputs across a series of transformations Transducers in the Real World break apart pallets
remove bags that smell like food
label heavy bags
Conveyances, sources, sinks are irrelevant rules don’t care (where something comes from) Transformation in the Programming World (comp
(partial map label-heavy)
(partial filter non-food?)
(partial mapcat unbundle-pallet))
Every new collection/process defines its own versions of map, filter, mapcat et all Composed algorithms are needlessly slow they can be used with many different types into, sequence, transduce, chan etc that accept transducers use the transducer to transform their (internal, encapsulated) reducing function do their job with the transformed reducing function Lectures on Constructive Functional Programming by R. S. Bird Many list fns can be defined in terms of foldr Similarly, you can define them in terms of foldl (aka reduce) The problem with regular map/filter/mapcat fns is that they have conj in them That complects these functions with sequences Transducers are fully decoupled Know nothing of process they modify May call step 0, 1 or more times Must pass previous result as next r otherwise must know nohting of r Transducer Types, Thus Far Clojure’s reduce supports early termination via reduced Transducers must support reduced Reducing processes must also support reduced Some transducers require state Might want to do a final transformation at end of input Spreadsheets for developers - Felienne Hermans Spreadsheets are code. (people think that spreadsheets are just data) Formulas are turing complete Suffers from same problems as code You should learn spreadsheets next Lingua Franca of computing (everyone else is doing it except programmers) There’s a notation for naming things. Cap’n Proto and Rust - Type Systems for Sharing - David Renshaw Sandstorm was the project that was the motivation for Cap’n Proto All communication in sandstorm must go through capn proto is a type system for fast, robust, secure, multi-language distributed computing Will be using Rust language to demonstrate Cap’n Proto. Eight Fallacies of Distributed Computing First heard of these 8 fallacies during Java One in 1994ish? Fallacy: a wrong belief, a false or mistaken idea Most of his statements are going to be false It’s not just about the speed of light; it’s about what the other node is doing; could be busy garbage collecting Bandwidth is data over time There is One Administrator The Network is Homogenous Corba and RMI are dead or almost dead Anything that hides the network from you hides the fallacies from you Timeouts - good, bad and ugly Read the docs and TEST if the docs are true go back and watch the Jepsen II Talk Let’s be careful out there. Clojure in Unity 3D: Functional Video Game this is pre-alpha software 