Skip to content

Instantly share code, notes, and snippets.

@candera
Created August 27, 2014 00:53
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save candera/d3d13172ca6275997ae3 to your computer and use it in GitHub Desktop.
Save candera/d3d13172ca6275997ae3 to your computer and use it in GitHub Desktop.

Transit

What Problem?

  • Sending values between applications
  • Written in different languages
  • Self-describing
  • With extensibility
  • Good performance

Why not just use…?

  • JSON
  • XML
  • ___
  • Fressian/EDN
{:foo 1
 :bar {:quux [2 3 4]
       :baaz #Point [1 2]}
 :beer-inst #inst "2014-08-26T18:45"}

Basic Idea

  • EDN/Fressian approach
  • Target existing high-performance parsers
    • JSON, msgpack
  • Minimal but rich core type set
  • Tagged extensions
  • Language mapping to representations

Core

  • Ground types
    • nil/null, string, boolean, integers, floats
    • vector, map (stringable keys)
  • Included taggged
    • keyword, symbols, bigdec, uuid, set, list, instant
    • maps with composite keys

Libraries

  • Clojure
  • Java
  • Javascript
  • ClojureScript
  • Python
  • Ruby
  • Non-Cognitect
    • OCaml
    • Erlang

Examples

(require '[cognitect.transit :as transit])
(import [java.io ByteArrayInputStream ByteArrayOutputStream])

(def out (ByteArrayOutputStream. 4096))
(def writer (transit/writer out :json))

(transit/write writer [1 nil "hi" :foo])
(.toString out)
(->  "[1,null,\"hi\",\"~:foo\"]"
     .getBytes
     java.io.ByteArrayInputStream.
     (transit/reader :json)
     transit/read)

Extensibility

  • Provide write handler
    • Implement WriteHandler
  • Write handler specifies name and encoding
  • Encoding specified in terms of base types
  • Can be ignored, flows
    • Unrecognized tag yields TaggedValue
(def out (java.io.ByteArrayOutputStream. 4096))
(defrecord Point [x y])
(defrecord Size [w h])
(defrecord Rectangle [origin size])
(defn size-rep [size]
  [(:w size) (:h size)])
(defn point-rep [pt]
  [(:x pt) (:y pt)])
(defn rectangle-rep [rect]
  [(:origin rect) (:size rect)])
(def writer
  (transit/writer out
                  :json
                  {:handlers {user.Point (transit/write-handler "Point"
                                                                point-rep)
                              user.Size (transit/write-handler "Size" size-rep)
                              user.Rectangle (transit/write-handler "Rect"
                                                                    rectangle-rep)}}))
(transit/write writer (Point. 3 17))
(transit/write writer (Rectangle. (Point. 0 1) (Size. 2 3)))
(.toString out)
(let [out (ByteArrayOutputStream. 4096)
      write-handlers {user.Point (transit/write-handler "Point"
                                                        point-rep)
                      user.Rectangle (transit/write-handler "Rect"
                                                            rectangle-rep)}
      read-handlers {"Rect" (transit/read-handler (fn [[origin size]]
                                                    {:o origin :s size}))
                     ;; "Point" (transit/read-handler (fn [[x y]]
                     ;;                         {:x x :y y}))
                     }
      writer (transit/writer out
                             :json
                             {:handlers write-handlers})]
  (transit/write writer (Rectangle. (Point. 0 1) (Point. 2 3)))
  (-> out
      .toByteArray
      java.io.ByteArrayInputStream.
      ;; Note: no Point reader
      (transit/reader :json {:handlers read-handlers})
      transit/read))

Dynamic Caching

  • Lots of repeated keys
  • Fixed-size circular rotation cache
    • size ???
  • Keywords/symbols cached everywhere
  • Strings cached when map keys
  • Reader must support, writer may cache
(def out (ByteArrayOutputStream. 4096))
(def writer (transit/writer out :json))
;; (def writer (transit/writer out :json-verbose))
(transit/write writer (into [] (repeat 3 {"long" "hi"
                                          "s" "bye"})))
(.toString out)

Examples

(defn transit [val]
  (let [out (ByteArrayOutputStream. 4096)
        writer (transit/writer out :json)]
    (transit/write writer val)
    (format "%s => %s" (pr-str val) (.toString out))))
(transit {"a" 1 "b" 2})

Level 1

Here is some prose. I have emphasis and italics. And I can do code.

NameAgeAge in Months
Craig42504
Jason35420
0

Org mode tricks

A little helper code:

(defn thresholds
  "Given a seq of thresholds in the range 0.0 to 1.0, and a collection
  of data, sorts the collection and returns the values that are at the
  positions indicated by the thresholds. E.g. (thresholds [0.5 0.75]
  (range 10)) returns {0.5 4, 0.75 6}."
  [tholds coll]
  (let [sorted (-> coll sort vec)
        n (count sorted)]
    (into {}
          (for [thold tholds]
            [thold (nth sorted (* thold n))]))))

I want to measure this function:

(defn bar [n]
  (rand-int (inc (* 2 n))))

Here’s our experiment

(->> (repeatedly 100 #(bar n))
     (thresholds [0.5 0.9 0.99])
     (into (sorted-map))
     vals
     (into [(str (java.util.Date.))
            n
            [:times-2 :plus-1]]))
Datenvariations50%90%99%
Tue Aug 26 20:33:06 EDT 2014478999
Tue Aug 26 20:34:04 EDT 2014438799
Tue Aug 26 20:34:05 EDT 2014548999
Tue Aug 26 20:34:06 EDT 2014549199
Tue Aug 26 20:35:33 EDT 2014(:times-2)100185198
Tue Aug 26 20:37:09 EDT 2014(:times-2 :plus-1)112182200
Tue Aug 26 20:38:12 EDT 2014100(:times-2 :plus-1)97185199
Tue Aug 26 20:38:17 EDT 20141000(:times-2 :plus-1)107218552000

Variations:

NameExplanation
times-2Multiply operand by two
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment