MichaelDrogalis

{"model":{"mapping":{"readyVMs":[3,2],"onlineNodes":{"0":{"sleepingVMs":[],"runningVMs":[0]},"1":{"sleepingVMs":[],"runningVMs":[1]},"2":{"sleepingVMs":[],"runningVMs":[4]},"3":{"sleepingVMs":[],"runningVMs":[]}},"offlineNodes":[]},"attributes":{"nodes":{},"vms":{}},"views":[]},"constraints":[{"vm":0,"continuous":false,"id":"running"},{"vm":1,"continuous":false,"id":"running"},{"vm":2,"continuous":false,"id":"running"},{"vm":3,"continuous":false,"id":"running"},{"vm":4,"continuous":false,"id":"running"},{"amount":1,"nodes":[0],"continuous":false,"id":"runningCapacity"},{"amount":1,"nodes":[1],"continuous":false,"id":"runningCapacity"},{"amount":1,"nodes":[2],"continuous":false,"id":"runningCapacity"},{"amount":2,"nodes":[3],"continuous":false,"id":"runningCapacity"}],"objective":{"id":"minimizeMTTR"}}

MichaelDrogalis

{
  "model": {
    "mapping": {
      "readyVMs": [
        2,
        1,
        0
      ],
      "onlineNodes": {
        "0": {

MichaelDrogalis

{
  "model": {
    "mapping": {
      "readyVMs": [
        2,
        1,
        0
      ],
      "onlineNodes": {
        "0": {

MichaelDrogalis

(->> {:onyx.core/batch [{:x 1}]}
     (t/apply-fn (λ [x] {:y 1}) false)
     :onyx.core/results
     (t/persistent-results!)
     (clojure.pprint/pprint))
;; =>                                                                                                                                                       
;; {:tree                                                                                                                                                   
;;  ({:root {:x 1}, :leaves ({:x 1, :message {:y 1}, :ack-val nil})}),                                                                                      
;;  :acks [],                                                                                                                                               
;;  :segments [],                                                                                                                                           

MichaelDrogalis

$ bin/onyx-etl --help
  -f, --from <medium>                           Input storage medium
  -t, --to <medium>                             Output storage medium
      --input-batch-size <n>                20  Batch size of the input task
      --transform-batch-size <n>            20  Batch size of the transformation task
      --output-batch-size <n>               20  Batch size of the output task
      --datomic-uri <uri>                       Datomic URI
      --datomic-partition <part>                Datomic partition to use
      --datomic-key-file <file>                 Absolute or relative path to a Datomic transformation file. See the project README for the spec.
      --sql-classname <JDBC classname>          The SQL JDBC spec classname

MichaelDrogalis

package onyxjava;

import java.util.Arrays;
import onyxjava.API;

public class OnyxJava {

    public static void main(String[] args) {
        EnvConfiguration envConfig = new EnvConfiguration();
        envConfig = envConfig.addParameter("zookeeper/address", "127.0.0.1:2188");

MichaelDrogalis

{:catalog-entry
 {:onyx/name
  {:doc "The name of the task that represents this catalog entry. Must correspond to a keyword in the workflow associated with this catalog."
   :type :keyword
   :choices :any
   :restrictions ["Must be unique across all catalog entries."
                  "Value cannot be `:none`."
                  "Value cannot be `:all`."]
   :optional? false}

MichaelDrogalis

Unison release process:

• Core project A, dependent projects B, C, and D tracking A through Voom
• [n] refers to a push

Day-to-day development:

• Push [1] to master on A
• CI triggers on push and builds [1]
• On success, lein-unison runs update-projects. Unison clones B, C, and D, checks out the freshen branch. If :merge is set, master is merged into the freshen branch. lein-unison then updates all of their versions of A to the latest version in [1]. lein-unison commits to projects B, C, and D.
• B [2], C [3], and D [4] are pushed
• CI triggers on push and builds [2]

MichaelDrogalis

91b0d12f82a3

MichaelDrogalis

Onyx 0.7.0 Benchmark Specification

Onyx SHA was 4530f76fa4644b48f2baef5d47fe103c4c0b0fec. Benchmark SHA was 3c5040dd89f3c52f5ef3b0f9a5783b44f6d76c1f. We are running on AWS to test Onyx's raw throughput independent of applications that run on top of it. We ran 1 job with 6 tasks and performance tuned it.

The job in question has 1 input task, 4 processing tasks, and 1 output task. Input messages were fabricated by a special plugin generating a 104 byte payload in each message. 4 bytes were allocated for a random integer, and 100 additional bytes were allocated as an opaque payload to simulate a medium sized message. These two values were put into a single map, which creates an Onyx segment. The segments flow through the workflow. The four processing tasks increment the integer by 1. When the message hits the output task, it is discarded. We do this because we're measuring Onyx's throughput independent of any other I/O. All benchmarks used the Aeron transport implementation with an external media driver.