stimata-debug

Requirements

• no upfront installation/agents on remote/slave machines - ssh should be enough
• application components should use third-party software, e.g. HDFS, Spark's cluster, deployed separately
• configuration templating
• environment requires/asserts, i.e. we need a JVM in a given version before doing deployment
• deployment process run from Jenkins

Solution

stimata-debug

Awesome Talks -- Watch it!

1. Deep Dive: Apache Spark Memory Management - An excellent talk about Spark's memory management in the past releases and the upcoming 2.0. No code. The slides were awesome with a superb presentation style. Very informatory.
2. A Deep Dive Into Structured Streaming -- a superb talk about the upcoming Structured Streaming in Spark 2.0.
3. Structuring Spark: Dataframes, Datasets And Streaming -- another superb talk about the reasons for structuring Spark using Datasets by the one and only Michael Armbrust.
4. Large-Scale Deep Learning with TensorFlow by Jeff Dean (Google) -- just yesterday I was thinking about feature vectors and how close they map to the real objects (they are supposed to represent) and that gave me the Aha moment that the more features the better but you need to be careful with over-featuring the m

stimata-debug

Introducting Apache Spark

• What use cases are a good fit for Apache Spark? How to work with Spark?
  • create RDDs, transform them, and execute actions to get result of a computation
  • All computations in memory = "memory is cheap" (we do need enough of memory to fit all the data in)
    • the less disk operations, the faster (you do know it, don't you?)
  • You develop such computation flows or pipelines using a programming language - Scala, Python or Java <-- that's where ability to write code is paramount
  • Data is usually on a distributed file system like Hadoop HDFS or NoSQL databases like Cassandra
  • Data mining = analysis / insights / analytics
• log mining

stimata-debug

How much of machine learning is statistics and vice versa?

Learning using https://www.coursera.org/learn/machine-learning/home/welcome

• machine learning = teaching a computer to learn concepts using data — without being explicitly programmed.
• Supervised learning = "right answers" given
• Regression problem
  • continuous valued output
  • deduce the function for a given data set and predict other values
• "in regression problems, we are taking input variables and trying to map the output onto a continuous expected result function."

stimata-debug

aka "Let's take some notes about using Docker on Mac OS X to turn deployment of Scala applications into a much better experience."

DISCLAIMER The doc is a compilation of different articles and videos found on the Internet. Almost nothing's mine - mostly layout. See CREDITS section below to know who to praise. All mistakes are mine and are not intended. Drop me an email at jacek@japila.pl if you spot any errors or just share what you think about the doc.

The document lives at https://gist.github.com/jaceklaskowski/ca55be80cb76e84ce478

Intro

I'm on Mac OS X and so you're going to see a lot of setup tweaks for the platform that are not necessarily needed for your environment, esp. Linux one. When you see boot2docker and you're on Linux, just disregard the line or even entire paragraph.

stimata-debug

About Cake Pattern

Below are the sources I took notes and hope to learn about Cake Pattern from.

Videos:

• SF Scala, Peter Potts: Cake Pattern in Practice
• Cake Pattern: The Bakery from the Black Lagoon
• Dead-Simple Dependency Injection

stimata-debug

Rough Notes about CQRS and ES

Once upon a time…

I once took notes (almost sentence by sentence with not much editing) about the architectural design concepts - Command and Query Responsibility Segregation (CQRS) and Event Sourcing (ES) - from a presentation of Greg Young and published it as a gist (with the times when a given sentence was heard).

I then found other summaries of the talk and the gist has since been growing up. See the revisions to know the changes and where they came from (aka the sources).

It seems inevitable to throw Domain Driven Design (DDD) in to the mix.

stimata-debug

Effective Engineer - Notes

• By Edmond Lau
• Highly Recommended 👍
• http://www.theeffectiveengineer.com/

What's an Effective Engineer?

• They are the people who get things done. Effective Engineers produce results.

Adopt the Right Mindsets

stimata-debug

SPC
    SPC: find file
    , switch buffer
    . browse files
    : MX
    ; EX
    < switch buffer
    ` eval
    u universal arg
    x pop up scratch