Skip to content

Instantly share code, notes, and snippets.

@rkaneko

rkaneko/ScalaMatsuriNotes.md

Last active August 29, 2015 14:06
Show Gist options
  • Save rkaneko/0c436183f0d9ef944c57 to your computer and use it in GitHub Desktop.
Save rkaneko/0c436183f0d9ef944c57 to your computer and use it in GitHub Desktop.
Download ZIP
ScalaMatsuri notes
Raw
ScalaMatsuriNotes.md

ScalaMatsuri

Twitter hash tag

#ScalaMatsuri
(addtional) #ra #rb
#ScalaMatsuri2
for unconference

Keynote by Martin Odersky

  • The Evolution Of Scala

10 years of Scala

from 04

Abstract type and mixin Pattern match

Pre history

80s Modula-2, Oberon 90s Functional 95 Pizza - Generics 以外の機能は評判が悪かった 98 99 Generic Java, javac 00 Functional Nets, Funnnel

Funnel 関数型ペトリネット　join計算の実装

Motivation for Scaka Grew out of Funnel Wanted to show that we can do a practical combinaaation of OOP and FP What got dropped

  • Concurrency was relegated to libraries
  • No tight connection between language and calculus What got added
  • Native object nad class model, Java interop, XML literal -> Crazy!

Why a New Language? The OO dogma ruled then: Encapsulate mutable data with methods

New at the time: WEbservices that process immutable (semi-)structured data.

  • Services sees the data "rom the outside"
  • FP is good for Webservices

Reeally, Why Two hypoothesi

  1. 汎用言語の記述能力は高くあるべき
  2. OOP と　FPを組み合わせれば高い記述力が得られるはず

Scala and Pizza Pizzawas another language on the JVM tga added functionnnal elements to Java Scala wa mmmpre ambitious More innovation on the OOP side More functional e.g. immutable values, by name parameters

Scala Beginnings

4W夜なべしながらコンパイラを作った。

Scala Reloaded 06 2.0

  • Compiler writtern in Scala
  • Followed the cake-pattern
  • Semicolon inference
  • Genreralization fo implicits and traits
  • Automcatically added empty parameter lists 2.1, 2.2
  • Qualified access
  • Multi-line string literal
  • Procedure sytax

Learning from Experience

Avoiding the Pitfalls

07 2.3-2.7 add lots of new features

  • Extractors
  • Tuples
  • Assugnment operators
  • "_" notation for functions
  • Early initialization
objecct Foo extends  {
  val x = 3
} with SomeTrait

Why the rapid growth? Peple asked for it Many valuteers help

Community Formation

07 Lift 08 First Scala liftoff uncanference 09 More Scala liftoffs 10-14 Scala Days

Gitbucket

@takezoen

  • based on Scala and JGit
  • Easy installation
  • Main target is intranet

About development

  • ssh
  • Apache MINA
  • front
  • Ace editor
  • Based on existing Java resources
  • NativeでGitとかsshのdaemonを提供している言語って他にある？
  • scalatra-forms
  • ajaxのバリデーションもPlayのFormみたいにできる。
  • アーキテクチャ的にはイケてないけど、C/Sで実装量が減る。
  • Slickはバージョンが上がるごとに、下位バージョンが切り捨てられていく。
  • Play はバックポートされる。セキュリティ面とか。
  • ZenHub
  • chrome extension
  • handle issues easier

Rapture

もともとIOライブラリ

raptureの考え方におけるリソースはなにかとれるもの？(自信ない)

Spark Session

SparkSQL

関係代数のためのexecution planning framework 論理実行計画を最適化して(Spark SQL)、物理実行計画を生成(Spark core)。

Catalyst

実行計画をたｔるためのFramework

  • Data types 通常のRDBMSと違い、入れ子のデータ形状も持てる。 Treesとか一般的にも便利！

  • Logical Operators

SparkSQL core

BigDecimalはサポートされていない。 can prune unnecessary columns when reading

Ref

Solid and Sustainable Developmnet in Scala

Scala use case in Hatena

Why Scala

  • Modern type system
  • Static typing
  • ALgebraic data type
  • Lots of libraries
  • Java libraries
  • Member's skill
  • some members had been
  • Java8
  • Not available yet
  • Haskell
  • Not many people want to program

2.1.1 -> 2.3.1までそんなに苦労なくバージョンアップ。

  • Modules コンパイルが遅いので、Moduleを細かく分けている。MVC的にも、機能的にも分けている。

  • Slickは2.03から2.1系にあげるときに苦労している。

  • QueryのLogが人に読めない。

  • git-pr-release (Jenkins plugin) deploy時の手順など、表示される。

  • Scalaんのsealed traitがId型とかUserName型とか気軽につくれて安全。Javaだと少し面倒。

Scalding, Storm & Summingbird

What's valus?

We can't survive with Rails.

Ruby -> Scala 5 years So long and Thanks 502.

Tweet per second How about handling data? Hadoop + Pig Giant UDF + hugely patched Pig -> hard! Pig -> Scalding

Scalding はググらビリティ低い。

Scaldingの良いトコロはlocalモードが合ってHadoopに使わなくても動かせる。

2種類のAPIがある。

  • Fields-base API
  • Type-safe API

Storm

Storm is a platform. Storm is similar to Hadoop, but for realtime. No data loss. Fault- blah

Storm は使いはじめるのがツライ。 niwさんのOSS使うと始めやすいらしい。

localの擬似クラスタ作って使える。

  • Spolt
  • Bolt
  • Topology
Summingbird

is a library.

Summingbird abstract platform Summingbird works for Scalding, Storm. Summingbird aggregates results from both batch and realtime jobs.

  • Producer: is a data flow abstraction
  • Platform: e.g. Storm, Scalding
  • Plan: is job -> Platform

Good point

merge results from Scalding + Storm

処理フローを単位時間にわけてそれぞれにJobを打つ。

同じデータ処理が使いまわせ結果を両方から取得できる！aggregation -> summingbird clientに入ってるかも。 Summingbirdでもlocalで試せる。けど大変。

ScalaのVersion上げるのはTwitter社でも大変。

Silkで編むデータフロー

# runnign failed tests only
sbt ~test-quick
# Monitor source code change. running specific tests.
sbt ~test-only

macroを使うと、コンパイル次のASTが取れる。

Operation in Silk from a DAG.

Silk needs declative programming. In 'Weave' step u can select platform.

Scalaのマクロに実用例から触れてみよう！

Using is easy, but developign is difficult.

Users of macro get big benefit.

macro is a Code that generates code.

Not tetual code.

Compiler invokes functions when running.

Before birth of macro ad-hoc textual code generation by sbt plugin

What's a benefit? Code is made simple. Efficiency improved

To use macro

  • import scala.language.experimental.macro
  • add a option

Notes

  • Macro are separte compilation.
  • Macro implementations need to be compiled before using macro.

Def macro prcatical example: logging logger.debug(s"Some $expensive message!") -> if (logger.isDebugEnabled) logger.debug(s"Some $expensive message!")

Macro definition

  • Starts with conditinal keyword macro
  • macro needs context as a first param.
  • as second param is c.Expr[T]. T is param u want to send

Quasquaptesを使うほうが、ASTを書くよりも楽。

Goodness

  • the hard to comprehend notion of meta programming.
  • DRY approach
  • Usage is fairly simple

Blackbox vs Whitebox why distinction? 継承を用いる場合、実行時に欲しい型予想と反する場合がある。 ->　現在の仕様では正しい。 でも気持ち悪いから、問題定期された。 その解決策がBlackbox restriction 型の厳密性がWhiteboxよりも強い。

Another example

  • Spire: is a library for numeric
  • Evaluate type idiom in string when compilation (not evaluate with string interporation)

Goodness

  • static (compiel-time) type check

Weak side current state is optimized for compiler developers not library users.

Future scala.meta

The goal of scala.meta

  • Metaprogramming is easy
  • All represented (Terms, Names, Operator) replace Trees

Manner of utilization

  • Code genration

  • Static type checking

  • etc

  • Macros are actively used in the OSS library

  • Can be more efficiently and safely programming.

  • Scala macros are evolving! (Problems are in macro between 2.10.x and 2.11)

from Ruby to Scala

一貫性と利便性のトレードオフ

The other

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment