Asakusa SCR #1 - draft

asakusa-scr_01.rst

• Tag
  • https://github.com/ashigeru/asakusafw/tree/SCR-01/

Table of Contents

• 1   DMDL Compiler
  • 1.1   Projects
  • 1.2   Preliminary
  • 1.3   Overview
  • 1.4   Entry Point
  • 1.5   Tasks
  • 1.6   Extension Points
  • 1.7   Syntax Analysis (JavaCC)
  • 1.8   Semantic Analysis
  • 1.9   Semantic Analysis Plug-ins
  • 1.10   Code Generation
  • 1.11   Java Syntax Library
  • 1.12   Code Generator Plug-ins
• 2   Operator DSL Compiler
  • 2.1   Projects
  • 2.2   Overview
  • 2.3   Preliminary
  • 2.4   Entry Point
  • 2.5   Operator Extensions
  • 2.6   Syntax Analysis
  • 2.7   Semantic Analysis
  • 2.8   Code Generation
    • 2.8.1   Code Generation (Factory)
    • 2.8.2   Code Generation (Impl)
  • 2.9   Other Extension Points
• 3   Flow DSL Compiler
  • 3.1   Projects
  • 3.2   Overview
  • 3.3   Entry Point
  • 3.4   Description Analysis (batch)
  • 3.5   Semantic Analysis (batch)
  • 3.6   Entry Point (flow)
  • 3.7   Description Analysis (flow)
  • 3.8   Semantic Analysis (flow)
  • 3.9   Code Generation (stage)
  • 3.10   Code Generation (flow)
  • 3.11   Code Generation (batch)
  • 3.12   Extensions Points

1   DMDL Compiler

1.1   Projects

• https://github.com/ashigeru/asakusafw/tree/SCR-01/asakusa-dmdl-java (entry point)
• https://github.com/ashigeru/asakusafw/tree/SCR-01/asakusa-dmdl-core

1.2   Preliminary

Flow:

1. Syntax Analysis
2. Semantic Analysis
3. Code Generation

Models:

1. Source Code
2. Abstract Syntax Tree
3. Semantic Graph
4. Target Code

1.3   Overview

• *.dmdl
• DmdlParser
• AstScript
• DmdlAnalyzer
• ModelDeclaration
• JavaModelClassGenerator
• *.java

1.4   Entry Point

• com.asakusafw.dmdl.java.Main (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/Main.java
• #main:L94
  • L96 -> #configure
  • L98 -> GenerateTask#process

1.5   Tasks

• com.asakusafw.dmdl.java.GenerateTask (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/GenerateTask.java
  • L83 -> AnalyzeTask#process
• com.asakusafw.dmdl.util.AnalyzeTask (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/util/AnalyzeTask.java

1.6   Extension Points

• com.asakusafw.dmdl.util.AnalyzeTask (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/util/AnalyzeTask.java
  • L108: ServiceLoader#load(TypeDriver.class)
  • L109: ServiceLoader#load(AttributeDriver.class)
• ServiceLoader: http://download.oracle.com/javase/6/docs/api/java/util/ServiceLoader.html

1.7   Syntax Analysis (JavaCC)

• com.asakusafw.dmdl.util.AnalyzeTask (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/util/AnalyzeTask.java
  • L119 -> DmdlParser#parse
• com.asakusafw.dmdl.parser.DmdlParser (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/parser/DmdlParser.java
  • L50 -> JjDmdlParser#parse
• src/main/javacc/DmdlParser.jj
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/javacc/DmdlParser.jj
  • JavaCC: https://javacc.dev.java.net/

1.8   Semantic Analysis

• com.asakusafw.dmdl.util.AnalyzeTask (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/util/AnalyzeTask.java
  • L122 -> DmdlAnalyzer#addModel
  • L81 -> DmdlAnalyzer#resolve
• com.asakusafw.dmdl.analyzer.DmdlAnalyzer (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/analyzer/DmdlAnalyzer.java
  • #addModel:L111
    • L128 -> #computeDependencies:L128->L132
  • #resolve:L148
    • L150 -> #resolveSymbol:L165 (using topological sort)
    • L152 -> #resolveAttribute:L573 (using AttributeDriver)

1.9   Semantic Analysis Plug-ins

• com.asakusafw.dmdl.analyzer.DmdlAnalyzer (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/analyzer/DmdlAnalyzer.java
  • L598 -> AttributeDriver#process
• com.asakusafw.dmdl.spi.AttributeDriver
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/spi/AttributeDriver.java
• src/main/resources/META-INF/services/com.asakusafw.dmdl.spi.AttributeDriver (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/resources/META-INF/services/com.asakusafw.dmdl.spi.AttributeDriver
• plug-in: com.asakusafw.dmdl.analyzer.driver.NamespaceDriver (core)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-core/src/main/java/com/asakusafw/dmdl/analyzer/driver/NamespaceDriver.java

1.10   Code Generation

• com.asakusafw.dmdl.java.GenerateTask (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/GenerateTask.java
  • L76 -> JavaModelClassGenerator#emit
• com.asakusafw.dmdl.java.emitter.JavaModelClassGenerator (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/emitter/JavaModelClassGenerator.java
  • L78 -> ConcreteModelEmitter#emit
• com.asakusafw.dmdl.java.emitter.ConcreteModelEmitter (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/emitter/ConcreteModelEmitter.java

1.11   Java Syntax Library

• javalang-tools
  • https://github.com/ashigeru/javalang-tools/tree/master/java-dom
  • Manipulates Java ASTs

1.12   Code Generator Plug-ins

• com.asakusafw.dmdl.java.emitter.ConcreteModelEmitter (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/emitter/ConcreteModelEmitter.java
  • L112 -> JavaDataModelDriver#getTypeAnnotations
  • L124 -> JavaDataModelDriver#getInterfaces
  • L133 -> JavaDataModelDriver#getMethods
  • L133 -> JavaDataModelDriver#getMemberAnnotations
• com.asakusafw.dmdl.java.spi.JavaDataModelDriver (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/spi/JavaDataModelDriver.java
• src/main/resources/META-INF/services/com.asakusafw.dmdl.java.spi.JavaDataModelDriver (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/resources/META-INF/services/com.asakusafw.dmdl.java.spi.JavaDataModelDriver
• example: com.asakusafw.dmdl.java.emitter.driver.WritableDriver (java)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dmdl-java/src/main/java/com/asakusafw/dmdl/java/emitter/driver/WritableDriver.java

2   Operator DSL Compiler

2.1   Projects

• https://github.com/ashigeru/asakusafw/tree/SCR-01/ashigel-compiler (entry point)
• https://github.com/ashigeru/asakusafw/tree/SCR-01/asakusa-dsl-vocabulary

2.2   Overview

• *.java
• javac (JDK)
• TypeElement (javax.lang.model.element)
• OperatorClassCollector
• OperatorClass
• OperatorClassEmitter
• *Factory.java, *Impl.java

2.3   Preliminary

• JSR 269: Pluggable Annotation Processing API
  • http://www.jcp.org/en/jsr/detail?id=269
• src/main/resources/META-INF/services/javax.annotation.processing.Processor
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/javax.annotation.processing.Processor

2.4   Entry Point

• com.asakusafw.compiler.operator.OperatorCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorCompiler.java
  • #init:L70
    • L77 -> #loadSubProcessors:L98
  • #process:L178
    • L188 -> #start:L198

2.5   Operator Extensions

• com.asakusafw.compiler.operator.OperatorProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorProcessor.java
  • #describe:L64
  • #implement:L72
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.operator.OperatorProcessor
• example: com.asakusafw.compiler.operator.processor.UpdateOperatorProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/processor/UpdateOperatorProcessor.java

2.6   Syntax Analysis

(by javac)

2.7   Semantic Analysis

• com.asakusafw.compiler.operator.OperatorCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorCompiler.java
  • L204 -> OperatorClassCollector#add
  • L206 -> OperatorClassCollector#collect
• com.asakusafw.compiler.operator.OperatorClassCollector (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorClassCollector.java
  • #add:L76
  • #collect:L141

2.8   Code Generation

• com.asakusafw.compiler.operator.OperatorCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorCompiler.java
  • L211 -> OperatorClassEmitter#emit
• com.asakusafw.compiler.operator.OperatorClassEmitter (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorClassEmitter.java
  • L121 -> OperatorFactoryClassGenerator#generate
  • L92 -> OperatorImplementationGenerator#generate

2.8.1   Code Generation (Factory)

• com.asakusafw.compiler.operator.OperatorFactoryClassGenerator (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorFactoryClassGenerator.java
  • L118 -> OperatorProcessor#describe

2.8.2   Code Generation (Impl)

• com.asakusafw.compiler.operator.OperatorImplementationClassGenerator (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/OperatorImplementationClassGenerator.java
  • L89 -> OperatorProcessor#implement

2.9   Other Extension Points

• com.asakusafw.compiler.operator.DataModelMirrorRepository
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/operator/DataModelMirrorRepository.java
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.operator.DataModelMirrorRepository

3   Flow DSL Compiler

3.1   Projects

• https://github.com/ashigeru/asakusafw/tree/SCR-01/ashigel-compiler-bootstrap (entry point)
• https://github.com/ashigeru/asakusafw/tree/SCR-01/ashigel-compiler
• https://github.com/ashigeru/asakusafw/tree/SCR-01/asakusa-dsl-vocabulary

3.2   Overview

4-Layered:

• Batch
  • Jobflow
    • External I/O
    • Stage (= MapReduce Job)
      • Map/Reduce
      • Shuffle

Flow:

• Description Analysis (Batch DSL)
• Semantic Analysis (transaction planning)
  • Description Analysis (Flow DSL)
  • Semantic Analysis (jobflow: stage planning)
    • Semantic Analysis (stage: shuffle planning)
      • Code Generation (Map/Reduce)
      • Code Generation (Shuffle)
    • Code Generation (stage)
    • Code Generation (external I/O)
  • Code Generation (jobflow)
• Code Generation (batch)

3.3   Entry Point

• com.asakusafw.compiler.bootstrap.BatchCompilerDriver (boot)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler-bootstrap/src/main/java/com/asakusafw/compiler/bootstrap/BatchCompilerDriver.java
  • L214 -> DirectBatchCompiler#compile
• com.asakusafw.compiler.testing.DirectBatchCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/testing/DirectBatchCompiler.java

3.4   Description Analysis (batch)

• com.asakusafw.compiler.testing.DirectBatchCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/testing/DirectBatchCompiler.java
  • L88 -> BatchDriver#analyze
• com.asakusafw.compiler.batch.BatchDriver (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/BatchDriver.java
  • L127 -> BatchDescription#start
• com.asakusafw.vocabulary.batch.BatchDescription (voc)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dsl-vocabulary/src/main/java/com/asakusafw/vocabulary/batch/BatchDescription.java
  • L73 -> #describe (DSL user code)
  • L164 -> #getWorks

3.5   Semantic Analysis (batch)

• com.asakusafw.compiler.testing.DirectBatchCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/testing/DirectBatchCompiler.java
  • L105 -> BatchCompiler.compile
• com.asakusafw.compiler.batch.BatchCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/BatchCompiler.java

3.6   Entry Point (flow)

• com.asakusafw.compiler.batch.BatchCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/BatchCompiler.java
  • L104 -> WorkDescriptionProcessor.process
• example: com.asakusafw.compiler.batch.processor.JobFlowWorkDescriptionProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/processor/JobFlowWorkDescriptionProcessor.java

3.7   Description Analysis (flow)

• com.asakusafw.compiler.batch.processor.JobFlowWorkDescriptionProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/processor/JobFlowWorkDescriptionProcessor.java
  • L108 -> JobFlowDriver.analyze
• java.com.asakusafw.compiler.flow.JobFlowDriver (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/JobFlowDriver.java
  • L112 -> FlowDescriptionDriver#createFlowGraph
• com.asakusafw.compiler.flow.FlowDescriptionDriver (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/FlowDescriptionDriver.java
  • L171 -> FlowDescription#start
• com.asakusafw.vocabulary.flow.FlowDescription (voc)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/asakusa-dsl-vocabulary/src/main/java/com/asakusafw/vocabulary/flow/FlowDescription.java
  • L40 -> #describe (DSL user code)

3.8   Semantic Analysis (flow)

• com.asakusafw.compiler.batch.processor.JobFlowWorkDescriptionProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/processor/JobFlowWorkDescriptionProcessor.java
  • L67 -> FlowCompiler#compile
• com.asakusafw.compiler.flow.FlowCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/FlowCompiler.java
  • L170 -> StagePlanner#plan
• com.asakusafw.compiler.flow.plan.StagePlanner (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/plan/StagePlanner.java
  • #validate:L801
    • #validateConnection
    • #validateAcyclic
  • #normalizeFlowGraph:L652
    • #inlineFlowParts
    • #insertCheckpoints
  • #buildStageGraph:L164
    • #buildComputationBlocks
    • #buildStageBlocks

3.9   Code Generation (stage)

• com.asakusafw.compiler.flow.FlowCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/FlowCompiler.java
  • L194 -> StageCompiler#compile
• com.asakusafw.compiler.flow.stage.StageCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/stage/StageCompiler.java
  • L188 -> ShuffleAnalyzer#analyze
  • L189 -> StageAnalyzer#analyze
  • L204* -> Shuffle*Emiter#emit
  • L154* -> *FragmentEmitter#emit
  • L136* -> *erEmitter#emit
• package: com.asakusafw.compiler.flow.stage (comp)
  • https://github.com/ashigeru/asakusafw/tree/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/stage

3.10   Code Generation (flow)

• com.asakusafw.compiler.flow.FlowCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/FlowCompiler.java
  • L203 -> JobflowCompiler#compile
• com.asakusafw.compiler.flow.jobflow.JobflowCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/jobflow/JobflowCompiler.java
  • L106 -> JobflowAnalyzer#analyze
  • L136* -> ExternalIoDescriptionProcessor#emit*
  • L276 -> StageClientEmitter#emit
• package: com.asakusafw.compiler.flow.jobflow (comp)
  • https://github.com/ashigeru/asakusafw/tree/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/jobflow

3.11   Code Generation (batch)

• com.asakusafw.compiler.batch.BatchCompiler (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/BatchCompiler.java
  • L79 -> WorkflowProcessor.process
• example: com.asakusafw.compiler.batch.experimental.ExperimentalWorkflowProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/experimental/ExperimentalWorkflowProcessor.java

3.12   Extensions Points

• com.asakusafw.compiler.batch.WorkflowProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/batch/WorkflowProcessor.java
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.batch.WorkflowProcessor
• com.asakusafw.compiler.flow.FlowElementProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/FlowElementProcessor.java
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.flow.FlowElementProcessor
• com.asakusafw.compiler.flow.ExternalIoDescriptionProcessor (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/ExternalIoDescriptionProcessor.java
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.flow.ExternalIoDescriptionProcessor
• com.asakusafw.compiler.flow.DataClassRepository (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/DataClassRepository.java
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.flow.DataClassRepository
• com.asakusafw.compiler.flow.FlowGraphRewriter (comp)
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/java/com/asakusafw/compiler/flow/FlowGraphRewriter.java
  • https://github.com/ashigeru/asakusafw/blob/SCR-01/ashigel-compiler/src/main/resources/META-INF/services/com.asakusafw.compiler.flow.FlowGraphRewriter