Incremental.md

What exists today

Running the incremental compiler

object Incremental (Incremental.scala)

• Used by object IncrementalCompile
• Delegates work to the incremental compilation strategy
• A mean to run the incremental compiler

Helper class to run incremental compilation algorithm.

/**
 * Runs the incremental compiler algorithm.
 * @param sources   The sources to compile
 * @param entry  The means of looking up a class on the classpath.
 * @param previous The previously detected source dependencies.
 * @param current  A mechanism for generating stamps (timestamps, hashes, etc).
 * @param doCompile  The function which can run one level of compile.
 * @param log  The log where we write debugging information
 * @param options  Incremental compilation options
 * @param equivS  The means of testing whether two "Stamps" are the same.
 * @return A flag of whether or not compilation completed succesfully, and the resulting dependency analysis object.
 */
def compile(sources: Set[File],
  entry: String => Option[File],
  previous: Analysis,
  current: ReadStamps,
  forEntry: File => Option[Analysis],
  doCompile: (Set[File], DependencyChanges) => Analysis,
  log: Logger,
  options: IncOptions)(implicit equivS: Equiv[Stamp]): (Boolean, Analysis)

object IncrementalCompile (Compile.scala)

• Used by object IC
• Uses object Incremental
• Another mean to run the incremental compiler

Helper methods for running incremental compilation. All this is responsible for is adapting any xsbti.AnalysisCallback into one compatible with sbt.inc.Incremental.

/**
 * Runs the incremental compilation algorithm.
 * @param sources The full set of input sources
 * @param entry A className -> source file lookup function.
 * @param compile The mechanism to run a single 'step' of compile, for ALL source files involved.
 * @param previous The previous dependency Analysis (or an empty one).
 * @param forEntry The dependency Analysis associated with a given file
 * @param output The configured output directory/directory mapping for source files.
 * @param log Where all log messages should go
 * @param options Incremental compiler options (like name hashing vs. not).
 * @return A flag of whether or not compilation completed succesfully, and the resulting dependency analysis object.
 */
def apply(sources: Set[File], entry: String => Option[File],
  compile: (Set[File], DependencyChanges, xsbti.AnalysisCallback) => Unit,
  previous: Analysis,
  forEntry: File => Option[Analysis],
  output: Output, log: Logger,
  options: IncOptions): (Boolean, Analysis)

object IC extends IncrementalCompiler[Analysis, AnalyzingCompiler] (IncrementalCompiler.scala)

• Uses object IncrementalCompile (which uses object Incremental)
• Another mean to run the incremental compiler

An implementation of the incremental compiler that can compile inputs and dump out source dependency analysis.

/**
 * Performs an incremental compilation as configured by `in`.
 * The returned Analysis should be provided to compilations depending on the classes from this compilation.
 */
override def compile(in: Inputs[Analysis, AnalyzingCompiler], log: Logger): Analysis

/**
 * This will run a mixed-compilation of Java/Scala sources
 * TODO - this is the interface sbt uses. Somehow this needs to be exposed further.
 *
 * @param scalac  An instances of the Scalac compiler which can also extract "Analysis" (dependencies)
 * @param javac  An instance of the Javac compiler.
 * @param sources  The set of sources to compile
 * @param classpath  The classpath to use when compiling.
 * @param output  Configuration for where to output .class files.
 * @param cache   The caching mechanism to use instead of insantiating new compiler instances.
 * @param progress  Progress listening for the compilation process.  TODO - Feed this through the Javac Compiler!
 * @param options   Options for the Scala compiler
 * @param javacOptions  Options for the Java compiler
 * @param previousAnalysis  The previous dependency Analysis object/
 * @param previousSetup  The previous compilation setup (if any)
 * @param analysisMap   A map of file to the dependency analysis of that file.
 * @param definesClass  A mehcnaism of looking up whether or not a JAR defines a particular Class.
 * @param reporter  Where we sent all compilation error/warning events
 * @param compileOrder  The order we'd like to mix compilation.  JavaThenScala, ScalaThenJava or Mixed.
 * @param skip  IF true, we skip compilation and just return the previous analysis file.
 * @param incrementalCompilerOptions  Options specific to incremental compilation.
 * @param log  The location where we write log messages.
 * @return  The full configuration used to instantiate this mixed-analyzing compiler, the set of extracted dependencies and
 *          whether or not any file were modified.
 */
def incrementalCompile(scalac: AnalyzingCompiler,
  javac: xsbti.compile.JavaCompiler,
  sources: Seq[File],
  classpath: Seq[File],
  output: Output,
  cache: GlobalsCache,
  progress: Option[CompileProgress] = None,
  options: Seq[String] = Nil,
  javacOptions: Seq[String] = Nil,
  previousAnalysis: Analysis,
  previousSetup: Option[CompileSetup],
  analysisMap: File => Option[Analysis] = { _ => None },
  definesClass: Locate.DefinesClass = Locate.definesClass _,
  reporter: Reporter,
  compileOrder: CompileOrder = Mixed,
  skip: Boolean = false,
  incrementalCompilerOptions: IncOptions)(implicit log: Logger): Result

interface IncrementalCompiler<Analysis, ScalaCompiler> (IncrementalCompiler.java)

• Pure interface to the incremental compiler
• We should define an interface for Analysis and ScalaCompile in Java so that we don't need to have them as type parameters.
• Clean and simple - What we should aim for?

public interface IncrementalCompiler<Analysis, ScalaCompiler> {
  /**
   * Performs an incremental compilation as configured by `in`.
   * The returned Analysis should be provided to compilations depending on the classes from this compilation.
   */
  Analysis compile(Inputs<Analysis, ScalaCompiler> in, Logger log);

  /**
   * Creates a compiler instance that can be used by the `compile` method.
   *
   * @param instance The Scala version to use
   * @param interfaceJar The compiler interface jar compiled for the Scala version being used
   * @param options Configures how arguments to the underlying Scala compiler will be built.
   */
  ScalaCompiler newScalaCompiler(ScalaInstance instance, File interfaceJar, ClasspathOptions options);

  /**
   * Compiles the source interface for a Scala version.  The resulting jar can then be used by the `newScalaCompiler` method
   * to create a ScalaCompiler for incremental compilation.  It is the client's responsibility to manage compiled jars for
   * different Scala versions.
   *
   * @param label A brief name describing the source component for use in error messages
   * @param sourceJar The jar file containing the compiler interface sources.  These are published as sbt's compiler-interface-src module.
   * @param targetJar Where to create the output jar file containing the compiled classes.
   * @param instance The ScalaInstance to compile the compiler interface for.
   * @param log The logger to use during compilation.
   */
  void compileInterfaceJar(String label, File sourceJar, File targetJar, File interfaceJar, ScalaInstance instance, Logger log);
}

class AggressiveCompile (AggressiveCompile.scala)

• Old mean to compile Scala and Java files, not incrementally (?)
• Deprecated.

Wrappers for the compilers

final class AnalyzingCompiler extends CachedCompilerProvider (AnalyzingCompiler.scala)

• Wraps an instance of scalac represented by xsbti.compile.ScalaInstance.
• Lots of overloads to trigger compilation.

Interface to the Scala compiler that uses the dependency analysis plugin. This class uses the Scala library and compiler provided by scalaInstance. This class requires a ComponentManager in order to obtain the interface code to scalac and the analysis plugin. Because these call Scala code for a different Scala version than the one used for this class, they must be compiled for the version of Scala being used.

def apply(sources: Seq[File],
  changes: DependencyChanges,
  classpath: Seq[File],
  singleOutput: File,
  options: Seq[String],
  callback: AnalysisCallback,
  maximumErrors: Int,
  cache: GlobalsCache,
  log: Logger): Unit

def compile(sources: Seq[File],
  changes: DependencyChanges,
  options: Seq[String],
  output: Output,
  callback: AnalysisCallback,
  reporter: Reporter,
  cache: GlobalsCache,
  log: Logger,
  progressOpt: Option[CompileProgress]): Unit

...

final class AnalyzingJavaCompiler (AnalyzingJavaCompiler.scala)

• Wraps JavaCompiler and provides helper functions to trigger compilation of Java files

This is a java compiler which will also report any discovered source dependencies/apis out via an analysis callback.

/**
 * Compile some java code using the current configured compiler.
 * @param sources  The sources to compile
 * @param options  The options for the Java compiler
 * @param output   The output configuration for this compiler
 * @param callback  A callback to report discovered source/binary dependencies on.
 * @param reporter  A reporter where semantic compiler failures can be reported.
 * @param log       A place where we can log debugging/error messages.
 * @param progressOpt An optional compilation progress reporter.  Where we can report back what files we're currently compiling.
 */
def compile(sources: Seq[File],
  options: Seq[String],
  output: Output,
  callback: AnalysisCallback,
  reporter: Reporter,
  log: Logger,
  progressOpt: Option[CompileProgress]): Unit

final class MixedAnalyzingCompiler (MixedAnalyzingCompiler.scala)

• Wraps AnalyzingCompiler and AnalyzingJavaCompiler and provides helper functions to compile both Scala and Java files.

An instance of an analyzing compiler that can run both javac + scalac.

/**
 * Compiles the given Java/Scala files.
 *
 * @param include  The files to compile right now
 * @param changes  A list of dependency changes.
 * @param callback  The callback where we report dependency issues.
 */
def compile(include: Set[File], changes: DependencyChanges, callback: AnalysisCallback): Unit

interface JavaCompiler (JavaCompiler.java)

• Simple interface for compilation of Java code`

/**
 * Interface to a Java compiler.
 */
public interface JavaCompiler {
  /**
   * Compiles java sources using the provided classpath, output directory and additional options.
   *
   * Output should be sent to the provided logger.
   * Failures should be passed to the provided Reporter.
   */
  void compileWithReporter(File[] sources, File[] classpath, Output output, String[] options, Reporter reporter, Logger log);
}

interface Compilers<ScalaCompiler> (Compilers.java)

• Interface for a wrapper around both Scala and Java compilers?
• Nobody uses it at the moment?

public interface Compilers<ScalaCompiler> {
  JavaCompiler javac();
  // should be cached by client if desired
  ScalaCompiler scalac();
}

sealed trait JavaTool & trait JavaCompiler (JavaCompiler.scala)

• Interface for a Java compiler

/**
 * An interface for on of the tools in the java tool chain.
 *
 * We assume the following is true of tools:
 * - The all take sources and options and log error messages
 * - They return success or failure.
 */
sealed trait JavaTool {
  /**
   * This will run a java compiler / or other like tool (e.g. javadoc).
   *
   *
   * @param sources  The list of java source files to compile.
   * @param options  The set of options to pass to the java compiler (includes the classpath).
   * @param log      The logger to dump output into.
   * @param reporter The reporter for semantic error messages.
   * @return true if no errors, false otherwise.
   */
  def run(sources: Seq[File], options: Seq[String])(implicit log: Logger, reporter: Reporter): Boolean
}

/** Interface we use to compile java code. This is mostly a tag over the raw JavaTool interface. */
trait JavaCompiler extends JavaTool {}

object JavaCompiler (JavaCompiler.scala)

• Factory methods for constructing a java compiler

/** Factory methods for constructing a java compiler. */
object JavaCompiler {
  /** Returns a local compiler, if the current runtime supports it. */
  def local: Option[JavaCompiler]
  /** Returns a local compiler that will fork javac when needed. */
  def fork(javaHome: Option[File] = None): JavaCompiler
}

Configuring a compilation run

interface Inputs<Analysis, ScalaCompiler> (Inputs.java)

• No implementation???
• There's a case class that has the same name and a very similar structure, but it doesn't implement this interface.
• Should be used to represent the configuration of a compiler run
• See also Options and Setup<Analysis>.

/** Configures a single compilation of a single set of sources.*/
public interface Inputs<Analysis,ScalaCompiler> {
 /** The Scala and Java compilers to use for compilation.*/
 Compilers<ScalaCompiler> compilers();
 /** Standard compilation options, such as the sources and classpath to use. */
 Options options();
 /** Configures incremental compilation.*/
 Setup<Analysis> setup();
}

interface Options (Options.java)

• No implementation???
• There's a case class that has the same name and a very similar structure, but it doesn't implement this interface.
• Standard compilation options

public interface Options {
  /** The classpath to use for compilation.
   * This will be modified according to the ClasspathOptions used to configure the ScalaCompiler.*/
  File[] classpath();
  /** All sources that should be recompiled.
   * This should include Scala and Java sources, which are identified by their extension. */
  File[] sources();
  /** Output for the compilation. */
  Output output();
  /** The options to pass to the Scala compiler other than the sources and classpath to use. */
  String[] options();
  /** The options to pass to the Java compiler other than the sources and classpath to use. */
  String[] javacOptions();
  /** Controls the order in which Java and Scala sources are compiled.*/
  CompileOrder order();
}

interface Setup<Analysis> (Setup.java)

• Configures incremental recompilation

public interface Setup<Analysis> {
  /** Provides the Analysis for the given classpath entry.*/
  Maybe<Analysis> analysisMap(File file);
  /** Provides a function to determine if classpath entry `file` contains a given class.
   * The returned function should generally cache information about `file`, such as the list of entries in a jar.
   */
  DefinesClass definesClass(File file);
  /** If true, no sources are actually compiled and the Analysis from the previous compilation is returned.*/
  boolean skip();
  /** The file used to cache information across compilations.
   * This file can be removed to force a full recompilation.
   * The file should be unique and not shared between compilations. */
  File cacheFile();
  GlobalsCache cache();
  /** If returned, the progress that should be used to report scala compilation to. */
  Maybe<CompileProgress> progress();
  /** The reporter that should be used to report scala compilation to. */
  Reporter reporter();
  /**
   * Returns incremental compiler options.
   * @see sbt.inc.IncOptions for details
   * You can get default options by calling <code>sbt.inc.IncOptions.toStringMap(sbt.inc.IncOptions.Default)</code>.
   * In the future, we'll extend API in <code>xsbti</code> to provide factory methods that would allow to obtain
   * defaults values so one can depend on <code>xsbti</code> package only.
   **/
  Map<String, String> incrementalCompilerOptions();
}

Retrieving compiler instances

interface GlobalsCache (GlobalsCache.java)

• Implemented (only?) by class CompilerCache

/** An interface which lets us know how to retrieve cached compiler instances form the current JVM. */
public interface GlobalsCache
{
  CachedCompiler apply(String[] args, Output output, boolean forceNew, CachedCompilerProvider provider, Logger log, Reporter reporter);
  void clear();
}

private final class CompilerCache (CompilerCache.scala)

• A map of CachedCompilers

interface CachedCompiler (CachedCompiler.java)

• Implemented by private final class CachedCompiler0
• Represents an actual instance of scalac?

public interface CachedCompiler {
  /** Returns an array of arguments representing the nearest command line equivalent of a call to run but without the command name itself.*/
  String[] commandArguments(File[] sources);
  void run(File[] sources, DependencyChanges cpChanges, AnalysisCallback callback, Logger log, Reporter delegate, CompileProgress progress);
}

interface CachedCompilerProvider (CachedCompilerProvider.java)

• Implemented by AnalyzingCompiler

public interface CachedCompilerProvider {
  ScalaInstance scalaInstance();
  CachedCompiler newCachedCompiler(String[] arguments, Output output, Logger log, Reporter reporter, boolean resident);
}

What do we want to do?

Ideas

• Completely hide CachedCompiler and CachedCompilerProvider inside the implementation of Compilers.
• There are a lot of nice interfaces that have been defined.

Provide a simple API that would allow to run the incremental compiler

Generated datatypes

{
  "types": [
    {
      "name": "Inputs",
      "namespace": "sbt.compiler",
      "doc": "Configures a single compilation of a single set of sources.",
      "type": "record",
      "target": "Scala",
      "fields": [
        {
          "name": "compilers",
          "type": "xsbti.compile.Compilers",
          "doc": "The Scala and Java compilers to use for compilation."
        },
        {
          "name": "options",
          "type": "xsbti.compile.Options",
          "doc": "Standard compilation options, such as the sources and classpath to use."
        },
        {
          "name": "setup",
          "type": "Setup",
          "doc": "Configures incremental compilation."
        }
      ]
    },
    {
      "name": "Options",
      "namespace": "sbt.compiler",
      "type": "record",
      "target": "Scala",
      "fields": [
        {
          "name": "classpath",
          "type": "java.io.File*",
          "doc": "The classpath to use for compilation This will be modified according to the ClasspathOptions used to configure the ScalaCompiler."
        },
        {
          "name": "sources",
          "type": "java.io.File*",
          "doc": "All sources that should be recompiled. This should include Scala and Java sources, which are identified by their extension."
        },
        {
          "name": "output",
          "type": "xsbti.compile.Output",
          "doc": "Output for the compilation."
        },
        {
          "name": "scalaOptions",
          "type": "String*",
          "doc": "The options to pass to the Scala compiler other than the sources and classpath to use."
        },
        {
          "name": "javaOptions",
          "type": "String*",
          "doc": "The options to pass to the Java compiler other than the sources and classpath to use."
        },
        {
          "name": "Order",
          "type": "xsbti.compile.CompileOrder",
          "doc": "Controls the order in which Java and Scala sources are compiled."
        }
      ]
    },
    {
      "name": "Compilers",
      "namespace": "sbt.compiler",
      "doc": "A wrapper around a Scala and a Java compiler",
      "type": "interface",
      "target": "Java",
      "fields": [
        {
          "name": "javac",
          "type": "AnalyzingJavaCompiler"
        },
        {
          "name": "scalac",
          "type": "AnalyzingScalaCompiler"
        }
      ]
    }
  ]
}

xsbti

interface Analysis {
  Stamps stamps();
  APIs apis();
  Relations relations();
  SourceInfos infos();
  Compilations compilations();

  Analysis addSource(
    File src,
    Source api,
    Stamp stamp,
    SourceInfo info,
    Iterable<Product> products,
    Iterable<InternalDependency> internalDeps,
    Iterable<ExternalDependency> externalDeps,
    Iterable<BinaryDependency> binaryDeps);

  Analysis $plus$plus(o: Analysis);
  Analysis $minus$minus(i: Analysis);

  // Missing: copy
}

interface Setup {
  /** Provides the Analysis for the given classpath entry.*/
  Maybe<Analysis> analysisMap(File file);

  /** The resulting Analysis from the previous compiler run */
  Maybe<Analysis> previousAnalysis();

  /** Provides a function to determine if classpath entry `file` contains a given class.
   * The returned function should generally cache information about `file`, such as the list of entries in a jar.
   */
  DefinesClass definesClass(File file);

  /** If true, no sources are actually compiled and the Analysis from the previous compilation is returned.*/
  boolean skip();

  /** The file used to cache information across compilations.
   * This file can be removed to force a full recompilation.
   * The file should be unique and not shared between compilations. */
  File cacheFile();

  /** If returned, the progress that should be used to report scala compilation to. */
  Maybe<CompileProgress> progress();

  /** The reporter that should be used to report scala compilation to. */
  Reporter reporter();

  /**
   * Returns incremental compiler options.
   * @see sbt.inc.IncOptions for details
   * You can get default options by calling <code>sbt.inc.IncOptions.toStringMap(sbt.inc.IncOptions.Default)</code>.
   * In the future, we'll extend API in <code>xsbti</code> to provide factory methods that would allow to obtain
   * defaults values so one can depend on <code>xsbti</code> package only.
   **/
  Map<String, String> incrementalCompilerOptions();
}

sbt

object Compiler {
  def compile(in: Inputs, log: Logger, reporter: xsbti.compiler.Reporter): CompileResult
}