namespace-management.md

namespace-management.md

Namespace Management (DRAFT)

Purpose

This document describes the guidelines for managing two important namespaces: the package scala._, and the Maven group org.scala-lang.

These questions become more important as we modularize the distribution, and as we publish new modules (such as ScalaJS, scala-async, scala-pickling.)

Goals

• Consistency: modules should follow the same conventions.
• Extensibility: What happens when a single JAR module evolves into a multi-JAR module?
• Backwards Compatibility: avoid requiring early adopters of a module to modify imports as the module graduates to non-experimental status.
• Cleanliness: avoid overloading global namespaces
• Independence: names added to one module should not interact with other modules

Maven Artifact Naming

The core of the scala distribution:

"org.scala-lang" % "scala-library" % "2.10.2"

Modules are cross versioned with an independent version number. The artifact name is prefixed with scala-.

"org.scala-lang.modules" %% "scala-parser-combinators" % "1.0"

Multi-JAR modules (or modules that could potentially become multi-JAR in the future) should include the module name in the Maven group ID. The artifact name is prefixed with scala-${moduleName}-.

"org.scala-lang.modules.js" %% "scala-js-compiler" % "0.1"
"org.scala-lang.modules.js" %% "scala-js-library" % "0.1"

These names should be reviewed and agreed upon during the SIP Process. If the project wants to make an alpha binary release before the module name has been agreed upon, it should do so under a completely different group ID (TODO example.) Rationale: if we release an artifact to Sonatype it lives there forever, and orphaned artificacts as are a visible distraction to anyone browsing /org/scala-lang in the repo.

Packaging

Use of java._ and javax._ in the JDK

• The JVM security manager actively prevents third party code from residing in java._.
• Standard extensions were placed in javax._ (e.g Swing)
• Discussion of java vs javax: http://stackoverflow.com/questions/727844/javax-vs-java-package

The JDK ships classes from both java._ and javax._ in classes.jar. Other JARs contribute to javax. The appendix lists the package structure.

Imports and Name binding in Scala

All Scala compilation units automatically include root imports:

import java.lang._
import scala._
import scala.Predef._

Scala differs from Java, which only includes java.lang._.

This begs the question: if we introduce scala.foo in a new release, will we change the meaning or break compilation of the following code?

// import scala._  // automatically added import
import O._
object Test { foo }
object O { def foo = () }

If the root imports were literally added to the source as shown, this would be a problem. But to avoid this fragility, the Scala compiler treats the root imports as though they were in an outer scope:

import java.lang._
{
   import scala._
   {
     import scala.Predef._
     {
        // YOUR CODE HERE
     }
   }
}

The rules of name binding then ensure that your imports take precendence. Similarly, if foo is defined in the same or an enclosing, nested package, it will bind with higher precedence than scala.foo.

The appendix contains the list of names in root imports that are shadowed by other root imports. Changes to that list will introduce source incompatibilities. For example, addition of scala.reflect in Scala 2.10.0 shadows java.lang.reflect.

% scala29 -nc  -e 'type t = reflect.Method'
% scala210 -nc  -e 'type t = reflect.Method'
error: type Method is not a member of package reflect

Scala package structure

The current Scala pacakge structure, and the full listing of the scala package is included in the appendix.

Packaging for modules:

TODO: discuss and debate the following and decide.

1. Modules that used to be in the standard library will retain the existing packaging (rationale: backwards compatibility.)

2. Modules that extend or complement existing code may package in a sub-package:

   scala.concurrent.$module._ // but, requires coordination to avoid clashes

3. Newly added modules

   1. scalax.$module._ // not sure if we want to follow javax, can be confusing // for people looking for the right import.
   2. scala.$module._ // convenient, but pollutes an important namespace
   3. scala.modules.$module // more verbose

Once again, the choice of package names should be discussed and agreed up in the SIP process. You may publish an early version of the package before this agreement is reached; unlike the choice of Group ID, some early fluctuations in your package names don't leave detritus in our repository, even if they do cause some rework for early adopters of the module.

Public API vs Implementation.

The Scala distribution places implementation details in be placed in sub-package(s) called internal (e.g. scala.reflect.internal.) Modules should follow this convention.

Rationale:

• such packages can be excluded from binary compatibility checking with MiMa. Simply making types private[mymodule] is not sufficient to exclude them from MiMa, as they are still bytecode-public.
• consistency helps users of our modules stick to the public API.

As always, modules should aim to present a minimal, coherent public API. This includes finalizing classes and methods unless they are designed with extension in mind, and using private access where applicable.

Open Issues

• Group ID and Package structure for SBT / Maven plugins

Appendix: Java Package Structure

classes.jar
    java
        applet
        awt
        beans
        io
        lang
        math
        net
        nio
        rmi
        security
        sql
        text
        util
    javax
        accessibility
        activation
        activity
        annotation
        imageio
        jws
        model
        management
        naming
        print
        rmi
        script
        security
        smartcardio
        sound
        sql
        swing
        tools
        transaction
        xml
dt.jar
    javax
        swing
jce.jar
    javax
        crypto

Appendix: Scala Package structure

scala-library.jar
    scala
        annotation
        beans
        collection
        compat
        concurrent
        io
        math
        ref
        reflect
        runtime
        sys
        text
        util

scala-reflect.jar
    scala
        reflect

scala-xml.jar
    scala
        xml

scala-parser-combinators.jar
    scala
        util
            parsing

Appendix: Contents of the scala._ provided by the standard library.

> println(ScalaPackageClass.info.decls.toList.filterNot(_.name.toString.contains('$')).mkString("\n"))
class AnyVal
object AnyVal
trait AnyValCompanion
object AnyValCompanion
class App
object App
class Array
object Array
class Boolean
object Boolean
class Byte
object Byte
class Char
object Char
trait Cloneable
object Cloneable
class Console
object Console
trait DelayedInit
object DelayedInit
class deprecated
object deprecated
class DeprecatedConsole
object DeprecatedConsole
class deprecatedInheritance
object deprecatedInheritance
class deprecatedName
object deprecatedName
class deprecatedOverriding
object deprecatedOverriding
trait DeprecatedPredef
object DeprecatedPredef
class Double
object Double
trait Dynamic
object Dynamic
class Enumeration
object Enumeration
trait Equals
object Equals
class FallbackArrayBuilding
object FallbackArrayBuilding
class Float
object Float
class Function
object Function
trait Function0
object Function0
trait Function1
object Function1
class Function10
object Function10
class Function11
object Function11
class Function12
object Function12
class Function13
object Function13
class Function14
object Function14
class Function15
object Function15
class Function16
object Function16
class Function17
object Function17
class Function18
object Function18
class Function19
object Function19
trait Function2
object Function2
class Function20
object Function20
class Function21
object Function21
class Function22
object Function22
trait Function3
object Function3
class Function4
object Function4
class Function5
object Function5
class Function6
object Function6
class Function7
object Function7
class Function8
object Function8
class Function9
object Function9
trait Immutable
object Immutable
class inline
object inline
class Int
object Int
class language
object language
class languageFeature
object languageFeature
class Long
object Long
class LowPriorityImplicits
object LowPriorityImplicits
class MatchError
object MatchError
trait Mutable
object Mutable
class native
object native
class noinline
object noinline
class None
object None
class NotImplementedError
object NotImplementedError
class NotNull
object NotNull
class Option
object Option
package package
package scala
trait PartialFunction
object PartialFunction
class Predef
object Predef
trait Product
object Product
class Product1
object Product1
class Product10
object Product10
class Product11
object Product11
class Product12
object Product12
class Product13
object Product13
class Product14
object Product14
class Product15
object Product15
class Product16
object Product16
class Product17
object Product17
class Product18
object Product18
class Product19
object Product19
trait Product2
object Product2
class Product20
object Product20
class Product21
object Product21
class Product22
object Product22
trait Product3
object Product3
class Product4
object Product4
class Product5
object Product5
class Product6
object Product6
class Product7
object Product7
class Product8
object Product8
class Product9
object Product9
trait Proxy
object Proxy
class remote
object remote
class Responder
object Responder
class ScalaReflectionException
object ScalaReflectionException
trait Serializable
object Serializable
class SerialVersionUID
object SerialVersionUID
class Short
object Short
class Some
object Some
trait Specializable
object Specializable
class specialized
object specialized
class StringContext
object StringContext
class Symbol
object Symbol
class throws
object throws
class transient
object transient
class Tuple1
object Tuple1
class Tuple10
object Tuple10
class Tuple11
object Tuple11
class Tuple12
object Tuple12
class Tuple13
object Tuple13
class Tuple14
object Tuple14
class Tuple15
object Tuple15
class Tuple16
object Tuple16
class Tuple17
object Tuple17
class Tuple18
object Tuple18
class Tuple19
object Tuple19
class Tuple2
object Tuple2
class Tuple20
object Tuple20
class Tuple21
object Tuple21
class Tuple22
object Tuple22
class Tuple3
object Tuple3
class Tuple4
object Tuple4
class Tuple5
object Tuple5
class Tuple6
object Tuple6
class Tuple7
object Tuple7
class Tuple8
object Tuple8
class Tuple9
object Tuple9
class unchecked
object unchecked
class UninitializedError
object UninitializedError
class UninitializedFieldError
object UninitializedFieldError
class UniquenessCache
object UniquenessCache
class Unit
object Unit
class volatile
object volatile
package annotation
package beans
package collection
package compat
package concurrent
package io
package math
package ref
package reflect
package runtime
package sys
package text
package util
package xml
package tools
package actors
type AnyRef
type Throwable
type Exception
type Error
type RuntimeException
type NullPointerException
type ClassCastException
type IndexOutOfBoundsException
type ArrayIndexOutOfBoundsException
type StringIndexOutOfBoundsException
type UnsupportedOperationException
type IllegalArgumentException
type NoSuchElementException
type NumberFormatException
type AbstractMethodError
type InterruptedException
value AnyRef
type TraversableOnce
type Traversable
value Traversable
type Iterable
value Iterable
type Seq
value Seq
type IndexedSeq
value IndexedSeq
type Iterator
value Iterator
type BufferedIterator
type List
value List
value Nil
type Stream
value Stream
type Vector
value Vector
type StringBuilder
value StringBuilder
type Range
value Range
type BigDecimal
value BigDecimal
type BigInt
value BigInt
type Equiv
value Equiv
type Fractional
value Fractional
type Integral
value Integral
type Numeric
value Numeric
type Ordered
value Ordered
type Ordering
value Ordering
type PartialOrdering
type PartiallyOrdered
type Either
value Either
type Left
value Left
type Right
value Right
class <repeated>
class Any
class Nothing
class <repeated...>
class <byname>
class Null
trait Singleton

Appendix: Root import shadowing

scala> def names(sym: Symbol) = sym.info.members.filter(x => x.info != NoType).map(_.name).toList.filterNot(_.encoded contains '$').toSet

scala> def shadow(sym1: Symbol, sym2: Symbol) = (names(sym1) intersect names(sym2)).toList.map(_.longString).sorted.mkString("\n")
shadow: (sym1: $r.intp.global.Symbol, sym2: $r.intp.global.Symbol)String

scala> shadow(JavaLangPackageClass, ScalaPackageClass)
res59: String =
term Boolean
term Byte
term Double
term Float
term Iterable
term Long
term Short
term StringBuilder
term annotation
term ref
term reflect
type AbstractMethodError
type ArrayIndexOutOfBoundsException
type Boolean
type Byte
type ClassCastException
type Cloneable
type Double
type Error
type Exception
type Float
type IllegalArgumentException
type IndexOutOfBoundsException
type InterruptedException
type Iterable
type Long
type NullPointerException
type NumberFormatException
type RuntimeException
type Short
type StringBuilder
type StringIndexOutOfBoundsException
type Throwable
type UnsupportedOperationException

scala> shadow(JavaLangPackageClass, PredefModule.moduleClass)
res60: String =
type Class
type String

scala> shadow(ScalaPackageClass, PredefModule.moduleClass)
res61: String = ""

Excellent write-up, @retronym! My vote is for the scalax root as an incubator, with modules graduating to scala once proven, and a deprecation cycle for the scalax-prefixed names (forwarding to scala._ during deprecation).

Why would we really want to graduate anything to scala? Wouldn't it be great if scala contained the least and the most stable part of core standard library that hardly ever changes for sake of long-term compatibility?

I would argue that having stuff in scalax permanently is nice as it would let projects live on their own (their own repos, versions, issues, jars, release schedules) without being slowed down by the arguably slow scala release cycle. It might be even appropriate to move the least favored parts of current standard library there to de-clutter current haphazardly organized scala namespace.

This split could also be a stepping stone between rock-solid core scala std lib and everything else.

Interesting analysis. While scalax is plain ugly, I can imagine it for "incubation" as Adriaan says. In general I think for "core functionality" it should stay noise free and be accessible directly via scala._ and not scala.modules._. For instance concurrent.stm or reflect. Some names are less fortunate, for example util for the parser combinators. I'm fine with the group-ID, although I don't understand why you need to duplicate "scala" again in the artifact-ID (e.g., why not just "org.scala-lang.modules" %% "parser-combinators" % "1.0").

If I'm not mistaken, project defined packages already shadow scala._, so if I have my own package xml this won't cause any trouble with scala.xml (the former will be visible). And if I don't depend on a module, it will also not pollute my namespace?

"If the project wants to make an alpha binary release before the module name has been agreed upon, it should do so under a completely different group ID."

This has to be before someone might publish a library against it. Otherwise, there is the problem of the dependency manager not applying conflict resolution because there are multiple group IDs. Macros/reflection are the typical example here. They were "experimental", but published libraries depended on them anyway.

" scala.modules.$module // more verbose"

This is otherwise attractive, so perhaps scala.ext.$module or scala.lib.$module would make it succinct enough?

@Sciss Right, not adding a module means it won't pollute your namespace. You might still need to care if you write a library, though. Your users might experience the conflict if they use the module.

@densh adds another aspect to the proposal, which is signaling compatibility timelines with the package name. Keeping scala._ minimal and only for the core, long-term stable libraries makes sense to me. Keeping it minimal makes sense because once something is in, it will be a long time before it can be removed.

I think that once you have a scalax. library incubated enough to be included as scala., a rename is a disruptive change. I'm not saying applying a regex is difficult. If the types in the module are passed between libraries, both libraries have to be updated to agree to use the same names. There might have been one or more existing libraries subsumed by this core/extension library (like with futures), so this would be a second rename. Overall, this seems to me to be a coordination/compatibility problem that is more trouble than it is worth.

You might also want to consider that there are a bunch of ad hoc "do things differently if this is somewhere in the scala package" points in the implementation, most or all of it completely unspecified but all with the tacit assumption that any code under scala.** can be trusted not to be doing anything unsavory or ill-advised.

See scala/scala#2803 for an example.

One thing I'm especially concerned about is that we seem to have started mixing packages in scala where some of them describe the functionality, and some use the project name.

Compare scala.collection, scala.concurrent, scala.math with scala.slick, scala.pickling, scala.async.
Consistency should be key and we are failing here.

In that sense, I don't have an issue if modules use the scala package as long as they follow the existing naming conventions.

Thanks for the inputs. This document has been revised and moved to a Google Doc.

The main difference is that I'm now in favour of scala._ for modules, rather than scalax._.