rupertlssmith/ElmPackageServer.md

## ElmPackageServer.md

      
    Raw
  

              ElmPackageServer.md
            
          
    Eco - An Alternate Elm Package Server

Eco is an alternative package management system for Elm. The name can stand for "Elm Compiler Offline".
The idea is to have a more configurable package management system for Elm, that allows for sharing private packages.
The main use case for this is intended to be within organizations, where a software developer or team can publish a private package that applications, or other private packages, in that organization can consume. This will support transitive dependency resolution in the same way that public Elm packages do.
The main repository and source of truth will be the existing package.elm-lang.org site.
This package management system can also act as a local cache of the package.elm-lang.org site, allowing work to continue if the central package site is down. This system will work in such a way that there can be multiple levels of caching, by chaining package sites together. There could be a public mirror of package.elm-lang.org, that acts as a backup, and private package repositories that feeds from the original or the mirror, as available.
There are 2 main software components, a server process that provides packages, and a command line tool that installs them into the local build environment.
Prior Work

This tool helps with private packages, by downloading them into elm-stuff and adding them to an
applications src directories in its elm.json file:
https://github.com/robinheghan/elm-git-install
It does not handle transitive dependency installation. It only works for applications, not packages. Both are limitations that this package system will overcome.
Building against the source makes modules that are not exposed by a package accessible, and you could write code that uses internals accidentally.
Command Line Tool

This will provide publish, install, bump, and diff commands.
There will be a configuration file, eco.json, that will provide the repository locations to install packages from.
Repository can either be a URL or a local directory.
One private repository must be defined in the config, and the CLI will publish to that location only.
The install, bump, and diff commands, will work as they do for the Elm compiler, but resolving through the configuration.
The Package Server

A package server can be configured with the same eco.json, as the CLI.
A package server may define one or more upstream repositories to mirror.
A package server cannot define an upstream repository to publish to, publishing must always be done with the CLI tool.
The same HTTP API as package.elm-lang.org site uses will be implemented by the package server:
GET: https://package.elm-lang.org/all-packages
{ "0ui/elm-task-parallel" : ["1.0.0","1.0.1","1.0.2","2.0.0"]
, ...
}

GET: https://package.elm-lang.org/all-packages/since/{seq}
["matheus23/elm-markdown-transforms@3.0.1", ...

POST: ...
When package is published, or cached from upstream, it will be screened before adding to the repository. These must all pass:

Checksum the .zip. This will differ to the GitHub assigned checksum used on the main package site, as it is by file contents only.
If cached from upstream, compare checksums with the upstream version. Checksums should always be validated.
Build it with elm install, using the appropriate compiler version. Confirm all dependencies are available, and it builds correctly.
Verify the elm.json matches the version to be published.

When a package is published, it will be screened before adding to the repository. These must all pass:

Its API will be compared to the previous version, if any. The correct semantic version will be confirmed depending on whether the API is breaking, changed but backwards compatible, identical. If there is no previous version, 1.0.0 will be used.
Confirm the package does not contain ports.
Confirm the package does not contain kernel code.

Any errors should result in an error state being recorded against the package, for manual inspection.
Allow a schedule to be set, to periodically check upstream for new packages to cache.
Package Domain Tags

Tags will be used to describe what the runtime requirements of packages are.
Tags will be inherited from package dependencies, by forming the super-set of all tags on the dependencies.
The following packages will be root tagged as org.elm-lang.core:
elm/core
elm/bytes
elm/file
elm/html
elm/http
elm/json
elm/parser
elm/project-metadata-utils
elm/random
elm/regex
The following packages will be root tagged as org.elm-lang.browser:
elm/browser
The following packages will be root tagged as org.elm-lang.vdom:
elm/virtual-dom
elm/html
elm/svg
The following packages will be root tagged as org.elm-lang.explorations
elm-explorations/benchmark
elm-explorations/linear-algebra
elm-explorations/markdown
elm-explorations/test
elm-explorations/webgl
All other packages will inherit some combination of org.elm-lang.core, org.elm-lang.browser, org.elm-lang.vdom and org.elm-lang.explorations tags.
** Note: These paragraphs describe the motivation for package domain tags. **
A potential alternate compiler for the Elm language, as opposed to The Elm Architecture, may support only a sub-set of these tags. This feature is not useful at the present, but is included now, so that it is part of the package server API from the start.
The intention is to be able to have an Elm build system that only supports say core, for running Elm code outside of the browser, with no virtual DOM to render as a view.
A future version may cater for private tags. For example ACME Corp can tag all its packages as com.acme, making it obvious when code depends on that private package domain. The private package domain at the head of those packages will not allow them to be published upstream, unless the private tag is removed by using only packages with public tags.
A build can be set to fail, if it includes packaging domains that it is not configured to allow.
For example, ACME Corp might publish a popular open source library that gives it kudos alongside its regular business. This would not include the com.acme domain, and if a developer accidentally starts including that domain, the build will fail.
The Package Server UI

The mobile friendly package site clone can be used as a starting point: https://elm.dmy.fr/
There needs to be some admin privileges for remote access to a package server, to adjust its configuration, to check for packages with errors, and so on.
How will it work with the Elm Compiler

Does the compiler always try to check the package site for updates? or can it be made to skip this step, and always build locally?
Ideally the existing Elm 0.19.x compilers should be able to build private packages without alteration - if they are inserted into either the ~/.elm folder, or the project elm-stuff folder.
If a hacked version of the compiler is needed, it must have the publish command removed, so as to ensure the package site is treated as read-only, and source of truth.
A hacked compiler should also have the install, bump, and diff commands removed, and these would be implemented by the CLI tool instead.
Example Use Cases

Working Offline

You grab your laptop and head out to the cabin in the woods, to get some peace and quiet and your work done. There is no internet there. Before leaving town you refresh the package server on your laptop, so that you have all the latest Elm packages - at that moment in time at least. As you work you are able to install more packages that your application comes to need as you work on it. They are cached on your laptop and you are able to browse their docs too through the package server UI.
Avoiding Productivity Outages

Back in town, a large development team at ACME corp is busy getting a new software product in shape for the pending launch date. You have a cache of all Elm packages running on the local network, and this is being updated frequently from the main public package site. You notice on Elm Slack that the main package site is down, users are getting anxious because the site admin is in the US Pacific Time Zone, and likely to be asleep. With a large team, ACME could be losing many man days of work in this scenario. The development manager at ACME is relaxed and happy today, her team are able to continue working from the local package server that she set up on the company network.
Trying out Work In Progress

You are developing a package at the same time as using it in an application you are writing. The code in the package may have started out as code in the application, but you decided to extract it as a useful package in its own right. The package has a large set of dependencies. The package is not really ready yet for releasing to the world. You create a version of the package called my-wonderful-lib-alpha and publish it to a private repository as 1.0.0. You can now import that package into your application and use it.
You continue to work on my-wonderful-lib-alpha until it gets to version 15.1.0, at the same time as using it in your application. It is now ready to reveal to the Elm community. You rename it and publish to package.elm-lang.org as my-wonderful-lib 1.0.0. You update your application to use this instead of the private alpha version.
Sharing Software Components

You are working for ACME Corp and there is a team that produces the companies look and feel across all of its products. This consists of several Elm packages; a UI framework and an ever growing set of standard widgets. The widgets package depends on the framework package, and both are published to the companies private Elm package server. ACME regards this material as proprietary and does not wish to share it and make it available as open source. ACME takes a structured approach to software engineering and likes all internal software releases to be versioned and made available using semantic versioning.
This package server makes the internal packages available when they are published. It provides a common place to find such packages, and to automatically resolve software dependencies against.
Supporting Larger Scale Engineering

You are a technical architect at ACME Corp, designing a number of software components that will work together. You design a server process with an HTTP interface, and figure out the structure of the messages it will consume and produce. You turn this into a data model as Elm code, and also give the
types of all the HTTP endpoints in Elm. You publish this internally as a private package, that the software developers writing a client will consume. As the API evolves, you can release new versions of it, controlling the lifecycle of its evolution, and with semantic versioning. You use GitLab permissions to ensure that only the architecture team can modify the API, avoiding design drift and enforcing the specification through Elm types.
Software modules help you to have control over data types that span multiple teams and software components. The package server also provides a place to share these interfaces and documentation for them.
-- This example also suggests that a useful feature might be to be able to control who can publish certain packages. On the main package site, packages are locked to github accounts, since you need to tag a package to publish it. This would need interfaces onto common source control systems to do. For example, a GitLab or Stash plugin, that checks you have credentials on those systems - or just checks the tags?