mxcl/Lockfile.md

## Lockfile.md

      
    Raw
  

              Lockfile.md
            
          
    Dependency Locking with the Swift Package Manager

Introduction

In a vibrant packaging ecosystem, dependencies update continuously with bug-fixes and feature improvements. When working against a collection of dependencies, especially in a team, it is vital to lock the graph and update it so that all team members receive the exact same graph at the same time.
The typical flow for this in dependency management is to create and use a dependency “Lockfile”.
Motivation

Our Lockfile system would be used for:

Ensuring that at any specific version-control revision the exact state of the dependency graph is recorded
Allowing users to lock, alter or “override” the URLs and or versions of packages in the graph. This is often required because:

They need to use a fork of a dependency that has a fix, or a private additional feature


Allowing users to commit and thus lock source alterations to their dependencies

Proposed Solution

A file: Packages/Lockfile exists in the source tree alongside the cloned packages.

The URL and versions of cloned packages
An inline diff of any modifications made to those packages relative to their pristine cloned states

This file should be checked-in with projects.
This file may be checked-in with packages designed for consumption in projects, if so it would allow the user to attempt to resolve a graph that exactly matches that of package author (though this will not always be possible, in some situations it is useful to try and replicate an exact dependency graph for reliability and bug fix purposes)
The Lockfile is generated by SwiftPM.
Any modifications made to the clones in Packages are recorded in the Lockfile as part of the flow described in the next section. Modifications here means: changes to git remotes and any local changes to the sources.
Detailed Design

In a fresh clone that does not contain a Packages directory swift build will determine the dependency graph, clone the packages into Packages and generate  Packages/Lockfile.
The user can now step into the Packages directory and modify package sources. If the user then runs swift build again the package manager will error out:
error: modified sources and unlocked sources
Execute `swift build --lock` or `swift build --ignore-lockfile`

It is an error to build against an unlocked dependency graph, but to facilitate fixing bugs etc. an ignore flag can be specified.
When swift build --lock is specified the package manager regenerates the lockfile
detailing the active git remote and the SHA that is checked-out. For local modifications it generates a diff against the check out and stores that in the Lockfile too.
Packages/Lockfile

There was concern with this feature using a file with lock in its name since this implies UNIX lockfiles. However, the emerging world of language dependency-managers has more or less settled on this term, so in order to be consistent our compromise is to call the file Lockfile and place it in the generated Packages directory. Thus it is clear that it is the lock for the cloned Packages and be not being named foo.lock it does not appear to be a UNIX lock.
The exact design of the contents of the Lockfile will be explored during iterative development, but here is a possible example (using TOML):
[package]
clone: Packages/PromiseKit-3.0.3
origin: https://github.com/mxcl/PromiseKit
version: 3.0.3

[package]
clone: Packages/Alamofire-1.2.3
origin: https://github.com/a-fork-somewhere/Alamofire
branch: crucial-fix

[package]
clone: Packages/Quick-1.2.3
origin: https://github.com/Quick/Quick
tag: 1.2.3
diff: INLINE-DIFF
Rationale for Local Diffs

Storing local diffs encourages users to edit, fix and improve their packages while preventing them from making changes that will not be stored as part of the dependency information revision history.
Workflow — Regular Build

The user is expected to commit the lock file into the git repo for others to reproduce the exact versions of dependencies on their system. This is optional, but strongly encouraged. Not checking it in is essentially saying: my project has uncontrolled dependencies: good luck!

User runs swift build
If Packages/ contains clones and a Lockfile SwiftPM skips to 7.
If Packages/ contains clones and no Lockfile the lockfile is generated from the clones
If Packages/ contains checked out sources without git information and no Lockfile SwiftPM fetches the git information and provided there is no diff, genereates the Lockfile, if there is variation it is an error *
If Packages/Lockfile is present its dependency graph is used
If Packages doesn't exist or is empty the depedency graph is resolved, packages are cloned and the Lockfile is generated
Build, if Packages are missing because we skipped from 2. the build will error, it is the user's responsibility to instruct SwiftPM to --update or to fix their dependency graph some other way.


This scenario is for users who choose to check in their complete dependency sources instead of a Lockfile, this is in fact a superior way to lock dependencies (because otherwise there is no gauarentee your dependencies will be unchanged since when you locked—it's the Internet dude) but to many it is distateful to check-in “derived data”.

Workflow — Making Modifications


User makes local modification to a dependency’s sources
User runs swift build
swift build errors out.
User must either lock the graph or run with --ignore-lock

Runing swift build --lock regenerates the lockfile, but doesn not build.
Workflow — Overriding Packages


User steps into a Package directory eg. Packages/Foo-1.2.3
User changes the origin of Foo to their own fork
User alters HEAD to point to a fix in their fork
swift build errors out.
User must either lock the graph or run with --ignore-lock

Running swift build --lock regenerates the lockfile, the new origin and tag is stored so if this project is freshly cloned it will use the overrides.
Workflow — Consuming a Package that wants you to override a dependency

A package, foo, depends on a package bar, bar has a bug, the author of foo fixes the bug in their own fork. The author of foo wants consumers of foo to use the bug fixed fork. What should happen here?
This is potentially a source of dependency hell, so we must be careful here to provide tooling that makes this situation tennable.
Solution: Dependent packages cannot cause overrides in root packages.
This is certainly unpleasant for package authors, but we have a responsibility to ensure a solid packaging ecosystem.
A package author would then specify the override that an end-user should configure for their own Lockfile.
As an alternative, the package author could change the dependency in their Package.swift to point to their own fork, and this is in fact fine. Especially if we follow through on our promise to lint the APIs of packages as part of a future publish step. As long as the API of a package is precise (and module collisions can be avoided via a namespacing system that is being proposed.) then we have avoided dependency hell.
Workflow — Updating Packages

SwiftPM has no update mechanism yet, but once it does running swift build --update will fetch the latest versions of all dependencies and update the lockfile.
Miscellaneous Details


The Lockfiles of dependencies are ignored and only the root Packages/Lockfile is used when resolving the graph (though we can add an optional feature to take into account other Lockfiles in the future)
The user is not expected to interact with this file as it'll always be generated by SwiftPM.

##Alternatives Considered
One alternative is to allow mentioning refs in manifest file while declaring a dependency but as discussed in this thread it might not be the best idea.
Using Git submodules for this feature was considered. It still could be implemented this way. We won't do it this way however because git-submodules are not widely understood and a Lockfile is clear. Ultimately it was firmly rejected because the local-diff feature could not combined.