nimble develop enhancements

workflow.md

Only new behavior is captured, everything else should work per usual.

Project local dependencies CLI flag

Nimble recently added the ability to work with project local dependencies. Simply by creating a nimbledeps directory in the project is sufficient to go into this mode. Once this directory exists, all information that typically goes into ~/.nimble is stored here, unaffected by any other projects or global changes.

This feature is being extended with a new -l | --local flag which sets up a project in localdeps mode. For most cases, all it does is creates a nimbledeps directory if it doesn't already exist.

E.g. nimble -l init will initialize a project in localdeps mode.

This flag is only useful when initially working with a project or a pre-existing project folder. Once the nimbledeps folder is setup, the flag is redundant.

nimble develop pkg

The nimble develop pkg command clones a new project so setting it up in localdeps mode requires some additional code changes.

Using nimble develop -l karax for example will result in the following actions:

git clone https://github.com/pragmagic/karax karax
cd karax
mkdir nimbledeps

After this, nimble will setup all dependencies in ./karax/nimbledeps since project is now in localdeps mode.

Develop dependencies

Larger Nim projects require code changes to dependencies as well. This requires the user to check out each dependency separately in nimble develop mode.

This workflow can be simplified with a new -d | --deps flag for nimble develop which will also set up all dependencies in develop mode.

Running nimble develop -d karax will result in the following directory structure:

./dotenv/...
./karax/...
./ws/...

The top level project karax and all its dependencies will be checked out with git clone at #head as sibling directories. This also means all requires statements with version specific information will be only used to identify dependencies but not their actual versions since you develop on #head.

Project local dependency mode

By default, nimble will use ~/.nimble and link all packages per usual.

If -l | --local is also specified, the project and all dependencies will be set up in project local dependency mode.

Running nimble develop -l -d karax will result in the following directory structure:

./nimbledeps/...						# contains all project local nimble metadata
./dotenv/nimbledeps/nimble.nimble-link	# links to ../../nimbledeps
./dotenv/...
./karax/nimbledeps/nimble.nimble-link	# links to ../../nimbledeps
./karax/...
./ws/nimbledeps/nimble.nimble-link		# links to ../../nimbledeps
./ws/...

This enables project isolation and the project and each dependency can be developed while being linked to their dependencies.

Any nimble commands to add/remove dependencies will work the same whether within the parent directory, the top level project ./karax or in a dependency directory.

Maybe instead of --local make it --localdeps?

Also -d might clash with --depsOnly, double check that isn't the case.

Running nimble develop -l -d karax will result in the following directory structure:

./nimbledeps/...						# contains all project local nimble metadata
./dotenv/nimbledeps/nimble.nimble-link	# links to ../../nimbledeps
./dotenv/...
./karax/nimbledeps/nimble.nimble-link	# links to ../../nimbledeps
./karax/...
./ws/nimbledeps/nimble.nimble-link		# links to ../../nimbledeps
./ws/...

This is a little weird. I would expect all of these to get cloned into ./karax/nimbledeps/... not in their own directory at $CWD.

I initially went with --localdeps but felt it was long. -d is also checked separately under each action so shouldn't conflict.

The directory structure is similar to what zah proposed in his forum reply and also similar to what Araq has mentioned - having dependencies accessible and not in a nested directory. By convention, it would all end up in $projectDir/nimbledeps/pkgs/dep.

Functionality-wise, this is close to what @bobeff has done with local develop files, but I'll describe the reasoning behind his design:

1) The "include" feature of the develop files makes it easy to manage large number of developed packages.

If every project had to reference all of its "develop mode" dependencies with separate links, the number of required links would have been N^2. Extracting some modules in a library would require you to create multiple links to the new library in all existing packages.

In contrast, the include feature makes it easy to create a single develop file that lists all of your libraries. With this file included in all of your top-level projects, both adding a new library or a new top-level project consist of a single operation.

Now, it's true that the scheme suggested above manages to reuse the entire nimbledeps directory and thus creates a similar setup, but this gets us to point 2)

2) You are not forced into a all-or-nothing decision regarding your dependencies.

In a more mature eco-system, a typical large project may have hundreds of dependencies. As an example, Lighthouse, our competing Ethereum 2 client implemented in Rust, has over 500 dependencies. From these, only 30 or so are packages developed by the Lighthouse team and thus are good candidates for the develop mode. In other words, I think most users will prefer to explicitly specify which packages are put in develop mode (either manually or through a script) than to create a directory with hundreds of third-party packages produced by a command such as nimble develop --deps.

The description above doesn't specify how the locally developed packages (e.g. ws) are discovered from the top-level project (e.g. karax), but I assume that there will be additional links in the shared ./nimbledeps/ folder. This graph of links can be equivalent to the tree of includes with the ".develop files" feature, but it may require the user to re-arrange the directory structure as she discovers new shared packages, new packages to be put in develop mode and so on. I feel the incremental steps with the .develop files are smoother and more natural to handle through the CLI or through manual editing of the contents of the .develop files.

3) Nimble should help the developer deal with the daily git pull.

When you run git pull every morning, you'll discover that most of your "develop mode" packages need to be updated (the lockfile of your top-level projects will point to new git revisions). This daily update must be handled in a highly interactive way because there are plenty of situations that require a decision from the user (e.g. a local dependency that must be updated may be in a dirty state, you may have started work on a diverging branch, etc).

The links in the nimbledeps directory don't carry enough semantic information for Nimble to be absolutely certain that the packages are actually in "develop mode". Without this explicit knowledge, it would be more difficult for Nimble to provide a highly specialized UI for all the complicated scenarios that may arise with the develop mode when you build, pull or push your code.

Hello @zah, thanks for taking the time to review this.

For #1, I think you already noticed but instead of introducing yet another nimble.develop file, I simply reused the .nimble-link implementation to point pkg/nimbledeps to ../nimbledeps. This reduces the amount of new code as well as avoiding yet another file that users need to worry about (gitignore, etc.). Note that if a user does not want to use localdeps mode, it will still work - you will end up with .nimble-link files stored in ~/.nimble/pkgs and no nimbledeps directories at all. So it minimizes the changes as well as the user impact.

You are not forced into a all-or-nothing decision regarding your dependencies.

For #2, I had the same concern and most users probably don't need all their deps in develop mode. But if this is the case, it would still be easier to simply do the following instead of adding all the flags proposed in the nimble.develop PR.

With current shipping behavior, where packages are specified in dep order:

nimble develop pkg1 pkg2 pkg3 pkg4 # uses ~/.nimble

For localdeps mode, we need a subset of the above proposal where we create nimbledeps links to the shared nimbledeps directory.

nimble develop -l pkg1 pkg2 pkg3 pkg4 # uses shared nimbledeps with nimble-link

Here, the packages identified would simply be those that the user really wants to develop and other dependencies will end up in the global ~/.nimble or shared nimbledeps directory. This would need a changes to nimble's processDeps() function where any dep identified out of order on the command line will need to be run through nimble develop. Without this change, they would end up with a version in $nimbleDir as well as #head checked out. It will work as expect though since #head will take precedence but would be messy.

Basically though, if the user needs to be specific about what they want, we don't even need the -d | --deps flag I proposed but the user would need to specify what they want.

The description above doesn't specify how the locally developed packages (e.g. ws) are discovered from the top-level project (e.g. karax)

When using ~/.nimble, nimble already knows where to find ws. For localdeps mode, ./karax/nimbledeps/nimble.nimble-link points to ./nimbledeps which contains a .nimble-link to ws in ./ws.

Adding / removing or converting packages between develop and regular mode would simply use standard nimble calls like nimble install, nimble remove or nimble develop. They would work whether run in the top level, the main project directory or a dep directory. No additional code is needed or a change in user workflow.

The links in the nimbledeps directory don't carry enough semantic information for Nimble to be absolutely certain that the packages are actually in "develop mode".

Nimble only creates links for develop packages - why is this not sufficient information? Further, the nimblemeta.json file also has an isLink: true value. If needed, we can even extend the metadata to state it is in develop mode.

The daily git pull can be discussed separately cause once we have the directory structure, it will just require the addition of some new flag or command like nimble develop -u in the top level directory which will then run through all the dependencies or something.

Let's not overvalue the "reduction of new code" and instead aim for a design with the best long-term properties.

I feel the main drawback of the local deps feature is that it tries to create a new proper way to use Nimble in which you can opt-in, but it still leaves other default behaviors that can lead you to problems down the road. Many users will fall into the trap of using the global cache for develop links or they will run with a single non-shared localdeps folder. Instead, @bobeff's approach aims to solve the problems with the global cache, so even the unexperienced users can get the best possible experience out of the box.

I'll address few particular critiques:

1) The .develop file clutters your working copy

I would say the same is even more true for the localdeps folder. I can name dozens of less sophisticated tools that can suffer from a large number of files in your project folder (e.g. text editors with less sophisticated "Find in files" features will give less relevant results; The Ctrl+P file navigation feature in a text editor can get slower; build systems that watch your project tree for modifications can run into file descriptor limits, etc). Hard disk space is indeed cheap nowadays, but if we can get a working system that needs only the global cache, why shouldn't we?

2) Links can do exactly the same as .develop files

This is not strictly true. The shared nimbledeps folder is roughly equivalent to having a shared .develop file that you include in your top-level projects. But in each top-level project's .develop file you can also easily temporary redirect the path only of a particular package. I can recall many situations where I needed to test how some new version of a particular component interoperates with an older working version of the same code and this extra flexibility of the .develop files makes it easier to setup small experiments like these.

You are also saying that having a link implies the so called "develop mode", but is this true? In the scheme you proposed, the entire shared nimbledeps directory is linked in each project. Shouldn't this imply that all of the packages in that directory are in develop mode, even the third party ones? It's true that metadata additions can address this.

3) Minimal version selection makes the lockfiles unnecessary

Again, this might look beneficial if the goal is to keep Nimble as small as possible, but it cuts into the flexibility that the lockfiles provide for the authors of libraries and for creating test setups. When @bobeff is so close to shipping the feature, why shouldn't we help him make the last few steps instead of pursuing alternative directions that will be more taxing for the end users and will require more development time at this point?

With lockfiles in place, it's clear that the interaction between Nim and Nimble will need to change somehow in order to provide the precise package selection in the presence of multiple package versions installed side-by-side (i.e. these are the nim.cfg plans). This gives us a chance to rethink some of the key operations concerning the global cache and allows us to solve the problems without introducing unnecessary fragmentation in the user experience (different users using Nimble in a different way, leading to "it works on my machine" type of problems)

I feel the main drawback of the local deps feature is that it tries to create a new proper way to use Nimble in which you can opt-in, but it still leaves other default behaviors that can lead you to problems down the road.

Localdeps mode was added because user wanted isolation for various reasons. As such, my proposal above works in both global as well as local mode because both are now available to nimble users.

but if we can get a working system that needs only the global cache, why shouldn't we?

As mentioned above, local mode is available to nimble users so any develop related changes need to work in this mode as well. If your approach does do away with localdeps mode completely (and the various reasons why it was added in the first place), it has not been clear.

Instead, @bobeff's approach aims to solve the problems with the global cache, so even the unexperienced users can get the best possible experience out of the box.

Can you please explain what these problems are and how bobeff’s approach solves them?

But in each top-level project's .develop file you can also easily temporary redirect the path only of a particular package.

If this is a temporary need then Nim's --path should be sufficient to pull this off - there's no need to maintain develop files when we already have cfg/nims to override what Nimble does.

What I'm really trying to get at is reusing existing functionality and user workflows within nimble/nim to solve this real requirement without introducing new concepts and commands that users need to deal with. I don't yet see anything fundamentally not possible with our existing concepts, hence, all these questions.

Minimal version selection makes the lockfiles unnecessary

I'll leave this out of this conversation since I do not want to mix it with this develop mode requirement. Frankly, minimum version selection is simpler in implementation as well as user education compared to lock files but I don't feel it is relevant in this conversation.

I see that bobeff has merged most of the latest nimble changes which is encouraging. However, I still implore more engagement on what the user workflow is going to be, what specific problems are being solved by all his work. Obviously, he started with lock files but there's so much more that's been done over this time that has resulted in 6.5k lines of difference.

Deferring the discussion until when the giant PR is ready is not prudent and will lead to delays and frustration on both sides for no good reason. Further, I have no clue what I can work on with Nimble while we wait.

There is a lot of discussion here, but this is the most important:

However, I still implore more engagement on what the user workflow is going to be, what specific problems are being solved by all his work. Obviously, he started with lock files but there's so much more that's been done over this time that has resulted in 6.5k lines of difference.

This worries me. Based on the discussion above it appears that there is a new feature being implemented that directly conflicts with the localdeps mode that was already implemented. I realise that this may be necessary to achieve the vision set out by you and @bobeff, but perfection is the enemy of the good, and in software development it's almost always much better to start with something small and iterate on it quickly.

So please, get these changes finalised and get them to us for review, even if they are not yet "perfect".