nimble develop enhancements

workflow.md

Only new behavior is captured, everything else should work per usual.

Project local dependencies CLI flag

Nimble recently added the ability to work with project local dependencies. Simply by creating a nimbledeps directory in the project is sufficient to go into this mode. Once this directory exists, all information that typically goes into ~/.nimble is stored here, unaffected by any other projects or global changes.

This feature is being extended with a new -l | --local flag which sets up a project in localdeps mode. For most cases, all it does is creates a nimbledeps directory if it doesn't already exist.

E.g. nimble -l init will initialize a project in localdeps mode.

This flag is only useful when initially working with a project or a pre-existing project folder. Once the nimbledeps folder is setup, the flag is redundant.

nimble develop pkg

The nimble develop pkg command clones a new project so setting it up in localdeps mode requires some additional code changes.

Using nimble develop -l karax for example will result in the following actions:

git clone https://github.com/pragmagic/karax karax
cd karax
mkdir nimbledeps

After this, nimble will setup all dependencies in ./karax/nimbledeps since project is now in localdeps mode.

Develop dependencies

Larger Nim projects require code changes to dependencies as well. This requires the user to check out each dependency separately in nimble develop mode.

This workflow can be simplified with a new -d | --deps flag for nimble develop which will also set up all dependencies in develop mode.

Running nimble develop -d karax will result in the following directory structure:

./dotenv/...
./karax/...
./ws/...

The top level project karax and all its dependencies will be checked out with git clone at #head as sibling directories. This also means all requires statements with version specific information will be only used to identify dependencies but not their actual versions since you develop on #head.

Project local dependency mode

By default, nimble will use ~/.nimble and link all packages per usual.

If -l | --local is also specified, the project and all dependencies will be set up in project local dependency mode.

Running nimble develop -l -d karax will result in the following directory structure:

./nimbledeps/...						# contains all project local nimble metadata
./dotenv/nimbledeps/nimble.nimble-link	# links to ../../nimbledeps
./dotenv/...
./karax/nimbledeps/nimble.nimble-link	# links to ../../nimbledeps
./karax/...
./ws/nimbledeps/nimble.nimble-link		# links to ../../nimbledeps
./ws/...

This enables project isolation and the project and each dependency can be developed while being linked to their dependencies.

Any nimble commands to add/remove dependencies will work the same whether within the parent directory, the top level project ./karax or in a dependency directory.

I feel the main drawback of the local deps feature is that it tries to create a new proper way to use Nimble in which you can opt-in, but it still leaves other default behaviors that can lead you to problems down the road.

Localdeps mode was added because user wanted isolation for various reasons. As such, my proposal above works in both global as well as local mode because both are now available to nimble users.

but if we can get a working system that needs only the global cache, why shouldn't we?

As mentioned above, local mode is available to nimble users so any develop related changes need to work in this mode as well. If your approach does do away with localdeps mode completely (and the various reasons why it was added in the first place), it has not been clear.

Instead, @bobeff's approach aims to solve the problems with the global cache, so even the unexperienced users can get the best possible experience out of the box.

Can you please explain what these problems are and how bobeff’s approach solves them?

But in each top-level project's .develop file you can also easily temporary redirect the path only of a particular package.

If this is a temporary need then Nim's --path should be sufficient to pull this off - there's no need to maintain develop files when we already have cfg/nims to override what Nimble does.

What I'm really trying to get at is reusing existing functionality and user workflows within nimble/nim to solve this real requirement without introducing new concepts and commands that users need to deal with. I don't yet see anything fundamentally not possible with our existing concepts, hence, all these questions.

Minimal version selection makes the lockfiles unnecessary

I'll leave this out of this conversation since I do not want to mix it with this develop mode requirement. Frankly, minimum version selection is simpler in implementation as well as user education compared to lock files but I don't feel it is relevant in this conversation.

I see that bobeff has merged most of the latest nimble changes which is encouraging. However, I still implore more engagement on what the user workflow is going to be, what specific problems are being solved by all his work. Obviously, he started with lock files but there's so much more that's been done over this time that has resulted in 6.5k lines of difference.

Deferring the discussion until when the giant PR is ready is not prudent and will lead to delays and frustration on both sides for no good reason. Further, I have no clue what I can work on with Nimble while we wait.

There is a lot of discussion here, but this is the most important:

However, I still implore more engagement on what the user workflow is going to be, what specific problems are being solved by all his work. Obviously, he started with lock files but there's so much more that's been done over this time that has resulted in 6.5k lines of difference.

This worries me. Based on the discussion above it appears that there is a new feature being implemented that directly conflicts with the localdeps mode that was already implemented. I realise that this may be necessary to achieve the vision set out by you and @bobeff, but perfection is the enemy of the good, and in software development it's almost always much better to start with something small and iterate on it quickly.

So please, get these changes finalised and get them to us for review, even if they are not yet "perfect".