Nimble RFC for lock file support

README.md

The following RFC aims to implement lock file support for Nimble. This is being tracked in issue 127 but this proposal is a simpler variation.

Intro:

• lock mode will be enabled by a flag in nimble.ini
  • Since it is a new feature, this will reduce impact on the existing user base
  • It can be turned on by default in the future when stable enough
• Lock information will be stored in $prj/nim.cfg
  • Nimble will append autogenerated section if existing
  • Nimble will update when any dep changes occur
  • Nimble will consume when having to install deps
  • Nim will consume per usual

Lock file format:

• Separate section in $prj/nim.cfg maintained by Nimble
• Nimble related metadata will be stored as nim comments
• E.g.

# Nimble autogenerated start
--noNimblePath
# url = https://...
# checksum = xyzxyz
--path:$nimbleDir/pkgs/package-1.0.0
# Nimble autogenerated end

• Format currently includes package name, version, url and checksum
• Nimble does not store source control hashes so that's not being considered for now

Nimble changes:

• Add code to write and read back from $prj/nim.cfg
• After any package install
  • Calculate and store checksum in either nimblemeta.json or a new file in pkg root
• processDeps()
  • Get packages from .nimble file
  • If $prj/nim.cfg exists
    • Get package versions from cfg file
  • For each package in .nimble requires
    • If package locked version exists
      • Install specific version of package if not already in ~/.nimble
    • Else
      • Install package using standard workflow today
      • Remember package version installed
      • Set create flag
  • If create flag set, create/update $prj/nim.cfg
• nimble install - use updated processDeps() workflow
• nimble install url|pkg - run same workflow as today
• nimble uninstall pkg - remove package per usual
• nimble build | docs | c | run
  • Use updated processDeps() workflow
  • Run Nim without --path: flags
• nimble develop - use updated processDeps() workflow
  • Impacted by develop --all RFC - TBD

Nim required changes:

• Replace instances of $nimbleDir in --path:$nimbleDir/xyz with path to OS specific Nimble directory
• Once lock files are always on and the only way to use Nimble, as part of issue 654
  • Deprecate and eventually remove --NimblePath and --cleanNimblePath
  • Remove code that searches packages in NimblePath

No changes:

• Dependencies will need to be added to the .nimble file in the requires section per usual to get installed and included in $prj/nim.cfg
  • Once nimble can update the .nimble file requires section automatically in the future during install/uninstall, the lock file can get auto-updated as well

Other issues addressed:

• Project isolation - issue 131
  • Since lock files point to specific versions, changes due to other projects will not matter
  • If someone does remove a dependency, it can be reinstalled with one of the nimble commands that runs processDeps()
• Reducing Nim's knowledge of Nimble - issue 654

Other issues expecting fix via lock files - not solved yet:

• Nimble develop of http package - issue 543
  • Nimble does not currently save sufficient info about packages to know for sure
• Skip dep checks - issue 589
  • Nimble will still run processDeps() - unclear if lock file will be faster

Notes:

• Does not include strictMode in original issue proposal
• URL is simply base URL to project and no commit hash - Nimble does not store this info today

This is as big of a workflow change - @dom96

It is step 1 in implementing lock files. It is turned off by default in nimble.ini and can be enhanced over time so as not to disrupt the community. Second, it is only relevant when you want lock files. If not, you still have the status quo. There is no workflow change if you don't want lock files. And if you want it, it is a new workflow anyway.

there is nothing about this proposal that enables a lock file implementation - @dom96

It charts a way to get there. Paths lock versions for you and that's all we have today. Nimble does not store enough data in its database to get a 100% solution in step 1.

So I see absolutely no point in hackishly putting it into the nim.cfg file. - @dom96

Agreed 100% it isn't pretty but a lock file does not have to be user facing. Why does the format matter if it functions correctly? It isn't ideal for sure but neither is adding more complexity into the tool for some ideal. The legacy is a .nimble and a nim.cfg so I'm trying to capitalize on that.

You are expecting people to run nimble setup before any nim execution(!) - @dom96

These people are showing willingness to using Nimble to setup their lock files before using Nim. Instead of recognizing that concession, you want them to change their workflows altogether. Why is nimble setup a bad idea when it can encourage more Nimble adoption?

This proposal adds another workflow to Nimble/Nim. I am not a fan of providing two different workflows which have the same use cases. - @dom96

Diversity is a fact of life. You can adapt it into Nimble or leave it for others to deal with. In either case, both will continue to exist.

We can do the same thing for nimsuggest.

No, we can't. And we won't. Already Nimble doesn't wrap other binaries well. And already nimsugggest is a fragile piece of software.

Already Nimble doesn't wrap other binaries well.

@Araq Yes, it does. I've already proven you wrong on this, why do you keep repeating this as fact?

there is nothing about this proposal that enables a lock file implementation - @dom96

Nimble does not store enough data in its database to get a 100% solution in step 1.

@genotrance I don't believe this is true, what information does Nimble not store in its database to make lock files possible?

Agreed 100% it isn't pretty but a lock file does not have to be user facing. Why does the format matter if it functions correctly? It isn't ideal for sure but neither is adding more complexity into the tool for some ideal.

@genotrance because that is in fact messy, I mean come on, you want to put lock-file-specific information into comments? I'm frankly shocked that @Araq is completely fine with this.

But forget about this seriously. You are misunderstanding what I'm saying, my point is that lock files are completely orthogonal to this proposal, so saying that this proposal is good because it "enables lock files" makes no sense. Let's stop talking about how this proposal enables lock files and focus on just the --path aspect.

It is turned off by default in nimble.ini and can be enhanced over time so as not to disrupt the community.

nimble c is already there! and it doesn't need to be used. People can choose to use nim c or nimble c. Once again, I do not want to force people to use nimble c, but that is the solution if using nim c as it is today is such a horrible thing (which I completely disagree that it is)

why do you keep repeating this as fact?

Because others chimed in and agreed with me. For instance, nimble build swallows the compiler's warnings.

I'm frankly shocked that @Araq is completely fine with this.

Actually I'm not, but I am not concerned with lock files.

Because others chimed in and agreed with me. For instance, nimble build swallows the compiler's warnings.

nimble build is a completely different topic. It doesn't replace any Nim commands, it's Nimble's own functionality.

I am still waiting to hear other's opinions about nimble c and why they dislike it. Can you point me to some comments from others about this? better yet, quote the passages here?

I'm frankly shocked that @Araq is completely fine with this.

Actually I'm not, but I am not concerned with lock files.

Whew. Thank goodness for that, this world hasn't turned upside down after all. This again goes to show that we shouldn't be conflating lock files with this proposal.

This proposal is off the table.

Despite being titled Nimble RFC for lock file support, the whole conversation here has gone on and on about Nim <=> Nimble interop.

The only real on-topic feedback I got was that you don't like the file format and this is not a full implementation, both of which are documented and explained multiple times. This was a simpler variation from the beginning, avoiding additional toml or json files, and leveraging existing infrastructure with minimal effort.

If this is just unbearable then there's nothing more to discuss.

Oh, in that case I must apologise. I didn't realise this proposal was primarily about lock files. :(