Grant Proposal - Make documentation installable and accessible

rakudoc-grant.md

Grant Proposal - Make documentation installable and accessible

Raku has code for handling the installation of distributions through CompUnit::Repository::Installation. It handles the part of the installation process that involves copying the files into their final location, path normalization, precompilation, as well as providing an interface for querying what is installed. Unfortunately it does not yet handle anything related to documentation.

Benefits to the Raku community

Developers can query for documentation

Currently there is no useful way of finding documentation for Raku distributions. There is a program called rakudoc, but it is generally only useful for core documentation. This would allow developers to query for documentation similar to how they can query for module names.

Developers more likely to write documentation

Since there is not a general way of accessing documentation, documentation in Raku distributions has been nearly non-existent compared to those in the Perl community.

Solves at least half of Raku/problem-solving/issues/393

The issue seen at Raku/problem-solving#393 is about how installing Raku distributions involves installing files that have not been declared in the META6.json file. While that issue affects more than documentation files, the solution implemented by this grant would act as prior art for what needs to be done to e.g. declaratively install bin/ files from a distribution.

Unblocks other RakuDoc related work

One example I can think of is related to the work being done on rakudoc renderers, particularly in regards to generating online html documenation a site like raku.land might use. These tools will be able to leverage the aforementioned META6.json updates.

Deliverables

• A META6.json spec for defining documentation related data is created.

• Updates to core Raku code that allow it to install, uninstall, and query-for documentation files as defined by the updated META6.json spec.

• Docs for CompUnit::Repository::Installation and modules are updated to mention the above changes.

• Examples are added (via the roast as spec tests) that show how to access documentation using core Raku functionality.

Note: Integration into rakudoc is not intendended to be part of this. Such an integration will be little more than copying and pasting the aforementioned examples, but there is some other unrelated work that would need to be done as well (making it installable via zef and how the user interacts with the CLI in regards to e.g. api/version/auth and multiple matching namespaces).

Project details

We need to update the META6.json spec to allow declaring our documentation files simlar to how we declare our source files with provides. Something like:

"provides-docs" : {
    "Text::Foo" : "docs/Text/Foo.rakudoc",
    "Text::Foo::DocInModule": "lib/Text/Foo/DocInModule.rakumod",
}

This will allow us to know where to look up the documentation for a given namespace, including know if the source is in a module or a stand alone documention file.

When installing a distribution Raku could then iterate through provides-docs to install (and precompile) the given files. There is a bit of nuance involved in this part due to the way rakudo hashes files to be used in namespace look ups. Rakudo does something like $filename = sha1("Namespace::Being::Installed") when installing modules which is also used when looking up namespaces, but that would be ambigious when we have something like:

"provides": {
    "Namespace::Being::Installed": "lib/Namespace/Being/Installed.rakumod"
},
"provides-docs": {
    "Namespace::Being::Installed": "docs/Namespace/Being/Installed.rakudoc"
}

The Raku logic for querying installed files could then be extended to also search documentation namespaces. This will involve adding a new method similar to the existing candidates method, or alternatively a new argument on the existing method. Again, this will be some nuance here involving the namespace normalization and file system location.

At last we can write the spec tests to show how these pieces are used. We'll want to ensure cases such as the ambigious namespace example above are covered.

Biography

I am Nick Logan, ugexe on github. I have been a co-author of the Raku module manager Zef, and a core developer of the Raku programming language for over 10 years. I am also a member of the Raku Steering Council.

Requested amount

$2500 USD

@finanalyst I believe what I have in mind would allow this. It would work with similar mechanisms to installed raku modules:

# Given the following in a META6.json:
#
# "provides-docs" : {
# 	"Foo::Bar": "docs/Foo/Bar.rakudoc"
# }

my $site-repo = CompUnit::RepositoryRegistry.repository-for-name('site');

my $doc-name-to-find = 'Foo::Bar';

# "document-candidates" (name TBD) emulates the existing "candidates" method for
# looking up a distribution by module name, except it searches "provides-docs"
# instead of "provides"
my $distribution = $site-repo.document-candidates($doc-name-to-find);

my Str $doc-source = do given $distribution { .content(.meta<provides-docs>{$doc-name-to-find}) };

say $doc-source;

you can see how this already works for modules with this snippet:

my $core-repo = CompUnit::RepositoryRegistry.repository-for-name('core');

my $name-to-find = 'NativeCall';

my $distribution = $core-repo.candidates($name-to-find).head;

my $source = do given $distribution { .content(.meta<provides>{$name-to-find}.keys[0]) };

say $source.open.slurp(:close);

@ugexe tl;dr The syntax seems fine.
A quibble and a request for confirmation, then some questions probably documented elsewhere

1. Why is it necessary to have a separate document-candidates method? candidates.head provides the first distribution in the chain that matches the search criterion. Once we have a distribution, surely all that is wanted would be do given $distribution { .content(.meta<provides-docs>{$doc-name-to-find}) } ?

• Separate candidates and document-candidates methods indicates there would be a distinction between a distribution that provides documentation, and one that does not.
• Suppose there is a fork of a distribution FOO with FOO:ver<1>auth<meticulous> and FOO:ver<2>:auth<fast&furious> in which meticulous has documentation while fast&furious has no documentation. A candidates method on FOO would presumably yield fast&furious, whilst a document-candidates would yield meticulous.
• Wouldn't that lead to a difference between the FOO being documented and the FOO being used?
• My thought would be that a documenting system should flag if the distribution that is used has no documentation, but serves the documentation for the distribution that does have documentation.

2. Am I right that .meta<provides-docs> without a key would produce a Hash with keys of all the file names associated with the distribution?

• This would be important in that documentation might be provided in several separate documents with extension .rakudoc or in the code with extension .rakumod.

Since I haven't looked at CompUnit for a while, please forgive me if these are silly questions.

3. Why is it necessary to have $source.open.slurp(:close) ? Isn't it sufficient now to have $source.slurp (presuming $source is a Path::IO object) ?
4. You provided core, and site as repository names, are there other possible names? Would * or () provide information about all repositories?
5. Suppose the program in which the code above is being invoked with raku -I. Is the distribution in . added to site?

Why is it necessary to have a separate document-candidates method? candidates.head provides the first distribution in the chain that matches the search criterion. Once we have a distribution, surely all that is wanted would be do given $distribution { .content(.meta{$doc-name-to-find}) } ?

Because that is a fairly significant change in behavior for existing uses. Consumers of candidates can know it is querying for module names and nothing else. If it also searched documentation then existing uses might start returning a different result based on a piece of documentation an unrelated distribution contains. We want to avoid affect module name resolution.

Separate candidates and document-candidates methods indicates there would be a distinction between a distribution that provides documentation, and one that does not.

I don't follow. One method searches provides, and one would search provides-docs. It does not imply anything about distributions themselves. Similarly if you want to find distributions that has a specific bin/foo file, you would use the existing (and unfortunately named) .files(...) method. I think I understand what you meant, and address this later on by mentioning searching without using a fully qualified name is always going to have problems like this.

---

At this point I get the impression you may be assuming every document namespace in the proposed provides-docs would have a corresponding provides entry, but I'm not sure that should be the case. For example zef might have a document namespace zef despite the fact there is no module with that namespace (and hence candidates won't find it by zef).

---

Suppose there is a fork of a distribution FOO with FOO:ver<1>auth and FOO:ver<2>:auth<fast&furious> in which meticulous has documentation while fast&furious has no documentation. A candidates method on FOO would presumably yield fast&furious, whilst a document-candidates would yield meticulous.

Yes, but I kind of think that is to be expected. Querying for namespaces without the name/api/auth/version has always been a potential source of pain. Ultimately this won't be a high level tool for end users, but something used to build those high level tools. For example a tool like rakudoc could call document-candidates first, then call candidates/resolve to see if the document candidate matches the module candidate. If it does not match it could warn the user or author to be more specific.

Wouldn't that lead to a difference between the FOO being documented and the FOO being used?

If you are querying for an exact namespace including name/api/auth/version then no. If you are not querying for an exact namespace then yes things might not do what you expect. The same problem exists for candidates and module names. The unfortunate solution is to always use the full long name.

My thought would be that a documenting system should flag if the distribution that is used has no documentation, but serves the documentation for the distribution that does have documentation.

I mentioned how that can be achieved earlier by end user tools like rakudoc via a call to document-candidates and candidates. I'm not sure I follow why there should be any additional logic occurring when something is used, and indeed I think we want to actively avoid adding any runtime penality to module loading just so the user can potentially query for documents. Maybe I'm misunderstanding this point though.

Am I right that .meta without a key would produce a Hash with keys of all the file names associated with the distribution?

It would be exactly like provides, except the file name the author lists would be for documentation files instead of module files. So yes, but note there is nothing automatic about this - the author will have to explicitly declare these similar to provides.

Why is it necessary to have $source.open.slurp(:close) ? Isn't it sufficient now to have $source.slurp (presuming $source is a Path::IO object) ?

Because it is not an IO::Path, it is an IO::Handle. Technically you shouldn't assume that you can access the path of $source and ideally you can just work with the source that gets slurped. This is because modules like CompUnit::Repository::Github can generate a distribution that isn't back by a file at all (rather a socket), and so something like $source.IO.e shouldn't be expected to work. In reality you might need a file path... this can still technically be figured out from the IO::Handle object but I'll leave that as an exercise to the read so as not to encourage that pattern.

You provided core, and site as repository names, are there other possible names? Would * or () provide information about all repositories?

$ raku -e 'say $*REPO.repo-chain.map(*.?name)'
(home site vendor core Nil Nil Nil)

$ raku -e '.say for $*REPO.repo-chain'
inst#/Users/nlogan/.raku
inst#/Users/nlogan/.rakubrew/versions/moar-blead/install/share/perl6/site
inst#/Users/nlogan/.rakubrew/versions/moar-blead/install/share/perl6/vendor
inst#/Users/nlogan/.rakubrew/versions/moar-blead/install/share/perl6/core
ap#
nqp#
perl5#

Additionally code (such as the aforementioned CompUnit::Repository::Github) can register other names at runtime. For example CompUnit::Repository::Github would show a github name and a line like github#repo<Raku-PathTools>#branch<main>#/

Suppose the program in which the code above is being invoked with raku -I. Is the distribution in . added to site?

Nothing is added to so a repository unless you explicitly install it. -I. type includes use CompUnit::Repository::FileSystem (name file), see:

$ pwd
/Users/nlogan/repos/zef/foo

$ raku -I. -e '.say for $*REPO.repo-chain'
file#/Users/nlogan/repos/zef/foo
inst#/Users/nlogan/.raku
inst#/Users/nlogan/.rakubrew/versions/moar-blead/install/share/perl6/site
inst#/Users/nlogan/.rakubrew/versions/moar-blead/install/share/perl6/vendor
inst#/Users/nlogan/.rakubrew/versions/moar-blead/install/share/perl6/core
ap#
nqp#
perl5#

So -I. in this case just puts a non-default repository to the front of the chain (and thus it will be queried first which affects resolution). Note that a CompUnit::Repository::FileSystem only ever represents a single distribution unlike CompUnit::Repository::Installation which may or may not be an important distinction for you.

@ugexe Thanks for the extensive response. I now appreciate the reasoning for document-candidates and candidates. I think the extra explanation about different use cases and the possibility of different name spaces might be usefully included in the grant proposal. It underlines the need for the new functionality.
All the other explanations and the examples were very useful.

@ugexe
I accidentally stumbled over this. (I hope hopping into this thread is not a faux-pas.)

Since there is not a general way of accessing documentation, documentation in Raku distributions has been nearly non-existent compared to those in the Perl community.

I can't stress enough how important I believe this topic is and thus how much I support this effort. (Doing something about ecosystem documentation has been on my list for a long time, it just didn't manage to make it to the top.)
If there is anything I can do to support you in doing this - do tell!

One question I have: If in a module I exclusively have documentation mixed into the code, will I have to have identical provides and provides-docs sections?

@patrickbkr

One question I have: If in a module I exclusively have documentation mixed into the code, will I have to have identical provides and provides-docs sections?

Yes, at least initially. There has to be a way to mark that a file contains documentation, as well as mapping that file to a namespace it is associated with. Theoretically it might be possible to detect documentation in a source file and automatically populate provides-docs at install time, but that will lead to various edge case issues. One might want to add documentation to their source code file (for internal documentation) while providing user-facing documentation via a rakudoc file. It is also useful to know what actions can be done with a distribution by just looking at the META6.json data instead of having to download, build, and precompile (and potentially install) a distribution.