Grant Proposal - Make documentation installable and accessible

rakudoc-grant.md

Grant Proposal - Make documentation installable and accessible

Raku has code for handling the installation of distributions through CompUnit::Repository::Installation. It handles the part of the installation process that involves copying the files into their final location, path normalization, precompilation, as well as providing an interface for querying what is installed. Unfortunately it does not yet handle anything related to documentation.

Benefits to the Raku community

Developers can query for documentation

Currently there is no useful way of finding documentation for Raku distributions. There is a program called rakudoc, but it is generally only useful for core documentation. This would allow developers to query for documentation similar to how they can query for module names.

Developers more likely to write documentation

Since there is not a general way of accessing documentation, documentation in Raku distributions has been nearly non-existent compared to those in the Perl community.

Solves at least half of Raku/problem-solving/issues/393

The issue seen at Raku/problem-solving#393 is about how installing Raku distributions involves installing files that have not been declared in the META6.json file. While that issue affects more than documentation files, the solution implemented by this grant would act as prior art for what needs to be done to e.g. declaratively install bin/ files from a distribution.

Unblocks other RakuDoc related work

One example I can think of is related to the work being done on rakudoc renderers, particularly in regards to generating online html documenation a site like raku.land might use. These tools will be able to leverage the aforementioned META6.json updates.

Deliverables

• A META6.json spec for defining documentation related data is created.

• Updates to core Raku code that allow it to install, uninstall, and query-for documentation files as defined by the updated META6.json spec.

• Docs for CompUnit::Repository::Installation and modules are updated to mention the above changes.

• Examples are added (via the roast as spec tests) that show how to access documentation using core Raku functionality.

Note: Integration into rakudoc is not intendended to be part of this. Such an integration will be little more than copying and pasting the aforementioned examples, but there is some other unrelated work that would need to be done as well (making it installable via zef and how the user interacts with the CLI in regards to e.g. api/version/auth and multiple matching namespaces).

Project details

We need to update the META6.json spec to allow declaring our documentation files simlar to how we declare our source files with provides. Something like:

"provides-docs" : {
    "Text::Foo" : "docs/Text/Foo.rakudoc",
    "Text::Foo::DocInModule": "lib/Text/Foo/DocInModule.rakumod",
}

This will allow us to know where to look up the documentation for a given namespace, including know if the source is in a module or a stand alone documention file.

When installing a distribution Raku could then iterate through provides-docs to install (and precompile) the given files. There is a bit of nuance involved in this part due to the way rakudo hashes files to be used in namespace look ups. Rakudo does something like $filename = sha1("Namespace::Being::Installed") when installing modules which is also used when looking up namespaces, but that would be ambigious when we have something like:

"provides": {
    "Namespace::Being::Installed": "lib/Namespace/Being/Installed.rakumod"
},
"provides-docs": {
    "Namespace::Being::Installed": "docs/Namespace/Being/Installed.rakudoc"
}

The Raku logic for querying installed files could then be extended to also search documentation namespaces. This will involve adding a new method similar to the existing candidates method, or alternatively a new argument on the existing method. Again, this will be some nuance here involving the namespace normalization and file system location.

At last we can write the spec tests to show how these pieces are used. We'll want to ensure cases such as the ambigious namespace example above are covered.

Biography

I am Nick Logan, ugexe on github. I have been a co-author of the Raku module manager Zef, and a core developer of the Raku programming language for over 10 years. I am also a member of the Raku Steering Council.

Requested amount

$2500 USD

@ugexe Thanks for the extensive response. I now appreciate the reasoning for document-candidates and candidates. I think the extra explanation about different use cases and the possibility of different name spaces might be usefully included in the grant proposal. It underlines the need for the new functionality.
All the other explanations and the examples were very useful.

@ugexe
I accidentally stumbled over this. (I hope hopping into this thread is not a faux-pas.)

Since there is not a general way of accessing documentation, documentation in Raku distributions has been nearly non-existent compared to those in the Perl community.

I can't stress enough how important I believe this topic is and thus how much I support this effort. (Doing something about ecosystem documentation has been on my list for a long time, it just didn't manage to make it to the top.)
If there is anything I can do to support you in doing this - do tell!

One question I have: If in a module I exclusively have documentation mixed into the code, will I have to have identical provides and provides-docs sections?

@patrickbkr

One question I have: If in a module I exclusively have documentation mixed into the code, will I have to have identical provides and provides-docs sections?

Yes, at least initially. There has to be a way to mark that a file contains documentation, as well as mapping that file to a namespace it is associated with. Theoretically it might be possible to detect documentation in a source file and automatically populate provides-docs at install time, but that will lead to various edge case issues. One might want to add documentation to their source code file (for internal documentation) while providing user-facing documentation via a rakudoc file. It is also useful to know what actions can be done with a distribution by just looking at the META6.json data instead of having to download, build, and precompile (and potentially install) a distribution.