Some words about a project idea codenamed Duchess - the goal is to make a replacement for Sandcastle and make creating documentation for open source project cool again

duchess.md

Burn Sandcastle to the Ground

Addressing The Documentation Problem

I've had lots of discussions with people over the past few days around how open source projects in the .NET space have poor documentation (if it exists). So how do we fix this?

At a high-level, I've got these goals in mind:

• make it easy
• make it cool
• leverage existing work in the .NET space
• bring in cool ideas from other ecosystems

I am going to codename this thing Duchess in lieu of a better name.

Make It Easy

One of the big reasons why Sandcastle has such a bad name is because it is really hard to setup (and automate).

Duchess should be one thing that can be run from the command line.

Duchess should be NuGet-able.

Duchess should be opinionated enough that you just point it at your project folder and it should just work. Opinions like:

• docs folder representing the site template
• Markdown for custom docs
• scan for //bin//*.xml (warning: probably not a real wildcard) files and use those for your APIs
• use magical unicorns to weave together custom Markdown files with generated documentation

Make It Cool

CHM files are a relic of a bygone era. Just let them go.

Duchess should generate a site or something that people actually like.

The default site should contain things like Bootstrap so that the site looks pretty out-of-the-box.

It should work with GitHub pages, or allow people to just deploy it like a static site.

Jekyll is a popular-enough static site generator that I'll generally defer to their conventions.

Leverage Existing Work

Developers still like using XML documentation.

We should parse that as a starting point, and that would require the output to behave something like CHM to help migrate existing projects across (unless they want to break their Google juice).

Cool ideas

• runnable code snippets (using LinqPad) which will fail the site generation if they don't work
• git push should let developers publish the site to Azure/Heroku (oh dear what am I thinking)
• annotate your source code -> documentation picks it up

What's next?

• spike something that'll take an XML documentation file and generate Markdown
• a simple site template using Bootstrap but with treeview-like behaviour (ew, maybe something better)
• people to contribute their thoughts on this

ETA: no clue. I'm exhausted and going on holidays in a few days.

Feel free to leave a comment or ping me on Twitter - I'm @shiftkey.

docs folder representing the site template
Markdown for custom docs
scan for //bin//*.xml (warning: probably not a real wildcard) files and use those for your APIs

A Thousand Times This. I've been thinking about this a lot lately actually, your timing is near-prescient.

So, the tricky part, is being able to merge the XML documentation with the Markdown documentation. I don't want to document API methods in VS XML. We almost need a way to do "Markdown Partial Classes™" - i.e. a way to merge the generated result of the XML files + the Markdown files that are hand-written.

@xpaulbettsx thanks for confirming i'm not crazy

I was imagining the site folder looking just like a Jekyll folder structure (hell, why not ship your blog from your source code?) and for those who are doing XML documentation the output of that goes under "/api/"

I'm mostly thinking from the XML documentation side but if you don't want that then going nuts with Markdown is something I need to flesh out.

"Markdown Partial Classes™"

Merging that's gonna be a real pain, but for a start why not introduce a convention for MyProject.Foo.md to be associated with the XML documentation for the MyProject.Foo type.

The JavaScript community have had some good improvements in this space in recent months (years?) that would be a good place to take ideas from. Two projects which come to mind are JSDoc and Docco.

These are both static site generators (well JSDoc uses 3rd party tools like JSDoc Toolkit to do that) which use JavaScript-friendly versions of XML comments.

Docco is a bit more advanced as it allows you to do comments anywhere in your source that you generate your documentation site from. Docco is just a command-line tool that you can easily plug into a build system so that's a good example of how to do the workflow you're talking about.

I really like Marginalia, it'd be awesome to generate this from NuGet packages (since you've got the deps info)

That's also a good question, philosophically - how does docs + NuGet fit together?

Definitely not crazy.

I think a key part is being able to merge XML docs with other docs (i.e. what we might currently include in github wiki's atm).

I also think having a good "linking" story is really important. For instance, if I have a supporting .md file, you should be able to link to docs for namespaces/types/methods/etc like you currently do with xml docs.

Further more, being able to have a reference to source code. When you reference source code, it should use the actual source from your repository (i.e. you shouldn't have to copy and paste code samples over from your code. Best example of this is GitHub API Docs where the JSON definitions they show actual are pulled out of the source code at run time. See - Finial output, http://developer.github.com/v3/repos/comments/ source https://github.com/github/developer.github.com/blob/master/content/v3/repos/comments.md (see the <%= headers 200 %> <%= json(:commit_comment) { |h| [h] } %> reference).

Definitely behind this and the fact that it should be able to run on build servers without out and "hard" dependencies (i.e. dependencies that can't be gather via nuget or similar). Sandcastle is a complete fail at this currently.

how does docs + NuGet fit together?

This is an excellent question. I've lost count of the number of times I'll curse a package's project URL for being unhelpful - this is probably part of the reason why people are hesistant towards OSS.

Some thoughts:

• this could be a baseline for getting documentation as a first-class citizen in NuGet (having just links and README files make me sad)
• if the documentation is inside the package then it could be run in offline mode (host the site in IIS Express or Katana)
• how would the user "view" (i.e. navigate to) the documentation inside VS? without VS?

run in offline mode (host the site in IIS Express or Katana)

Don't forget that frameworks like Nancy/WebAPI can now run self hosted in their own process.

@avanderhoorn this is a development-time thing, so it would just be a simple webserver listening on port XYZ. It doesn't care about what web technology you use in your app (I did this with Katana on a related project) as long as it runs everywhere (see Mono support).

I really like Marginalia, it'd be awesome to generate this from NuGet packages (since you've got the deps info)

Yep, it's brilliant. Definitely something I'd like to emulate.

Totally agree about being technology agnostic atm. Was more trying to re-enforce that I agree about having a simple webserver running the site as an option.

My thoughts:

• There should be an MSDN lite HTML output option. I haven't got a strong opinion on server generated ASP.NET pages vs static HTML pages but other people might. Either way some degree of HTML customization is important. Maybe HTML isn't a V1 feature but it should be designed with multiple outputs in mind
• It is important to remember when prioritizing that XML docs are primarily for intelli-sense. Custom high level doc pages are more useful to people learning a framework than method descriptions.
• As well as markdown for custom doc pages there is an XML standard called MAML - http://en.wikipedia.org/wiki/Microsoft_Assistance_Markup_Language - Sandcastle and I think MSDN uses this format for defining custom doc pages. Probably not a V1 feature but it should be designed with multiple inputs in mind
• Need to support table of contents. Up to you how you represent it but Sandcastle has a format you can look at for ideas - https://github.com/JamesNK/Newtonsoft.Json/blob/master/Doc/doc.content
• Need to support extracting code from a file and inserting it into the documentation. Sandcastle has a good plugin for this. Anything inside a #region is automatically inserted

If you want to see a semi-complex Sandcastle example the source code for the Json.NET documentation is here - https://github.com/JamesNK/Newtonsoft.Json/tree/master/Doc - and the final website is here - http://james.newtonking.com/projects/json/help/

I don't have time to work on the core of another OSS project but if in 6 months this project looks promising for me to use I'm happy to contribute back what extensions I make ✨

@JamesNK Thoughts on thoughts:

• Extensions for outputs/input is certainly something i'll keep in mind going forward - the defaults should "just work" but should be easy to opt-in for different things
• Migrating existing content is something I need to scope for V1 - I really want to focus on the "new project" use case first.
• Table of Contents - yes (would prefer Markdown nested lists)
• Extracting code from a file - hell yes, this is something I want to do (but regions? maaaaaan)

I'm just shaking out the concept at the moment, no clear timelines yet. @avanderhoorn showed me something cool which is in this space that he's looking to unveil soon which might help steer some of these things...

Random ideas from a brainstorming session over dinner this evening:

• extend the VS IDE to allow users to annotate the class/method (which are stored in a different location to the code) as a simpler version of XML documentation
• some developers open a browser to the documentation site - it would be cool to do this inside a different VS panel. For additional awesome the browser could navigate to the current API you're focused on automagically (could be annoying)

Moving this discussion to here so I can write proper specs and allow others to contribute directly.