dustinlacewell/sip-1.org

## sip-1.org

      
    Raw
  

              sip-1.org
            
          
    Generalizing Styx

Motivation

Currently Styx bakes in a number of concepts and abstractions typical of
  “static site generators”. Some of these ideas are:

  The main build target is a “site”
  The main assets of the build are “pages”
  The ideas of “layout”, “templates” and “themes” are central

There are a number of advantages to following these standard conventions:

  User familiarity with the ideas
  Association with existing tools

However, being based on Nix, Styx has the room be much more flexible, and
  general than that. This document describes a plan to simplify the Styx core and
  move it towards more towards something that would be described as a
  “programmable content pipeline”.
Programmable Content Pipeline

The phrase “programmable content pipeline” attempt to denote a system that is
  useful for any kind of static content processing:

  Arbitrary data comes in as assets
  Assets are processed arbitrarilly
    
      transforming asset content
      deriving new assets from existing ones
      aggregating assets into taxonomies
      associating assets to each other
    
  
  Assets are targetted to disk

By narrowing Styx’s core to just this kind of system we both simplify it’s
  implementation but also open it up to more kinds of content processing such as
  audio, video, text analysis, and so on. Anything really.
Nomenclature

This document will use revised nomenclature for the parts of Styx to reflect
  its aims. The prominent changes are:

  Site -> Task: The build target is now a more generalized “task”
  Page -> Output: The artifacts of the build are now “outputs”
  Data -> Input: The structure feeding the outputs is now the “inputs”
  Environment -> Context The structure made available to library functions
  Layout -> Renderer: The function that finalizes output content is now the
    “renderer”
  Template -> removed: It is expected that output’s input and renderer attributes
    are sufficient to produce the final content
  Theme -> Module: Themes become modules which are packages which are passed to
    the styx import. These module’s can contribute their own library functions,
    inputs and outputs

Mock tasks.nix (minimal)

The following is a minimal example showing the new pipeline of tasks.nix:
{ pkgs ? import <nixpkgs> { }, extraConf ? { } }:

let
  styx = import pkgs.styx { config = [ ./conf.nix extraConf ]; };

  inputs.name = "ldlework";

  outputs.index = {
    path = "/example.txt";
    renderer = o: "Hello ${inputs.name}!";
  };

in styx.mkTask { inherit inputs ouputs; }
Mock tasks.nix (simple)

{ pkgs ? import <nixpkgs> { }, extraConf ? { } }:

let
  styx = import pkgs.styx {
    config = [ ./conf.nix extraConf ];
    context = { inherit inputs outputs; };
    modules = [
      pkgs.styx-templates
      pkgs.styx-sass
    ];
  };

  inherit (styx.lib) templates sass;

  inputs.message = name: "Hello ${name}!";

  outputs.index = {
    path = "/index.html";
    name = "ldlework";
    content = ''<div class="message">${inputs.message name}</div>'';
    renderer = templates.render ./templates/layout.html;
  };

  outputs.css = sass.load ./sass/site.sass // {
    path = "/site.css";
  };

in styx.mkTask { inherit inputs ouputs; }
Mock tasks.nix

Given that the following example only produces a single output, =about.html,= it
  is true that it could be a lot more simple.
However, it has been written in a “refactored” style to demonstrate that the
  user is now free to create their own abstractions in order to produce their
  inputs and outputs however they desire.
These helper functions could be tucked away in a local utils.nix, a local Styx
  module, or even a published Styx module package.
We don’t have to keep putting things into Styx to support more and more. We
  should just make it easy to grab those things and use them.
{ pkgs ? import <nixpkgs> { }, extraConf ? { } }:

let
  styx = import pkgs.styx { ①
    config = [ ./conf.nix extraConf ];
    context = { inherit inputs outputs; }; ②
    modules = [ ③
      pkgs.styx-markdown ④
      pkgs.styx-theme-generic ⑤
    ];
  };

  inherit (styx.lib) markdown templates; ⑥

  htmlRenderer = template: output: ⑦
    templates.layout (template output);

  pageDefaults = { ⑧
    renderer = htmlRenderer template.page.full;
  };

  page = attrs: pageDefaults // attrs; ⑨

  markdownPage = { file, ... }@args: Ⓐ
    let
      data = markdown.load ({ inherit file; }); Ⓑ
      attrs = data // args; Ⓒ
    in page attrs; Ⓓ

  inputs = {
    navbar = with outputs; [ about ]; Ⓔ
  };

  outputs = rec {
    about = markdownPage rec { Ⓕ
      file = ./data/sample/pages/about.md;
      title = "About";
      path = "/about.html";
      navbarTitle = title;
    };
  };

in styx.mkTask { inherit inputs ouputs; } Ⓖ
Description


  1. The pkgs.styx package must be called to be initialized. This performs
    configuration loading and type-checking, and initializes the library.
  2. The context is exposed to all library functions upon initialization.
  3. Module packages are passed to Styx so that their configuration can be loaded
    and their library initialized.
  4. Specific data transformation functionality is now externalized to
    modules. Modules can contain configuration, library functions, and their own
    outputs.
  5. Themes are now just normal modules. Modules can provide templates as library
    functions with a convention of using the shared styx.lib.templates namespace.
  6. Raising the styx.lib.markdown and styx.lib.templates namespaces for
    convenience.
  7. htmlRenderer is a helper function that takes a template function and an
    output attrset and applies template to output and then applies
    templates.layout to the result. This produces the final output content.
  8. pageDefaults is an attrset which sets the renderer to the htmlRenderer
    function so every “page-like” output works the same.
  9. page is a helper function that takes an attribute set attrs and merges them
    over the pageDefaults.
  A. markdownPage is a helper function to create outputs based on markdown
    sources.
  B. The markdown file is loaded and returned as an attrset. Internally,
    markdown.load uses styx.core.load to load the textfile, its metadata, and
    apply nix-lang interpolations.
  C. Merge in any custom attribute overrides.
  D. Call page attrs to produce an output with the pageDefaults. In otherwords,
    setting the output’s renderer.
  E. Our templates use inputs.navbar to generate a navigation bar.
  F. outputs.about is set to calling markdownPage to produce the final output
    attrset.
  G. styx.mkTask takes in the inputs and outputs. It will flatten the outputs,
    call the renderer for each, and write the result to the location in the
    path attribute.

pkgs.styx { …} startup

When the user calls pkgs.styx a number of things happen:

  Each module’s option-declarations are loaded and merged
  The merged option-declarations are resolved to defaults and type-checked
  The configuration sources are loaded and merged
  The merged configuration is applied to the merged option-declarations and
    type-checked
  Every module library is loaded with the following arguments:
    
      configuration set
      user supplied context
      the fixed-point of the library itself, the merged result of every module
        library
    
  
The returned attrset

An attribute set is returned to the user containing:

  core: styx.core
  conf: the loaded configuration
  lib: the merged module libraries
  context: the library context
  modules: the actual modules
  mkTask: the function that will perform the build

styx.mkTask function

The styx.mkTask function is much like mkSite with some differences:

  It takes both inputs and outputs
  It flatten outputs for the user
  It stores its full argument set to the returned derivation’s passthru
    attribute

passthru Attribute

The derivation’s passthru attribute allows us to store arbitrary Nix data on
  the derivation without affecting the derivation’s build environment.
This can be used for the documentation generator to get the information it
  needs without having to have the user return an attrset with the attributes we
  need.
styx.core

styx.core will contain much of what the built-in library contains today. The
  only difference is that it is no longer dependant on going through the startup
  process to import.
styx.lib

styx.lib will contain an attrset of functions, merged from all of the loaded
  module libraries. The libraries themselves have access to:

  the configuration
  the context
  the fixed-point lib itself

To make the library itself available to module libraries on loading, we use
  pkgs.lib.fix in order to produce a fixed-point version the library.
Styx Modules

Styx modules are packages which can contribute to a task. Their primary
  advantage is that they can provide their own configuration interfaces which can
  change their internal behaviors.
Module can contribue the following things:

  Option Declarations
  Outputs which get merged with task outputs
  Library functions

Module initialization

There is a bit of a chicken and egg problem with module initialization. While
  modules can provide both configuration and library functions, we need a
  complete configuration in order to load their library functions.
Another chicken an egg problem is that module libraries should be able to
  access functions in the library from other modules. But how can we already have
  the library on hand to pass to modules, before we’ve loaded them?
Configuration Initialization

Modules will be loaded in two phases. When calling styx.core.modules.load you
  will get back an attrset containing decls and partial.
The decls can be used to merge with the declarations of other modules and the
  user configuration to produce the complete configuration.
The complete configuration can then be used to call partial, which will load
  the rest of the module.
Library Initialization

Calling the module partial does not however fully load its library. Afterall we
  have to pass the merged library to it, in order to load it.
Therefore we load each module’s library as a function taking the following
  arguments:

  context: the user context attrset
  lib: the merged module libraries

To solve the infinite recursion we utilize nixpkgs.lib.fix in order to
  calculate the fixed-point of the merged library.
Styx CLI

The CLI will be updated to allow the user to build any task defined by their
  tasks.nix By returning an attrset of styx.mkTask calls, the user can provide
  named tasks. The CLI should be able to determine whether tasks.nix returns a
  single or multiple tasks.
In the case that the user defines multiple tasks, but no task name is provided
  to the CLI, the CLI should look for a task named ‘default’. It should provide
  an error if it can’t be found.
Local Dev, src/nixpkgs and themes/versions.nix

Currently in order to facilitate local development there is a
  src/nixpkgs/default.nix. There is also a themes/versions.nix which contains the
  github details and hashes for pinned versions of various themes that we want to
  build documentation for in the official docs.
Both of these mechanisms are dropped in favor of normal overlays. With an
  overlay like the one below at ~/.config/nixpkgs/overlays/styx.nix tools like
  nix-build will find the versions of the packages desire. At this point, Styx
  can just refer to the normal package names, and the correct local ones, or even
  github pinned ones, will be found and used (and documented)
self: super: {
  styx = super.callPackage /home/ldlework/src/styx/styx {};
  styx-theme-generic = super.callPackage /home/ldlework/src/styx/themes/generic-templates {};
  styx-theme-agency = super.callPackage /home/ldlework/src/styx/themes/agency {};
  styx-theme-showcase = super.callPackage /home/ldlework/src/styx/themes/showcase {};
}