sip-1.org

Generalizing Styx

Motivation

Currently Styx bakes in a number of concepts and abstractions typical of “static site generators”. Some of these ideas are:

• The main build target is a “site”
• The main assets of the build are “pages”
• The ideas of “layout”, “templates” and “themes” are central

There are a number of advantages to following these standard conventions:

• User familiarity with the ideas
• Association with existing tools

However, being based on Nix, Styx has the room be much more flexible, and general than that. This document describes a plan to simplify the Styx core and move it towards more towards something that would be described as a “programmable content pipeline”.

Programmable Content Pipeline

The phrase “programmable content pipeline” attempt to denote a system that is useful for any kind of static content processing:

• Arbitrary data comes in as assets
• Assets are processed arbitrarilly
  • transforming asset content
  • deriving new assets from existing ones
  • aggregating assets into taxonomies
  • associating assets to each other
• Assets are targetted to disk

By narrowing Styx’s core to just this kind of system we both simplify it’s implementation but also open it up to more kinds of content processing such as audio, video, text analysis, and so on. Anything really.

Nomenclature

This document will use revised nomenclature for the parts of Styx to reflect its aims. The prominent changes are:

• Site -> Task: The build target is now a more generalized “task”
• Page -> Output: The artifacts of the build are now “outputs”
• Data -> Input: The structure feeding the outputs is now the “inputs”
• Environment -> Context The structure made available to library functions
• Layout -> Renderer: The function that finalizes output content is now the “renderer”
• Template -> removed: It is expected that output’s input and renderer attributes are sufficient to produce the final content
• Theme -> Module: Themes become modules which are packages which are passed to the styx import. These module’s can contribute their own library functions, inputs and outputs

Mock tasks.nix (minimal)

The following is a minimal example showing the new pipeline of tasks.nix:

{ pkgs ? import <nixpkgs> { }, extraConf ? { } }:

let
  styx = import pkgs.styx { config = [ ./conf.nix extraConf ]; };

  inputs.name = "ldlework";

  outputs.index = {
    path = "/example.txt";
    renderer = o: "Hello ${inputs.name}!";
  };

in styx.mkTask { inherit inputs ouputs; }

Mock tasks.nix (simple)

{ pkgs ? import <nixpkgs> { }, extraConf ? { } }:

let
  styx = import pkgs.styx {
    config = [ ./conf.nix extraConf ];
    context = { inherit inputs outputs; };
    modules = [
      pkgs.styx-templates
      pkgs.styx-sass
    ];
  };

  inherit (styx.lib) templates sass;

  inputs.message = name: "Hello ${name}!";

  outputs.index = {
    path = "/index.html";
    name = "ldlework";
    content = ''<div class="message">${inputs.message name}</div>'';
    renderer = templates.render ./templates/layout.html;
  };

  outputs.css = sass.load ./sass/site.sass // {
    path = "/site.css";
  };

in styx.mkTask { inherit inputs ouputs; }

Mock tasks.nix

Given that the following example only produces a single output, =about.html,= it is true that it could be a lot more simple.

However, it has been written in a “refactored” style to demonstrate that the user is now free to create their own abstractions in order to produce their inputs and outputs however they desire.

These helper functions could be tucked away in a local utils.nix, a local Styx module, or even a published Styx module package.

We don’t have to keep putting things into Styx to support more and more. We should just make it easy to grab those things and use them.

{ pkgs ? import <nixpkgs> { }, extraConf ? { } }:

let
  styx = import pkgs.styx { ①
    config = [ ./conf.nix extraConf ];
    context = { inherit inputs outputs; }; ②
    modules = [ ③
      pkgs.styx-markdown ④
      pkgs.styx-theme-generic ⑤
    ];
  };

  inherit (styx.lib) markdown templates; ⑥

  htmlRenderer = template: output: ⑦
    templates.layout (template output);

  pageDefaults = { ⑧
    renderer = htmlRenderer template.page.full;
  };

  page = attrs: pageDefaults // attrs; ⑨

  markdownPage = { file, ... }@args: Ⓐ
    let
      data = markdown.load ({ inherit file; }); Ⓑ
      attrs = data // args; Ⓒ
    in page attrs; Ⓓ

  inputs = {
    navbar = with outputs; [ about ]; Ⓔ
  };

  outputs = rec {
    about = markdownPage rec { Ⓕ
      file = ./data/sample/pages/about.md;
      title = "About";
      path = "/about.html";
      navbarTitle = title;
    };
  };

in styx.mkTask { inherit inputs ouputs; } Ⓖ

Description

• 1. The pkgs.styx package must be called to be initialized. This performs configuration loading and type-checking, and initializes the library.
• 2. The context is exposed to all library functions upon initialization.
• 3. Module packages are passed to Styx so that their configuration can be loaded and their library initialized.
• 4. Specific data transformation functionality is now externalized to modules. Modules can contain configuration, library functions, and their own outputs.
• 5. Themes are now just normal modules. Modules can provide templates as library functions with a convention of using the shared styx.lib.templates namespace.
• 6. Raising the styx.lib.markdown and styx.lib.templates namespaces for convenience.
• 7. htmlRenderer is a helper function that takes a template function and an output attrset and applies template to output and then applies templates.layout to the result. This produces the final output content.
• 8. pageDefaults is an attrset which sets the renderer to the htmlRenderer function so every “page-like” output works the same.
• 9. page is a helper function that takes an attribute set attrs and merges them over the pageDefaults.
• A. markdownPage is a helper function to create outputs based on markdown sources.
• B. The markdown file is loaded and returned as an attrset. Internally, markdown.load uses styx.core.load to load the textfile, its metadata, and apply nix-lang interpolations.
• C. Merge in any custom attribute overrides.
• D. Call page attrs to produce an output with the pageDefaults. In otherwords, setting the output’s renderer.
• E. Our templates use inputs.navbar to generate a navigation bar.
• F. outputs.about is set to calling markdownPage to produce the final output attrset.
• G. styx.mkTask takes in the inputs and outputs. It will flatten the outputs, call the renderer for each, and write the result to the location in the path attribute.

pkgs.styx { …} startup

When the user calls pkgs.styx a number of things happen:

• Each module’s option-declarations are loaded and merged
• The merged option-declarations are resolved to defaults and type-checked
• The configuration sources are loaded and merged
• The merged configuration is applied to the merged option-declarations and type-checked
• Every module library is loaded with the following arguments:
  • configuration set
  • user supplied context
  • the fixed-point of the library itself, the merged result of every module library

The returned attrset

An attribute set is returned to the user containing:

• core: styx.core
• conf: the loaded configuration
• lib: the merged module libraries
• context: the library context
• modules: the actual modules
• mkTask: the function that will perform the build

styx.mkTask function

The styx.mkTask function is much like mkSite with some differences:

• It takes both inputs and outputs
• It flatten outputs for the user
• It stores its full argument set to the returned derivation’s passthru attribute

passthru Attribute

The derivation’s passthru attribute allows us to store arbitrary Nix data on the derivation without affecting the derivation’s build environment.

This can be used for the documentation generator to get the information it needs without having to have the user return an attrset with the attributes we need.

styx.core

styx.core will contain much of what the built-in library contains today. The only difference is that it is no longer dependant on going through the startup process to import.

styx.lib

styx.lib will contain an attrset of functions, merged from all of the loaded module libraries. The libraries themselves have access to:

• the configuration
• the context
• the fixed-point lib itself

To make the library itself available to module libraries on loading, we use pkgs.lib.fix in order to produce a fixed-point version the library.

Styx Modules

Styx modules are packages which can contribute to a task. Their primary advantage is that they can provide their own configuration interfaces which can change their internal behaviors.

Module can contribue the following things:

• Option Declarations
• Outputs which get merged with task outputs
• Library functions

Module initialization

There is a bit of a chicken and egg problem with module initialization. While modules can provide both configuration and library functions, we need a complete configuration in order to load their library functions.

Another chicken an egg problem is that module libraries should be able to access functions in the library from other modules. But how can we already have the library on hand to pass to modules, before we’ve loaded them?

Configuration Initialization

Modules will be loaded in two phases. When calling styx.core.modules.load you will get back an attrset containing decls and partial.

The decls can be used to merge with the declarations of other modules and the user configuration to produce the complete configuration.

The complete configuration can then be used to call partial, which will load the rest of the module.

Library Initialization

Calling the module partial does not however fully load its library. Afterall we have to pass the merged library to it, in order to load it.

Therefore we load each module’s library as a function taking the following arguments:

• context: the user context attrset
• lib: the merged module libraries

To solve the infinite recursion we utilize nixpkgs.lib.fix in order to calculate the fixed-point of the merged library.

Styx CLI

The CLI will be updated to allow the user to build any task defined by their tasks.nix By returning an attrset of styx.mkTask calls, the user can provide named tasks. The CLI should be able to determine whether tasks.nix returns a single or multiple tasks.

In the case that the user defines multiple tasks, but no task name is provided to the CLI, the CLI should look for a task named ‘default’. It should provide an error if it can’t be found.

Local Dev, src/nixpkgs and themes/versions.nix

Currently in order to facilitate local development there is a src/nixpkgs/default.nix. There is also a themes/versions.nix which contains the github details and hashes for pinned versions of various themes that we want to build documentation for in the official docs.

Both of these mechanisms are dropped in favor of normal overlays. With an overlay like the one below at ~/.config/nixpkgs/overlays/styx.nix tools like nix-build will find the versions of the packages desire. At this point, Styx can just refer to the normal package names, and the correct local ones, or even github pinned ones, will be found and used (and documented)

self: super: {
  styx = super.callPackage /home/ldlework/src/styx/styx {};
  styx-theme-generic = super.callPackage /home/ldlework/src/styx/themes/generic-templates {};
  styx-theme-agency = super.callPackage /home/ldlework/src/styx/themes/agency {};
  styx-theme-showcase = super.callPackage /home/ldlework/src/styx/themes/showcase {};
}

Well, it hasn't be to be a store path, it really depends of the mkTask implementation.
For example this implementation link storepaths and create file if source is a text (the polymorphic approach):

{ pkgs ? import <nixpkgs> {} }:

let
  mkTask = { outputs }: 
    let
      name = "styx-product";
      env = {
        meta = { platforms = pkgs.lib.platforms.all; };
        preferLocalBuild = true;
        allowSubstitutes = false;
      };
    in
    pkgs.runCommand name env ''
      mkdir $out
      ${pkgs.lib.concatMapStringsSep "\n" (output: 
        if builtins.isString output.source then ''
          echo "${output.source}" > "$out/${output.path}"
        '' else ''
          ln -s "${output.source}" "$out/${output.path}"
        ''
      ) outputs}
    '';
in {
  result = mkTask {
    outputs = [{
      source = pkgs.writeText "foo" "foo";
      path = "foo.txt";
    } {
      source = "bar";
      path = "bar.txt";
    }];
  };
}

That is a reason why I suggested to fix mkTask API first, as it will determine many other things.

Quite unrelated, but I had a few other nix experiments that seem a good match for stix:

• mkPresentation: generating html slides from mardown files
  • this is more or less just site generation
• machine learning: training a machine learning model and run it
  • learning configuration (epochs and backend) would be in a conf module, and there would be a task for the model and another for the frontend