Skip to content

Instantly share code, notes, and snippets.

@supersym
Last active December 10, 2015 12:28
Show Gist options
  • Save supersym/4434011 to your computer and use it in GitHub Desktop.
Save supersym/4434011 to your computer and use it in GitHub Desktop.
Some ideas on what we can do with remote data stored as GitHub Gists or git repositories.

DocPad configuration file

The DocPad configuration file may take one of the following file names:

  • docpad.cson
  • docpad.json
  • docpad.js
  • docpad.coffee

This file defines several key aspects of the running (or to be ran) DocPad process / program instance.

  • the global configuration object docpadConfig
  • The global template data objects as literals such as site or site.url
  • The global methods accessible from within document instances as JS/Coffee helper functions

This allows us to use system-wide data that doesn't specifically belong to document instances inside the entire docpad environment during build time. Since in essence a document/site generator, the process of rendering (or compiling, processing) our source code files (Stylus, Jade, Coffeekup etc) requires a minimum one-time pass over the source 'filling in' the data from the helpers/fields defined in docpad.cson.

By it's nature, this limits the dynamic configuration methods and values we have available during run-time inside the browser (which is your browser rendering HTML documents, style sheets and scripts). This may be easily forgotten. The following will thus not work:

h1 This is a header on level one

:coffeescript
  for value,key in getDocuments('html').toJSON()
    $('#somediv').innerHTML value

This due to the logical fact that the browser side script does not have any knowledge about historic DocPad rendering operations and variables available at that time.

Still we have other cool means to asynchronously operate DocPad together with GitHub remote data storage and their API's.

Below I will illustrate some possible use-cases, continue after you have review the stub source at example-document.html.jade gist.

Naive implementation

For the uninitiated the example shown below would make perfect sense. Anyone who has used DocPad beyond superficial point-click-fire docpad run will know that this would, currently, trigger errors due to the Document class not knowing about any built-in collection named 'gists'.

The helpers section

First, lets explore the other essential piece when trying to perform code blocks during DocPad rendering: functions. More specifically DocPad TemplateData Helpers help us in a way that it allows us to do complex operations on data, transform it's output and return that value to us in a desired fashion.

I imagine you would like to have helpers for the most common, frequently used operations upon the API.

sendAuthentication: ->
   # step 1: create token programatically see http://stackoverflow.com/a/1626933 for oauth tokens and your options
   # a) perhaps use a 'createOauth2Token()' method as a helper for proper subject abstraction and using in other OAuth enabled services
   # step 2: obtain the token value and set the token in our computer memory
   # step 3: send the token over HTTP to the API to authenticate
   # ...

getGistById = (id) ->
   # obtain gist by reference of it's id
   
getGistByName = (filename) ->
   # Overloaded method by filename parameter signature / identifier (no)
   # Idea: using the gist name: we can have structured file name conventions for templating say LICENSE information -> LICENSE.md file output logic

setLicenseInfo = (cb) ->
   getGistByName('LICENSE.md')
   if err cb(err)
     log 'Warning', 'The file name specified is not a file in this gist'

...

Use these helpers internally and optionally reference at any object literal you might want to expose publically (and keep the processes private).

Refactoring

But I feel we can do some refactoring on the design. Some more abstraction would perhaps be at place. One thing I could think about is the idea of Gists as remote storage facilities just like any other data source in the more traditional sense of say RDBMS like MySQL, NoSQL or MongoDB. The first ones, all using Standard Query Language to retrieve and manipulate data inside the storage (database tables with rows and fields or columns), traditional record set like structures with high concurrency and proper multi-user locking and cursoring. The second perhaps more like MongoDB with its Document-Oriented structure of notating the data and consequently be better (or worse) at specific types of data in your set. Or take even GraphDB's, data stored as hypergraphs based upon relations and which address the paradigm of the index-free adjacency.

Abstraction

We have can have a commonly used 'proper' abstraction of verbs for all of these storage mechanism operations which we call CRUD

  • CREATE We define the data 'scheme' and insert it's (field) values in the storage container
  • READ We call upon previously saved, possibly serialized, remote data and retrieve these values from the remote storage to use locally
  • UPDATE We selected upon a existing record much like read, however wish to modify the remote values, and thus edit and change the data
  • DELETE We select records and will remove them from the set, this does not always have to mean permanently, many CSM will actually 'archive' rows to designated tables to keep the frequently accessed tables somewhat clean (perhaps archive per year posts).

These would constitute the proper data (base/set/row/field) operations abstractions to it's generic form that we find desirable here.

One could think of this in a OOP like fashion to layout the framework in coffeescript pseudo code

# enumerate the different notations of data that we can work with
StorageFormats = [ 'json', 'ngraph', 'binary blob', 'plain text', etc... ]
BaseOperations = [ 'read', 'create', 'update', 'delete' ]

# Parent class following nested classes
class Data

# Some class
# @param...etc
class Data::Storage

   # default values
   name: 'github-gist-plain-text-1'
   location: 'http://gist.github.com/12345'
   description: 'This is a Gist as remote datasource'
   note: 'Omitted Coda document decorator comments'
   
   # constructor etc

class Data::Operation

   read = (src) ->
     ...
     
   update = (key, value) ->  
     # update fields/columns/objects as a key:value-pair
   
   update = (sql) ->
     # or overload with a SQL query string when we have a storage that uses query language
     
   # and there might be more storage mechanism or vendor specific operations and options that we don't know of.

Model-first approach

Of course we could also go back to the basis of DocPad, which is data as Backbone.Model to represent it's structure, logic and values to view. The most probable course of action would be to have a Model object import the (gist) data and add it as Document.Source, which in turn is obtained through a helper method. This would result in a more generic model for data which we already abstracted and use as DocPad Models (derived from Backbone.Model) yet would naturally limit the possible additional, more fine-grained, configurations of specific data storage facilities. With the latter one could essentially think of these in the traditional OOP like structure as illustrated in the stub above.

In the model first approach we defined the Backbone.Model and import data later, this is a very dynamic way of importing unstructured/structured data and have it abby to your model.

Data-first approach

In this approach we first handle the data from the perspective of the remote storage down to docpad, work with it and finally just apply the Model????

---
layout: default
---
hgroup
h1 Appliance
h2 of build-time remote data storage and retrieval over HTTP GitHub API v3
h3 Regular DocPad pattern to obtain a list of static assets (src/files)
code
- for file in docpad.getCollection('files').toJSON()
p= file.url
p Actual execution block during build-time
- for file in docpad.getCollection('files').toJSON()
p= file.url
pre This would render a list of all file urls under src/files as a DocPad Collection derived from Backbone.Collection
h3 The following would allow for usage of GitHub remote data storage retrieval during build-time
code
- for gist in getDocument().getCollection('gists').toJSON()
code= gist.source
em (omitted the execution block)
pre This would render the data set of gists as code (example) blocks (quote like notation for illustration purposes)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment