Skip to content

Instantly share code, notes, and snippets.

@tonyfast

tonyfast/index.html

Last active August 29, 2015 14:22
Show Gist options
  • Save tonyfast/c8fa98cd19a0edf55181 to your computer and use it in GitHub Desktop.
Save tonyfast/c8fa98cd19a0edf55181 to your computer and use it in GitHub Desktop.
Download ZIP
Literate Presentation Streams with Markdown
Display the source blob
Display the rendered blob
Raw
readme.ipynb
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Raw
readme.md

Literate Presentation Streams

{{title}} combine semantic text and code as structured text to communicate information. Presentation streams provide a minimal abstraction that can display information on many popular services like Github, nbviewer, speakerdeck, Disqus, Jekyll, bl.ocks, blogs, RSS...

The static presentation of information should promote maturity to an interaction, app, download. Static information that is compliant with HTML and Javascript canp be accessed most broadly.

Markdown is the ideal candidate to represent this type of information

# Jekyll blog information
layout: post
title: Literate presentation streams

What is a (markdown)stream?

A stream is a Github Flavored Markdown document. Code fences separate streams with ```. Four-tab indents are parts of semantic text. A markdown stream always starts with semantic text. A markdown stream always alternates between semantic text and code. GFM adds syntax highlighting by default, but it also indicates the language a code block is written in.

  • Standard markdown only contains code and not code.
  • RMarkdown code-fence parameters do not have meaning outside of R. RMarkdown does very much influence the ability to create many presentation formats, but there needs to be a heavier focus on the web.
  • Matlab markup sucks, most people don't even know it exists.
  • GFM has a large user base and has proven to be language agnostic.
print('This is python code and output')
This is python code and output

The Meaning of the Presentation Layer

There are many methods to present information, but each method can infer the information in the stream differently.

For example, in the two code cells above:

  • The python indicates syntax highlighting and Python code
  • There is a blank markdown cell in between
  • One can imagine creating a parser that uses that pattern to control a JSON-ified notebook with input and output code.
  • Even without IPython notebook the document renders as Markdown in gist.
some code
<!-- some Bokeh output -->

Bokeh can embed beautiful HTML5 visualizations

Haha, how is this going to work? I think it can render in bl.ocks or python notebooks.

extensions:
  blocks: 
  gist:
  nbviewer:
  notebook: # download nbviwer link
  jekyll:

HTML Presentation

There doesn't always need to be markdown It can be HTML 

# references at the bottom
references:
  jekyll: www.jekyllrb.com
  rmarkdown: http://rmarkdown.rstudio.com/

Now what?

All the major data has been displayed. The most accessible presentation of data is the underlying motive. After the data is presented we make it dance.

console.log('coffeescript')
### Update the list of extensions in this readme ###
d3.select 'ul#ext-list'
  .selectAll 'li'
  .data d3.entries document.__data__.extensions
  .call (s)->
    s.enter().append 'li'
  .each (d,i)->
    d3.select @
      .append 'a'
      .text d.key
Raw
index.html
<!doctype html>
<!--
index.html is a naming convention adopted by blocks single gist paages
the languages shown in this example can all be parsed with cell
magics in the ipython notebooks
yaml, coffeescript, javascript, json, html, python
-->
<head>
<script src="http://cdn.jsdelivr.net/g/marked,d3js,handlebarsjs">
</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/js-yaml/3.3.1/js-yaml.min.js">
</script>
<script src="http://coffeescript.org/extras/coffee-script.js">
</script>
</head>
<body>
<script type="text/coffeescript">
d3.text 'readme.md', (readme)->
markdown = d3.select 'body'
.append 'div'
.attr 'classed','markdown'
.html marked readme
### Store yaml and json data on document object ###
document.__data__ = {}
gfm = (lang)->
['','lang-','highlight-'].map (prefix)-> ".#{prefix}#{lang}"
.join ','
.slice 0,-1
markdown.selectAll gfm 'json'
.each ()->
d3.entries JSON.parse @innerText
.forEach (d)->
document.__data__[d.key] = d.value
markdown.selectAll gfm 'yaml'
.each ()->
d3.entries jsyaml.load @innerText
.forEach (d)->
document.__data__[d.key] = d.value
### run any parts of the stream in that are javascript and coffeescript ###
markdown.selectAll gfm 'javascript'
.each ()->
CoffeeScript.run @innerText
### Run coffeescript cells ###
markdown.selectAll gfm 'coffeescript'
.each ()->
CoffeeScript.run @innerText
###
Template with Handlebars
Handlebars is not compatiable with Jekyll's liquid syntax so this example stays simple
Template languages insert data into the semantic text.
###
markdown.html ()->
template = Handlebars.compile @innerHTML
template document.__data__
</script>
</body>
Display the source blob
Display the rendered blob
Raw
markdown2ipynb.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Can Markdown describe an IPython notebook?\n",
"\n",
"* Use Github Flavored Markdown code-fences to separate structured code and unstructured text\n",
"\n",
" * Extend language hightlighting to cell magics\n",
" \n",
"* Markdown is portable and diffable.\n",
"\n",
"### Misc\n",
"\n",
"* Written in coffeescript\n",
"* Use as much existing data as possible."
]
},
{
"cell_type": "code",
"execution_count": 31,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The coffeetools extension is already loaded. To reload it, use:\n",
" %reload_ext coffeetools\n"
]
}
],
"source": [
"%load_ext coffeetools"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Existing Data\n",
"\n",
"* A [Markdown Stream]('https://gist.githubusercontent.com/tonyfast/c8fa98cd19a0edf55181/raw/readme.md') designed to be presented in the notebook. The design has very little opinion.\n",
"* [Phantom notebook json]('https://gist.githubusercontent.com/tonyfast/49bcb30e9005f3a242b4/raw/NotebookJSON.ipynb') "
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"application/javascript": [
"// Generated by CoffeeScript 1.9.1\n",
"(function() {\n",
" require([\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\"], function(d3) {\n",
"\n",
" /* An example markdown stream meant to comply with the notebook opinions */\n",
" d3.text('https://gist.githubusercontent.com/tonyfast/c8fa98cd19a0edf55181/raw/readme.md', function(e, markdown) {\n",
" return d3.select(document).datum({\n",
" markdown: markdown\n",
" });\n",
" });\n",
"\n",
" /* Phantom output for all the states of the notebook cells */\n",
" return d3.json('https://gist.githubusercontent.com/tonyfast/49bcb30e9005f3a242b4/raw/NotebookJSON.ipynb', function(e, notebook) {\n",
" return document.__data__.notebook = notebook;\n",
" });\n",
" });\n",
"\n",
"}).call(this);\n"
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%coffeescript\n",
"require [\n",
" \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\"\n",
"], (d3)->\n",
" ### An example markdown stream meant to comply with the notebook opinions ###\n",
" d3.text 'https://gist.githubusercontent.com/tonyfast/c8fa98cd19a0edf55181/raw/readme.md', (e,markdown)->\n",
" d3.select(document).datum {markdown: markdown}\n",
" ### Phantom output for all the states of the notebook cells ###\n",
" d3.json 'https://gist.githubusercontent.com/tonyfast/49bcb30e9005f3a242b4/raw/NotebookJSON.ipynb', (e, notebook) ->\n",
" document.__data__.notebook = notebook"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Splitting the streams\n",
"\n",
"Split the markdown by <code>```</code> that are supported by GFM markdown. This constraint means that the first cell is ALWAYS unstructured text then cells alternate between code and text. \n",
"\n",
"GFM accepts a language after the code fences. This information can be used to drive connections between input and output along with cell magics."
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"application/javascript": [
"// Generated by CoffeeScript 1.9.1\n",
"(function() {\n",
" require([\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\"], function(d3) {\n",
" return document.__data__.parsed = document.__data__.markdown.split('```').reduce(function(prev, next, index, array) {\n",
" var code, ref, ref1;\n",
" if (index % 2 === 0) {\n",
" if (next !== '\\n') {\n",
"\n",
" /* markdown type :: order, emphasis, hierarchy */\n",
" prev.push({\n",
" cell_type: 'markdown',\n",
" source: next.replace(/\\n*$/g, \"\").split('\\n').map(function(s) {\n",
" return s + \"\\n\";\n",
" })\n",
" });\n",
" }\n",
" } else {\n",
"\n",
" /* monospaced type */\n",
" code = next.replace(/\\n*$/g, \"\").split('\\n');\n",
" prev.push({\n",
" cell_type: (ref = code[0]) !== '\\n' ? 'code' : 'raw',\n",
" magic: (ref1 = code[0]) !== '\\n' ? code[0] : null,\n",
" source: code.slice(1).map(function(s, i) {\n",
" if (i === code.length - 2) {\n",
" return \"\" + s;\n",
" } else {\n",
" return s + \"\\n\";\n",
" }\n",
" })\n",
" });\n",
" }\n",
" return prev;\n",
" }, []);\n",
" });\n",
"\n",
"}).call(this);\n"
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%coffeescript\n",
"require [\n",
" \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\"\n",
"], (d3)->\n",
" document.__data__.parsed = document.__data__.markdown.split '```'\n",
" .reduce (prev, next, index, array)->\n",
" if index % 2 == 0\n",
" unless next in ['\\n']\n",
" ### markdown type :: order, emphasis, hierarchy ###\n",
" prev.push \n",
" cell_type: 'markdown'\n",
" source: next.replace(/\\n*$/g, \"\").split('\\n').map (s)->\"#{s}\\n\"\n",
" else\n",
" ### monospaced type ###\n",
" code = next.replace(/\\n*$/g, \"\").split '\\n'\n",
" prev.push\n",
" cell_type: unless code[0] in ['\\n'] then 'code' else 'raw'\n",
" magic: unless code[0] in ['\\n'] then code[0] else null\n",
" source: code[1..].map (s,i)-> \n",
" if i == code.length-2\n",
" \"#{s}\"\n",
" else\n",
" \"#{s}\\n\"\n",
" prev\n",
" , []"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## JSON-ified notebook data\n",
"\n",
"Map minimal states of notebook cells to an object. The sources, text, and cell_types are modified so the output is compliant with the IPython notebook template.\n",
"\n",
"### Types of Cells\n",
"\n",
"* Markdown Cells\n",
"* Code Cells\n",
"\n",
" * Rich Display Code Cells - present javascript or html\n",
" * stdout - present normal python outputs\n",
" * No Ouputs - unexecuted or non-python cells\n",
" \n",
"* Raw Cells - monospaced, uncompiled code"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"application/javascript": [
"// Generated by CoffeeScript 1.9.1\n",
"(function() {\n",
" document.__data__.cells = {\n",
" markdown: document.__data__.notebook.cells[0],\n",
" code: {\n",
" noOutput: document.__data__.notebook.cells[1],\n",
" stdout: document.__data__.notebook.cells[2],\n",
" html: document.__data__.notebook.cells[3]\n",
" },\n",
" raw: document.__data__.notebook.cells[4]\n",
" };\n",
"\n",
"}).call(this);\n"
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%coffeescript\n",
"document.__data__.cells = \n",
" markdown: document.__data__.notebook.cells[0]\n",
" code:\n",
" noOutput: document.__data__.notebook.cells[1]\n",
" stdout: document.__data__.notebook.cells[2]\n",
" html: document.__data__.notebook.cells[3]\n",
" raw: document.__data__.notebook.cells[4]"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"## Impose some Rules\n",
"\n",
"* code with ``python`` as a language followed immediately by a ``raw``, ``html``, or ``javascript`` has output.\n",
"\n",
" * ``raw`` is ``stdout``, otherwise rich display\n",
" \n",
"* code cells with a language other than ``python`` that are not output are prepended with cell magic."
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"application/javascript": [
"// Generated by CoffeeScript 1.9.1\n",
"(function() {\n",
" require([\"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\"], function(d3) {\n",
" var merge;\n",
" merge = function(obj2, obj1) {\n",
" var output;\n",
" output = {};\n",
" d3.entries(obj2).forEach(function(d, i) {\n",
" return output[d.key] = d.value;\n",
" });\n",
" d3.entries(obj1).forEach(function(d, i) {\n",
" return output[d.key] = d.value;\n",
" });\n",
" return output;\n",
" };\n",
" document.__data__.reduce = document.__data__.parsed.reduce(function(prev, next, index, array) {\n",
" var key, ref, ref1, ref2, ref3, ref4, ref5, temp;\n",
" if ((ref = next.cell_type) === 'markdown' || ref === 'raw') {\n",
" temp = merge(document.__data__.cells[next.cell_type], next);\n",
" prev.parsed.push(temp);\n",
" return prev;\n",
" } else if (prev.skipNext) {\n",
" prev.skipNext = false;\n",
" return prev;\n",
" } else if ((ref1 = next.cell_type) === 'code') {\n",
" if ((ref2 = next.magic) === 'python') {\n",
" if ((array[index + 1]['magic'] != null) && ((ref3 = array[index + 1]['magic']) === 'html' || ref3 === 'javascript' || ref3 === '')) {\n",
" delete next.magic;\n",
" prev.skipNext = true;\n",
" temp = merge(document.__data__.cells.code.html, next);\n",
" key = (ref4 = array[index + 1]['magic']) === 'html' ? 'text/html' : (ref5 = array[index + 1]['magic']) === 'javascript' ? 'application/javascript' : null;\n",
" if (key != null) {\n",
" temp.outputs[0].data[key] = array[index + 1].source;\n",
" prev.parsed.push(temp);\n",
" } else {\n",
" temp = merge(document.__data__.cells.code.stdout, next);\n",
" temp.outputs[0].text = array[index + 1].source;\n",
" prev.parsed.push(temp);\n",
" }\n",
" } else {\n",
" prev.parsed.push(merge(document.__data__.cells.code.noOutput, next));\n",
" }\n",
" } else {\n",
" temp = merge(next, {});\n",
" delete temp.magic;\n",
" if (next.magic !== '') {\n",
" temp.source.unshift(\"%%\" + next.magic + \"\\n\");\n",
" }\n",
" prev.parsed.push(merge(document.__data__.cells.code.noOutput, temp));\n",
" }\n",
" return prev;\n",
" }\n",
" }, {\n",
" skipNext: false,\n",
" parsed: []\n",
" });\n",
" document.__data__.output = merge(document.__data__.notebook, {});\n",
" return document.__data__.output.cells = document.__data__.reduce.parsed;\n",
"\n",
" /* Show the output */\n",
" });\n",
"\n",
"}).call(this);\n"
],
"text/plain": [
"<IPython.core.display.Javascript object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"%%coffeescript\n",
"require [\n",
" \"https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js\"\n",
"], (d3)->\n",
" merge = ( obj2,obj1) ->\n",
" output = {}\n",
" d3.entries(obj2).forEach (d,i)-> output[d.key] = d.value\n",
" d3.entries(obj1).forEach (d,i)-> output[d.key] = d.value\n",
" output\n",
" \n",
" document.__data__.reduce = document.__data__.parsed.reduce (prev, next, index, array)->\n",
" if next.cell_type in ['markdown','raw']\n",
" temp = merge document.__data__.cells[next.cell_type], next\n",
" prev.parsed.push temp\n",
" prev\n",
" else if prev.skipNext\n",
" prev.skipNext = false\n",
" prev\n",
" else if next.cell_type in ['code']\n",
" if next.magic in ['python']\n",
" if array[index+1]['magic']? and array[index+1]['magic'] in ['html','javascript','']\n",
" delete next.magic\n",
" prev.skipNext = true\n",
" temp = merge document.__data__.cells.code.html, next\n",
" key = if array[index+1]['magic'] in ['html'] \n",
" 'text/html'\n",
" else if array[index+1]['magic'] in ['javascript']\n",
" 'application/javascript'\n",
" else null\n",
" if key?\n",
" temp.outputs[0].data[key] = array[index+1].source \n",
" prev.parsed.push temp\n",
" else\n",
" temp = merge document.__data__.cells.code.stdout, next\n",
" temp.outputs[0].text = array[index+1].source \n",
" \n",
" prev.parsed.push temp\n",
" else\n",
" prev.parsed.push merge document.__data__.cells.code.noOutput, next\n",
" else\n",
" temp = merge next, {}\n",
" delete temp.magic\n",
" unless next.magic == ''\n",
" temp.source.unshift \"%%#{next.magic}\\n\"\n",
" prev.parsed.push merge document.__data__.cells.code.noOutput, temp\n",
" prev\n",
" , {\n",
" skipNext: false\n",
" parsed: []\n",
" }\n",
" \n",
" document.__data__.output = merge document.__data__.notebook, {}\n",
" document.__data__.output.cells = document.__data__.reduce.parsed\n",
" ### Show the output ###\n",
" "
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from IPython.core import display"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"<h3>Transformed Notebook Metadata</h3>\n",
"<pre><code id='json'></code></pre>\n",
"<script>\n",
" d3.select('#json').text( JSON.stringify(document.__data__.output, '', 2) )\n",
"</script>\n"
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"display.HTML(\"\"\"\n",
"<h3>Transformed Notebook Metadata</h3>\n",
"<pre><code id='json'></code></pre>\n",
"<script>\n",
" d3.select('#json').text( JSON.stringify(document.__data__.output, '', 2) )\n",
"</script>\n",
"\"\"\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.4.3"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment