Created
January 31, 2019 16:09
-
-
Save jwhitlock/e0b26e586676add748dff54321cd50a2 to your computer and use it in GitHub Desktop.
Rendering document from PR 5216
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" | |
| "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> | |
| <html xmlns="http://www.w3.org/1999/xhtml"> | |
| <head> | |
| <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> | |
| <title>Rendering Documents — Kuma Documentation</title> | |
| <link rel="stylesheet" href="_static/alabaster.css" type="text/css" /> | |
| <link rel="stylesheet" href="_static/pygments.css" type="text/css" /> | |
| <script type="text/javascript"> | |
| var DOCUMENTATION_OPTIONS = { | |
| URL_ROOT: './', | |
| VERSION: 'latest', | |
| COLLAPSE_INDEX: false, | |
| FILE_SUFFIX: '.html', | |
| HAS_SOURCE: true | |
| }; | |
| </script> | |
| <script type="text/javascript" src="_static/jquery.js"></script> | |
| <script type="text/javascript" src="_static/underscore.js"></script> | |
| <script type="text/javascript" src="_static/doctools.js"></script> | |
| <link rel="top" title="Kuma Documentation" href="index.html" /> | |
| <link rel="prev" title="Deploying MDN" href="deploy.html" /> | |
| <link rel="stylesheet" href="_static/custom.css" type="text/css" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" /> | |
| </head> | |
| <body role="document"> | |
| <div class="document"> | |
| <div class="documentwrapper"> | |
| <div class="bodywrapper"> | |
| <div class="body" role="main"> | |
| <div class="section" id="rendering-documents"> | |
| <h1>Rendering Documents<a class="headerlink" href="#rendering-documents" title="Permalink to this headline">¶</a></h1> | |
| <p>Kuma’s wiki format is a restricted subset of HTML, augmented with KumaScript | |
| macros <code class="docutils literal"><span class="pre">{{likeThis}}</span></code> that render to HTML for display. Rendering is done | |
| in several steps, many of which are stored in the database for read-time | |
| performance.</p> | |
| <img src="_images/rendering.svg" /><p>A revision goes through several rendering steps before it appears on the site:</p> | |
| <ol class="arabic simple"> | |
| <li>A user submits new content, and Kuma stores it as <a class="reference internal" href="#revision-content"><span>Revision content</span></a></li> | |
| <li>Kuma bleaches and filters the content to create <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a></li> | |
| <li>KumaScript renders macros and returns <a class="reference internal" href="#kumascript-content"><span>KumaScript content</span></a></li> | |
| <li>Kuma bleaches and filters the content again to create <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a></li> | |
| <li>Kuma divides and processes the content into <a class="reference internal" href="#body-html"><span>Body HTML</span></a>, <a class="reference internal" href="#quick-links-html"><span>Quick links HTML</span></a>, <a class="reference internal" href="#toc-html"><span>ToC HTML</span></a>, <a class="reference internal" href="#summary-text-and-html"><span>Summary text and HTML</span></a></li> | |
| </ol> | |
| <p>There are other rendered outputs:</p> | |
| <ul class="simple"> | |
| <li>Kuma normalizes the <a class="reference internal" href="#revision-content"><span>Revision content</span></a> into the <a class="reference internal" href="#diff-format"><span>Diff format</span></a> to comparing revisions</li> | |
| <li>Kuma filters the <a class="reference internal" href="#revision-content"><span>Revision content</span></a> and adds section IDs to create the <a class="reference internal" href="#triaged-content"><span>Triaged content</span></a> for updating pages</li> | |
| <li>KumaScript renders the <a class="reference internal" href="#triaged-content"><span>Triaged content</span></a> as <a class="reference internal" href="#preview-content"><span>Preview content</span></a></li> | |
| <li>Kuma filters the <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a> and adds section IDs to publish the <a class="reference internal" href="#raw-content"><span>Raw content</span></a></li> | |
| <li>Kuma extracts code sections from the <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a> to flesh out a <a class="reference internal" href="#live-sample"><span>Live sample</span></a></li> | |
| <li>Kuma extracts the text from the <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a> for the <a class="reference internal" href="#search-content"><span>Search content</span></a></li> | |
| </ul> | |
| <div class="section" id="revision-content"> | |
| <span id="id1"></span><h2>Revision content<a class="headerlink" href="#revision-content" title="Permalink to this headline">¶</a></h2> | |
| <p><a class="reference internal" href="ckeditor.html"><em>CKEditor</em></a> provides a visual HTML editor for MDN writers. The | |
| raw HTML returned from CKEditor is stored in the Kuma database for further | |
| processing.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd><p class="first">User-entered content, usually via CKEditor from the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$edit">edit view</a> (URLs ending with <code class="docutils literal"><span class="pre">$edit</span></code>)</p> | |
| <p class="last">Developer-submitted content via an HTTP <code class="docutils literal"><span class="pre">PUT</span></code> to the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$api">API view</a> (URLs ending in <code class="docutils literal"><span class="pre">$api</span></code>)</p> | |
| </dd> | |
| <dt>displayed on MDN</dt> | |
| <dd>“Revision Source” section of the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$revision/1454597">revision detail view</a> (URLs ending with | |
| <code class="docutils literal"><span class="pre">$revision/<id></span></code>), in a <code class="docutils literal"><span class="pre"><pre></span></code> tag</dd> | |
| <dt>database</dt> | |
| <dd><code class="docutils literal"><span class="pre">wiki_revision.content</span></code></dd> | |
| <dt>code</dt> | |
| <dd><code class="docutils literal"><span class="pre">kuma.wiki.models.Revision.content</span></code></dd> | |
| </dl> | |
| <p>To illustrate rendering, consider a new document published at | |
| <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple">/en-US/docs/Sandbox/simple</a> with this <em>Revision content</em>:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">p</span><span class="p">></span>{{ CSSRef }}<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">></span>I am red.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">></span>{{HTMLElement('div')}}<span class="p"></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span> <span class="na">onclick</span><span class="o">=</span><span class="s">"alert('hacked!');"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">script</span><span class="p">></span> | |
| <span class="nx">alert</span><span class="p">(</span><span class="s1">'How about this?'</span><span class="p">);</span> | |
| <span class="p"></</span><span class="nt">script</span><span class="p">></span> | |
| </pre></div> | |
| </div> | |
| <p>This document has elements that highlight different areas of rendering:</p> | |
| <ul> | |
| <li><p class="first">A sidebar macro <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/CSSRef.ejs">CSSRef</a>, which will be rendered by KumaScript and extracted for display</p> | |
| </li> | |
| <li><p class="first">A <code class="docutils literal"><span class="pre"><h2></span></code> tag, which will gain an <code class="docutils literal"><span class="pre">id</span></code> attribute</p> | |
| </li> | |
| <li><dl class="first docutils"> | |
| <dt>A list of three links:</dt> | |
| <dd><ol class="first last arabic simple"> | |
| <li>An HTML link to an existing document</li> | |
| <li>A reference macro <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/HTMLElement.ejs">HTMLElement</a> which will be rendered by KumaScript</li> | |
| <li>An HTML link to a new document, which will get <code class="docutils literal"><span class="pre">rel="nofollow"</span></code> and <code class="docutils literal"><span class="pre">class="new"</span></code> attributes</li> | |
| </ol> | |
| </dd> | |
| </dl> | |
| </li> | |
| <li><p class="first">An <code class="docutils literal"><span class="pre">onclick</span></code> attribute, added in Source mode, which will be removed</p> | |
| </li> | |
| <li><p class="first">A <code class="docutils literal"><span class="pre"><script></span></code> section, added in Source mode, which will be escaped</p> | |
| </li> | |
| </ul> | |
| <p>CKEditor has partial support for restricting content to the HTML subset | |
| allowed for display. It also enforces a style where paragraphs (<code class="docutils literal"><span class="pre"><p></span></code>) | |
| are split by empty lines, start at the first column, and are closed on | |
| the same line. Nested elements are indented one space. Plain text is wrapped | |
| in <code class="docutils literal"><span class="pre"><p></span></code> tags by default. KumaScript macros, such as <code class="docutils literal"><span class="pre">{{CSSRef}}</span></code>, are | |
| treated as plain text by CKEditor, so they are also wrapped in <code class="docutils literal"><span class="pre"><p></span></code> tags.</p> | |
| <p>Writers can also switch to “Source” mode, which permits direct editing of the | |
| HTML, avoiding formatting and content restrictions. This can be used to attempt | |
| to inject scripts like a <code class="docutils literal"><span class="pre">onclick</span></code> attribute or a <code class="docutils literal"><span class="pre"><script></span></code>. These | |
| attempts are stored in the revision content.</p> | |
| <p>The <a class="reference external" href="https://developer.mozilla.org/en-US/docs/MDN/Contribute/Tools/PUT_API">PUT API</a> can also be used to add new revisions. This experimental API is | |
| for staff only at this time.</p> | |
| </div> | |
| <div class="section" id="bleached-content"> | |
| <span id="id2"></span><h2>Bleached content<a class="headerlink" href="#bleached-content" title="Permalink to this headline">¶</a></h2> | |
| <p>A revision can contain invalid markup, or elements that are not allowed on | |
| MDN. When a new revision is created, the related document is updated in | |
| <code class="docutils literal"><span class="pre">revision.make_current()</span></code>. This includes updating the title, path, and | |
| tags, and also cleaning the content and saving it on the Document record.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd><a class="reference internal" href="#revision-content"><span>Revision content</span></a>, processed with multiple filters</dd> | |
| <dt>displayed on MDN</dt> | |
| <dd>The <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$api">API view</a> (URLs ending in <code class="docutils literal"><span class="pre">$api</span></code>)</dd> | |
| <dt>database</dt> | |
| <dd><code class="docutils literal"><span class="pre">wiki_document.html</span></code> for current revision, not stored for historical revisions</dd> | |
| <dt>code</dt> | |
| <dd><p class="first"><code class="docutils literal"><span class="pre">kuma.wiki.models.Document.get_html()</span></code> (current revision, cached)</p> | |
| <p class="last"><code class="docutils literal"><span class="pre">kuma.wiki.models.Revision.content_cleaned</span></code> (any revision, dynamically generated)</p> | |
| </dd> | |
| </dl> | |
| <p>The <em>Bleached content</em> of the simple document looks like this:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">p</span><span class="p">></span>{{ CSSRef }}<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color: red;"</span><span class="p">></span>I am red.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">></span>{{HTMLElement('div')}}<span class="p"></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="ni">&lt;</span>script<span class="ni">&gt;</span> | |
| alert('How about this?'); | |
| <span class="ni">&lt;</span>/script<span class="ni">&gt;</span> | |
| </pre></div> | |
| </div> | |
| <p>The first step of cleaning is “bleaching”. The <a class="reference external" href="https://github.com/mozilla/bleach">bleach</a> library parses the | |
| raw HTML and drops any tags, attributes, or styles that are not on the | |
| <a class="reference external" href="https://github.com/mozilla/kuma/blob/master/kuma/wiki/constants.py">allowed lists</a>. In the simple document, this step drops the <code class="docutils literal"><span class="pre">onclick</span></code> | |
| attribute from the <code class="docutils literal"><span class="pre"><div></span></code>, and escapes the <code class="docutils literal"><span class="pre"><script></span></code> section.</p> | |
| <p>Next, the HTML is tokenized by <a class="reference external" href="https://github.com/html5lib/html5lib-python">html5lib</a>. The content is parsed for <code class="docutils literal"><span class="pre"><iframe></span></code> | |
| elements, and any <code class="docutils literal"><span class="pre">src</span></code> attributes that refer to disallowed domains are | |
| dropped.</p> | |
| <p>The tokenized document is serialized back to HTML, which may make | |
| changes to whitespace or attribute order. In the simple document, this step | |
| adds the extra space in <code class="docutils literal"><span class="pre">style="color:</span> <span class="pre">red"</span></code>.</p> | |
| </div> | |
| <div class="section" id="kumascript-content"> | |
| <span id="id3"></span><h2>KumaScript content<a class="headerlink" href="#kumascript-content" title="Permalink to this headline">¶</a></h2> | |
| <p>KumaScript macros are represented by text content in two curly braces, and | |
| <code class="docutils literal"><span class="pre">{{lookLike('this')}}</span></code>. The KumaScript service processes these macros and | |
| replaces them with plain HTML. This intermediate representation is not stored, | |
| but instead is further processed to generate the <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd><a class="reference internal" href="#bleached-content"><span>Bleached content</span></a>, processed by KumaScript</dd> | |
| <dt>displayed on MDN</dt> | |
| <dd><em>not published</em></dd> | |
| <dt>database</dt> | |
| <dd>Errors at <code class="docutils literal"><span class="pre">wiki_document.rendered_errors</span></code>, content not stored</dd> | |
| <dt>code</dt> | |
| <dd>Errors at <code class="docutils literal"><span class="pre">kuma.wiki.models.Document.rendered_errors</span></code>, content not stored</dd> | |
| </dl> | |
| <p>The <em>KumaScript content</em> for the simple document looks like this:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">p</span><span class="p">><</span><span class="nt">section</span> <span class="na">class</span><span class="o">=</span><span class="s">"Quick_links"</span> <span class="na">id</span><span class="o">=</span><span class="s">"Quick_Links"</span><span class="p">><</span><span class="nt">ol</span><span class="p">><</span><span class="nt">li</span><span class="p">><</span><span class="nt">strong</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/CSS"</span><span class="p">></span>CSS<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">strong</span><span class="p">></</span><span class="nt">li</span><span class="p">><</span><span class="nt">li</span><span class="p">><</span><span class="nt">strong</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/CSS/Reference"</span><span class="p">></span>CSS Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">strong</span><span class="p">></</span><span class="nt">li</span><span class="p">></</span><span class="nt">ol</span><span class="p">></</span><span class="nt">section</span><span class="p">></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color: red;"</span><span class="p">></span>I am red.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML/Element/div"</span> <span class="na">title</span><span class="o">=</span><span class="s">"The HTML Content Division element (&lt;div&gt;) is the generic container for flow content. It has no effect on the content or layout until styled using CSS."</span><span class="p">><</span><span class="nt">code</span><span class="p">></span><span class="ni">&lt;</span>div<span class="ni">&gt;</span><span class="p"></</span><span class="nt">code</span><span class="p">></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="ni">&lt;</span>script<span class="ni">&gt;</span> | |
| alert('How about this?'); | |
| <span class="ni">&lt;</span>/script<span class="ni">&gt;</span> | |
| </pre></div> | |
| </div> | |
| <p>In the sample document, the <code class="docutils literal"><span class="pre">{{CSSRef}}</span></code> macro renders a sidebar. It uses | |
| data from the <a class="reference external" href="https://github.com/mdn/data">mdn/data project</a> (fetched from GitHub), and the child pages of | |
| the CSS topic index (fetched from <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/CSS$children">Web/CSS$children</a> on the Kuma API server).</p> | |
| <p>Because the sample document isn’t a real CSS reference page, the sidebar is | |
| smaller than usual. The data may specify that a page is in one or | |
| more groups, and a cross-reference should be added to the sidebar. For example, | |
| on <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/CSS/@media">Web/CSS/@media</a>, the <a class="reference external" href="https://github.com/mdn/data/blob/master/css/at-rules.json">mdn/data JSON</a> says it is in the “Media Queries” | |
| group, and the cross-reference is populated from API data feteched from | |
| <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/CSS/Media_Queries$children">Web/CSS/Media_queries$children</a>. These data-driven elements can cause the | |
| sidebar to grow to several kilobytes.</p> | |
| <p>The <code class="docutils literal"><span class="pre">{{HTMLElement('div')}}</span></code> macro also requires metadata from the <code class="docutils literal"><span class="pre"><div></span></code> | |
| page, fetched from <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Web/HTML/Element/div$json">Web/HTML/Element/div$json</a> on the Kuma API server, to | |
| populate the <code class="docutils literal"><span class="pre">title</span></code> attribute of the link.</p> | |
| <p>Macros are implemented as <a class="reference external" href="https://www.ejs.co/">Embedded JavaScript templates</a> (<code class="docutils literal"><span class="pre">.ejs</span></code> files), | |
| which mix JavaScript code with HTML output. The <a class="reference external" href="https://developer.mozilla.org/en-US/dashboards/macros">macro dashboard</a> has a list | |
| of macros, provided by the KumaScript service, as well as the count of pages | |
| using the macros, populated from site search. The macro source is stored in | |
| the KumaScript repo, such as <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/CSSRef.ejs">CSSRef.ejs</a> and <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/HTMLElement.ejs">HTMLElement.ejs</a>. Macro names are | |
| case-insenstive, so <code class="docutils literal"><span class="pre">{{CSSRef}}</span></code> is the same as <code class="docutils literal"><span class="pre">{{cssref}}</span></code>.</p> | |
| <p>If KumaScript encounters an issue during rendering, the error | |
| is encoded and returned in an HTTP header, in a format compatible with <a class="reference external" href="https://firelogger.binaryage.com">FireLogger</a>. | |
| These errors are stored as JSON in <code class="docutils literal"><span class="pre">wiki_document.rendered_errors</span></code>. The | |
| rendered HTML isn’t stored, but it passed for further processing. Moderators | |
| frequently review <a class="reference external" href="https://developer.mozilla.org/en-US/docs/with-errors">documents with errors</a>, and fix those that they can fix.</p> | |
| <div class="section" id="environment-variables"> | |
| <h3>Environment variables<a class="headerlink" href="#environment-variables" title="Permalink to this headline">¶</a></h3> | |
| <p>KumaScript macros often vary on page metadata, stored in the <code class="docutils literal"><span class="pre">env</span></code> object in | |
| the render context. The render call is a <code class="docutils literal"><span class="pre">POST</span></code> where the body is the | |
| <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a>, and the headers include the encoded page metadata:</p> | |
| <dl class="docutils"> | |
| <dt>id</dt> | |
| <dd>The database ID of the document, like <code class="docutils literal"><span class="pre">233925</span></code></dd> | |
| <dt>locale</dt> | |
| <dd>The locale of the page, like <code class="docutils literal"><span class="pre">"en-US"</span></code></dd> | |
| <dt>modified</dt> | |
| <dd>The timestamp of the document modification time, like <code class="docutils literal"><span class="pre">1548278930.0</span></code></dd> | |
| <dt>path</dt> | |
| <dd>The URL path of the page, like <code class="docutils literal"><span class="pre">/en-US/docs/Sandbox/simple</span></code></dd> | |
| <dt>review_tags</dt> | |
| <dd>A list of review tags, like <code class="docutils literal"><span class="pre">["technical",</span> <span class="pre">"editorial"]</span></code></dd> | |
| <dt>revision_id</dt> | |
| <dd>The database ID of the revision, like <code class="docutils literal"><span class="pre">1438410</span></code></dd> | |
| <dt>slug</dt> | |
| <dd>The slug section of the URL, like <code class="docutils literal"><span class="pre">Sandbox/simple</span></code></dd> | |
| <dt>tags</dt> | |
| <dd>A list of document tags for the page, like <code class="docutils literal"><span class="pre">[]</span></code> or <code class="docutils literal"><span class="pre">["CSS"]</span></code></dd> | |
| <dt>title</dt> | |
| <dd>The document title, like <code class="docutils literal"><span class="pre">"A</span> <span class="pre">simple</span> <span class="pre">page"</span></code></dd> | |
| <dt>url</dt> | |
| <dd>The full URL of the page, forced to <code class="docutils literal"><span class="pre">http</span></code>, like <code class="docutils literal"><span class="pre">http://developer.mozilla.org/en-US/docs/Sandbox/simple</span></code>.</dd> | |
| </dl> | |
| </div> | |
| <div class="section" id="macro-rendering-speed"> | |
| <h3>Macro rendering speed<a class="headerlink" href="#macro-rendering-speed" title="Permalink to this headline">¶</a></h3> | |
| <p>It is unpredictable how long it will take to render the macros on a page. | |
| After editing, a render is requested, and if it returns quickly, then the | |
| rendered page is displayed. Otherwise, rendering is queued as a background | |
| task, and the user sees a message that rendering is in progress.</p> | |
| <p>Macros vary on rendering time, stability, and ease of testing based on where | |
| they get their data. From simplest to most complex:</p> | |
| <dl class="docutils"> | |
| <dt>functional</dt> | |
| <dd>The output varies only on the macro inputs, like <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/SimpleBadge.ejs">SimpleBadge</a></dd> | |
| <dt>environment data</dt> | |
| <dd>The output varies on the environment variables, like <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/ObsoleteBadge.ejs">ObsoleteBadge</a></dd> | |
| <dt>local data</dt> | |
| <dd>The output varies on data packaged with KumaScript, like <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/SpecName.ejs">SpecName</a> | |
| (from <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/SpecData.json">SpecData.json</a>) or <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/Compat.ejs">Compat</a> (from the npm-installed | |
| <a class="reference external" href="https://github.com/mdn/browser-compat-data">browser-compat-data project</a>)</dd> | |
| <dt>Kuma data</dt> | |
| <dd>The output varies on data gathered from <a class="reference external" href="https://developer.mozilla.org/en-US/docs/MDN/Contribute/Tools/Document_parameters#Document_metadata_resources">Kuma API calls</a> to an | |
| in-cluster dedicated Kuma API server, like <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/Index.ejs">Index</a>, which calls | |
| the <code class="docutils literal"><span class="pre">$children</span></code> API, or <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/HTMLElement.ejs">HTMLElement</a>, which calls the | |
| <code class="docutils literal"><span class="pre">$json</span></code> API.</dd> | |
| <dt>external data</dt> | |
| <dd>The output varies on data from an external data source, like | |
| <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/bug.ejs">Bug</a> (loads data from the <a class="reference external" href="https://bugzilla.mozilla.org">Bugzilla</a> API) or <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/CSSRef.ejs">CSSRef</a> (loads data from the | |
| <a class="reference external" href="https://github.com/mdn/data">mdn/data project</a> via the GitHub API)</dd> | |
| </dl> | |
| </div> | |
| </div> | |
| <div class="section" id="rendered-content"> | |
| <span id="id5"></span><h2>Rendered content<a class="headerlink" href="#rendered-content" title="Permalink to this headline">¶</a></h2> | |
| <p><em>Rendered content</em> is <a class="reference internal" href="#kumascript-content"><span>KumaScript content</span></a> that has been cleaned up | |
| using the same process as <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a>. This ensures that escaping | |
| issues in KumaScript macros do not affect the security of users on displayed | |
| pages.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd>Bleached <a class="reference internal" href="#kumascript-content"><span>KumaScript content</span></a></dd> | |
| <dt>displayed on MDN</dt> | |
| <dd><em>not published</em></dd> | |
| <dt>database</dt> | |
| <dd><code class="docutils literal"><span class="pre">wiki_document.rendered_html</span></code></dd> | |
| <dt>code</dt> | |
| <dd><code class="docutils literal"><span class="pre">kuma.wiki.models.Document.get_rendered()</span></code></dd> | |
| </dl> | |
| <p>The <em>Rendered content</em> for the simple document looks like this:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">p</span><span class="p">></</span><span class="nt">p</span><span class="p">><</span><span class="nt">section</span> <span class="na">class</span><span class="o">=</span><span class="s">"Quick_links"</span> <span class="na">id</span><span class="o">=</span><span class="s">"Quick_Links"</span><span class="p">><</span><span class="nt">ol</span><span class="p">><</span><span class="nt">li</span><span class="p">><</span><span class="nt">strong</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/CSS"</span><span class="p">></span>CSS<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">strong</span><span class="p">></</span><span class="nt">li</span><span class="p">><</span><span class="nt">li</span><span class="p">><</span><span class="nt">strong</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/CSS/Reference"</span><span class="p">></span>CSS Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">strong</span><span class="p">></</span><span class="nt">li</span><span class="p">></</span><span class="nt">ol</span><span class="p">></</span><span class="nt">section</span><span class="p">><</span><span class="nt">p</span><span class="p">></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color: red;"</span><span class="p">></span>I am red.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML/Element/div"</span> <span class="na">title</span><span class="o">=</span><span class="s">"The HTML Content Division element (&lt;div>) is the generic container for flow content. It has no effect on the content or layout until styled using CSS."</span><span class="p">><</span><span class="nt">code</span><span class="p">></span><span class="ni">&lt;</span>div<span class="ni">&gt;</span><span class="p"></</span><span class="nt">code</span><span class="p">></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="ni">&lt;</span>script<span class="ni">&gt;</span> | |
| alert('How about this?'); | |
| <span class="ni">&lt;</span>/script<span class="ni">&gt;</span> | |
| </pre></div> | |
| </div> | |
| <p>The parser doesn’t allow <code class="docutils literal"><span class="pre"><section></span></code> as a child element of <code class="docutils literal"><span class="pre"><p></span></code>, so the | |
| serializer closes the tag with a <code class="docutils literal"><span class="pre"></p></span></code>, and adds another empty paragraph | |
| element after the section. This is a side-effect of the differences between the | |
| editing format, where <code class="docutils literal"><span class="pre">{{CSSRef}}</span></code> is text that needs to be in a paragraph | |
| element, and the rendered content, where the macro is expanded as a | |
| <code class="docutils literal"><span class="pre"><section></span></code>.</p> | |
| </div> | |
| <div class="section" id="body-html"> | |
| <span id="id6"></span><h2>Body HTML<a class="headerlink" href="#body-html" title="Permalink to this headline">¶</a></h2> | |
| <p>The “middle” of a wiki document is populated by the <em>Body HTML</em>.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd>Extracted from <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>, cached in the database</dd> | |
| <dt>displayed on MDN</dt> | |
| <dd>On the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple">displayed page</a>, in an <code class="docutils literal"><span class="pre"><article></span></code> element</dd> | |
| <dt>database</dt> | |
| <dd><code class="docutils literal"><span class="pre">wiki_document.body_html</span></code></dd> | |
| <dt>code</dt> | |
| <dd><code class="docutils literal"><span class="pre">kuma.wiki.models.Document.get_body_html()</span></code></dd> | |
| </dl> | |
| <p>The <em>Body HTML</em> for the simple document looks like this:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">p</span><span class="p">></</span><span class="nt">p</span><span class="p">><</span><span class="nt">p</span><span class="p">></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color: red;"</span><span class="p">></span>I am red.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span> <span class="na">id</span><span class="o">=</span><span class="s">"Some_Links"</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML/Element/div"</span> <span class="na">title</span><span class="o">=</span><span class="s">"The HTML Content Division element (&lt;div>) is the generic container for flow content. It has no effect on the content or layout until styled using CSS."</span><span class="p">><</span><span class="nt">code</span><span class="p">></span><span class="ni">&lt;</span>div<span class="ni">&gt;</span><span class="p"></</span><span class="nt">code</span><span class="p">></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">rel</span><span class="o">=</span><span class="s">"nofollow"</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span> <span class="na">class</span><span class="o">=</span><span class="s">"new"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="ni">&lt;</span>script<span class="ni">&gt;</span> | |
| alert('How about this?'); | |
| <span class="ni">&lt;</span>/script<span class="ni">&gt;</span> | |
| </pre></div> | |
| </div> | |
| <p>The section <code class="docutils literal"><span class="pre"><section</span> <span class="pre">id="Quick_links"></span></code> is discarded, leaving the empty | |
| <code class="docutils literal"><span class="pre"><p></p></span></code> elements from the <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>. This can cause annoying | |
| empty space at the top of a document.</p> | |
| <p>IDs are injected into header elements (such as <code class="docutils literal"><span class="pre">id="Some_Links"</span></code>), | |
| based on the header text.</p> | |
| <p>Any links on the page are checked to see if they are links to other wiki | |
| pages, and if the destination page exists. The link to <code class="docutils literal"><span class="pre">a_new_document</span></code> | |
| gains a <code class="docutils literal"><span class="pre">rel="nofollow"</span></code> as well as <code class="docutils literal"><span class="pre">class="new"</span></code>, to tell crawlers | |
| and humans that the link is to a page that hasn’t been written yet.</p> | |
| </div> | |
| <div class="section" id="quick-links-html"> | |
| <span id="id7"></span><h2>Quick links HTML<a class="headerlink" href="#quick-links-html" title="Permalink to this headline">¶</a></h2> | |
| <p>The sidebar, on pages that include it, is populated from the <em>quick links html</em>.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd>Extracted from <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>, cached in the database</dd> | |
| <dt>displayed on MDN</dt> | |
| <dd>On the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple">displayed page</a>, in a <code class="docutils literal"><span class="pre"><div</span> <span class="pre">class="quick-links"</span> <span class="pre">id="quick-links"></span></code> element</dd> | |
| <dt>database</dt> | |
| <dd><code class="docutils literal"><span class="pre">wiki_document.quick_links_html</span></code></dd> | |
| <dt>code</dt> | |
| <dd><code class="docutils literal"><span class="pre">kuma.wiki.models.Document.get_quick_links_html()</span></code></dd> | |
| </dl> | |
| <p>For the simple document, the <em>Quick links HTML</em> looks like this:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">ol</span><span class="p">><</span><span class="nt">li</span><span class="p">><</span><span class="nt">strong</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/CSS"</span><span class="p">></span>CSS<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">strong</span><span class="p">></</span><span class="nt">li</span><span class="p">><</span><span class="nt">li</span><span class="p">><</span><span class="nt">strong</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/CSS/Reference"</span><span class="p">></span>CSS Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">strong</span><span class="p">></</span><span class="nt">li</span><span class="p">></</span><span class="nt">ol</span><span class="p">></span> | |
| </pre></div> | |
| </div> | |
| <p>The content of <code class="docutils literal"><span class="pre"><section</span> <span class="pre">id="Quick_Links"></span></code> is extracted from the rendered | |
| HTML. It is processed to annotate any new links with <code class="docutils literal"><span class="pre">rel="nofollow"</span></code> and | |
| <code class="docutils literal"><span class="pre">class="new"</span></code>.</p> | |
| </div> | |
| <div class="section" id="toc-html"> | |
| <span id="id8"></span><h2>ToC HTML<a class="headerlink" href="#toc-html" title="Permalink to this headline">¶</a></h2> | |
| <p>The table of contents is populated from the <code class="docutils literal"><span class="pre"><h2></span></code> elements of the | |
| <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>, if any, and appears as a floating “Jump to” bar when | |
| included. The “Jump to” bar can be supressed in editing mode by opening “Edit | |
| Page Title and Properties”, and setting TOC to “No table of contents”. | |
| The JavaScript can also decide to keep the bar hidden, such as when there | |
| is a single heading. Even when not shown, the <em>ToC HTML</em> is generated and cached.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd>Extracted from <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>, cached in the database</dd> | |
| <dt>displayed on MDN</dt> | |
| <dd>On the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple">displayed page</a>, in an <code class="docutils literal"><span class="pre"><ol</span> <span class="pre">class="toc-links"></span></code> element</dd> | |
| <dt>database</dt> | |
| <dd><code class="docutils literal"><span class="pre">wiki_document.toc_html</span></code></dd> | |
| <dt>code</dt> | |
| <dd><code class="docutils literal"><span class="pre">kuma.wiki.models.Document.get_toc_html()</span></code></dd> | |
| </dl> | |
| <p>For the simple document, the <em>ToC HTML</em> looks like this:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">rel</span><span class="o">=</span><span class="s">"internal"</span> <span class="na">href</span><span class="o">=</span><span class="s">"#Some_Links"</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">a</span><span class="p">></span> | |
| </pre></div> | |
| </div> | |
| </div> | |
| <div class="section" id="summary-text-and-html"> | |
| <span id="id9"></span><h2>Summary text and HTML<a class="headerlink" href="#summary-text-and-html" title="Permalink to this headline">¶</a></h2> | |
| <p>Summary text is used for SEO purposes. An editor can specify the summary text | |
| by adding an <code class="docutils literal"><span class="pre">id="Summary"</span></code> attribute to the element that contains the | |
| summary. Otherwise, the code extracts a summary from the first non-empty | |
| paragraph.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd>Extracted from <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>, cached in the database</dd> | |
| <dt>displayed on MDN (text)</dt> | |
| <dd><p class="first">On the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple">displayed page</a>, in the <code class="docutils literal"><span class="pre"><meta</span> <span class="pre">name"description"></span></code> element and other elements</p> | |
| <p>In <a class="reference external" href="https://developer.mozilla.org/en-US/search?q=%22I+am+a+simple+document%22&none=none">internal search results</a>, as the search hit summary</p> | |
| <p class="last">On some document lists, like <a class="reference external" href="https://developer.mozilla.org/en-US/docs/tag/CSS">Documents by tag</a></p> | |
| </dd> | |
| <dt>displayed on MDN (HTML)</dt> | |
| <dd><p class="first">The <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$json">page metadata view</a> (URLs ending in <code class="docutils literal"><span class="pre">$json</span></code>)</p> | |
| <p>The <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple?summary=1">summary view</a> (URLs with <code class="docutils literal"><span class="pre">?summary=1</span></code>) (currently broken, see <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1523955">bug 1523955</a>)</p> | |
| <p class="last">KumaScript macros that use page metadata, for example to populate <code class="docutils literal"><span class="pre">title</span></code> attributes</p> | |
| </dd> | |
| <dt>database</dt> | |
| <dd><p class="first"><code class="docutils literal"><span class="pre">wiki_document.summary_text</span></code></p> | |
| <p class="last"><code class="docutils literal"><span class="pre">wiki_document.summary_html</span></code></p> | |
| </dd> | |
| <dt>code</dt> | |
| <dd><p class="first"><code class="docutils literal"><span class="pre">kuma.wiki.models.Document.get_summary_text()</span></code></p> | |
| <p class="last"><code class="docutils literal"><span class="pre">kuma.wiki.models.Document.get_summary_html()</span></code></p> | |
| </dd> | |
| </dl> | |
| <p>For the simple document, the summary text is:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span>I am a simple document with a CSS sidebar. | |
| </pre></div> | |
| </div> | |
| <p>The summary HTML is:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar. | |
| </pre></div> | |
| </div> | |
| </div> | |
| <div class="section" id="diff-format"> | |
| <span id="id10"></span><h2>Diff format<a class="headerlink" href="#diff-format" title="Permalink to this headline">¶</a></h2> | |
| <p>MDN moderators and localization leaders are interested in the changes to wiki | |
| pages. They want to revert spam and vandalism, enforce documentation standards, | |
| and learn about the writer community. They are focused on what changed between | |
| document revisions. The differences format, or <em>Diff format</em>, is used to | |
| highlight content changes.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd><a class="reference internal" href="#revision-content"><span>Revision content</span></a>, pretty-printed with <a class="reference external" href="http://www.html-tidy.org/developer/">tidylib</a>, and | |
| compared to other revisions.</dd> | |
| <dt>displayed on MDN</dt> | |
| <dd><p class="first"><a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$compare?locale=en-US&to=1454597&from=1454596">Revision comparison views</a> (URLs ending in <code class="docutils literal"><span class="pre">$compare</span></code>)</p> | |
| <p>The <a class="reference external" href="https://developer.mozilla.org/en-US/dashboards/revisions">Revision dashboard</a></p> | |
| <p><a class="reference external" href="https://developer.mozilla.org/en-US/docs/MDN/Contribute/Tools/Page_watching">Page watch emails</a></p> | |
| <p>First edit emails, sent to content moderators</p> | |
| <p class="last"><a class="reference external" href="https://developer.mozilla.org/en-US/docs/MDN/Contribute/Tools/Feeds">RSS and Atom feeds</a></p> | |
| </dd> | |
| <dt>database</dt> | |
| <dd><code class="docutils literal"><span class="pre">wiki_revision.tidied_content</span></code></dd> | |
| <dt>code</dt> | |
| <dd><code class="docutils literal"><span class="pre">kuma.wiki.models.Revision.get_tidied_content()</span></code></dd> | |
| </dl> | |
| <p>The simple document in <em>Diff format</em> looks like this:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="cp"><!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"></span> | |
| <span class="p"><</span><span class="nt">html</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">head</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">title</span><span class="p">></</span><span class="nt">title</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">head</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">body</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span> | |
| {{ CSSRef }} | |
| <span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span> | |
| I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar. | |
| <span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">></span> | |
| I am red. | |
| <span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span><span class="p">></span> | |
| Some Links | |
| <span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">></span>{{HTMLElement('div')}} | |
| <span class="p"></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span> <span class="na">onclick</span><span class="o">=</span><span class="s">"alert('hacked!');"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">script</span><span class="p">></span> | |
| <span class="nx">alert</span><span class="p">(</span><span class="s1">'How about this?'</span><span class="p">);</span> | |
| <span class="p"></</span><span class="nt">script</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">body</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">html</span><span class="p">></span> | |
| </pre></div> | |
| </div> | |
| <p>The <a class="reference internal" href="#revision-content"><span>Revision content</span></a> is normalized using <a class="reference external" href="https://pypi.org/project/pytidylib/">pytidylib</a>, a Python interface | |
| to the C <a class="reference external" href="http://www.html-tidy.org/developer/">tidylib</a> library, which turns the content into a well-structured HTML | |
| 4.01 document.</p> | |
| <p>Content difference reports, or “diffs”, are generated by a line-by-line | |
| comparison of the content in <em>Diff format</em> of two revisions. Lines that differ | |
| are dropped, so that the reports focus on just the changed content, often | |
| without the wrapping HTML tags like <code class="docutils literal"><span class="pre"><p></p></span></code>. These diffs often contain line | |
| numbers from the <em>Diff format</em>, which do not correspond to the line numbers in | |
| the <a class="reference internal" href="#revision-content"><span>Revision content</span></a> because of differences in formatting and | |
| whitespace.</p> | |
| <p>Because the <em>Diff format</em> can contain unsafe content, it is not displayed | |
| directly on MDN. On <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$compare?locale=en-US&to=1454597&from=1454596">Revision comparison views</a>, the <a class="reference external" href="https://developer.mozilla.org/en-US/dashboards/revisions">Revision dashboard</a>, | |
| and in feeds, two <em>Diff formats</em> are processed by <a class="reference external" href="https://docs.python.org/2/library/difflib.html#difflib.HtmlDiff">difflib.HtmlDiff</a> to | |
| generate an HTML <code class="docutils literal"><span class="pre"><table></span></code> showing only the changed lines, and with HTML | |
| escaping for the content.</p> | |
| <p>For emails, <a class="reference external" href="https://docs.python.org/2/library/difflib.html#difflib.HtmlDiff">difflib.unified_diff</a> generates a text-based difference | |
| report, and it is sent as a plain-text email without escaping.</p> | |
| </div> | |
| <div class="section" id="triaged-content"> | |
| <span id="id11"></span><h2>Triaged content<a class="headerlink" href="#triaged-content" title="Permalink to this headline">¶</a></h2> | |
| <p>When a document is re-edited, the <a class="reference internal" href="#revision-content"><span>Revision content</span></a> of the current | |
| revision is processed before being sent to the editor. This is a lighter | |
| version of the full bleaching process used to create <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a> | |
| and <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd><a class="reference internal" href="#revision-content"><span>Revision content</span></a>, with further processing in <code class="docutils literal"><span class="pre">RevisionForm</span></code>.</dd> | |
| <dt>displayed on MDN</dt> | |
| <dd><p class="first">Editing <code class="docutils literal"><span class="pre"><textarea></span></code> in the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$edit">edit view</a> (URLs ending with <code class="docutils literal"><span class="pre">$edit</span></code>)</p> | |
| <p class="last">Editing <code class="docutils literal"><span class="pre"><textarea</span></code> in the <a class="reference external" href="https://developer.mozilla.org/en-US/docs/Sandbox/simple$translate?tolocale=fr">translate view</a> (URLs ending with <code class="docutils literal"><span class="pre">$translate</span></code>)</p> | |
| </dd> | |
| <dt>database</dt> | |
| <dd><em>not stored</em></dd> | |
| <dt>code</dt> | |
| <dd><em>not available</em></dd> | |
| </dl> | |
| <p>For the simple document, this is the <em>Triaged content</em>:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">p</span><span class="p">></span>{{ CSSRef }}<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color:red"</span><span class="p">></span>I am red.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span> <span class="na">id</span><span class="o">=</span><span class="s">"Some_Links"</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">></span>{{HTMLElement('div')}}<span class="p"></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">script</span><span class="p">></span> | |
| <span class="nx">alert</span><span class="p">(</span><span class="s1">'How about this?'</span><span class="p">);</span> | |
| <span class="p"></</span><span class="nt">script</span><span class="p">></span> | |
| </pre></div> | |
| </div> | |
| <p>The headers get IDs, based on the content, if they did not have them before. | |
| For example, <code class="docutils literal"><span class="pre">id="Some_Links"</span></code> is added to the <code class="docutils literal"><span class="pre"><h2></span></code>.</p> | |
| <p>A simple filter is applied that strips any attributes that start with | |
| <code class="docutils literal"><span class="pre">on</span></code>, such as the scripting attempt <code class="docutils literal"><span class="pre">onclick</span></code>. Further bleaching, | |
| for example to remove the <code class="docutils literal"><span class="pre"><script></span></code>, is not applied.</p> | |
| <p>CKEditor will perform additional parsing and formatting at load time. It will | |
| sometimes notice the empty <code class="docutils literal"><span class="pre"><div></span></code> and replace it with | |
| <code class="docutils literal"><span class="pre"><div</span> <span class="pre">class="button">&nbsp;</div></span></code>, especially if it is the last element | |
| on the page. It may also remove the <code class="docutils literal"><span class="pre"><script></span></code> element entirely.</p> | |
| <p>If a writer makes a change, these backend and CKEditor changes will be | |
| reflected in the new <a class="reference internal" href="#revision-content"><span>Revision content</span></a>. This can confuse writers | |
| (“I didn’t add those IDs!”).</p> | |
| </div> | |
| <div class="section" id="preview-content"> | |
| <span id="id12"></span><h2>Preview content<a class="headerlink" href="#preview-content" title="Permalink to this headline">¶</a></h2> | |
| <p>When editing, a user can request a preview of the document. This sends the | |
| in-progress document to editing, with a smaller list of environment variables.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd><a class="reference internal" href="#triaged-content"><span>Triaged content</span></a>, with CKEditor parsing, passed through KumaScript</dd> | |
| <dt>output</dt> | |
| <dd>HTML content at <code class="docutils literal"><span class="pre">/<locale>/docs/preview-wiki-content</span></code></dd> | |
| <dt>database</dt> | |
| <dd><em>not stored</em></dd> | |
| <dt>code</dt> | |
| <dd><em>not available</em></dd> | |
| </dl> | |
| <p>The <em>Preview content</em> for the simple document is:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">p</span><span class="p">></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color: red;"</span><span class="p">></span>I am red.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML/Element/div"</span> <span class="na">title</span><span class="o">=</span><span class="s">"The HTML Content Division element (&lt;div>) is the generic container for flow content. It has no effect on the content or layout until styled using CSS."</span><span class="p">><</span><span class="nt">code</span><span class="p">></span><span class="ni">&lt;</span>div<span class="ni">&gt;</span><span class="p"></</span><span class="nt">code</span><span class="p">></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="ni">&lt;</span>script<span class="ni">&gt;</span> | |
| alert('How about this?'); | |
| <span class="ni">&lt;</span>/script<span class="ni">&gt;</span> | |
| </pre></div> | |
| </div> | |
| <p>Fewer environment variables are passed to the KumaScript server for preview | |
| than when generating the <a class="reference internal" href="#kumascript-content"><span>KumaScript content</span></a>:</p> | |
| <dl class="docutils"> | |
| <dt>url</dt> | |
| <dd>The base URL of the website, like <code class="docutils literal"><span class="pre">https://developer.mozilla.org/</span></code></dd> | |
| <dt>locale</dt> | |
| <dd>The locale of the request, like <code class="docutils literal"><span class="pre">"en-US"</span></code></dd> | |
| </dl> | |
| <p>Some macros use the absence of an environment variable to detect preview mode, | |
| and change their output. For example, <code class="docutils literal"><span class="pre">{{CSSRef}}</span></code> notices that <code class="docutils literal"><span class="pre">env.slug</span></code> | |
| is not defined, and outputs an empty string, leaving <code class="docutils literal"><span class="pre"><p></p></span></code> in the | |
| preview output.</p> | |
| <p>Other macros don’t have specific code to detect preview mode, and have | |
| KumaScript rendering errors in preview.</p> | |
| <p>Some macros, like <code class="docutils literal"><span class="pre">{{HTMLElement}}</span></code>, work as expected in preview.</p> | |
| </div> | |
| <div class="section" id="raw-content"> | |
| <span id="id13"></span><h2>Raw content<a class="headerlink" href="#raw-content" title="Permalink to this headline">¶</a></h2> | |
| <p>A <code class="docutils literal"><span class="pre">?raw</span></code> parameter can be added to the end of a document to request the | |
| source for a revision. This is processed in a similar way to the | |
| <a class="reference internal" href="#triaged-content"><span>Triaged content</span></a>, but from the <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a>.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd><a class="reference internal" href="#bleached-content"><span>Bleached content</span></a>, with filters</dd> | |
| <dt>output</dt> | |
| <dd>The page with a <code class="docutils literal"><span class="pre">?raw</span></code> query parameter</dd> | |
| <dt>database</dt> | |
| <dd><em>not stored</em></dd> | |
| <dt>code</dt> | |
| <dd><em>not available</em></dd> | |
| </dl> | |
| <p>For the simple document, this is the <em>raw content</em>:</p> | |
| <div class="highlight-html"><div class="highlight"><pre><span></span><span class="p"><</span><span class="nt">p</span><span class="p">></span>{{ CSSRef }}<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span><span class="p">></span>I am a <span class="p"><</span><span class="nt">strong</span><span class="p">></span>simple document<span class="p"></</span><span class="nt">strong</span><span class="p">></span> with a CSS sidebar.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">p</span> <span class="na">style</span><span class="o">=</span><span class="s">"color: red;"</span><span class="p">></span>I am red.<span class="p"></</span><span class="nt">p</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">h2</span> <span class="na">id</span><span class="o">=</span><span class="s">"Some_Links"</span><span class="p">></span>Some Links<span class="p"></</span><span class="nt">h2</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/Web/HTML"</span><span class="p">></span>The HTML Reference<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">></span>{{HTMLElement('div')}}<span class="p"></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">li</span><span class="p">><</span><span class="nt">a</span> <span class="na">href</span><span class="o">=</span><span class="s">"/en-US/docs/NewDocument"</span><span class="p">></span>A new document<span class="p"></</span><span class="nt">a</span><span class="p">></</span><span class="nt">li</span><span class="p">></span> | |
| <span class="p"></</span><span class="nt">ul</span><span class="p">></span> | |
| <span class="p"><</span><span class="nt">div</span> <span class="na">class</span><span class="o">=</span><span class="s">"button"</span><span class="p">></</span><span class="nt">div</span><span class="p">></span> | |
| <span class="ni">&lt;</span>script<span class="ni">&gt;</span> | |
| alert('How about this?'); | |
| <span class="ni">&lt;</span>/script<span class="ni">&gt;</span> | |
| </pre></div> | |
| </div> | |
| <p>The <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a> is parsed for filtering . The headers get IDs, based | |
| on the content, if they did not have them before. For example, | |
| <code class="docutils literal"><span class="pre">id="Some_Links"</span></code> is added to the <code class="docutils literal"><span class="pre"><h2></span></code>.</p> | |
| <p>A simple filter is applied that strips any attributes that start with | |
| <code class="docutils literal"><span class="pre">on</span></code>, such as the scripting attempt <code class="docutils literal"><span class="pre">onclick</span></code>. However, this step should | |
| do nothing, since these attribute are dropped when creating the | |
| <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a>.</p> | |
| </div> | |
| <div class="section" id="live-sample"> | |
| <span id="id14"></span><h2>Live sample<a class="headerlink" href="#live-sample" title="Permalink to this headline">¶</a></h2> | |
| <p><a class="reference external" href="https://developer.mozilla.org/en-US/docs/MDN/Contribute/Structures/Live_samples">Live samples</a> are stored in document content. The content is then processed | |
| to extract the CSS, JS, and HTML, and reformat them as a stand-alone HTML | |
| document suitable for displaying in an <code class="docutils literal"><span class="pre"><iframe></span></code>.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd>A section extracted from <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>, with further processing</dd> | |
| <dt>output</dt> | |
| <dd>Live sample documents on a separate domain, such as <a class="reference external" href="https://mdn.mozillademos.org">https://mdn.mozillademos.org</a></dd> | |
| <dt>database</dt> | |
| <dd>Not stored in the database, but cached</dd> | |
| <dt>code</dt> | |
| <dd><code class="docutils literal"><span class="pre">kuma.wiki.Document.extract.code_sample(section_id)</span></code></dd> | |
| </dl> | |
| <p>The simple document does not include one of these samples. | |
| The <a class="reference external" href="https://developer.mozilla.org/en-US/docs/MDN/Contribute/Structures/Live_samples">Live samples</a> page on MDN describes how the system works for content | |
| authors, and includes a <a class="reference external" href="https://developer.mozilla.org/en-US/docs/MDN/Contribute/Structures/Live_samples#Live_sample_demo">live sample demo</a>.</p> | |
| <p>Most live samples are loaded in an <code class="docutils literal"><span class="pre"><iframe></span></code>, inserted by the macro | |
| <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/EmbedLiveSample.ejs">EmbedLiveSample</a>. If the sample doesn’t work as an <code class="docutils literal"><span class="pre"><iframe></span></code>, | |
| <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/LiveSampleLink.ejs">LiveSampleLink</a> can be used instead. The <code class="docutils literal"><span class="pre"><iframe</span> <span class="pre">src=</span></code> URL is Kuma, running | |
| on a different domain, such as <a class="reference external" href="https://mdn.mozillademos.org">https://mdn.mozillademos.org</a>, and configured | |
| to serve live samples (the <a class="reference external" href="https://mdn.mozillademos.org/en-US/docs/MDN/Contribute/Structures/Live_samples$samples/Live_sample_demo?revision=1438808">code sample view</a>) and attachments. A separate | |
| domain for user-created content, often served in an <code class="docutils literal"><span class="pre"><iframe></span></code>, mitigates | |
| many security issues.</p> | |
| <p>The live sample is cached on first access, and generated when requested. The | |
| extractor looks for <code class="docutils literal"><span class="pre"><pre></span></code> sections with <code class="docutils literal"><span class="pre">class="brush:</span> <span class="pre">html"</span></code>, | |
| <code class="docutils literal"><span class="pre">"brush:</span> <span class="pre">css"</span></code>, and <code class="docutils literal"><span class="pre">"brush:</span> <span class="pre">js"</span></code>, to find the sample content, and then | |
| selectively un-escapes some HTML and CSS. These sections are used to | |
| populate a basic HTML file.</p> | |
| <p>There are other sample types that are not derived from wiki content. | |
| These are out-of-scope for this document, but the most significant are listed | |
| here for the curious:</p> | |
| <ul class="simple"> | |
| <li><strong>Legacy samples</strong>, like <a class="reference external" href="https://developer.mozilla.org/samples/cssref/background-attachment.html">cssref/background-attachment.html</a>, are no longer maintained | |
| and are planned for removal (see <a class="reference external" href="https://bugzilla.mozilla.org/show_bug.cgi?id=1076893">bug 1076893</a> and related bugs).</li> | |
| <li><strong>GitHub Live Samples</strong>, like the <a class="reference external" href="https://mdn.github.io/css-examples/shapes/overview/circle.html">CSS circle demo</a>, are maintained in an | |
| MDN repo like <a class="reference external" href="https://github.com/mdn/css-examples">mdn/css-examples</a>, served by GitHub pages, | |
| and inserted with <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/EmbedGHLiveSample.ejs">EmbedGHLiveSample</a>.</li> | |
| <li><strong>Interactive examples</strong> are sourced in the | |
| <a class="reference external" href="https://github.com/mdn/interactive-examples">mdn/interactive-examples repository</a>, deployed as a static website, | |
| inserted with the <a class="reference external" href="https://github.com/mdn/kumascript/blob/master/macros/EmbedInteractiveExample.ejs">EmbedInteractiveExamples</a> macro near the top of the page, | |
| and are displayed in an <code class="docutils literal"><span class="pre"><iframe></span></code>.</li> | |
| </ul> | |
| </div> | |
| <div class="section" id="search-content"> | |
| <span id="id15"></span><h2>Search content<a class="headerlink" href="#search-content" title="Permalink to this headline">¶</a></h2> | |
| <p>Wiki documents are converted to a JSON format and indexed by | |
| <code class="xref doc docutils literal"><span class="pre">ElasticSearch</span></code> for <a class="reference external" href="https://developer.mozilla.org/en-US/search">internal search</a>. This allows searching | |
| for words in the wiki content.</p> | |
| <dl class="docutils"> | |
| <dt>source</dt> | |
| <dd>text extracted from <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a></dd> | |
| <dt>output</dt> | |
| <dd>text sections from in-content results from <a class="reference external" href="https://developer.mozilla.org/en-US/search">internal search</a></dd> | |
| <dt>database</dt> | |
| <dd>stored in ElasticSearch</dd> | |
| <dt>code</dt> | |
| <dd><code class="docutils literal"><span class="pre">kuma.wiki.search.WikiDocumentType.from_django(Document)</span></code></dd> | |
| </dl> | |
| <p>The Django utility <a class="reference external" href="https://docs.djangoproject.com/en/1.11/ref/utils/#django.utils.html.strip_tags">strip_tags</a> is used to quickly remove HTML tags. This | |
| utility is not guarenteed to generate an HTML safe string, as highlighted | |
| in a <a class="reference external" href="https://www.djangoproject.com/weblog/2014/mar/22/strip-tags-advisory/">security advisory</a>. Kuma does not redisplay this string. ElasticSearch | |
| applies the <a class="reference external" href="https://www.elastic.co/guide/en/elasticsearch/reference/5.6/analysis-htmlstrip-charfilter.html">HTML Strip Char Filter</a> to this and other content, which also | |
| strips tags and replaces HTML entities like <code class="docutils literal"><span class="pre">&amp;</span></code> with the character | |
| equivalents like <code class="docutils literal"><span class="pre">&</span></code>.</p> | |
| <p>When a search result is picked because of a content match, ElasticSearch | |
| returns the matching section, highlighting the matching terms in bold. This | |
| HTML is redisplayed on the search results page.</p> | |
| <p>Documents are indexed when created and when updated, as an asyncronous process. | |
| Documented are removed from the index when deleted. Administrators can also | |
| re-create the entire index, for ElasticSearch upgrades or to freshen the data.</p> | |
| <p>There is additional page metadata sent to ElasticSearch to power internal | |
| search. This includes page titles, tags, and locales. It also includes | |
| KumaScript macros, CSS class names, and HTML attributes, to allow | |
| <a class="reference external" href="https://developer.mozilla.org/en-US/docs/MDN/Contribute/Tools/Search">advanced search queries</a> and to power the <a class="reference external" href="https://developer.mozilla.org/en-US/dashboards/macros">macro dashboard</a>.</p> | |
| </div> | |
| <div class="section" id="future-changes"> | |
| <h2>Future Changes<a class="headerlink" href="#future-changes" title="Permalink to this headline">¶</a></h2> | |
| <p>Rendering evolved over years, and this document describes how it works, rather | |
| than how it was designed. There are some potential changes that would simplify | |
| rendering:</p> | |
| <ul class="simple"> | |
| <li>Sidebar macros are heavy users of API data and require post-processing of the | |
| <a class="reference internal" href="#rendered-content"><span>Rendered content</span></a>. Sidebar generation could be moved into Kuma instead | |
| of being specified by a macro.</li> | |
| <li>The <a class="reference internal" href="#diff-format"><span>Diff format</span></a> could be replaced by the <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a> | |
| format, which would be stored for each revision rather than just for the most | |
| recent document.</li> | |
| <li>Content from editing could be normalized and filtered before storing as the | |
| <a class="reference internal" href="#revision-content"><span>Revision content</span></a>. This may unify the <a class="reference internal" href="#triaged-content"><span>Triaged content</span></a>, | |
| <a class="reference internal" href="#diff-format"><span>Diff format</span></a>, and <a class="reference internal" href="#bleached-content"><span>Bleached content</span></a>.</li> | |
| <li>The views that accept new revisions could add IDs to the content before | |
| storing the <a class="reference internal" href="#revision-content"><span>Revision content</span></a>, rather than wait for the | |
| <a class="reference internal" href="#triaged-content"><span>Triaged content</span></a> or <a class="reference internal" href="#body-html"><span>Body HTML</span></a>.</li> | |
| <li>Developers could refactor the code to consistently access and generate | |
| content, rather than repeat filter logic in different forms and views.</li> | |
| </ul> | |
| </div> | |
| <div class="section" id="history"> | |
| <h2>History<a class="headerlink" href="#history" title="Permalink to this headline">¶</a></h2> | |
| <p>MDN has used different rendering processes in the past.</p> | |
| <p>Prior to 2004, Netscape’s DevEdge was a statically-generated website, with | |
| content stored in a revision control system (<a class="reference external" href="https://en.wikipedia.org/wiki/Concurrent_Versions_System">CVS</a> or similar). This was | |
| shut down for a while, until Mozilla was able to negotiate a license for the | |
| content.</p> | |
| <p>From 2005 to 2008, <a class="reference external" href="https://en.wikipedia.org/wiki/MediaWiki">MediaWiki</a> was used as the engine of Mozilla Developer | |
| Center. The DevEdge content was converted to <a class="reference external" href="https://en.wikipedia.org/wiki/MediaWiki#Markup">MediaWiki Markup</a>.</p> | |
| <p>From 2008 to 2011, <a class="reference external" href="https://en.wikipedia.org/wiki/MindTouch">MindTouch DekiWiki</a> was used as the engine. MindTouch | |
| migrated the MediaWiki content to the DekiWiki format, a restricted subset of | |
| HTML, augmented with macros (“DekiScript”). During this period, the site was | |
| rebranded as Mozilla Developer Network.</p> | |
| <p>In 2011, Kuma was forked from <a class="reference external" href="https://github.com/mozilla/kitsune">Kitsune</a>, the Django-based platform for | |
| <a class="reference external" href="https://support.mozilla.org/en-US/">support.mozilla.org</a>. The wiki format was as close as possible to the | |
| DekiWiki format. A new service <a class="reference external" href="https://github.com/mdn/kumascript">KumaScript</a> was added to implement | |
| DekiScript-style macros. The macros, also known as templates, were stored | |
| as content in the database. The service had a <code class="docutils literal"><span class="pre">GET</span></code> API to render pages, | |
| and a <code class="docutils literal"><span class="pre">POST</span></code> API to render previews.</p> | |
| <p>In 2013, content zones were added, which allowed a “zone” of pages to have a | |
| different style from the rest of the site. For example, the Firefox Zone of | |
| all the documents under <code class="docutils literal"><span class="pre">/Mozilla/Firefox</span></code> had a logo and a shared | |
| sub-navigation sidebar. Sub-navigation was similar to quick links, identified | |
| by <code class="docutils literal"><span class="pre"><section</span> <span class="pre">id="Subnav"></span></code>, but stored on the “zone root” | |
| (<code class="docutils literal"><span class="pre">/Mozilla/Firefox</span></code>) rather than generated by a macro. Zones were part of an | |
| effort to consolidate developer documentation on MDN.</p> | |
| <p>In 2016, the macros were exported from the Kuma database into the | |
| <a class="reference external" href="https://github.com/mdn/kumascript/commits/master/macros">macros folder in the KumaScript repository</a>. The historical changes were | |
| exported to <a class="reference external" href="https://github.com/mdn/archived_kumascript">mdn/archived_kumascript</a>. This made rendering faster, and | |
| allowed code reviews and automated tests of macros, at the cost of requiring | |
| review and a production push to deploy macro changes.</p> | |
| <p>In 2018, the content zones feature was removed. This was part of an effort | |
| to focus MDN Web Docs on common web platform technologies, and away from | |
| Mozilla-specific documentation. The sub-navigation feature was dropped.</p> | |
| <p>In 2019, the KumaScript engine and macros were modernized to use current | |
| features of JavaScript, such as <code class="docutils literal"><span class="pre">async</span></code> / <code class="docutils literal"><span class="pre">await</span></code>, rather than | |
| libraries common in 2011. The API was also unified, so that both previews | |
| and standard renders required a <code class="docutils literal"><span class="pre">POST</span></code>.</p> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="sphinxsidebar" role="navigation" aria-label="main navigation"> | |
| <div class="sphinxsidebarwrapper"> | |
| <h1 class="logo"><a href="index.html">Kuma</a></h1> | |
| <p class="blurb">The platform that powers <a href="https://developer.mozilla.org/en-US/">MDN</a></p> | |
| <h3>Navigation</h3> | |
| <ul class="current"> | |
| <li class="toctree-l1"><a class="reference internal" href="installation.html">Installation</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="development.html">Development</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="troubleshooting.html">Troubleshooting</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="tests.html">The Kuma test suite</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="tests-ui.html">Client-side testing</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="tests-performance.html">Performance testing</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="feature-toggles.html">Feature toggles</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="celery.html">Celery and async tasks</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="elasticsearch.html">Search</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="localization.html">Localization</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="ckeditor.html">CKEditor</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="data.html">Getting Data</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="documentation.html">Documentation</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="docker.html">Docker</a></li> | |
| <li class="toctree-l1"><a class="reference internal" href="deploy.html">Deploying MDN</a></li> | |
| <li class="toctree-l1 current"><a class="current reference internal" href="">Rendering Documents</a><ul> | |
| <li class="toctree-l2"><a class="reference internal" href="#revision-content">Revision content</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#bleached-content">Bleached content</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#kumascript-content">KumaScript content</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#rendered-content">Rendered content</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#body-html">Body HTML</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#quick-links-html">Quick links HTML</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#toc-html">ToC HTML</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#summary-text-and-html">Summary text and HTML</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#diff-format">Diff format</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#triaged-content">Triaged content</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#preview-content">Preview content</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#raw-content">Raw content</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#live-sample">Live sample</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#search-content">Search content</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#future-changes">Future Changes</a></li> | |
| <li class="toctree-l2"><a class="reference internal" href="#history">History</a></li> | |
| </ul> | |
| </li> | |
| </ul> | |
| <hr /> | |
| <ul> | |
| <li class="toctree-l1"><a href="https://developer.mozilla.org">MDN</a></li> | |
| <li class="toctree-l1"><a href="https://developer.allizom.org">MDN Staging</a></li> | |
| <li class="toctree-l1"><a href="https://github.com/mozilla/kuma">Kuma on GitHub</a></li> | |
| <li class="toctree-l1"><a href="https://github.com/mdn/kumascript">KumaScript on GitHub</a></li> | |
| </ul> | |
| <div class="relations"> | |
| <h3>Related Topics</h3> | |
| <ul> | |
| <li><a href="index.html">Documentation overview</a><ul> | |
| <li>Previous: <a href="deploy.html" title="previous chapter">Deploying MDN</a></li> | |
| </ul></li> | |
| </ul> | |
| </div> | |
| <div id="searchbox" style="display: none" role="search"> | |
| <h3>Quick search</h3> | |
| <form class="search" action="search.html" method="get"> | |
| <input type="text" name="q" /> | |
| <input type="submit" value="Go" /> | |
| <input type="hidden" name="check_keywords" value="yes" /> | |
| <input type="hidden" name="area" value="default" /> | |
| </form> | |
| <p class="searchtip" style="font-size: 90%"> | |
| Enter search terms or a module, class or function name. | |
| </p> | |
| </div> | |
| <script type="text/javascript">$('#searchbox').show(0);</script> | |
| </div> | |
| </div> | |
| <div class="clearer"></div> | |
| </div> | |
| <div class="footer"> | |
| ©Mozilla. | |
| | | |
| Powered by <a href="http://sphinx-doc.org/">Sphinx 1.3.5</a> | |
| & <a href="https://github.com/bitprophet/alabaster">Alabaster 0.7.10</a> | |
| | | |
| <a href="_sources/rendering.txt" | |
| rel="nofollow">Page source</a> | |
| </div> | |
| </body> | |
| </html> |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment