Skip to content

Instantly share code, notes, and snippets.

@johnloy
Created August 23, 2016 12:59
Show Gist options
  • Save johnloy/260b1e7792b3fa0ad36250415a08d570 to your computer and use it in GitHub Desktop.
Save johnloy/260b1e7792b3fa0ad36250415a08d570 to your computer and use it in GitHub Desktop.
Reusable parsing algorithm using pure functions

There's no need to fully parse the xml of entire clearleap api responses to JSON. And there doesn't seem to be a compelling reason to use xml2json, unless api responses need to be rendered on the back end in an isomorphic app. Usually, we just want to pluck certain values out, massage them a bit, and add them as properties of an object/structure better suited to presentation.

At its simplest, this can be done with a function that takes an instance of Node, performs DOM querying logic to retrieve and aggregate values from elements and attributes, and then returns a single value.

This gist shows a way to reuse and compose pure functions of this variety to converge parsing operations for invidual values into a final object. Because only pure functions are involved, presumably parsing could be parallelized with web workers (https://adambom.github.io/parallel.js/). Each pure function involved is also automatically memoized and passed within an aggregate object into every function when invoked. This allows those functions to potentially make use of one another internally without having to concern themselves about redundant effort.

To follow the flow of how this all works, I recommend reading the source in the following order:

  1. parse-next-asset.js — The high-level operation of using a composed parser.
  2. parsers.js — An example of how different parsers might be composed from individual functions.
  3. converge-parsers.js — The curried function that produces composed parsers.
  4. play-response-parse-fns.js — The pure functions that return a single value after parsing a source value or object.

This technique is possibly a replacement for https://github.com/willowtreeapps/cl2json, and would likely result in less bundled code — xml2json wouldn't be necessary at all — greater flexibility for reuse. It also looks like R.createMapEntry has been removed from recent releases of Ramda, and that function is heavily relied upon by cl2json.

There's also no reason why this technique couldn't be applied to parsing JSON, as long as the parser functions accept an Object type as their first argument.

A caveat regarding this technique that should be mentioned is that it requires that parser functions be defined using function declarations so that they have a .name property. These names are used as key names in the final object produced from parsing. To ensure this works, uglify, or whichever minification and obfuscation tool is used, needs to be configured to not mangle symbols. To my knowledge, this is the default for the uglify webpack loader, as well as for grunt-contrib-uglify.

import R from 'ramda';
function convergeToObj (buildTuple, list) {
const converged = R.pipe(
R.curry(buildTuple),
R.flip(R.chain)(list),
R.splitEvery(2),
R.fromPairs
)();
return converged;
}
const convergeParsers = R.curry((parseFns, source) => {
const buildMemoizedTuple = (parseFn) => {
const memoized = R.memoize(parseFn);
memoized._name = parseFn.name;
return [ parseFn.name, memoized ];
};
const memoizedParseFns = convergeToObj(buildMemoizedTuple, parseFns);
const buildParsedTuple = (parseFn) => {
const parsedValue = parseFn(source, memoizedParseFns);
return [ parseFn._name, parsedValue ];
};
const converged = convergeToObj(buildParsedTuple, R.values(memoizedParseFns));
return converged;
});
export default convergeParsers;
import * as parse from 'parsers';
const playResponse = // in play-response.xml
// No need to for SAX parsing of XML (e.g. xml2json). Just use DOM querying methods.
const domParser = new window.DOMParser();
const xml = domParser.parseFromString(playResponse, 'text/xml');
// The magic happens here...
const episode = parse.episode(xml);
// Parse xml to an object like:
// {
// guid: '1f10ced-0049a1b0561',
// seriesTitle: 'Rome'
// title: 'Philippi',
// description: 'Two armies clash with the future of Rome at stake and Pullo is sent on a brutal mission.',
// episodeInSeason: 6,
// images: [
// {
// profile: 'NORDIC-POSTER'
// url: 'https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357594.jpg',
// width: 540,
// height: 540
// }
// ]
// }
import * as p from 'play-response-parse-fns';
import convergeParsers from 'converge-parsers';
// Defining the shape of a final parsed object is as simple as
// making a list/array of the parsers used to extract the object's
// property values. The names of the parser functions will be used
// as keys in the final object.
const episodeParsers = [
p.guid,
p.seriesTitle,
p.title,
p.description,
p.episodeInSeason,
p.images
];
// Define other parser lists below and use simple Array methods
// to compose and blend them.
// ...
// Finally, export a curried final parser function which takes a
// single argument, a value to parse. The value can be xml, json,
// or anything else that's parseable.
export { convergeParsers(episodeParsers) as episode };
import R from 'ramda';
const CLEARLEAP_XML_NS = 'http://www.clearleap.com/namespace/clearleap/1.0/';
export function guid (xml, parseFns) {
const value;
try {
value = xml.querySelector('channel > guid').firstChild.nodeValue;
} catch (e) {
throw new Error('Expected guid element as direct child of the channel element.');
}
return value;
}
export function seriesTitle (xml, parseFns) {
const value;
try {
value =
xml.getElementsByTagNameNS(CLEARLEAP_XML_NS, 'series')
[0]
.firstChild
.nodeValue;
} catch (e) {
throw new Error('Expected clearleap:series element as an ancestor of the channel element.');
}
return value;
}
export function title(xml, parseFns) {
const value;
try {
value = xml.querySelector('channel title').firstChild.nodeValue;
} catch (e) {
throw new Error('Expected title element as an ancestor of the channel element.');
}
return value;
}
export function description(xml, parseFns) {
const value;
try {
value = xml.querySelector('channel description').firstChild.nodeValue;
} catch (e) {
throw new Error('Expected description element as an ancestor of the channel element.');
}
return value;
}
export function episodeInSeason(xml, parseFns) {
const value;
try {
value =
xml.getElementsByTagNameNS(CLEARLEAP_XML_NS, 'episodeInSeason')
[0]
.firstChild
.nodeValue;
} catch (e) {
throw new Error('Expected clearleap:episodeInSeason element as an ancestor of the channel element.');
}
return value;
}
export function images(xml, parseFns) {
const thumbnailEls = xml.getElementsByTagNameNS('http://search.yahoo.com/mrss/', 'thumbnail');
const value = R.reduce(function (images, el) {
const url = el.getAttribute('url');
const profile, width, height;
images.alreadyFound = images.alreadyFound || {};
if (!(url in images.alreadyFound)) {
images.push({
url: url,
profile: el.getAttribute('profile'),
width: parseInt(el.getAttribute('width'), 10),
height: parseInt(el.getAttribute('height'), 10)
});
images.alreadyFound[url] = null;
}
return images;
}, [], thumbnailEls);
return value;
}
<result version='2.0' xmlns:media='http://search.yahoo.com/mrss/' xmlns:dcterms='http://purl.org/dc/terms/' xmlns:clearleap='http://www.clearleap.com/namespace/clearleap/1.0/'>
<status>success</status>
<url>http://hbonordic-production-vod.hds.adaptive.level3.net/stream/1f10ced-004d8f0641c/HBON-AAFGQ-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6448919.zip/HBON-AAFGQ-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6448919.f4m</url>
<clearleap:chainplayTimer>10</clearleap:chainplayTimer>
<clearleap:cuePoints>
<clearleap:cuePoint type="end_credits">3384000</clearleap:cuePoint>
</clearleap:cuePoints>
<rss version='2.0'>
<channel>
<title>Philippi</title>
<description>Two armies clash with the future of Rome at stake and Pullo is sent on a brutal mission.</description>
<guid>1f10ced-0049a1b0561</guid>
<clearleap:analyticsLabel>Philippi</clearleap:analyticsLabel>
<language>en_hbon</language>
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-7362457.jpg' profile='HBON-MOVIE-BANNER-550-221' width='221' height='221' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357594.jpg' profile='NORDIC-POSTER' width='540' height='540' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357599.jpg' profile='HBON-MOVIE-POSTER-720-405' width='405' height='405' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357601.jpg' profile='HBON-MOVIE-POSTER-640-360' width='360' height='360' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6549781.jpg' profile='HBON-SAMSUNG-ASSET-THUMB-176-99' width='99' height='99' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710867.jpg' profile='HBON-XBOX360-ASSET-THUMB-263-148' width='148' height='148' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-7362459.jpg' profile='HBON-MOVIE-BANNER-640-257' width='257' height='257' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710862.jpg' profile='HBON-XBOX360-MOVIE-POSTER-533-300' width='300' height='300' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-7362458.jpg' profile='HBON-MOVIE-BANNER-720-290' width='290' height='290' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6549783.jpg' profile='HBON-SAMSUNG-ASSET-THUMB-192-108' width='108' height='108' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710864.jpg' profile='HBON-XBOX360-RECWATCHEDLIST-208-156' width='156' height='156' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357597.jpg' profile='NORDIC-POSTER-LARGE' width='630' height='630' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-7362456.jpg' profile='HBON-MOVIE-CAROUSEL-640-268' width='268' height='268' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710865.jpg' profile='HBON-XBOX360-CAROUSEL-422-317' width='317' height='317' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357600.jpg' profile='NORDIC-THUMB' width='126' height='126' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6549784.jpg' profile='HBON-SAMSUNG-ASSET-FEAT-720-405' width='405' height='405' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6549782.jpg' profile='HBON-SAMSUNG-ASSET-THUMB-272-153' width='153' height='153' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710863.jpg' profile='HBON-XBOX360-ASSET-FEAT-522-294' width='294' height='294' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710866.jpg' profile='HBON-XBOX360-EPISODE-POSTER-400-300' width='300' height='300' />
<ttl>180</ttl>
<link>https://api-hbon.hbo.clearleap.com:443/cloffice/client/web/browse/</link>
<pubDate>Tue Feb 16 14:40:02 UTC 2016</pubDate>
<clearleap:totalResults>1</clearleap:totalResults>
<clearleap:paymentMethodRequired>false</clearleap:paymentMethodRequired>
<item>
<guid>1f10ced-0049a1b0561</guid>
<title>Philippi</title>
<description>Two armies clash with the future of Rome at stake and Pullo is sent on a brutal mission.</description>
<media:keywords>Drama</media:keywords>
<clearleap:titleSortName>Rome Season 2:006,Philippi</clearleap:titleSortName>
<clearleap:shortTitle>Rome Season 2: 006,</clearleap:shortTitle>
<houseId>AAFGR</houseId>
<clearleap:episode>Philippi</clearleap:episode>
<clearleap:series>Rome</clearleap:series>
<clearleap:episodeInSeries>2006</clearleap:episodeInSeries>
<clearleap:season>2</clearleap:season>
<clearleap:episodeInSeason>6</clearleap:episodeInSeason>
<clearleap:bookmark>3243</clearleap:bookmark>
<clearleap:tagline>
<position>Upper Left</position>
<color>#000000</color>
</clearleap:tagline>
<clearleap:analyticsLabel>Rome Season 2:006,Philippi</clearleap:analyticsLabel>
<media:credit role='actor'>Ray Stevenson</media:credit>
<media:credit role='creator'>John Milius</media:credit>
<media:credit role='actor'>Polly Walker</media:credit>
<media:credit role='year'>2007</media:credit>
<media:credit role='creator'>William J. Macdonald</media:credit>
<media:credit role='actor'>Kevin McKidd</media:credit>
<media:credit role='actor'>Kerry Condon</media:credit>
<media:credit role='creator'>Bruno Heller</media:credit>
<clearleap:itemType>media</clearleap:itemType>
<media:price type='RENTAL' price='0.00' currency='USD' />
<media:status status='ACTIVE' />
<dcterms:available>start=2012-10-14T22:00:00;new=0d;end=2019-12-31T22:59:59;lastChance=0d;scheme:W3C-DTF</dcterms:available>
<media:rating scheme='urn:hbon'>15</media:rating>
<media:subTitle type='text/xml' lang='da_hbon' href='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-SUB-01-01-XML-DA.xml' profile='NORDIC-SUBTITLE-DA.xml' />
<media:subTitle type='text/xml' lang='no_hbon' href='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-SUB-01-01-XML-NO.xml' profile='NORDIC-SUBTITLE-NO.xml' />
<media:subTitle type='text/xml' lang='fi_hbon' href='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-SUB-01-01-XML-FI.xml' profile='NORDIC-SUBTITLE-FI.xml' />
<media:subTitle type='text/xml' lang='sv_hbon' href='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-SUB-01-01-XML-SV.xml' profile='NORDIC-SUBTITLE-SV.xml' />
<media:group>
<media:content url='https://api-hbon.hbo.clearleap.com:443/cloffice/client/web/play/?contentId=d6f5c59b-2c97-4131-83a9-06df50e8706e&amp;categoryId=1f10ced-0049a1b0561' type='video/mp4' duration='3249' medium='video' isDefault='false' expression='full' profile='HBON-SMOOTH-PlayReady' />
<media:content url='https://api-hbon.hbo.clearleap.com:443/cloffice/client/web/play/?contentId=bdb42b62-518c-4afb-b7a4-516587278e4f&amp;categoryId=1f10ced-0049a1b0561' type='video/mp4' duration='3249' medium='video' isDefault='false' expression='full' profile='HBON-HDS-AdobeAccess' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-7362457.jpg' medium='image' width='221' height='221' profile='HBON-MOVIE-BANNER-550-221' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357594.jpg' medium='image' width='540' height='540' profile='NORDIC-POSTER' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357599.jpg' medium='image' width='405' height='405' profile='HBON-MOVIE-POSTER-720-405' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357601.jpg' medium='image' width='360' height='360' profile='HBON-MOVIE-POSTER-640-360' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6549781.jpg' medium='image' width='99' height='99' profile='HBON-SAMSUNG-ASSET-THUMB-176-99' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710867.jpg' medium='image' width='148' height='148' profile='HBON-XBOX360-ASSET-THUMB-263-148' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-7362459.jpg' medium='image' width='257' height='257' profile='HBON-MOVIE-BANNER-640-257' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710862.jpg' medium='image' width='300' height='300' profile='HBON-XBOX360-MOVIE-POSTER-533-300' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-7362458.jpg' medium='image' width='290' height='290' profile='HBON-MOVIE-BANNER-720-290' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6549783.jpg' medium='image' width='108' height='108' profile='HBON-SAMSUNG-ASSET-THUMB-192-108' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710864.jpg' medium='image' width='156' height='156' profile='HBON-XBOX360-RECWATCHEDLIST-208-156' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357597.jpg' medium='image' width='630' height='630' profile='NORDIC-POSTER-LARGE' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-7362456.jpg' medium='image' width='268' height='268' profile='HBON-MOVIE-CAROUSEL-640-268' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710865.jpg' medium='image' width='317' height='317' profile='HBON-XBOX360-CAROUSEL-422-317' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6357600.jpg' medium='image' width='126' height='126' profile='NORDIC-THUMB' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6549784.jpg' medium='image' width='405' height='405' profile='HBON-SAMSUNG-ASSET-FEAT-720-405' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6549782.jpg' medium='image' width='153' height='153' profile='HBON-SAMSUNG-ASSET-THUMB-272-153' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710863.jpg' medium='image' width='294' height='294' profile='HBON-XBOX360-ASSET-FEAT-522-294' alt='Rome Season 2 Episode 6 - Philippi' />
<media:thumbnail url='https://static.hbonordic.com/1f10ced-0049a1b0561/HBON-AAFGR-000-PGM-01-01-2500-HD-169-SR-1920x1080-50-6710866.jpg' medium='image' width='300' height='300' profile='HBON-XBOX360-EPISODE-POSTER-400-300' alt='Rome Season 2 Episode 6 - Philippi' />
</media:group>
<link>https://api-hbon.hbo.clearleap.com:443/cloffice/client/web/browse/1f10ced-0049a1b0561</link>
<clearleap:parentFolderUri>https://api-hbon.hbo.clearleap.com:443/cloffice/client/web/browse/f5c42d9c-ac1d-406e-b7d6-30b505295a3e</clearleap:parentFolderUri>
<clearleap:parentGuid>f5c42d9c-ac1d-406e-b7d6-30b505295a3e</clearleap:parentGuid>
<clearleap:meta title='Rome Season 2 Episode 6 - Watch on HBO Nordic' description='Watch season 2 episode 6 of Rome on HBO Nordic right now. You can stream to your laptop, tablet, mobile device, game console, and TV. Try one month for free.' />
<clearleap:displayName>Rome Season 2 Episode 6</clearleap:displayName>
</item>
</channel>
</rss>
</result>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment