Skip to content

Instantly share code, notes, and snippets.

@tracker1
Created August 1, 2019 00:14
Show Gist options
  • Save tracker1/84fd0c1a31b3d17b176495f37b3431ba to your computer and use it in GitHub Desktop.
Save tracker1/84fd0c1a31b3d17b176495f37b3431ba to your computer and use it in GitHub Desktop.
Custom encoded data into JSON

For encoding custom/complex data in JSON for generic detection, I suggest the following methodology.

// USE ASCII Characters for kind of what they are meant for.
const DLE = String.fromCharCode(16); // Data Link Escape
const SOH = String.fromCharCode(1); // start of heading
const STX = String.fromCharCode(2); // start ot text
const ETX = String.fromCharCode(3); // end of text
const EOT = String.fromCharCode(4); // end of transmission

An encoded data record should consist of...

   // Encoded record ...
   `${DLE}${SOH}TYPE${STX}DATA${ETX}${EOT}${DLE}`

If the string starts with DLE + SOH and ends with ETX + EOT + DLE then it can be checked for an encoding. Further split on STX into header and body parts.

In this case TYPE could be "DATE/RFC3339" and the DATA would be a properly encoded string.

In this way, we could use existing escape characters for encoded data, and if there's a supported decoder, it can be added and handle any encoded complex type, while still complying with original JSON.


For more complex data types, there's also...

const FS = String.fromCharCode(28); // file separator
const GS = String.fromCharCode(29); // group separator
const RS = String.fromCharCode(30); // record separator
const US = String.fromCharCode(31); // unit separator

For further reading, see ASCII Control Codes: https://text-symbols.com/ascii/

@tracker1
Copy link
Author

tracker1 commented Aug 1, 2019

If the actual data is binary, make the DATA section Base-64 text.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment