An example of parsing log data for later transformation, analysis and visualization with Data-Forge

parsing log data.md

This markdown was exported from Data-Forge Notebook

Example of parsing log files so you have structured data to use with Data-Forge

First define some test data:

const data =
`{"@t":"2019-08-21T10:49:46.9587554Z","@mt":"Some message 1"}
{"@t":"2019-08-21T10:49:48.1290648Z","@mt":" Some message 2"}
{"@t":"2019-08-21T10:49:51.9312112Z","@mt":" Some message 3"}`;
display.text(data);

{"@t":"2019-08-21T10:49:46.9587554Z","@mt":"Some message 1"}
{"@t":"2019-08-21T10:49:48.1290648Z","@mt":" Some message 2"}
{"@t":"2019-08-21T10:49:51.9312112Z","@mt":" Some message 3"}

Split the data

Now split the data into separate lines:

const lines = data.split("\n");
display(lines);

Parse JSON data

Each line of data is a JSON object, so iterate the lines and parse each one:

const parsedLines = lines.map(line => JSON.parse(line));

Now you have structured data

You can use it however you want.

display(parsedLines);

Now you can use Data-Forge

Data-Forge isn't really useful to load or parse this data, because the data isn't in a standard format.

But after loading and parsing the data yourself you can use Data-Forge now for data transformation, analysis and visualization.

const dataForge = require("data-forge");

const dataFrame = new dataForge.DataFrame(parsedLines);
display(dataFrame);

index	@t	@mt
0	2019-08-21T10:49:46.9587554Z	Some message 1
1	2019-08-21T10:49:48.1290648Z	Some message 2
2	2019-08-21T10:49:51.9312112Z	Some message 3

Note that I have written a book on working with data in JavaScript!

There's heaps of good stuff like this in the book.

You can find it here:

http://bit.ly/2t2cJu2

This markdown was exported from Data-Forge Notebook