GSoC 2018 - Kibana: Calendar Visualization and Filtering

gsoc.md

Google Summer of Code with Elastic: Calendar Visualization and Filtering

Introduction

Friday, 31 August, 2018 Aaron Zhao

Kibana now has various visualizations that can apply data aggregations and filtering based on index pattern fields. It is currently missing a calendar heatmap visualization that shows field aggregation results, e.g. number of classes per day throughout a year. It is requested by my mentor Daniel that during Kibana teaching, there's no calendar-like visualizations to track the total timeline. After GSoC is over, I released a working prototype which can be found at Kibana Calendar Vis.

This would allow users to create and manipulate heatmap-like calendar visualization based on index pattern data retrieved from Elasticsearch engine. It could show year, month or day overview calendar visualizations based on the time span user provides.

Demonstration

Installation

For a demonstration to work, make sure you have Kibana version >= 6.4 and Elasticsearch version >= 6.4.

Follow the instructions to install Calendar Visualization as a plugin to Kibana by typing the following command under the root of Kibana:

bin/kibana-plugin install <installation file's url>

For this demo, the installation path would be https://github.com/aaronoah/kibana_calendar_vis/releases/download/v6.4.0/kibana_calendar_vis-6.4.0.zip

Playing Around

Kick start your Kibana instance after starting off Elasticsearch by the following command:

bin/kibana

The plugin now supports index pattern based on a specific time fileld when you create an index pattern:

Then, navigate to Visualize -> Create a visualization -> Other (Calendar), choose the index you just created and adjust the time filter located at the top toolbar to Last 1 year. What you would see is something like below:

Adjust the Metrics aggregation at the left sidebar to make more sense of available data; e.g. showing number of classes (alias to Average of bytes) for the last year.

Click on a specific month label located at the top of a month grid, e.g. Sept of year 2017, it will re-render a month view:

Click on a specific day cell on either the year view or month view will re-render a day view:

You can save the visualization for later access. e.g. press Save button on the top and name it demo. In Dashboard, you can add this visualization:

If expand the demo to full screen, the figure will also be enlarged to accommodate the container:

Challenges

Due to its nature as a plugin of Kibana, we faced various challenges along the summer. These include finding the proper way to color the day cells, rewriting existing legends in React, porting test infrastructures from Kibana, etc.

One particular challenge that I faced was reusing the field formats throughout the testing phase of calendar visualization plugin. For instance, the Kibana browser tests rely on pre-registered field formats such as IpFormat, NumberFormat, StringFormat etc. However, the plugin test environment does not have exposure to any pre-registration like Kibana main platform, issue#21273. The current workaround was creating a hack file against field formats registry in test environment;

function createStringFormat(FieldFormat) {
    return class StringFormat extends FieldFormat {
        _convert(val) {
            return String(val);
        }

        static id = 'string';
        static title = 'String';
        static fieldType = [
            'number',
            'boolean',
             ...
        ]
    }
}

fieldFormats.register(createStringFormat);
fieldFormats._defaultMap = {
    _default_: { id: 'string', params: {} }
};

It registers a default StringFormat during the tests and it worked for now. But it does not make much sense for all plugin dev environments in the long run, which means further discussions and explorations are required in the future.

During this journey, my mentors Tim and Thomas were very helpful in providing necessary hands-on assistance to make sure I fulfilled the tasks. Despite various active evolvements in Kibana repo, I managed to make a working prototype at the end.

Further Improvements

I would like to further the maintenance and development of Calendar Visualization during the journey of contributing to Kibana. Some of the main challenges ahead include:

• Refactor the editor implementations in React when the Elastic EUI components are ready to be ported in the majority of Kibana Visualizations.

• Add a sophisticated index pattern handling mechanism as to automatically add additional filters if index data does not has a specific time field.

• Devise a proper way for Kibana to setup the field formats of index pattern for all plugin testing environments.

Thoughts

Apart from implementing a calendar visualization during the summer, I enjoyed a lot from contributing to Kibana repo as well. By asking questions in the discuss forum and discussing issues on Github, I got to know more intelligent and warm-hearted Elastic engineers. I appreciate the opportunity GSoC and Kibana team gave me to expolre the world of open source and I'd like to contribute more to the communities!

Acknowledgements

First and foremost I'd like to thank my mentors who provided great help along the journey. Desipte the overwhelming meetings Tim has, He could take time answering my questions about the platforms. Thomas helped me made my first PR to the Kibana and ported it into production and is very responsive about questions I have during the development. Thanks Daniel, who originally requested the calendar visualization, discuss with me thoroughly about the design ideas and how it all makes sense. Thanks Philipp who helped my visa issue and touchbased with me the progress of the project.

All right, editing on the way :)