kylebrandt

Frame encoding within Grafana

By Destination

Case Backend Plugin:

• If going to frontend leave it encoded
• If going to alerting, decode it

Case Core (Internal) Datasource:

kylebrandt

{"results":{"A":{"refId":"A","series":null,"tables":null,"dataframes":["QVJST1cxAAD/////sBAAABAAAAAAAAoADgAMAAsABAAKAAAAFAAAAAAAAAEDAAoADAAAAAgABAAKAAAACAAAAJQAAAADAAAAVAAAACgAAAAEAAAA3O///wgAAAAMAAAAAQAAAEEAAAAFAAAAcmVmSWQAAAD87///CAAAABQAAAAKAAAAbWFueV90eXBlcwAABAAAAG5hbWUAAAAAJPD//wgAAAAkAAAAGQAAAHsiY3VzdG9tIjp7IkhpIjoidGhlcmUifX0AAAAEAAAAbWV0YQAAAAAdAAAAFA8AANQMAAAYDAAAlAsAACALAACcCgAAKAoAAKQJAAAwCQAArAgAADQIAAC0BwAARAcAAMQGAABUBgAA1AUAAGQFAADkBAAAbAQAAOwDAAB8AwAA/AIAAJgCAAAcAgAAtAEAAEwBAADUAAAAbAAAAAQAAACu8///FAAAAEAAAABAAAAAAAASAUAAAAABAAAABAAAAPTw//8IAAAAFAAAAAgAAABkdXJhdGlvbgAAAAAEAAAAbmFtZQAAAAAAAAAAuvv//wAAAwAIAAAAZHVyYXRpb24AAAAA1vH//xQAAABAAAAAQAAAAAAAABJAAAAAAQAAAAQAAABY8f//CAAAABQAAAAIAAAAZHVyYXRpb24AAAAABAAAAG5hbWUAAAAAAAAAAB78//8AAAMACAAAAGR1cmF0aW9uAAAAAHb0//8UAAAASAAAAEgAAAAAAAoBSAAAAAEAAAAEAAAAvPH//wgAAAAcAAAAEwAAAG51bGxhYmxlX3RpbWVzdGFtcHMABAAAAG5hbWUAAAAAAAAAAIr8//8AAAMAEwAAAG51bGxhYmxlX3RpbWVzdGFtcHMArvL//xQAAABAAAAAQAAAAAAAAApAAAAAAQAAAAQAAAAw8v//CAAAABQAAAAKAAAAdGltZXN0YW1wcwAABAAAAG5h

kylebrandt

Why data frames notes

Grafana's "wide tent" philosophy means we want Grafana to work with our users' data, and not make our users work to use Grafana. The data frame structure is a concept borrowed from data analysis like R and Pandas to allow us to do this.

Data frames allow users' data to be more accurately represented and with less effort.

• The data binding throughout Grafana's is now shared. Data moved between Grafana's backend plugins, Grafana server, and frontend is now less destructive. The means there is less data munging and more supported data types such as time, nullable types, various number types and other primitives.
• Time series data can be returned in different shapes that match those more commonly found.
• Time series and table data now exist in the same structure.
• Tables exist in the same structure as time series.

kylebrandt

Why data frames notes

Grafana's "wide tent" philosophy means we want Grafana to work with our users' data, and not make our users work to use Grafana. The data frame structure is a concept borrowed from data analysis like R and Pandas to allow us to do this.

Data frames allow users' data to be more accurately represented and with less effort.

• The data binding throughout Grafana's is now shared. Data moved between Grafana's backend plugins, Grafana server, and frontend is now less destructive. The means there is less data munging and more supported data types such as time, nullable types, various number types and other primitives.
• Time series data can be returned in different shapes that match those more commonly found.
• Time series and table data now exist in the same structure.
• Tables exist in the same structure as time series.

kylebrandt

=== RUN   TestLongToWide/sparse:_one_value,_two_factor
Name: long_to_wide_test
Dimensions: 4 Fields by 6 Rows
+-------------------------------+---------------------+---------------------+-------------------------+
| Name: Time                    | Name: Values Floats | Name: Animal Factor | Name: Location          |
| Labels:                       | Labels:             | Labels:             | Labels:                 |
| Type: []time.Time             | Type: []float64     | Type: []string      | Type: []string          |
+-------------------------------+---------------------+---------------------+-------------------------+
| 2020-01-02 03:04:00 +0000 UTC | 1                   | cat                 | Florida                 |
| 2020-01-02 03:04:00 +0000 UTC | 2                   | sloth               | Central & South America |

kylebrandt

Support Graphing of Long (a.k.a. Tall) and Wide Format Time Series Dataframes

Use Case

Currently our SQL plugins derive things from the special names of the columns. Instead we can use more conventional Long and Wide formats and derive the format from the schema. This will result in a more conventional SQL experience for users. The amount of code plugin authors need to create will be reduced.

Taking a MySQL Example:

SELECT

kylebrandt

$ ./run.sh -d 5m -v 150 -c annotations_by_tag_test -u http://localhost:3000

          /\      |‾‾|  /‾‾/  /‾/   
     /\  /  \     |  |_/  /  / /    
    /  \/    \    |      |  /  ‾‾\  
   /          \   |  |‾\  \ | (_) | 
  / __________ \  |__|  \__\ \___/ .io

  execution: local
     output: -

kylebrandt

package dataframe

//go:generate genny -in=$GOFILE -out=vector_vector.gen.go gen "gen=byte"

type genVectorVector [][]gen

func newgenVectorVector(n int) *genVectorVector {
	v := genVectorVector(make([][]gen, n))
	return &v
}

kylebrandt

// Code generated by protoc-gen-go. DO NOT EDIT.
// source: Flight.proto

package arrow_flight_protocol

import (
	fmt "fmt"

	proto "github.com/golang/protobuf/proto"

kylebrandt

diff --git a/pkg/dataframe/csv.go b/pkg/dataframe/csv.go
new file mode 100644
index 0000000000..0838a7c413
--- /dev/null
+++ b/pkg/dataframe/csv.go
@@ -0,0 +1,48 @@
+package dataframe
+
+import (
+	"encoding/csv"