Skip to content

Instantly share code, notes, and snippets.

@mattijn

mattijn/mark_array-progress-report.md

Last active June 27, 2024 10:54
Show Gist options
  • Save mattijn/e0700e48a0a54977809109f239e100e2 to your computer and use it in GitHub Desktop.
Save mattijn/e0700e48a0a54977809109f239e100e2 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
mark_array-progress-report.md

variants explored so far

  1. heatmap transform single array only
  2. heatmap transform with color scale
  3. heatmap transform with color scale and axis
  4. heatmap transform double array faceted with color scale and axis
  5. heatmap transform single array with non-zero x and y scale
  6. heatmap transform double array with non-zero x and y scale

Note: in the specs below, I've reduced the length of the grid values. In the accompanying Vega-Editor links are all values of the grids.

heatmap transform values only

Let's start with a basic example in numpy.

import numpy as np
import matplotlib.pyplot as plt
from skimage import data
from skimage.transform import rescale
import pyperclip

array = data.camera()

array_small = rescale(array, 0.245, anti_aliasing=False)
array_round = (array_small * 255).astype(np.uint8)

plt.imshow(array_round, cmap='gray')
print('shape', array_round.shape)

array_as_flatlist = array_round.flatten(order='C').tolist()  # row-major

print('head', array_as_flatlist[0:5])
pyperclip.copy(str(array_as_flatlist))

image

We can make it work using the heatmap transform in Vega, using the following specification (Vega-Editor):

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",  
  "data": [
    {
      "name": "GRID_ARRAY",
      "values": [
        {
          "width": 125,
          "height": 125,
          "values": [199, 200, 200, 198, 198, 118, 135, 161, 161, 140]
        }
      ]
    },
    {
      "name": "GRID_IMAGE",
      "source": "GRID_ARRAY",
      "transform": [{"type": "heatmap"}]
    }
  ],
  "marks": [
    {
      "type": "image",
      "from": {"data": "GRID_IMAGE"},
      "encode": {
        "update": {
          "x": {"value": 0},
          "y": {"value": 0},
          "image": {"field": "image"},
          "width": {"signal": "datum.width"},
          "height": {"signal": "datum.height"}
        }
      }
    }
  ]
}

The result looks like this:

image

It seems this is the image drawn with opacity levels only.

heatmap transform with color scale

Let's add a color scale (Vega-Editor):

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",  
  "data": [
    {
      "name": "GRID_ARRAY",
      "values": [
        {
          "width": 125,
          "height": 125,
          "values": [199, 200, 200, 198, 198, 130, 118, 135, 161, 161, 140]
        }
      ]
    },
    {
      "name": "GRID_IMAGE",
      "source": "GRID_ARRAY",
      "transform": [
        {
          "type": "heatmap",
          "color": {"expr": "scale('COLOR_SCALE', datum.$value / datum.$max)"},
          "opacity": 1
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "COLOR_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 1],
      "range": {"scheme": "viridis"}
    }
  ],
  "marks": [
    {
      "type": "image",
      "from": {"data": "GRID_IMAGE"},
      "encode": {
        "update": {
          "x": {"value": 0},
          "y": {"value": 0},
          "image": {"field": "image"},
          "width": {"signal": "datum.width"},
          "height": {"signal": "datum.height"}
        }
      }
    }
  ]
}

The result will look like this:

image

Using this approach, I also can reproduce the grayscale image like in python using plt.imshow().

By modifying the color scale as such (Vega-Editor):

{
  "name": "COLOR_SCALE",
  "type": "linear",
  "zero": true,
  "domain": [0, 1],
  "range": {"scheme": "greys"},
  "reverse": true
}

image

heatmap transform with color scale and axis

Next step is to add axis to the image. The Vega specification now looks as such (Vega-Editor):

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "width": 250,
  "height": 250,
  "data": [
    {
      "name": "GRID_ARRAY",
      "values": [
        {
          "width": 125,
          "height": 125,
          "values": [199, 200, 200, 198, 198, 118, 135, 161, 161, 140]
        }
      ]
    },
    {
      "name": "GRID_IMAGE",
      "source": "GRID_ARRAY",
      "transform": [
        {
          "type": "heatmap",
          "color": {"expr": "scale('COLOR_SCALE', datum.$value / datum.$max)"},
          "opacity": 1
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "COLOR_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 1],
      "range": {"scheme": "viridis"}
    },
    {
      "name": "X_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 125],
      "range": "width"
    },
    {
      "name": "Y_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 125],
      "range": "height"
    }
  ],
  "axes": [
    {
      "scale": "X_SCALE",
      "domain": false,
      "orient": "bottom",
      "tickCount": 5,
      "labelFlush": true
    },
    {
      "scale": "Y_SCALE",
      "domain": false,
      "orient": "left",
      "titlePadding": 5,
      "offset": 2
    }
  ],
  "marks": [
    {
      "type": "image",
      "from": {"data": "GRID_IMAGE"},
      "encode": {
        "update": {
          "x": {"value": 0},
          "y": {"value": 0},
          "image": {"field": "image"},
          "width": {"signal": "width"},
          "height": {"signal": "height"}
        }
      }
    }
  ]
}

image

So far so good.

heatmap transform double array faceted with color scale and axis

Are we able to facet grids, if we have for example two grids as input?

I've adapted my python code to prepare the data arrays:

import numpy as np
from skimage import data
from skimage import color
from skimage.transform import rescale
import pyperclip
import json

def array2vega(array):
    grid = {
        'height': array.shape[0],
        'width': array.shape[1],
        'values': array.flatten(order='C').tolist()  # row-major
    }
    return grid

array = data.camera()
array_small = rescale(array, 0.245, anti_aliasing=False)
array_round = np.round(array_small, 2)

grid0 = array2vega(array_round)
grid1 = array2vega(1 - array_round)
arrays = [{'grid':grid0, 'variant': 'A'}, {'grid':grid1, 'variant': 'B'}]

pyperclip.copy(json.dumps(arrays))

And modified the Vega specification. This now looks as such (Vega-Editor):

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "width": 250,
  "height": 250,
  "data": [
    {
      "name": "GRID_ARRAY",
      "values": [{"grid": {"width": 125, "height": 125, "values": [0.78, 0.78, 0.78, 0.78, 0.78, 0.46, 0.53, 0.63, 0.63, 0.55]}, "variant": "A"}, {"grid": {"width": 125, "height": 125, "values": [0.21999999999999997, 0.21999999999999997, 0.21999999999999997, 0.21999999999999997, 0.21999999999999997, 0.54, 0.47, 0.37, 0.37, 0.44999999999999996]}, "variant": "B"}]
    },
    {
      "name": "GRID_IMAGE",
      "source": "GRID_ARRAY",
      "transform": [
        {
          "type": "heatmap",
          "field": "grid",
          "color": {"expr": "scale('COLOR_SCALE', datum.$value / datum.$max)"},
          "opacity": 1
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "COLOR_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 1],
      "range": {"scheme": "viridis"}
    },
    {
      "name": "X_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 125],
      "range": "width"
    },
    {
      "name": "Y_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 125],
      "range": "height"
    }
  ],
  "axes": [
    {
      "scale": "Y_SCALE",
      "domain": false,
      "orient": "left",
      "offset": 2
    }
  ],
  "layout": {
    "columns": 2
  },
  "marks": [
    {
      "type": "group",
      "from": {
        "facet": {
          "name": "facet",
          "data": "GRID_IMAGE",
          "groupby": "variant"
        }
      },
      "title": {
        "text": {"signal": "parent.variant"}
      },
      "encode": {
        "update": {
          "width": {"signal": "width"},
          "height": {"signal": "height"}
        }
      },
      "axes": [
        {
          "scale": "X_SCALE",
          "domain": false,          
          "orient": "bottom"
        }
      ],
      "marks": [
        {
          "type": "image",
          "from": {"data": "facet"},
          "encode": {
            "update": {
              "x": {"value": 0},
              "y": {"value": 0},
              "image": {"field": "image"},
              "width": {"signal": "width"},
              "height": {"signal": "height"}
            }
          }
        }
      ]
    }
  ]
}

image

Not bad!

heatmap transform single array with non-zero x and y scale

This variant is still a bit difficult. The array is in unit degrees and goes on the x-axis from -180 to 180 longitude and on the y-axis from -81 to 87 latitude. The step-size is 1 degrees in both directions.

See Vega-Editor:

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "width": 360,
  "height": 168,
  "data": [
    {
      "name": "GRID_ARRAY",
      "values": [{
        "year":2016,
        "grid":{
          "x1_":-180,
          "x2_":180,
          "y1_":-81,
          "y2_":87,
          "height":168,
          "width":360,
          "values":[392,392,392,392,393,166,163,165,168,169]
        }
      }]
    },
    {
      "name": "GRID_IMAGE",
      "source": "GRID_ARRAY",
      "transform": [
        {
          "type": "heatmap",
          "field": "grid",
          "color": {"expr": "scale('COLOR_SCALE', datum.$value / datum.$max)"},
          "opacity": 1
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "COLOR_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 1],
      "range": {"scheme": "viridis"}
    },
    {
      "name": "X_SCALE",
      "type": "linear",
      "zero": false,
      "domain": [-180, 180],
      "range": "width"
    },
    {
      "name": "Y_SCALE",
      "type": "linear",
      "zero": false,
      "domain": [-81, 87],
      "range": "height"
    }
  ],
  "axes": [
    {
      "scale": "X_SCALE",
      "domain": false,
      "orient": "bottom"
    },
    {
      "scale": "Y_SCALE",
      "domain": false,
      "orient": "left",
      "titlePadding": 5,
      "offset": 2
    }
  ],
  "marks": [
    {
      "type": "image",
      "from": {"data": "GRID_IMAGE"},
      "encode": {
        "update": {
          "x": {"value": 0},
          "y": {"value": 0},
          "image": {"field": "image"},
          "width": {"signal": "datum.grid.width"},
          "height": {"signal": "datum.grid.height"}
        }
      }
    }
  ]
}

This results in:

image

Basically, for the grid only use the height and width to allocate the canvas size and iterate over the 1D array to colorize each pixel. For the X_SCALE and Y_SCALE we use the information of x1/x2 and y1/y2 (still manually). We use the "datum.grid.width" and "datum.grid.height" as signal for within the image mark encoding. Since the scales also need a width and height, the global width/height are currently still set to the same witdth and height of the grid.

But if I change the grid input object to:

"x1":-180,
"x2":180,
"y1":-81,
"y2":87,
"height":168,
"width":360,

(removing the appended _ from x1/x2/y1/y2) The result is this:

image

I've the feeling all negative values of our scales malfunction in the iterator within heatmap.js (here). But then it seems the drawn y-axis is reversed for the canvas iterator. If I add a "reverse":true to the scale Y_SCALE then it becomes more clear that only positive values are colorized in the canvas:

image

But then the latitude values on the y-axis does not match the input array.

heatmap transform double array with non-zero x and y scale

Lets make it a bit more complex. A facetted chart with non-zero x and y scales. Lets start with data preparation in python:

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import pyperclip

import urllib.request
import json
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import pyperclip

# define data
source = 'https://raw.githubusercontent.com/vega/vega-datasets/main/data/annual-precip.json'
with urllib.request.urlopen(source) as url:
    data = json.load(url)
values = data['values']
width = data['width']  # 360
height = data['height']  # 168
extent = [-180, 180, -81, 87]  # xmin, xmax, ymin, ymax

# prepare array and plot
array = np.array(values).reshape(height, width)
plt.imshow(array, extent=extent)

image

def array2vega(array, extent):
    grid = {
        'extent': extent,
        'height': array.shape[0],
        'width': array.shape[1],
        'values': array.flatten(order='C').tolist()  # row-major
    }
    return grid

grid0 = array2vega(array, extent)
grid1 = array2vega(1 - array, extent)
arrays = [{'grid': grid0, 'variant': 'A'}, {'grid': grid1, 'variant': 'B'}]
df = pd.DataFrame.from_dict(arrays)

# copy and display
pyperclip.copy(df.to_json(orient='records'))
df

image

When prepararing a vega chart for this as such, See Vega-Editor:

{
  "$schema": "https://vega.github.io/schema/vega/v5.json",
  "width": 250,
  "height": 250,
  "data": [
    {
      "name": "GRID_ARRAY",
      "values": [
        {
          "grid": {
            "extent": [-180, 180, -81, 87],
            "height": 168,
            "width": 360,
            "values": [392, 392, 392, 169, 187, 196]
          },
          "variant": "A"
        },
        {
          "grid": {
            "extent": [-180, 180, -81, 87],
            "height": 168,
            "width": 360,
            "values": [-391, -391, -391, -164, -167, -168]
          },
          "variant": "B"
        }
      ]
    },
    {
      "name": "GRID_IMAGE",
      "source": "GRID_ARRAY",
      "transform": [
        {
          "type": "heatmap",
          "field": "grid",
          "color": {"expr": "scale('COLOR_SCALE', datum.$value / datum.$max)"},
          "opacity": 1
        }
      ]
    }
  ],
  "scales": [
    {
      "name": "COLOR_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [0, 1],
      "range": {"scheme": "viridis"}
    },
    {
      "name": "X_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [-180, 180],
      "range": "width"
    },
    {
      "name": "Y_SCALE",
      "type": "linear",
      "zero": true,
      "domain": [-81, 87],
      "range": "height"
    }
  ],
  "axes": [
    {"scale": "Y_SCALE", "domain": false, "orient": "left", "offset": 2}
  ],
  "layout": {"columns": 2},
  "marks": [
    {
      "type": "group",
      "from": {
        "facet": {"name": "facet", "data": "GRID_IMAGE", "groupby": "variant"}
      },
      "title": {"text": {"signal": "parent.variant"}},
      "encode": {
        "update": {"width": {"signal": "width"}, "height": {"signal": "height"}}
      },
      "axes": [{"scale": "X_SCALE", "domain": false, "orient": "bottom"}],
      "marks": [
        {
          "type": "image",
          "from": {"data": "facet"},
          "encode": {
            "update": {
              "x": {"value": 0},
              "y": {"value": 0},
              "image": {"field": "image"},
              "width": {"signal": "datum.grid.width"},
              "height": {"signal": "datum.grid.height"}
            }
          }
        }
      ]
    }
  ]
}

image

Two issues become clear from this:

  • We see the interference of a global-defined width and height and the array-defined grid.width and grid.height.
  • Another issue that becomes apparent is that currently the color scale is not applied independent.
@melonora
Copy link

melonora commented Jun 27, 2024 •
edited
Loading

I looked also more in blending mode, so the main problem seems to be that the default blending mode is not blending in isolation of the background and thus the background and image get blended together. Setting isolation to isolate which is supported in css is not supported in vega. For more information see: https://vega.github.io/vega/docs/marks/image/ and https://developer.mozilla.org/en-US/docs/Web/CSS/mix-blend-mode#examples

@melonora
Copy link

melonora commented Jun 27, 2024 •
edited
Loading

example here

{
  "$schema": "https://vega.github.io/schema/vega/v5.json", 
  "signals": [
    {
      "name": "blendMode",
      "value": "destination-out",
      "bind": {
        "input": "select",
        "options": [
          "normal",
          "multiply",
          "screen",
          "overlay",
          "darken",
          "lighten",
          "color-dodge",
          "color-burn",
          "hard-light",
          "soft-light",
          "difference",
          "exclusion",
          "hue",
          "saturation",
          "color",
          "luminosity",
          "destination-out",
          "destination-in"
        ]
      }
    }
  ], 
  "data": [
    {
      "name": "GRID_ARRAY",
      "values": [
        {
          "width": 125,
          "height": 125,
          "values": [199, 200, 200, 198, 198, 198, 198, 198, 198, 198, 1198, 199, 198, 199, 199, 198, 198, 198, 198, 197, 197, 198]
        }
      ]
    },
    {
      "name": "GRID_IMAGE",
      "source": "GRID_ARRAY",
      "transform": [{"type": "heatmap"}]
    }
  ],
  "marks": [
    {
      "type": "image",
      "from": {"data": "GRID_IMAGE"},
      "encode": {
        "update": {
          "x": {"value": 0},
          "y": {"value": 0},
          "image": {"field": "image"},
          "width": {"signal": "datum.width"},
          "height": {"signal": "datum.height"},
          "blend": {"signal":"blendMode"},
          "isolation": {"value": "isolate"}
        }
      }
    }
  ]
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment