MetaMaps formalized spec

METAMAPS.md

Overview

There are several changes required in order for MetaMaps to be accepted into the MetaGame codebase. While the most important tasks involve QA and formatting. This is also a good time to augment the existing implementation with changes we are planning in the future.

Requirements

Hasura + Table Structure

The Maps table needs to be regenerated. It would be a good idea to reset all diffs in Hasura and update it to use Player.id instead of an Ethereum address. The address that is pulled from 3box should be used to find the player's id on MetaGame.

Tasks

• Reset all diffs in Hasura.

• Have maps associated with Player.id instead of an Ethereum address.

• Map data should be json instead of text.

React/Redux Refactor

The current implementation of Redux needs to be updated to: https://redux-toolkit.js.org/. There are general refactors required such as needing more efficient mappings for component renders. This would be pretty important for larger map datasets.

Tasks

• Refactor to redux toolkit.

• Improve component renderings such as not have components inside redux stores.

Updating Component Libraries

We should be utilizing the @metafam/ds component libraries. We may want to also implement new context specific components for example the shapes and context menus used in metamaps.

Tasks

• Update and migrate components to @metafam/ds.

• Copy theme elements from other MetaGame's designs. Can use Figma as a reference: https://www.figma.com/file/RWVfMGvXDAX74fQexze8uO/Meta-Game-Copy-Copy?node-id=58%3A2

• Improve overall code structure to jsx/tsx.

Z Component

As per comments from @heterotic, we should have a z component. Where one can toggle the z index of the map. This would be a +/- button with an input at the top of the map.

Tasks

• Create a z layer for maps.

• Maps can be toggled with a -/+ button at the top with a number input.

• Map renders are based on the z layer.

So, I guess this goes back to the use case... but my understanding is that we are trying to build a data visualizer. In that sense, it needs to extract Schemas from the data sources it is configured to connect to, and then allow the user to input how the schema should be rendered... this requires an intermediate layer to the UX which allows the schema objects to be displayed, interacted with, and ultimately 'pulled into' the map area for the renderer to plot... It should be possible to customize the display of elements, but that customization should effect all elements of a similar typing...

I will address the current contents of @metamaps/alpha for these edits...

The loading of the data: String! in most of the Map tables in the proposed hasura should only be in the parser, most data should remain in cache as json. Time travel is something that we can do, but the resource considerations are prohibitive, and we either need that BZZ balancer, or some other distributed low-cost data store for a fully decentralized model, to create records of data streams. The system only needs to be able to save the schema variables that define the objects in the map renderer. We should give Players the ability to export the data along with the view, but the data file should be a local backup created from the current json in the renderer.

Each view in the map is defined by the data sources it is designed to load, the schemas extracted from that data source on load, and the design elements assigned to instances of the schemas from that data source as defined by the Player designing the view. The parsing engine driving the map needs three basic classes. The 1st class is a class that loads the data connections and data connection types (graphQL, *sql, API, etc) as well as any access protocol data that is required to establish the connection. The user should be required to complete this information the first time a new data source is loaded, but the default data source of the MetaGame hasura could be added as a connection available to a default view that we added as the initial row to the Map connection table(s). The connection data for a particular view could be stored in a user table as indicated in the Map tables so far, to manage permissions and the data source connection to load the initial schemas for the second class in the map parsing engine. The last part of the first class is a function to query the datasources based on their data type to extract the schemas from the data source. Once the schema information has been extracted for each data source in a view, it should first be added to cache in a JSON string, for use in the 2nd class in the map parsing engine.

The 2nd class in the map parsing engine starts with a function that takes the schema stored in the JSON string for a particular data source and uses that schema to construct a query as indicated by the data source type to extract the necessary information for use in instancing the view design tools of the 3rd class. My experience is that table definitions are often sparse and incomplete, even things like primary keys can be neglected in database design, and foreign keys are often unidentifiable without testing values across tables, although sometimes column names give some indication of a foreign key, even that cannot be easily trusted in any concrete sense... in my opinion, it is better to obtain the following items to further inform a user of the likelihood of the availability for a join (and also a method of determining normalization constraints for bad table design) for each 'table/(array)' in a data source, we need at least: a tableName, the number of potential fields/columns, and the number of rows/keyValueSets, in addition, for each 'field/(column)' in each 'table/(array)' of a data source we need, at a minimum: a Count of NotNulls (or nulls, whichever is easier based on the source), a Count of Unique values, and any field specific schema data available from the data source schema like data type, etc... all of this information should be loaded into a final JSON string and sent to the 3rd class, but storing it in a Map table in hasura for each instance of a view would not be a bad idea, as these queries are not likely to run in a user friendly amount of time, and could potentially take several minutes (or even longer) to resolve in real compute time. This class should be re-run for a map view whenever there is an update supplied by the user to the view which adds a new data source, or as a refresh option selectable by the user, and/or run by default to check for changes to the existing schema prior to any re-rendering of the map after restoring a saved view to check for renderer breaking changes to the schema.

The 3rd class in the parsing engine is a visualizer for the schema data provided in the JSON string from the 2nd class. This class also contains any tools needed for visually constructing the queries that produce the data to be displayed in the view of the metaMaps rendering engine, and runs the queries prior to sending the results back to the map for rendering <3 Basically, we need something like this: https://apis.guru/graphql-voyager/ <3 https://github.com/APIs-guru/graphql-voyager ....perhaps showing each table as a stripped down rectangle/square with the basic table/array data from the stored JSON (tableName, # col, # row) , but allowing the full field data to be shown/hidden on a click behavior. Also, in addition to displaying the schema information visually, the class should be able to allow a user to select elements (nodes and edges) to be displayed in the map and construct the queries that will poll data from the datasources to be displayed by the rendering engine. Whenever the map is running a view and has passed the check in the 2nd class, this visualizer should be accessible as a pull out drawer from the map. Whenever an element is selected from the drawer and dragged over to the view of the map, this class should check the current elements displayed in the map as edges or nodes and add the newly selected element to the query so that data from the new element can be added in the map rendering engine. Finally, there should be a way to set whether a filter is required for a particular element when loaded, and the type of filter that is to be added (type, multitype, specific value, multi-value, value range, etc...). The metadata used by the tools and/or the current query to poll data from the datasources for the map rendering engine should be saved both in a JSON string, and optionally as part of the saved view in the Map table in hasura.

Since it looks like dysbulic has got a lot of this going in the most recent PR, I would like to propose that the next steps we take in developing the metamaps involve publishing this Introspection Query to the 1st Class as a method to run when the default user loads the default metamap and selects the defualt system, so that it can use the default connection from the default user to run the query against our local data source in the 2nd Class:
query IntrospectionQuery {
__schema {

    queryType { name }
    mutationType { name }
    subscriptionType { name }
    types {
      ...FullType
    }
    directives {
      name
      description
      
      locations
      args {
        ...InputValue
      }
    }
  }
}

fragment FullType on __Type {
  kind
  name
  description
  fields(includeDeprecated: true) {
    name
    description
    args {
      ...InputValue
    }
    type {
      ...TypeRef
    }
    isDeprecated
    deprecationReason
  }
  inputFields {
    ...InputValue
  }
  interfaces {
    ...TypeRef
  }
  enumValues(includeDeprecated: true) {
    name
    description
    isDeprecated
    deprecationReason
  }
  possibleTypes {
    ...TypeRef
  }
}

fragment InputValue on __InputValue {
  name
  description
  type { ...TypeRef }
  defaultValue
}

fragment TypeRef on __Type {
  kind
  name
  ofType {
    kind
    name
    ofType {
      kind
      name
      ofType {
        kind
        name
        ofType {
          kind
          name
          ofType {
            kind
            name
            ofType {
              kind
              name
              ofType {
                kind
                name
              }
            }
          }
        }
      }
    }
  }
}

The 2nd Class should run the query passed by the 1st class, and the results should be published to a JSON string, and then pushed to the metamap in the 3rd class to be displayed in a simplified manner similar to the image that was rendered in https://apis.guru/graphql-voyager/ from our query result shown here:

However, instead of all of the additional trappings that are presented in the Voyager, we want it to look like the MetaGame, use the design elements from our own repo, and instead of displaying the full schema logic, just show the links between tables as single edges, and show the tables as labeled square.tsx objects, which on-click load a popup.tsx that contains the schema details for that table... Finally, we do not need to load the object called 'query_root' here, which I believe is an abstraction based on all of the actual tables spelled out in the diagram... this will further reduce the number of edges displayed in the initial load of the schema data, and clean things up a bit visually to leave space
for the query builder...

Once we get that far, we can talk more about what additional tools we will need to create a 'component builder' and/or 'query builder' for the metamaps <3