custom providers and dbs draft.md

Custom Functions and modules in Eve

Note: We plan to do a thorough documentation pass in the coming weeks to clean up and document the codebase. This document is a very rough draft intended to help intrepid adventurers navigate the jungle until then.

Terms Sheet

• Cardinality - The number of results a thing would return.
• Intermediate - A value which is necessary to derive the final result, but not part of the result.
• Referential Transparency - The property of always returning exactly the same output given an input.

Evaluating Eve

To understand the API for creating Databases (Eve modules) and Providers (Eve functions), you need to understand a little about our evaluation strategy. In many ways, it is more like a database engine than a conventional language interpreter. The compiler breaks searches down into a list of relational joins called Scans. Scans represent index lookups, constraints, and even functions. To Eve, a function is just a join of its arguments to its outputs. These joins are computed using an algorithm called Generic Join, which looks at the set of Scans and tries to solve each individual variable as efficiently as possible. It does so by making ordering choices based on the cardinality of each variable, which allows it to avoid large intermediate sets and alleviates the need for the advanced query planners found in other relational databases.

Providers

In order to let Generic Join work its magic, our function providers (henceforth, providers) need to do a little bit more than just evaluate a function. A provider must be able to:

1. Return an output for an input.
2. Calculate the number of outputs for an input.
3. Test whether an input would have produced a given output

A provider must also have the following properties or very bad things will happen:

1. It must be synchronous
2. It must have a finite cardinality
3. It must be referentially transparent.

If any of these are untrue, there's no guarantee what result you will get. The executor schedules Scans assuming they can be run whenever it thinks is best. As such if any of these aren't true, Eve may successfully execute a block but end up with the wrong answer.

Implementation

A Provider is created by subclassing the Constraint class. It must implement all of the following abstract methods:

class MyProvider extends Constraint {
  // This can optionally be implemented to do any initial setup that may be required.
  // Just pass the arguments through in a call to super
  constructor(id: string, args: any[], returns: any[]);

  // Maps input attributes to array indexes in `this.args`.
  static AttributeMapping:{[attribute:string]: number};
  // Maps output attributes to the array indexes in `this.returns`.
  static ReturnMapping:{[attribute:string]: number};

  // Given a "prefix" - the variables that have already been solved - this
  // returns a `resolved` object with the `this.args` and `this.returns` arrays to their current values.
  resolve(prefix):{args: any[], returns?: any[]};

  // Given a variable to solve for and a prefix of solved variables, return
  // a proposal for that variable.
  abstract getProposal(tripleIndex: TripleIndex, proposed: Variable, prefix: any) : Proposal | undefined;

  // Resolve a proposal into values for a variable.
  abstract resolveProposal(proposal: Proposal, prefix: any[]) : any[];

  // Test if a prefix adheres to the constraint being implemented (e.g., 1 + 1 = 2).
  abstract test(prefix: any) : boolean;
}

The resolve function maps the "prefix" of solved variables into the args of the provider. As such, to retrieve the inputs for your function you do this:

  // Let's say you have the following Attribute/ReturnMappings:
  static AttributeMapping = {
    "value": 0,
    "to": 1,
  }
  static ReturnMapping = {
    "converted": 0,
  }

  // You can retrieve the value for the "to" attribute with:
  let {args, returns} = this.resolve(prefix);
  let to = args[this.AttributeMapping["to"]];

You'll notice we can retrieve the return values here as well. While that may sound strange, with Eve's unordered semantics, it's possible that another provider got first dibs on setting the output.

The three abstract methods that need to be implemented are:

The getProposal method, which determines what the cardinality of the result will be for the given inputs. The provider then updates and returns its proposalObject. If the provider would fail or otherwise have no result, it should assign a cardinality of zero.

The resolveProposal method takes a proposal generated from getProposal and returns an array of outputs. This is where the function gets evaluated.

The test method allows a proposal to accept or reject a proposed output for the given inputs. The returns attribute on object returned by this.resolve(prefix) is guaranteed to be available here. If the returns array contains only valid outputs for each of its attributes, the test returns true, otherwise false. This method is also used for providers which only filter rather than introducing new variables (e.g., >).

Let's walk through an existing provider as an example:

// Urlencode a string
class Urlencode extends Constraint {
  static AttributeMapping = {
    "text": 0
  };
  static ReturnMapping = {
    "value": 0
  };

  // To resolve a proposal, we urlencode a text
  resolveProposal(proposal, prefix) {
    let {args, returns} = this.resolve(prefix);
    let value = args[this.AttributeMapping["text"]];
    let converted;
    converted = encodeURIComponent(value);
    return [converted];
  }

  test(prefix) {
    let {args, returns} = this.resolve(prefix);
    let value = args[this.AttributeMapping["text"]];

    let converted = encodeURIComponent(value);

    return converted === returns[this.ReturnMapping["value"]];
  }

  // Urlencode always returns cardinality 1
  getProposal(tripleIndex, proposed, prefix) {
    let proposal = this.proposalObject;
    proposal.cardinality = 1;
    proposal.providing = proposed;
    return proposal;
  }
}

// ...

providers.provide("urlencode", Urlencode);

This provider implements the urlencode function. It defines a single input attribute, text, and an output attribute value.

• The resolveProposal method retrieves the input as above, feeds it through the native encodeUriComponent function, and returns its value in the ReturnMapping["value"] slot (0).
• The test function also runs encodeUriComponent on its input, but compares it to the already proposed value in returns. In some cases it's not necessary to actually evaluate the function to test validity. E.g., square root can't possibly work on a string, so the specific correct value doesn't matter. However, evaluating the function and comparing the result will always be correct and is a good place to start.
• The getProposal function here is pretty boring. Since it is valid to urlencode any eve value (including numbers and booleans), the provider has a constant cardinality of 1. In the case where a value may be invalid (e.g., requiring a number but receiving a string), the getProposal function should be the one to catch this. In that case, it should return a cardinality of zero. In the future, we plan to also have a channel available for providers to warn users about bad inputs, but this does not yet exist.

Finally, the Provider is registered in the global providers registry, which makes it available for documents to use.

Databases

Databases are Eve's version of modules. They include an Eve document to run, which by convention searches in the Database of the same name. E.g., the editor DB imports a document named editor.eve whose blocks look at the @editor Database for input. All of this is technically configurable, but its good practice to be as obvious as possible.

In the future, it will be possible to bundle a set of native providers into your DB, but this currently isn't really possible for third parties.

NOTE: Databases are in need of a refactor for extensibility. Creating a custom Database is currently a very invasive process. For this reason we are unlikely to accept pull requests for new Databases until the refactor happens except in exceptional cases.

Implementation

A custom Database is created by subclassing the Database class.

export class MyDB extends Database {
  blocks: Block[];

  // Used to build the document this Database contains, if any.
  // In the future, there may be a more elegant mechanism for both this and
  // supplying providers from within the DB.
  constructor();

  // Invoked when a new evaluation using this DB is opened.
  register(evaluation: Evaluation);

  // Invoked when an evaluation using this DB is closed.
  unregister(evaluation: Evaluation);

  // Invoked when an evaluation using this DB has completed a change.
  // `changes` contains the changing state.
  onFixpoint(currentEvaluation: Evaluation, changes: Changes);
}

Unless otherwise specified, be sure to invoke the super's method when overriding a method.

Let's look at some examples. For a simple, "pure" Eve DB, let's look at @editor in src/runtime/databases/browserSession.ts.

export class BrowserEditorDatabase extends Database {
  constructor() {
    super();
    let source = eveSource.get("/examples/editor.eve");
    if(source) {
      let {results, errors} = parser.parseDoc(source, "editor");
      if(errors && errors.length) console.error("Editor DB Errors", errors);
      let {blocks, errors: buildErrors} = builder.buildDoc(results);
      if(buildErrors && buildErrors.length) console.error("Editor DB Errors", buildErrors);
      this.blocks = blocks;
    }
  }
}

We only need to override the constructor here, and the entirety of that is some boilerplate for retrieving and building the source for the document editor.eve which powers @editor. We unfortunately don't have a nice way to send out errors here yet, so we console.error any unexpected errors to at provide some warning of foul play. Finally, we attach the built document's blocks to the DB's blocks attribute. Running evaluations using this DB will take these into account automatically.

Next, let's look at a Database that provides native functionality. @http provides a simple interface for sending and receiving JSON requests in src/runtime/databases/http.ts.

export class HttpDatabase extends Database {

  sendRequest(evaluation, requestId, request) {
    var oReq = new XMLHttpRequest();

     // ...
  }

  onFixpoint(evaluation: Evaluation, changes: Changes) {
    let name = evaluation.databaseToName(this);
    let result = changes.result({[name]: true});
    let handled = {};
    let index = this.index;
    let actions = [];

    // @NOTE: The API for matching records from TS is currently very low level
    // and needs some love.
    for(let insert of result.insert) {
      let [e,a,v] = insert;
      if(!handled[e]) {
        handled[e] = true;
        if(index.lookup(e,"tag", "request") && !index.lookup(e, "tag", "sent")) {
          let request = index.asObject(e);
          if(request.url) {
            actions.push(new InsertAction("http|sender", e, "tag", "sent", undefined, [name]));
            this.sendRequest(evaluation, e, request);
          }
        }
      }
    }
    if(actions.length) {
      setTimeout(() => {
        // console.log("actions", actions);
        evaluation.executeActions(actions);
      })
    }
  }

The important bit here is an override on onFixpoint, which looks for new records in @http that match the signature for a new request. For each of these it adds an InsertAction to mark the request as sent and then actually does so. At the end, we asynchronously execute any actions we generated to avoid changing the current state until after all Databases have used it. This is vitally important to upholding Eve's guarantee that changes are always executed in a series of timesteps, and without it very bad things could happen.

Finally, we need to tell the evaluation to use our new database. In the future this will be detected by searching a DB registry (much like the provider registry) for DBs used in an evaluation's blocks, but for now we hardwire it by adding an instance of the class in src/runtime/runtimeClient.ts in the extraDBs argument of the RuntimeClient constructor.