benjie/RELEASE_NOTES_BETA22.md

## RELEASE_NOTES_BETA22.md

      
    Raw
  

              RELEASE_NOTES_BETA22.md
            
          
    Release Notes 2024-05-09

This is a huge release, folks! Our last release was 12th Feb, so this covers
the 3 month period since then. I finally caught up enough to attack some of the
bigger issues in Grafast (specifically the
"early exit" and
"global dependencies"
issues discussed at the first Grafast WG); this has required some breaking
changes to the API but as always I've kept these to an absolute minimum;
migration steps are outlined below.
Crowd-funded open source software

We ask all individuals and businesses that use our software to support its
ongoing maintenance and development via sponsorship. Sponsorship comes with many
perks, including full release notes like these for each release of the Crystal
software. (Changelogs are
openly available in the source repository,
but release notes are a maintainer editorial that gathers together the main
things you need to know from the changelogs, providing highlights, a narrative,
and helping to reduce your workload.)
Since this release is so significant and includes breaking changes, we have
decided to make its release notes public. Please consider
sponsoring us to help support our open source
endeavors, and of course to get access to release notes like these for all
Crystal releases via the sponsors-only 🔮 channel
on our Discord - we have over 50 previous
releases documented in there already!
Breaking changes

A quick migration guide to how to deal with the breaking changes in this
release. Note that if you don't write your own plan resolvers, and don't write
your own step classes, then you can skip to the next section.
Plan resolvers

If you use connection() in a plan resolver, the signature has changed; the
second and third arguments now should be detailed via an options object:
-return connection($list, nodePlan, cursorPlan);
+return connection($list, { nodePlan, cursorPlan });
If you only pass the first argument then no action is necessary.
Step classes

If you have implemented your own step classes, you need to make the following
changes:
ExecutableStep.execute signature has changed

The signature of ExecutableStep.execute has changed, we now pass a "details
object" rather than multiple arguments, and the values arg is now a list of
ExecutionValues (see below) rather than a list of lists as before.
ExecutionValues come in two varieties: batch and unary, where a unary value only
ever represents a single value no matter how deep in the plan you are.
If you don't care about the details and performance, this change to each of your
execute methods will get you up and running again:
- async execute(count: number, values: any[][], extra: ExecutionExtra) {
+ async execute({ count, values: newValues, extra }: ExecutionDetails) {
+   const values = newValues.map((dep) =>
+     dep.isBatch ? dep.entries : new Array(count).fill(dep.value)
+   );
    // REST OF YOUR FUNCTION HERE
  }
For more details, see: https://err.red/gev2
this.addDependency(step, true) no longer supported

Instead, use this.addDependency({ step, skipDeduplication: true }) (this is
equivalent to previous behavior).
Note: this.addDependency(step) is unaffected.
New indexMap helper

You don't have to make this change, but it will make your life easier! Where
previously your execute function built an array via a for loop:
async execute(count, values) {
  const [allA, allB] = values;
  const stuff = await doTheThing(allA, allB);

  const results = [];
  for (let i = 0; i &lt; count; i++) {
    const a = allA.at(i);

    const b = allB.at(i);
    results.push(stuff.getResultFor(a, b));
  }
  return results;
}
You can now instead use the indexMap helper:
async execute({ indexMap, values }) {
  const [allA, allB] = values;
  const stuff = await doTheThing(allA, allB);

  return indexMap(i => {
    const a = allA.at(i);
    const b = allB.at(i);
    return stuff.getResultFor(a, b);
  });
}
Saves you a couple lines, and helps ensure that the result at a given index is
related to the input at the given index.
Returning errors no longer works

You must throw an error, return a rejected promise, or you can
return flagError(new Error("...")). Just returning an error will result in the
error being treated as a value (not a thrown error).
 // Turns even indexes into errors
 execute({ count }) {
   return indexMap(i => {
     if (i % 2 === 1) {
       return i;
     } else {
-      return new Error("No even numbers!");
+      return flagError(new Error("No even numbers!"));
     }
   });
 }
Notable changes

Three months of work, the Grafast changes, albeit the most significant in terms
of thinking, planning, writing, and re-rewriting, were not the only things we
changed! Many of these changes overlap multiple projects, but to roughly break
them down:
Grafast


Unary steps introduced to solve the "global dependencies" problem.
Complete overhaul of Grafast's internals relating to step results, error
handling, null handling, output plans, and essentially the whole of execution
to address the "early exit" problem.
inhibitOnNull(), assertNotNull() and trap() steps added (see "Nitty
gritty")
loadOne() and loadMany() can leverage the new unary dependencies to make
life easier if you want to accept an input argument (or arguments) into a
loadOne/loadMany call (e.g. to add a filter or limit to all queries) - no
longer a need to group by values in this case. To use this, add another step
(a unary step) as the second argument to the function. If you need to load
multiple, use list() or object() to turn them into a single unary.
connection() step can now have edgeDataPlan specified as part of the
configuration object, enabling edges to represent data not just about the node
itself but about its relationship to the connection
makeGrafastSchema completely reworked

Fixes bugs with enums
Enables Interface Object inputPlan


execute() and subscribe() call signatures now deprecate additional
arguments, preferring them to be merged into the original arguments.
envelop peerDependency upgraded to V5; drops support for Node versions we
don't support anyway.

Grafserv


Grafserv internals overhauled.
Now compatible with Envelop via GrafservEnvelopPreset.
Possible to add validation rules (e.g. @graphile/depth-limit) to Grafserv
(see example at bottom).
Fixes the bug where Grafserv would sometimes keep Node alive for 15 seconds
after it shut down (particularly in tests) due to not correctly releasing a
timeout.

Graphile Export


Massive Graphile Export overhaul.
Properties on functions are now automatically exported (e.g.
fn.isSyncAndSafe = true).
Input object inputPlan is now exported in typeDefs mode.
Fixes export of scalars in typeDefs mode.
Expose exportValueAsString so that values (e.g. the registry) can be
exported, not just schemas.
Automatic inference of enableDeferStream via directive detection.

PostGraphile


Better handling of Relay-style Node IDs, particularly when the expected type
is known (thanks to Grafast's inhibit features).
PostGraphile now runs integration tests against the exported schema; and we've
thus fixed a huge number of issues related to schema exports.
More PgSelectSteps are able to inline themselves into their parents thanks to
generic (rather than specific) checks against dependencies resulting in fewer
queries to the database (and significant performance fixes in some cases).
PgSelectStep::clone() is no longer @internal; you can use it to build a
PgSelectStep in aggregate mode via: pgResource.find().clone('aggregate').
Authorization check on PgResource (selectAuth()) is now able to call other
steps, e.g. allows reading from context().
When using orderByAscDesc() in makeAddPgTableOrderByPlugin, nullable is
now accepted as an option, and will default to true if nulls was set as an
option. Fixes pagination over nullables with custom orders.
Fixes sending record types to/from the database, particularly when they have
custom casting and to/fromPg implementations; this fixes a bug with certain
computed column function arguments.
constant() now has .get(key) and .at(idx) methods.
New condition() step representing mathematical and logical comparisons (e.g.
condition(">", $a, $b) represents "a > b")

Minor changes


ExecutableStep::canAddDependency($a) added to determine if adding $a as a
dependency would be allowed or not.
No longer outputs non-async warning in production.
No longer warns about loadOne()/loadMany() callback being non-async.
GRAPHILE_ENV now supports test, and doesn't output warning in test mode.
te.debug helper added to help debug tamedevil expressions
pg-sql2 gains the ability to automatically cast certain values to SQL if
they implement the SQLable interface (like .toString() but for SQL) - in
particular this means you can use a PgSelect step directly in sql
expressions rather than having to extract it's .alias property… So there's 6
characters you're saving each time: you're welcome.
DEBUG="grafast:OutputPlan:verbose" can be used to debug the output plan and
buckets.
Inflector replacements no longer need to pass this explicitly via
previous.call(this, arg1, arg2); they can now just call
previous(arg1, arg2).
Better inflector typings.
pg-introspection now exports PgEntity.
Better detection of invalid presets/plugins, to try and reduce Benjie's
ESM-related support burden.
Error messages output during planning now reflect list positions (e.g. if an
error occurred whilst planning the list item rather than the field itself).
Some identifiers in the SQL are now more descriptive (i.e. not just t
anymore!).
Doesn't run EXPLAIN on error when DEBUG="@dataplan/pg:PgExecutor:explain"
is set.
You can now configure how many documents Grafserv will cache the parsing of
via preset.grafserv.parseAndValidateCacheSize (default 500).

Nitty-gritty

If you're interested in exactly what's changed and why, here's some
explanations! (You may want to read over the
"early exit" and
"global dependencies"
issues too, if you really want to geek out!)
Grafast

Execution values

Grafast previously modeled step results as simple lists of values. When it came
time to run the next step, it would look at the lists representing its
dependencies, filter out the errors (reducing batch size) and then send them
through to the step's execute() method. This had a number of issues:

Errors were always treated as "thrown" errors, they could never be treated as
values (e.g. if you wanted to represent details about them in a mutation
payload).
Nulls were always simple null values, you could never tell a step not to
execute if it received null, instead each step had to decide what to do with
nulls.
Joining these together: there was no way to reliably "ignore" an invalid Node
ID. If you want to filter Posts by an Author ID, but the user feeds in a
Product ID, the system would just treat the ID as if it were null. This meant
that feeding in an invalid ID would be equivalent to asking for the posts with
no author (anonymous posts), which is not the same thing. Another option would
have been to throw an error, but really what we want is to return an empty
list of posts without consulting the database. There was no way to do this.
Another issue was that every time we went through a layer boundary we'd have
to "multiply up" values so that they matched the batch size, even if we knew
that there was only one of certain values (e.g. input arguments, context()
and derivatives, etc). This means that certain types of step would then have
to group by the different values in this "multiplied up" list to see what
unique values it contained, e.g. if you wanted to add a LIMIT to an SQL
query based on an input argument, you'd have to group by all the limit
values ultimately to determine that there was only one. It's inefficient and
annoying.
When we didn't want to execute things any more because the polymorphism didn't
line up, we'd use an error (POLY_SKIPPED) to represent this, because it was
the only way of preventing more execution. Super ugly workaround!
Errors would all be wrapped in GrafastError to make detection faster, since
instanceof is slow.

This latest release overhauls this entire system, and step results are now
stored in a structure called an ExecutionValue. There are two forms of
execution value: batch values, and unary values. Batch values contain an array
of results, as before. Unary values are a solution to the "global dependencies"
problem - they never get "multiplied up", and when a step adds a dependency on
another step it may choose to require that it is a unary step (a step that would
result in a unary value) via this.addUnaryDependency($step). This fixes our
issue with LIMIT above since we can require there's exactly one value, and
just write that into the SQL at execution time.
You can use the .at(idx) method on both batch and unary values to get the
value at the specified index if the fact of whether they are unary or not is
unimportant to you (for unary values, .at(idx) will always return the same
value, no matter what the value of idx is).
In addition to their two forms, ExecutionValues now store flags (a bit map)
that indicate special properties of the values. Mostly this is an internal
feature that makes it faster for us to filter errors/polymorphism/etc since we
can use bitwise operations rather than instanceof/property checks, but it can
also be useful in user space. In particular it gives you the ability to "inhibit
on null", and to "trap" inhibits and errors.
Inhibit on null: in the Author ID scenario outlined in the third bullet above,
we don't want to run any SQL code if a Product ID is received instead of an
Author ID, but we also don't want to throw an error. What we can do is to turn
the Product ID null into an "inhibit null" so steps depending on it will
automatically skip execution (effectively an "inhibit null" is a "super" null —
it is null, and anything that depends on it automatically becomes null without
executing). We can combine this with some logic to say to inhibit it only if the
id itself is non-null; so $id = "Author:7" would be coerced as expected and
we'd look for all posts by author 7, $id = null would be coerced as expected
and we'd look for all anonymous posts (posts with a null author_id), and
finally $id = "Product:12" would be coerced into an "inhibit null", preventing
fetches of posts at all).
Trapping: when an inhibit or error occurs, you might want to handle it
specially. In the case described above where an invalid ID "Product:12" is
used, we probably don't want to return null for the posts, instead we probably
want to return an empty array []. We can use trap() to do this, it lets us
indicate the flags on values that we will accept, in this case we'd tell the
system that we don't care if the value is inhibited, we'd like to receive it
into our execute() method anyway (and then we decide what to do with it).
Internally, errors no longer need to be wrapped in GrafastError — the "error
bit" being set in the flags is sufficient for us to know a value represents an
error. This means that the value itself and whether it's treated as an error or
not are disconnected: anything can be an error (throw "sandwich" would mark
the string "sandwich" as an error) and, conversely, an instance of Error can
be treated as a regular value if desired.
All in all the overhaul of this system, daunting as it was to pull off, has made
Grafast significantly more powerful and able to express and handle more
complex patterns. This is particularly good news for people using the
PostGraphile Relay preset who want to filter by IDs or use them as inputs to
database functions.
Documentation for all of this is on the TODO list, and will be arriving over the
coming weeks. I didn't want to delay this release even longer just waiting for
docs, especially because most people don't need to know these details (at least,
not yet!)
Grafserv

Envelop

To use Envelop with Grafserv, you can import the GrafservEnvelopPreset and
then pass your getEnveloped function as part of your preset:
import { GrafservEnvelopPreset } from "grafserv/envelop";
import { envelop, useMaskedErrors } from "@envelop/core";

const getEnveloped = envelop({
  plugins: [useMaskedErrors()],
});

const preset = {
  extends: [GrafservEnvelopPreset],
  grafserv: {
    getEnveloped,
  },
};
Validation rules

If you want to load custom validation rules into Grafserv, you can do so with a
plugin generator like this:
let counter = 0;
function addValidationRules(
  newValidationRules: ValidationRule | ValidationRule[],
): GraphileConfig.Plugin {
  return {
    name: `AddValidationRulesPlugin_${++counter}`,
    grafserv: {
      hooks: {
        init(info, event) {
          const { validationRules } = event;
          if (Array.isArray(newValidationRules)) {
            validationRules.push(...newValidationRules);
          } else {
            validationRules.push(newValidationRules);
          }
        },
      },
    },
  };
}
which you can call like this:
import { depthLimit } from "@graphile/depth-limit";

const preset: GraphileConfig.Preset = {
  plugins: [addValidationRules(depthLimit())],
};
Releases

Releases:
  @dataplan/json@0.0.1-beta.16
  @dataplan/pg@0.0.1-beta.18
  @grafserv/persisted@0.0.0-beta.19
  @graphile/simplify-inflection@8.0.0-beta.5
  grafast@0.1.1-beta.7
  grafserv@0.1.1-beta.9
  graphile-build-pg@5.0.0-beta.21
  graphile-build@5.0.0-beta.17
  graphile-config@0.0.1-beta.8
  graphile-export@0.0.2-beta.12
  graphile-utils@5.0.0-beta.21
  graphile@5.0.0-beta.22
  pg-introspection@0.0.1-beta.8
  pg-sql2@5.0.0-beta.6
  pgl@5.0.0-beta.22
  postgraphile@5.0.0-beta.22
  ruru-components@2.0.0-beta.15
  ruru@2.0.0-beta.12
  tamedevil@0.0.0-beta.7

What now?

Try it out via yarn add postgraphile@beta or equivalent, and provide feedback
via GitHub Issues, or our
Discord server.
Help out on some of the
issues left to complete before V5.0.0 release
— we're particularly keen to see improvements to the documentation, don't worry
that you're not an expert yet, even just adding a small code sample in the right
place can be hugely helpful to seed a documentation change (we always edit it
before it goes out anyway!), and even as a PR it acts as a useful resource for
other community members.
Please consider sponsoring us, so we can spend more time on this (and hopefully
get V5 out sooner, and with better documentation and typing!):
https://www.graphile.org/sponsor/