totten/00-moved.md Secret

## 00-moved.md

      
    Raw
  

              00-moved.md
            
          
    This document has been moved to https://lab.civicrm.org/dev/core/-/issues/4356.

  
## draft.md

      
    Raw
  

              draft.md
            
          
    (This message is generally written for a technical audience -- people who are familiar with many technologies, who want to bring them to Civi, and who find themselves asking, "Why can't we have nice things?")
[[TOC]]
* Preliminaries
    * Why replace AngularJS?
    * What is "$XXX"?
    * What is CiviCRM?
* Challenges
    * Linker-Loader and Modularity
        * Definition
        * What kind of linking/loading does CiviCRM require?
        * What does linking/loading look like in Javascript?
        * Challenges of linking/loading for Javascript
        * Approaches to linking/loading ES6 for CiviCRM
    * Angular, TypeScript, ng
    * ReactJS and JSX

Preliminaries

Why replace AngularJS?


The upstream "AngularJS" project (a Javascript framework) re-made itself as "Angular" (a Typescript framework).


This reflects important changes in the platform, and it is not a straight-forward upgrade.  Adopting newer "Angular" versions should be approached as a migration.


The upstream "AngularJS" project has been proclaiming: "We don't like our JS product anymore. Don't use that."


One original goal of adopting "AngularJS" was to get some bandwagon benefits. Those benefits are no longer there.


What is "$XXX"?

$XXX is a stand-in for "Angular 14 or ReactJS or some other client-side page-rendering system". The specific value is a matter for consideration.
Some migration issues are generally applicable to any alternative, and we may use $XXX as stand-in (because it gets tiresome to write "Angular 14 or ReactJS or some other client-side page-rendering system").
What is CiviCRM?

For purposes of this evaluation, CiviCRM is an ecosystem -- a collection of packages developed and maintained by many people. It includes a significant core package and a significant set of extensions. If we adopt $XXX, then we ae encouraging this ecosystem to write their screens and UI components with $XXX.
Today, CiviCRM's ecosystem is quite varied:

You have civicrm-core -- which includes the framework, several common business modules, APIs, and various UIs for admins+staff+consumers.
You have extensions like CiviVolunteer or CiviGrant -- which define new business modules (including their APIs and UIs).
You have extensions like CiviRules or Search Kit -- which build generic functionality (that works with many business modules).
You have extensions like Mosaico or CiviDiscount -- which significantly improve the features of an existing business-module.
You have extensions like AngularProfiles or Email API -- which add extra building-blocks or fill-in gaps.
You have bespoke extensions for specific sites -- extensions which fine-tune screens, workflows, settings, for a specific organizational context.

Each package may specify some mix of:

Redistributed artifacts (like CSS files, APIv3 files, APIv4 files, Smarty templates, etc) defined by developers.
Local artifacts (like Profiles and SavedSearches and Webforms and Afforms) defined by administrators for a specific site (with support from the extensions).

These packages are distributed in various ways, eg:

Published as part of civicrm-core
Published in a public directory for impromptu download/installation
Organized as in-house collections

Is this the optimal shape for the ecosystem? I'm not sure how you would answer that.
Instead, for this discussion, I simply accept this shape for the ecosystem. CiviCRM's ecosystem should continue to have similar kinds of packages and similar kinds of distribution -- even if some specific packages change, and even if it adopts another tool like $XXX. This shape is not perfect or static - but it is realistic and representative.
Challenges

Perhaps we just drop-in some new files for the latest version of Angular?  Or, failing that, drop-in ReactJS?  Surely, at the end of the day, these are JS files used by the browser.  You just need to put them on a web-server.  The browser will run them, and that's it.  What's wrong with that?
Let's break-down a few specific challenges.

Linker-Loader and Modularity
Angular 14 and TypeScript
ReactJS and JSX

The challenges can be addressed -- the question is what trade-offs, costs, or compromises we accept.
Linker-Loader and Modularity

Definition

Sane software is built in pieces.  To run the software, you have to put the pieces together. This requires a linking or loading mechanism.
Just to clarify what I mean, let's use a basic example.  Here we have some classes for an application where the user draws rectangles on a canvas:
// FILE: Canvas.js
class Canvas { 
  constructor() {
    // A list of visual objects to display in this presentation
    this.objects = [];
  }
  addRectangle(x, y, width, height) {
    this.objects.push({
      pos: new Coordinate(x, y),
      obj: new Rectangle(width, height)
    });
  }
}
// FILE: Rectangle.js
class Rectangle {
  constructor(width, height) {
    this.width = width;
    this.height = height;
  }
}
// FILE: Coordinate.js
class Coordinate {
  constructor(x, y) {
    this.x = x;
    this.y = y;
  }
}
N.B. To use Canvas, you must also use Rectangle and Coordinate. There must be a mechanism to say: "We are loading Canvas.js and also Rectangle.js plus Coordinate.js."
If you're reading this, then I'm sure you're already familiar with several such mechanisms, though they go by many names. Perhaps you've encountered PHP's "autoloader"; Javascript's "webpack"; bash's "PATH"; C's "static-linking" and "libtool". All of these address the need to combine/resolve related pieces of software.
The design of the link-load mechanism can have effects on:

Performance - Load times, download sizes, cache efficacy, update frequency, etc. (A link-load mechanism that performs nicely for small apps might perform badly for large apps, and vice-versa.)
Workflow - The ease or difficulty of installing, developing, customizing, upgrading, etc.
Modularity - How you design/frame/distribute the packages of the system.

In the phrase "link-load", I wish to evoke the general mechanism of "getting the stuff we need" -- with an understanding that different systems have different phrases, tools, and flows.
What kind of linking/loading does CiviCRM require?

Recall that the CiviCRM ecosystem has a mix of extensions providing different kinds of value (business modules, APIs, UIs, etc) with various artifacts (CSS files, APIv4 files, Quickform files). If we adopt $XXX, then $XXX will provide more kinds of artifacts (JS-based pages; Angular components; React components; etc). You should be able to define these in Civi extensions. Thus, you might create files like:
## Angular component defined in core on Drupal 9
/var/www/vendor/civicrm/civicrm-core/ang/*.component.ts

## React component defined in an extension on Drupal 7
/var/www/sites/default/files/ext/myextension/react/*.jsx

The linker/loader mechanism used for $XXX in CiviCRM should be able to:

Load screens defined by core+extensions.
Load components defined by core+extensions (across extension boundaries).
Load transitive dependencies required by the screens+components in core+extensions (across extension boundaries).

What does linking/loading look like in Javascript?

Javascript is a big ecosystem! And it's been around for 25 years. We will not try to discuss all the techniques of the many applications/frameworks/tools. Rather, let's note a couple relevant practices - the baseline; the current mechanisms in CiviCRM; and the mechanisms favored in contemporary Angular+React ecosystems.
First, as a baseline, the traditional mechanism (available to all web developers since 1995) is to write HTML: you create a webpage with a list of <script> tags containing URLs of all your JS files.
<head>
<script type="text/javascript" src="./js/Rectangle.js"></script>
<script type="text/javascript" src="./js/Coordinate.js"></script>
<script type="text/javascript" src="./js/Canvas.js"></script>
</head>
This is (roughly) how JS files get into CiviCRM's PHP screens - except that the <script> tags are written as PHP statements (addScriptFile($ext, $file)). But it's similar: it's a top-level list of JS files, and they're printed as a list of <script> tags in the main HTML document. On a case-by-case basis, some screens make extra calls to addScriptFile/<script>.
CiviCRM's AngularJS screens use another mechanism -- a dependency-graph. It works as follows:

Each major Angular module has an explicit metadata file (*.ang.php) with a list of inputs and outputs (components/tags/etc).
If a site-builder/developer makes a custom form (*.aff.html), then we scan the HTML and infer its dependencies (based on the tags and attributes).
When a user requests an Angular-based screen (eg civicrm/admin/search), then we use this metadata to identify dependencies and send them all as a bundle.

In contemporary Angular and ReactJS ecosystems, it is also conventional to use a dependency-graph -- but now based on ES6 modules. ECMAScript 6 defines a language-level standard with import/export statements. For example, Canvas.js could be updated to:
// FILE: Canvas.js
import { Coordinate } from 'Coordinate.js';
import { Rectangle } from 'Rectangle.js';

export class Canvas { 
  constructor() {
    // A list of visual objects to display in this presentation
    this.objects = [];
  }
  addRectangle(x, y, width, height) {
    this.objects.push({
      pos: new Coordinate(x, y),
      obj: new Rectangle(width, height)
    });
  }
}
For pure JS development, this represents a significant improvement over the baseline -- you do not need to maintain a global list of scripts. You don't need custom metadata (like *.ang.php). You describe the JS dependencies within the JS files. And unlike Civi's AngularJS loader, this mechanism is a widely known standard. ES6 modules have become nearly ubiquitous in documentation for contemporary JS libraries.
Challenges of linking/loading for Javascript

ES6 imports are part of the language standard, but you cannot take this mechanism for granted. There are multiple implementations with important differences. To see this, let's do an example.
Suppose you are writing an extension like "Mosaico" which builds on top of CiviMail. You need to use a (Angular/React/etc) component from CiviMail. So you import it:
import ReceipientSelector from '../../../sites/all/modules/civicrm/components/Mailing/RecipientSelector.js';
const MosaicoEditor = () => <div>....<ReceipientSelector />...</div>;
export default MosaicoEditor;
This import (with a physical path) is compatible with most implementations of import, but the path itself is untenable -- it will only work on (say) 50% of CiviCRM deployments. For broad support, you need to hide some of these physical details and use a simpler (logical) path -- such as:
-import ReceipientSelector from '../../../sites/all/modules/civicrm/components/Mailing/RecipientSelector.js';
+import ReceipientSelector from 'civicrm/components/Mailing/RecipientSelector.js';
This can be done - and I suspect that most pure-JS apps benefit from simpler expressions. However, this depends on the particular implementation of import. Implementations fall into a few major groups:

Browsers: Since ~2019, most major web-browsers have had support for imports with physical paths. However, support for logical paths ("import maps") is gradually rolling out. At time of writing, Chrome has support; Firefox requires an experimental opt-in (about:config); and Safari has no support. (There may be a polyfill.)
Bundlers: Bundlers (like webpack or rollup) have provided support for a longer period, and they appear in many web-app project-templates. Bundlers scan JS code, examine imports, and build aggregated JS files. They have more features -- such  as customized path resolution and build-optimizations ("minification", "tree shaking"). However, most (all?) popular bundlers are designed for NodeJS.

(Question: Is the choice of import mechanism all-or-nothing? Or do they interoperate? For example, suppose you use webpack to make a base-bundle $XXX + common widgets -- and then you load a few extra JS modules in the browser. Will imports in the browser find resources from webpack's bundle?)
Approaches to linking/loading ES6 for CiviCRM


Don't support ES6 modules

Pro: No new code!
Con: You will find it confusing to read documentation+examples in the $XXX ecosystem. Authors tend to assume you have figured out ES6. (If you read import foo from bar.js, then you will need to mentally substitute calls like addScriptFile('bar.js').)
Con: Some libraries may be difficult/impossible to assimilate (because the code is sprinkled with unresolvable imports).


Rely on browser support. All imports in Civi-land use physical paths. Rearrange Civi.

How: Rearrange the file-tree (for every Civi-CMS integration) to make physical paths cleaner and more predictable.
Pro: Clean, predictable paths are pleasant.
Pro: Standards-based import mechanism. No special tools/libraries.
Con: Major compatibility break with existing CiviCRM deployments/hosts/scripts.
Con: Doesn't support bundling/aggregation.


Rely on browser support. All imports in Civi-land use physical paths. Sync files during installation.

How: Define a data folder like [civicrm.files]/js. Whenever you install/upgrade/remove an extension, synchronize the JS resources to this folder -- and ensure that each file lands in a clean/predictable subdir. The web-browser requests JS files from [civicrm.files]/js.
Pro: No major compatibility break.
Pro: Clean, predictable paths are pleasant.
Con: Large file sync's are unpleasant (especially during development).
Con: Doesn't support bundling/aggregation.


Rely on browser support. All imports in Civi-land use import-maps.

How: In PHP, define a registry of import-paths. Fill this based on the list of installed extensions. Output the import-map to the HTML <HEAD>.
Pro: Works with existing file-tree.
Pro: Standards-based import mechanism. No special tools/libraries.
Con: Firefox+Safari support is... eh... "pending". There may be a polyfill (not yet assessed).
Con: Doesn't support bundling/aggregation.
Comment: If we think that Firefox+Safari will have good support in (say) 2023, then we could proceed down this path and simply keep this functionality in "alpha" until 2023.


Rely on a bundler. Every extension calls the bundler on its own.

How: In each extension that uses $XXX for a UI, define package.json (et al) to download a bundler (eg webpack) and build a custom bundle. To include resources from another extension, ask Civi for the paths (eg cv path -x $EXT). Publish the compiled bundle as part of the extension.
Pro: Use a popular bundler with advanced features (like tree-shaking).
Pro: Each extension developer can choose their own bundler. (Freeedom!!)
Con: All such extensions will have independent versions of $XXX (and independent versions of common utilities/widgets/etc).
Con: Upgrades/bugfixes are onerous (must upgrade every extension; they're all statically-linked).
Con: If you have a screen (e.g. "The Dashboard") that mixes elements from multiple extensions, then you will get multiple copies of $XXX.
Con: Impossible for one extension to add to another extension (eg CiviDiscount adding elements to CiviEvent).
Con: Strange bootstrap process (similar to civicrm_generated.mysql; you need a working Civi deployment with the extension enabled before you can write the *.js file that will be redistributed.)


Rely on a bundler. Integrate a popular bundler (like webpack/rollup) into Civi. Run it whenever you install/upgrade/remove extensions.

How: Make a JS script which calls webpack (etc). Pass key configuration-data (e.g. the list of bundles and JS files) from Civi to webpack.
Pro: Use a popular bundler with advanced features (like tree-shaking).
Pro: Works with existing file-tree.
Con: Requires JS runtime -- eg node subprocess, node daemon, or browser-based queue-worker.
Con: Vendor-specific import mechanism.
Con: No dynamic bundling (based on user-defined forms, settings, etc)


Rely on a bundler. Use/improve a PHP-based bundler for JS files.

How: There's an alpha-stage JS bundler at https://bitbucket.org/fugu-fuman-library/es6bundler/src/main/. Add it to core. As necessary, contribute to it.
Pro: Works with existing source-tree.
Pro: Supports bundling
Pro: Supports dynamic bundling (based on user-defined forms, settings, etc)
Con: It's alpha-stage with no major user-base. It likely needs work.
Con: There is not much of an ecosystem for parsing JS in PHP. You're likely to use approximations (regex) or something slow.
Con: Limited optimizations (Never as good as webpack. Ex: You can do minification but not tree-shaking.*)
Con: Vendor-specific import mechanism.


Rely on 2-step bundler. Use JS to build dependency graph. Use PHP to concatenate final bundle(s).

How: If an extension publishes JS files, then it should also publish a JSON index (listing the files+dependencies). Provide JS devtool to generate the index. In Civi, use the PHP bundler.
Pro: Works with existing source-tree.
Pro: Supports bundling
Pro: Supports dynamic bundling (based on user-defined forms, settings, etc)
Pro: JS scanning is done by JS code (should be able to find a robust JS parser)
Con: No one else is doing it.
Con: Need to include JS indexer as part of extension-development workflow.
Con: Limited optimizations (Never as good as webpack. Ex: You can do minification but not tree-shaking.*)
Con: Vendor-specific import mechanism.


Angular, TypeScript, ng

Angular 14 appears to parallel the original AngularJS in that it has two main file-types:


Version
Logic/Behavior-Implementation
Layout/Component-Graph


AngularJS
Javascript (*.js)
XHTML (*.html)


Angular
Typescript (*.ts)
XHTML (*.html)


The good news is that the layout/component-graph is broadly the same format - HTML provides the canvas for organizing the choice of components that you want on the screen. This canvas still uses a well-known format which is amenable to reading/filtering/writing with many tools. This gives us a good prognosis for migrating data (immediately) and maintaining it (future updates).
But... the logic/behavior files changed to Typescript. This requires a TypeScript compiler. Like with linker-loader, the compiler needs to be taught to load classes from different extensions (ie the type-checker needs full visibility on the list of types - and types can originate in different places).
I haven't examined of how Angular 14 implements its compiler/linker/loader mechanisms, so the list of approaches here is nowhere near as detailed as the link-loader discussion above. But loosely, the documented/supported mechanisms are built around a CLI tool named ng (implemented with node). The general options appear to be:

Integrate with ng. (This will have similar pro/con as the integrating with the webpack bundler.)
Examine+decompose the functionality of ng. It may be possible to combine this with one of our linker-loader approaches.
Don't support TypeScript at all. (This will likely be very difficult. It's true that earlier versions of Angular presented JS options, and it's true that Angular compiles down to JS, but contemporary documentation is exclusively geared toward TS. Even if we figure out how, they're liable to change it.)

The adoption of ng represents another significant change between AngularJS and contemporary Angular. This is likely very nice for people who write pure-Angular applications -- the ng tool provides more functionality and hides more details. Thus, they can make big changes in the mechanics (eg switching from "Just in Time" compilation to "Ahead of Time" compilation) without significantly changing the DX. But I worry about the breadth of ng's scope -- that your choice is to either (a) do everything "the ng way" or (b) continuously reengineer your process to stay aligned with ng.
I have a theory (but can't fully back it up, as I haven't been following along with Angular 2+ closely) -- by introducing ng, Angular began asserting more ownership over the stack+workflow. This has probably helped pure-Angular development work (because lots of changes can be encapsulated within ng) -- but it's has made it harder to integrate with other applications (like Civi and the CMSs). It doesn't seem like a coincidence that AngularJS (JS library) had a fairly long/stable 1.x series -- while Angular (Typescript platform) has done frequent major-version bumps.
ReactJS and JSX

(Sketch/Notes)

The graph of ReactJS components is defined as a series of JS functions. These functions can be written in pure JS, but they are typically written in "JSX" (JS with inline HTML).

From POV of an HTML developer, the extra power of JSX is that it provides a full (Turing complete) language -- where you can mingle function calls, loops, etc.
From POV of a JS developer, the extra ease of JSX is that embeds HTML (rather than using verbose DOM calls).
These things are exactly what makes it hard to process in other ways -- eg if the component-graph is defined with JS function-calls, then you're tooling must model "JS function-calls".


To provide tooling around JSX, you need the ability to compile/parse/generate JSX (aka "encode/decode"; aka "use the abstract syntax tree (AST)").

Here are some of the things that would require AST/encode-decode capability:

Drag/drop editor (let a web-user compose the graph of widgets)
Incremental-upgrades (automatically swap deprecated code with supported code)
Make client+server logic match-up (use the same document to determine entity-bindings and validations)


The standard tooling for JSX (compilers/parsers/ASTs) are implemented in JS themselves. But if the server and dev-workflow are in PHP, then you have to either (a) live with constrained callouts between PHP+JS or (b) reimplement JSX tooling in PHP.


It is possible to use ReactJS only as a runtime (in the same sense that Civi Profiles (UFGroups) use Quickform as the runtime).

You might do this if you want to get some functional aspect (or branding aspect) of ReactJS.
ReactJS without JSX is not the "ReactJS experience". For a developer who's experienced in ReactJS, it will feel alien.
d
Version	Logic/Behavior-Implementation	Layout/Component-Graph
AngularJS	Javascript (`*.js`)	XHTML (`*.html`)
Angular	Typescript (`*.ts`)	XHTML (`*.html`)