NamelessCoder/GridTCA.md

## GridTCA.md

      
    Raw
  

              GridTCA.md
            
          
    Vision: Grid as TCA type for TYPO3

Synopsis

This little gist is an attempt to describe my vision for a nested content / grid feature for TYPO3.
It contains a description of the vision I have and relates this vision to the current blueprint and WIP
which are available at https://wiki.typo3.org/Blueprints/StructuredContentContainers and https://review.typo3.org/#/c/41936/.
My vision

My vision is to create a completely new TCA type which will function in two ways:

It will enable TYPO3 to maintain strict relations in SQL tables between parent, grid column and child element.
It will enable developers to define grids to contain content elements by writing TCA

Both capabilities are essential; the second capability is different from currently proposed WIP and blueprint in that it
uses a new TCA type to declare that table X has a grid containing records from table Y. Current WIP and blueprint suggests
that this capability is being tied to a new CType and use pageTSconfig to define the structure; with file based storages
and version control being an afterthought rather than the basis for the feature. But, having this as a new type of TCA
field allows the grid concept to be used in custom plugins and make the rendering less dependent on a specific cObject.
My vision is to provide the same capabilities that are mentioned in the WIP and blueprint but do so in a way that allows
grids to be used for anything - not just structured content elements of one particular type. To achieve this it is ideal
to use a new TCA type for managing the relations and configuration, apply that same relationship to pages<->content and
finally provide a new CType and cObject along the lines of the current WIP, which contains a basic working column
splitter with minimal configuration that is easy to modify (since column splitting is very much a CSS framework dictated
feature).
The UX perspective

I agree with the UX decisions about how to display and define grids when you are an editor. Being based on TCA the UX may
need to be slightly different than the current one, but the essential parts are absolutely agreed upon and required:

Displaying nested content in list view
Rendering grids in the page module
Defining grids using a wizard (storage arbitrary; not UX related, more on this below)

Since I agree with these aspects there is no need to change any of it.
The storage perspective

The current WIP suggests that grids are stored as pageTSconfig (stored in a record, file, etc.) or provided from a
provider implementation under the BackendLayoutDataProvider pattern. My vision differs significantly from this suggested
functionality in the following ways:

I intend to store the full representation and relations of a grid; every row and every column, in relational
SQL tables as default method and allow an API-based way to create completely custom storage methods to define and
save the grid.
I intend to implement a backend layout compatible storage method which can be used as backwards compatibility as well
as legacy support.

So rather than implement backend layout and backend layout providers, which carry with them some problems
when you begin to add things like dynamic grids, a new and more generic wrapper is created which does not suffer from the
issues that the pattern built exclusively for backend_layouts currently does. This wrapper is then compatible with the
previous DataProviders but does not solve their internal issues like the one in the link above. A full switch to the
new approach is required for those issues to go away (note: this can be seen as opinion and you should not feel obligated
to agree. In fact, if you have a better idea, I want to hear it!)
Most of the current issues arising from dynamic provisioning of backend_layouts and other related features such as using
itemsProcFunc to fill additional colPos entries, stem from the work done on FormEngine. While the resulting logic is
a big improvement, it has resulted in some regressions which may not be easy fixes. My vision for this grid TCA feature
centers on new FormEngine components and using as much as possible of the existing relation capable fields. And rather
than merely providing new values for selectors, the vision is to allow grid storages to perfectly control both how they
work with other grid storages (for example: multiple grid providers adding page columns; exclusivity issues with current
logic) and give them a true API to create columns and manipulate their behaviours in terms of translation, ACLs,
restrictions and more.
How will it work?

The following describes the default working mode (as in: not legacy/compatibility mode or custom implementation).
Developers are allowed to move outside this working mode but are encouraged not to do so.
The feature will create strict relationships between the parent (for example, a page record), the columns the parent has
and the content mapped to each of those columns. Meaning it will be possible to select "all content elements from column
foobar in parent XYZ" (and the kicker is: the parent can be any record type, not just pages or content). The database
stores only column relations, not rows. Instead the row index is defined in the column and moving a column to a new row
thus changes only the (numeric) row index - not the column identifier. Columns then contain an additional sorting flag
which puts the columns in the right order in each row. All relationships are preserved regardless of how the grid is
structured or restructured.
To render a grid structure one then only needs to select all columns ordered by row and column index, then iterate those
and insert whichever breaks are required (bootstrap css divs, table cells, ASCII art, ...) when the row index changes.
Or it can be done directly in the page template (which can use any arbitrary template system) and each column rendered
with a direct reference, such as it is when you for example define a TS object which renders all content in colPos zero.
In the latter case one only needs to care about the column's identifier, not the actual structure of rows and columns. To
put it differently about rendering: you can choose automatic or manual rendering strategies; both of which are supported
by the framework on equal terms.
All current methods of rendering content are supported like this - with only minimal changes required to the query that
selects content records based on a clause.
Which parts of TYPO3 will be changed?

In terms of UX all the parts mentioned in the current WIP will change to handle the
new nested structures. In terms of TCA, the following will be done:

The central new TCA type grid will be added along with required companion FormEngine components etc. (inner API).
A companion TCA type columnSelector will be added which is specifically built to read possible column identifiers
and which fully supports any dynamic processing of the values. It has the function and form of a selectSingle which
is preconfigured and restricted to only work for (and work ideally for) retrieving column identifiers. It wraps around
API which can also be used from third party code to read such identifiers.
pages will be fitted with new TCA defining a grid which uses an adapter that loads the grid from the previously used
backend_layout records by default. The intention is to later change this, or provide a toggle, to move to the new way
of storing grid definitions.
tt_content will have a new column relationship identifier that will take over the old colPos purpose.
tt_content will have a new column of grid type which will contain the default implementation we ship, and which can
be easily used in plugins for example - without having to worry about the DB schema.
New table(s) will be added to store grid definitions in ways that allow SQL JOINs and is "parsing friendly" (see
above regarding row/column index).
A new CType will be added with a sufficiently generic column splitter type element (configurable or with presets for
common column-capable CSS frameworks). It will make use of the new grid field added on tt_content.

In addition but not specific to TCA:

A new cObject will be added which renders the CType (via an overridable Fluid template, possibly with MVC)

The scope of new additions is fairly limited and exists peacefully with current features, with the notable caveat that
the pages and tt_content tables will receive columns to store the real relations between page, page grid, content and
content grids. The added TCA type will allow third party developers to make use of the grid for their own page/content
types and control access to it like any other TCA field (showitem for record type, in palette, etc.).
What will it break?

In very few words: as little as possible, nearly nothing in vanilla core setups but unfortunately requiring significant
migration for third party code which uses SQL to select content elements (which is, presumably, every third party code
that operates with content elements since there is no Repository for tt_content). To list the most important expected
breaking changes and their migrations:

Obviously, selecting content based on colPos will break (and there is currently no sane way to force rewriting of
queries, nor - to my knowledge - any way of logging queries which include this colPos without big performance hits.
Migration will require SQL queries which include colPos clauses to switch to clauses using the new column-ID column;
if necessary including JOIN statements to also select the column definition record. For compatibility purposes the
compatibility layer will name columns using the same integers as colPos did and store those as column ID.
The intention is to deprecate the backend_layout table or convert it to become the parent of grid definition records
(not yet decided). However, the naming may change as the TCA grid feature is not just a backend feature (as well as
considering the likely direction TYPO3 takes towards proper FE editing which further removes the FE/BE distinction).
The UX of the backend layout wizard will be reused, possibly extended with additional column configuration options.
The intention is to deprecate BackendLayoutDataProviders and associated patterns. This pattern is to be replaced by
GridStorage implementations or signals to modify the returns from those.
Third party implementations of custom content grids are not affected directly (due to the compatibility layer) but
should in time migrate to GridStorage and other patterns. Such implementations are likely also subject to the colPos
changes mentioned above and would require migration in either case.

The main part of breaking changes then should almost exclusively affect third party column splitting / nested content
solutions like Gridelements, DCE and Flux - and will require those extensions to migrate. The effect on users on the other
hand is minimal, since the same UX exists and the core now provides an alternative to nested content.
The examples

ASCII: column definitions table
------------------------------------------------------------
|GridID | ID | Name/label | Row index | Column index | ... |
------------------------------------------------------------
| 1     | 0  | Legacy "0" | 0         | 0            |     |
------------------------------------------------------------
| 1     | AB | New type   | 1         | 0            |     |
------------------------------------------------------------
| 1     | C  | New type 2 | 1         | 1            |     |
------------------------------------------------------------

Any parent that uses the grid can reference the grid ID and this becomes the binding between parent, grid definition and
children inserted into the grid columns - all in strict SQL relations.
Given this structure it is possible to select with queries:

The entire structure of a grid from a single table
The column definition along with any content inserted into that column on any page
Content and structure of a grid ordered by row, column and position (yielding records that can be processed in one loop)

It also becomes possible to delete or trigger other actions to be performed on content if a grid column of a grid that is
currently referenced gets deleted (for example: moving content to recycler, moving it to an "unused content" area, etc.).
All such orphans can be detected also with a single SQL query.
The columns structure can be created using that single table; the GridStorage is responsible for handling all column
definition records belonging to a particular grid definition. Or alternatively the column definitions can be given a
parent - for example the existing backend_layout record although naming then no longer is correct - which can then be
the aggregate root of each grid definition.
TCA array (as YAML for brevity) 
tt_content
   columns
      column_id
        label: Column
        config
          type: columnSelector
          from_field: grid
      grid (or other name)
        label: A grid field
        config
          type: grid
          mode: static
          foreign_table: tt_content
          foreign_table_field: column_id
          grid
            columnOne
              label: First column
              columnIndex: 0
              rowIndex: 0
              options
                allowedContentTypes: media,my_type,third_type
                deniedContentTypes: table,other_type
                allowLocalization: false
            columnTwo:
              label: Second column, second row
              columnIndex: 0
              rowIndex: 1
              options
                allowLocalization: true
                
This example of statically defined TCA illustrates how a grid might be defined using nothing but simple values in a plain
old array that is then cached/stored as TCA. The entire grid definition is contained in a sub-property in the config
of the parent table's field that associates nested content with the parent. In other words the configuration of how
the grid layout is, exists in the same place as the relationship is defined.
Each column's configuration is provided inside an options sub-array with the intention that options are arbitrary and
can for example be extended by third parties to support new options, which can then be handled by custom implementations
for either the GridStorage or custom Column types, or by any signal in the runtime. There will be a proper API and
interfaces for creating custom implementations of all of these parts (though the default implementations should be good
enough for the vast majority of cases).
When configuration is provided statically it can be persisted to DB to ensure a common storage. This design decision is
not yet settled.
Alternatives for "config" array in a "grid" TCA column
  usingWizard
    label: Grid configured entirely by editor using grid wizard in backend, saved as definition in DB
    config
      type: grid
      mode: user
In this mode, the editor is alone responsible for defining and maintaining the grid. He does so with the already known
backend layout wizard and the resulting grid definition records are saved. A shortcut to create or edit the grid's
definition is then rendered (like the shortcuts you see in IRRE fields).
Alternatives for "config" array in a "grid" TCA column
  usingDelegated
    label: Grid definition entirely generated by our delegate class
    config
      type: grid
      mode: delegate
      delegate: My\Class\Implementing\Proper\Interface
In this mode the entire responsibility for generating (and persisting, if required to be different from default method
of storing in DB - for example to store in files!) every piece of the grid structure: all columns with every option.
A proper API will be provided to make it easier for such implementations to create new structures programatically.
WORK IN PROGRESS - TO BE CONTINUED