Vision: Grid as TCA type for TYPO3
This little gist is an attempt to describe my vision for a nested content / grid feature for TYPO3. It contains a description of the vision I have and relates this vision to the current blueprint and WIP which are available at https://wiki.typo3.org/Blueprints/StructuredContentContainers and https://review.typo3.org/#/c/41936/.
My vision is to create a completely new TCA type which will function in two ways:
- It will enable TYPO3 to maintain strict relations in SQL tables between parent, grid column and child element.
- It will enable developers to define grids to contain content elements by writing TCA
Both capabilities are essential; the second capability is different from currently proposed WIP and blueprint in that it
uses a new TCA type to declare that table X has a grid containing records from table Y. Current WIP and blueprint suggests
that this capability is being tied to a new
CType and use pageTSconfig to define the structure; with file based storages
and version control being an afterthought rather than the basis for the feature. But, having this as a new type of TCA
field allows the grid concept to be used in custom plugins and make the rendering less dependent on a specific
My vision is to provide the same capabilities that are mentioned in the WIP and blueprint but do so in a way that allows
grids to be used for anything - not just structured content elements of one particular type. To achieve this it is ideal
to use a new TCA type for managing the relations and configuration, apply that same relationship to pages<->content and
finally provide a new
cObject along the lines of the current WIP, which contains a basic working column
splitter with minimal configuration that is easy to modify (since column splitting is very much a CSS framework dictated
The UX perspective
I agree with the UX decisions about how to display and define grids when you are an editor. Being based on TCA the UX may need to be slightly different than the current one, but the essential parts are absolutely agreed upon and required:
- Displaying nested content in list view
- Rendering grids in the page module
- Defining grids using a wizard (storage arbitrary; not UX related, more on this below)
Since I agree with these aspects there is no need to change any of it.
The storage perspective
The current WIP suggests that grids are stored as pageTSconfig (stored in a record, file, etc.) or provided from a provider implementation under the BackendLayoutDataProvider pattern. My vision differs significantly from this suggested functionality in the following ways:
- I intend to store the full representation and relations of a grid; every row and every column, in relational SQL tables as default method and allow an API-based way to create completely custom storage methods to define and save the grid.
- I intend to implement a backend layout compatible storage method which can be used as backwards compatibility as well as legacy support.
So rather than implement backend layout and backend layout providers, which carry with them some problems
when you begin to add things like dynamic grids, a new and more generic wrapper is created which does not suffer from the
issues that the pattern built exclusively for
backend_layouts currently does. This wrapper is then compatible with the
previous DataProviders but does not solve their internal issues like the one in the link above. A full switch to the
new approach is required for those issues to go away (note: this can be seen as opinion and you should not feel obligated
to agree. In fact, if you have a better idea, I want to hear it!)
Most of the current issues arising from dynamic provisioning of
backend_layouts and other related features such as using
itemsProcFunc to fill additional
colPos entries, stem from the work done on FormEngine. While the resulting logic is
a big improvement, it has resulted in some regressions which may not be easy fixes. My vision for this grid TCA feature
centers on new FormEngine components and using as much as possible of the existing relation capable fields. And rather
than merely providing new values for selectors, the vision is to allow grid storages to perfectly control both how they
work with other grid storages (for example: multiple grid providers adding page columns; exclusivity issues with current
logic) and give them a true API to create columns and manipulate their behaviours in terms of translation, ACLs,
restrictions and more.
How will it work?
The following describes the default working mode (as in: not legacy/compatibility mode or custom implementation). Developers are allowed to move outside this working mode but are encouraged not to do so.
The feature will create strict relationships between the parent (for example, a page record), the columns the parent has
and the content mapped to each of those columns. Meaning it will be possible to select "all content elements from column
foobar in parent XYZ" (and the kicker is: the parent can be any record type, not just pages or content). The database
stores only column relations, not rows. Instead the row index is defined in the column and moving a column to a new row
thus changes only the (numeric) row index - not the column identifier. Columns then contain an additional sorting flag
which puts the columns in the right order in each row. All relationships are preserved regardless of how the grid is
structured or restructured.
To render a grid structure one then only needs to select all columns ordered by row and column index, then iterate those
and insert whichever breaks are required (bootstrap css divs, table cells, ASCII art, ...) when the row index changes.
Or it can be done directly in the page template (which can use any arbitrary template system) and each column rendered
with a direct reference, such as it is when you for example define a TS object which renders all content in
In the latter case one only needs to care about the column's identifier, not the actual structure of rows and columns. To
put it differently about rendering: you can choose automatic or manual rendering strategies; both of which are supported
by the framework on equal terms.
All current methods of rendering content are supported like this - with only minimal changes required to the query that selects content records based on a clause.
Which parts of TYPO3 will be changed?
In terms of UX all the parts mentioned in the current WIP will change to handle the new nested structures. In terms of TCA, the following will be done:
- The central new TCA type
gridwill be added along with required companion FormEngine components etc. (inner API).
- A companion TCA type
columnSelectorwill be added which is specifically built to read possible column identifiers and which fully supports any dynamic processing of the values. It has the function and form of a
selectSinglewhich is preconfigured and restricted to only work for (and work ideally for) retrieving column identifiers. It wraps around API which can also be used from third party code to read such identifiers.
pageswill be fitted with new TCA defining a grid which uses an adapter that loads the grid from the previously used
backend_layoutrecords by default. The intention is to later change this, or provide a toggle, to move to the new way of storing grid definitions.
tt_contentwill have a new column relationship identifier that will take over the old
tt_contentwill have a new column of
gridtype which will contain the default implementation we ship, and which can be easily used in plugins for example - without having to worry about the DB schema.
- New table(s) will be added to store grid definitions in ways that allow SQL JOINs and is "parsing friendly" (see above regarding row/column index).
- A new
CTypewill be added with a sufficiently generic column splitter type element (configurable or with presets for common column-capable CSS frameworks). It will make use of the new
gridfield added on
In addition but not specific to TCA:
- A new
cObjectwill be added which renders the
CType(via an overridable Fluid template, possibly with MVC)
The scope of new additions is fairly limited and exists peacefully with current features, with the notable caveat that
tt_content tables will receive columns to store the real relations between page, page grid, content and
content grids. The added TCA type will allow third party developers to make use of the grid for their own page/content
types and control access to it like any other TCA field (showitem for record type, in palette, etc.).
What will it break?
In very few words: as little as possible, nearly nothing in vanilla core setups but unfortunately requiring significant
migration for third party code which uses SQL to select content elements (which is, presumably, every third party code
that operates with content elements since there is no Repository for
tt_content). To list the most important expected
breaking changes and their migrations:
- Obviously, selecting content based on
colPoswill break (and there is currently no sane way to force rewriting of queries, nor - to my knowledge - any way of logging queries which include this
colPoswithout big performance hits. Migration will require SQL queries which include
colPosclauses to switch to clauses using the new column-ID column; if necessary including JOIN statements to also select the column definition record. For compatibility purposes the compatibility layer will name columns using the same integers as
colPosdid and store those as column ID.
- The intention is to deprecate the
backend_layouttable or convert it to become the parent of grid definition records (not yet decided). However, the naming may change as the TCA grid feature is not just a backend feature (as well as considering the likely direction TYPO3 takes towards proper FE editing which further removes the FE/BE distinction). The UX of the backend layout wizard will be reused, possibly extended with additional column configuration options.
- The intention is to deprecate
BackendLayoutDataProvidersand associated patterns. This pattern is to be replaced by
GridStorageimplementations or signals to modify the returns from those.
- Third party implementations of custom content grids are not affected directly (due to the compatibility layer) but
should in time migrate to
GridStorageand other patterns. Such implementations are likely also subject to the
colPoschanges mentioned above and would require migration in either case.
The main part of breaking changes then should almost exclusively affect third party column splitting / nested content solutions like Gridelements, DCE and Flux - and will require those extensions to migrate. The effect on users on the other hand is minimal, since the same UX exists and the core now provides an alternative to nested content.
ASCII: column definitions table ------------------------------------------------------------ |GridID | ID | Name/label | Row index | Column index | ... | ------------------------------------------------------------ | 1 | 0 | Legacy "0" | 0 | 0 | | ------------------------------------------------------------ | 1 | AB | New type | 1 | 0 | | ------------------------------------------------------------ | 1 | C | New type 2 | 1 | 1 | | ------------------------------------------------------------
Any parent that uses the grid can reference the grid ID and this becomes the binding between parent, grid definition and children inserted into the grid columns - all in strict SQL relations.
Given this structure it is possible to select with queries:
- The entire structure of a grid from a single table
- The column definition along with any content inserted into that column on any page
- Content and structure of a grid ordered by row, column and position (yielding records that can be processed in one loop)
It also becomes possible to delete or trigger other actions to be performed on content if a grid column of a grid that is currently referenced gets deleted (for example: moving content to recycler, moving it to an "unused content" area, etc.). All such orphans can be detected also with a single SQL query.
The columns structure can be created using that single table; the GridStorage is responsible for handling all column
definition records belonging to a particular grid definition. Or alternatively the column definitions can be given a
parent - for example the existing
backend_layout record although naming then no longer is correct - which can then be
the aggregate root of each grid definition.
TCA array (as YAML for brevity) tt_content columns column_id label: Column config type: columnSelector from_field: grid grid (or other name) label: A grid field config type: grid mode: static foreign_table: tt_content foreign_table_field: column_id grid columnOne label: First column columnIndex: 0 rowIndex: 0 options allowedContentTypes: media,my_type,third_type deniedContentTypes: table,other_type allowLocalization: false columnTwo: label: Second column, second row columnIndex: 0 rowIndex: 1 options allowLocalization: true
This example of statically defined TCA illustrates how a grid might be defined using nothing but simple values in a plain
old array that is then cached/stored as TCA. The entire grid definition is contained in a sub-property in the
of the parent table's field that associates nested content with the parent. In other words the configuration of how
the grid layout is, exists in the same place as the relationship is defined.
Each column's configuration is provided inside an
options sub-array with the intention that options are arbitrary and
can for example be extended by third parties to support new options, which can then be handled by custom implementations
for either the GridStorage or custom Column types, or by any signal in the runtime. There will be a proper API and
interfaces for creating custom implementations of all of these parts (though the default implementations should be good
enough for the vast majority of cases).
When configuration is provided statically it can be persisted to DB to ensure a common storage. This design decision is not yet settled.
Alternatives for "config" array in a "grid" TCA column usingWizard label: Grid configured entirely by editor using grid wizard in backend, saved as definition in DB config type: grid mode: user
In this mode, the editor is alone responsible for defining and maintaining the grid. He does so with the already known backend layout wizard and the resulting grid definition records are saved. A shortcut to create or edit the grid's definition is then rendered (like the shortcuts you see in IRRE fields).
Alternatives for "config" array in a "grid" TCA column usingDelegated label: Grid definition entirely generated by our delegate class config type: grid mode: delegate delegate: My\Class\Implementing\Proper\Interface
In this mode the entire responsibility for generating (and persisting, if required to be different from default method of storing in DB - for example to store in files!) every piece of the grid structure: all columns with every option. A proper API will be provided to make it easier for such implementations to create new structures programatically.
WORK IN PROGRESS - TO BE CONTINUED