weierophinney/rfc-expressive-programmatic.md

## rfc-expressive-programmatic.md

      
    Raw
  

              rfc-expressive-programmatic.md
            
          
    [Expressive] RFC: Programmatic pipelines

When we originally created the API for Expressive, it was programmatic:
We had pipelines:
$pipeline->pipe($loggingMiddleware);
$pipeline->pipe($serverUrlHelperMiddleware);
$pipeline->pipeRoutingMiddleware();
$pipeline->pipe($urlHelperMiddleware);
$pipeline->pipeDispatchMiddleware();
And routed middleware, using HTTP verbs:
$pipeline->get('/ping', $pingMiddleware, 'ping');
$pipeline->get('/status', $statusMessagesMiddleware, 'status');
$pipeline->post('/status', $createStatusMiddleware);
$pipeline->get('/status/{id:\d+}', $statusMessageMiddleware, 'status.message');
$pipeline->patch('/status/{id:\d+}', $statusUpdateMiddleware);
$pipeline->delege('/status/{id:\d+}', $statusDeleteMiddleware);
Along the road to 1.0, we added configuration-driven piping and routing,
and ended up using this as the default approach for the 1.0 release of the
skeleton.
What we have learned since is that the configuration approach is often difficult
to understand, as users do not understand what the configuration maps to
programmatically. This was fully evident after I wrote a blog post on programmatic Expressive;
reactions online and in-person were overwhelmingly in favor of this approach.
This RFC details the steps proposed to make programmatic Expressive the default
in the skeleton, and the recommended path for existing users.
Backwards Compatibility concerns

First, we do not want to break backwards compatibility for existing users.
As such, the current Application factory and use case MUST continue to
work.
Any changes introduced will need to be:

Additional factories, classes, or configuration that are opt-in for existing
users.
Changes to the skeleton to opt-in by default to these features for new
projects.

Middleware piping

In Expressive, you may pipe service names for both pipeline and routed
middleware. This is problematic with Stratigility 1.3+, as we have no way of
knowing what type of middleware — legacy callable double-pass or http-interop
— will be provided by the service until it is retrieved.
In order to handle either type, I propose updating
MarshalMiddlewareTrait::marshalLazyMiddlewareService to continue using the
legacy callable signature, as that signature is also supported in Stratigility
2.0.  The closure would then test the middleware retrieved from the container to
see what signature it uses in order to determine how to dispatch it:

If it is double-pass callable middleware, it will invoke it using the same
arguments provided to it.
If it is http-interop middleware, callable or direct implementation, it will:

determine if the $next argument is a DelegateInterface; if not, it will
wrap it in a Stratigility CallableDelegateDecorator instance.
process the middleware using the provided request and the delegate.


This approach retains backwards compatibility with previous versions, while
providing forwards-compatible support for http-interop middleware services.
Additionally, we'd update the pipeErrorMiddleware() method to emit a
deprecation notice, informing users they should consider updating to the new
error handling paradigm detailed below.
Error handling

One aspect of using programmatic pipelines that is so compelling is that it
allows users to inject their own error handling middleware, and thus create
fully custom workflows.
Currently, this is not possible, because of the following:

Zend\Stratigility\Dispatch includes a try/catch block that, when a
throwable/exception is caught, triggers error middleware.
If no error middleware is able to handle the error and return a response,
delegation falls back to the final handler.

The first case means you cannot have outer middleware act as an error handler;
you are, in fact, required to use the Stratigility-specific error middleware
and/or final handler system. As such, generic middleware written to work with
other middleware dispatchers that includes a try/catch block... will never catch
a throwable or exception, as the Dispatch class will catch it first. That
makes this pattern unusable in Stratigility/Expressive:
$app->pipe(XClacksOverhead::class);
$app->pipe(ErrorHandlerMiddleware::class);
$app->pipeRoutingMiddleware();
$app->pipeDispatchMiddleware();
$app->pipe(NotFoundMiddleware::class);
In the second case, where no error middleware is present (which is the norm with
the default Expressive skeleton pipeline configuration), normal workflows are
completely bypassed, which means that you cannot have outer middleware react to
the returned response.
$app->pipe($middlewareThatInjectsResponseHeaders);
$app->pipeRoutingMiddleware();
$app->pipeDispatchMiddleware();
In the above example, if dispatched, routed middleware raises an exception, the
final handler gets invoked, and returns a response that bypasses the other
layers.
Stratigility 1.3 offers a solution to this problem, which is fully realized in
Stratigility 2.0.
In 1.3, you can opt-in to the new system by calling raiseThrowables() on a
MiddlewarePipe instance (from which Expressive's Application class derives);
once called, Dispatch no longer uses the try/catch block, allowing exceptions
to bubble up.
Stratigility 1.3 also offers a new final handler implementation, the
NoopFinalHandler; it's job is to simply return the response provided to it. In
2.0, this means returning the response provided when a MiddlewarePipe is
called via its __invoke() method. Since this handler is only supposed to be
called if no other middleware was able to return a response, this means that you
can use it to provide a generic error response.
The upshot of all of this is that, by adopting Stratigility 1.3 and up as the
base for Expressive, we can recommend pipelines like the following in our
skeleton application:
$app->raiseThrowables();
$app->pipe(XClacksOverhead::class);
$app->pipe(ErrorHandler::class);
$app->pipeRoutingMiddleware();
$app->pipeDispatchMiddleware();
$app->pipe(NotFoundHandler::class);
The above provides:

Middleware that triggers on every request and which writes to the response.
Error handler middleware that wraps the call to $next (or $delegate,
depending on whether or not we dispatch using http-interop semantics) in a
try/catch block, and which generates an error response if an exception is
caught.
Routing middleware, which attempts to route the incoming request.
Dispatch middleware, which dispatches routed middleware, if a match was found.
Middleware that dispatches only if no other middleware was, and which then
returns a 404 response.

The above flow is quite visual, more succinct than the equivalent
configuration-driven pipeline, and makes the order of operations more easily
predictable (vs priority numbers, and potential re-ordering when configuration
is merged).
Stratigility 1.3 provides both error handling and 404 middleware currently. The
404 middleware would need an Expressive-specific solution, as it only updates
the status, and returns a text response at this time. The error middleware,
however, allows developers to provide a response generator; Expressive could
use this middleware and configure that generator.
Proposed workflow for existing users

Existing users should be able to upgrade to Expressive 1.1, and thus
Stratigility 1.3, without impact.  Additionally, with the upgrade, they'll
immediately be able to use http-interop middleware in their applications.
The only impact will be whether or not they used functionality specific to the
Stratigility request/response decorators (e.g., getOriginalRequest(),
getOriginalResponse(), etc.). These will now emit deprecation notices,
detailing the changes the user will need to make to their code; making those
changes is both backwards- and forwards-compatible.
From here, users can either continue using Expressive and Stratigility as-is, or
begin preparing for the 2.0 releases/begin using the new Stratigility features.
These new features are two-fold:

Opting into the new error handling.
Opting into programmatic pipelines.

The second essentially incorporates the first, but does not have to; it could
continue using the final handler. This proposal suggests that if a developer
opts into the second, however, they should also be opting into the first.
Considering that the new error handling paradigm works best in an explicit,
layered approach, and requires adding middleware in the innermost layer (for
providing 404 responses), this proposal will combine the two as a single
migration step.
Error middleware and generator

We will add:

a "templated" ErrorResponseGenerator implementation, which will allow
optionally passing a template renderer and a template that will receive the
error details, and a flag indicating whether or not to display them
(essentially "development mode?"). Both will be optional, defaulting to no
template, and no display of error details.
a factory for the ErrorResponseGenerator, which will allow
retrieving configuration, and thus the "development mode" flag, based on
configuration we already accept, and testing for a template renderer interface
to pass as well.
a factory for the ErrorHandler middleware, which will be injected with the
ErrorResponseGenerator service.
a "templated" NotFoundHandler middleware, where the template renderer and
template are optional; if present, however, the URL and request method will be
provided to the template when rendering it for the response. It will require
and compose a response prototype.
a factory for the NotFoundHandler middleware, which will inject the response
prototype, and optionally inject the template renderer interface if present.

The above services will make it possible to have default error and 404 handling
for programmatic workflows.
For development purposes, we will provide a new WhoopsErrorResponseGenerator;
this will accept configuration for the Whoops instance and its handlers, and
then return the value from handleException(). This would be accompanied by a
WhoopsErrorResponseGeneratorFactory class.
Users will need to provide the following new service definitions in their
configuration; these would likely be in
config/autoload/dependencies.global.php or
config/autoload/middleware-pipeline.global.php.
use Zend\Expressive\Container;
use Zend\Expressive\Middleware\ErrorResponseGenerator;
use Zend\Expressive\Middleware\NotFoundHandler;
use Zend\Stratigility\Middleware\ErrorHandler;

return [
    'dependencies' => [
        'factories' => [
            ErrorHandler::class => Container\ErrorHandlerFactory::class,
            ErrorResponseGenerator::class => Container\ErrorResponseGeneratorFactory::class,
            NotFoundHandler::class => Container\NotFoundHandlerFactory::class,
        ],
    ],
];
Additionally, the following may need to be configured:
// config/autoload/local.php:

return [
    'debug' => true, // For displaying stack traces
];

// config/autoload/templates.global.php:

use Zend\Expressive\Template\TemplateRendererInterface;

return [
    'dependencies' => [
        'factories' => [
            TemplateRendererInterface::class => /* factory */,
        ],
    ],
];

// config/autoload/errorhandler.local.php:
// Replace the 'Zend\Expressive\FinalHandler` line with:
\Zend\Expressive\Middleware\ErrorResponseGenerator::class => 
    \Zend\Expressive\Container\WhoopsErrorResponseGeneratorFactory::class
Application factory

We will also update the ApplicationFactory to allow providing an additional
configuration flag, programmatic_pipeline. When that flag is enabled, the
factory will do the following:

Create the Application instance, using the configured router, the container,
a NoopFinalHandler, and an emitter.
Set the raiseThrowables flag.
Return the created Application instance.

It WILL NOT inject any configured pipeline or route middleware when the flag
is set.
It WILL NOT set the response prototype. Setting the response prototype may
have unexpected consequences for end-users if they were previously providing a
manipulated response to inner layers; those changes will no longer be
propagated. As such, we cannot set it for an initial release providing
Stratigility 1.3 support.
The following would be added to config/autoload/global.php or
config/autoload/dependencies.global.php:
return [
    'zend-expressive' => [
        'programmatic_pipeline' => true,
    ],
];
Pipeline and route injection via configuration

We will provide a utility class defining two methods, each accepting an
Application instance an a traversable of configuration. These methods will be:

injectPipelineFromConfig(): this will take the middleware_pipeline
configuration and pipe items to the application instance.
injectRoutesFromConfig(): this will take the routes configuration and
inject the items to the application instance.

The ApplicationFactory will be updated to use these utilities internally, but
users can also use these to create their pipeline and routed middleware if
desired.
This might look like:
$app->pipe(ErrorHandler::class);
ApplicationUtils::injectPipelineFromConfig($app, $container->get('config'));
$app->pipe(NotFoundHandler::class);

ApplicationUtils::injectRoutesFromConfig($app, $container->get('config'));
This would be recommended primarily as a migration measure, but could also be
used when the pipeline or routes may vary from one environment to the next.
Recommended pipeline

Once the configuration pieces have been put into place, the user would update
their public/index.php to create the application pipeline, as well as provide
routed middleware. The recommended pipeline will be:
use App\Action\HomePageAction;
use App\Action\PingAction;
use Zend\Expressive\Helper\UrlHelperMiddleware;
use Zend\Expressive\Helper\ServerUrlMiddleware;
use Zend\Expressive\Middleware\NotFoundHandler; // This will be new
use Zend\Stratigility\Middleware\ErrorHandler;
use Zend\Stratigility\Middleware\OriginalMessages;

$app->pipe(OriginalMessages::class);
$app->pipe(ServerUrlMiddleware::class);
$app->pipe(ErrorHandler::class);
$app->pipeRoutingMiddleware();
$app->pipe(UrlHelperMiddleware::class);
$app->pipeDispatchMiddleware();
$app->pipe(NotFoundHandler::class);
We will note in the migration guide that if other middleware is defined in their
pipeline, they should port it into this in the appropriate order. Alternately,
they can use the utilities from the previous section to create the pipeline, so
long as they either add the error and not found handlers to the configuration,
or outside the calls to the configuration utilities.
Proposed default pipeline for new users

The following is the proposed default pipeline for the next minor version of the
skeleton:
use App\Action\HomePageAction;
use App\Action\PingAction;
use Zend\Expressive\Helper\UrlHelperMiddleware;
use Zend\Expressive\Helper\ServerUrlMiddleware;
use Zend\Expressive\Middleware\NotFoundHandler; // This will be new
use Zend\Stratigility\Middleware\ErrorHandler;
use Zend\Stratigility\Middleware\OriginalMessages;

$app->pipe(OriginalMessages::class);
$app->pipe(ServerUrlMiddleware::class);
$app->pipe(ErrorHandler::class);
$app->pipeRoutingMiddleware();
$app->pipe(UrlHelperMiddleware::class);
$app->pipeDispatchMiddleware();
$app->pipe(NotFoundHandler::class);

$app->get('/', HomePageAction::class, 'home');
$app->get('/api/ping', PingAction::class, 'api.ping');
The above essentially mimics the existing 1.0 workflow, but using a programmatic
paradigm. Users can then edit this in order to alter the pipeline or add routed
middleware.
Additionally, configuration will use the new application factory as described in
the previous section.
Requirements for Expressive 1.1

The following requirements changes will occur:

zendframework/zend-stratigility will be bumped to ^1.3.

The following new classes will need to be created:

Zend\Expressive\Middleware\ErrorResponseGenerator, which will implement
__invoke($e, ServerRequestInterface $request, ResponseInterface $response),
and accept the following constructor arguments:

boolean flag indicating development/debug mode
optional TemplateRendererInterface instance
optional string $template, defaulting to error::error


Zend\Expressive\Middleware\WhoopsErrorResponseGenerator, which will
implement the identical signature to the above, and accept the following to
its constructor:

Whoops\Run instance


Zend\Expressive\Middleware\NotFoundHandler, which will accept the following
constructor arguments:

required ResponseInterface instance (a response prototype)
optional TemplateRendererInterface instance
optional string $template, defaulting to error::404


Zend\Expressive\Container\ErrorResponseGeneratorFactory for generating the
ErrorResponseGenerator instance.
Zend\Expressive\Container\ErrorHandlerFactory, for creating a
Zend\Stratigility\Middleware\ErrorHandler instance, using the configured
ErrorResponseGenerator.
Zend\Expressive\Container\NotFoundHandlerFactory, for generating the
NotFoundHandler instance.
Zend\Expressive\ApplicationUtils, with the methods:

injectPipelineFromConfig(Application $application, array $config)
injectRoutesFromConfig(Application $application, array $config)


The following changes will be made to existing classes:

Zend\Expressive\Container\ApplicationFactory:

will need to check for the zend-expressive.programmatic_pipeline setting.

When present, a NoopFinalHandler will be provided as the final handler
to the Application instance, and not the configured FinalHandler
service.
When present, it will return the instance before doing any injection of
the pipeline or routes from configuration.


will use the new ApplicationUtils methods to inject pipeline and routed
middleware, eliminating the current set of internal methods.


Zend\Expressive\Application:

Update pipeErrorMiddleware() to emit a deprecation notice.


Zend\Expressive\MarshaMiddlewareTrait:

update marshalLazyMiddlewareService() as detailed under the "Middleware
piping" section above.


Requirements for Expressive Skeleton 1.1

The following changes will be made to the skeleton:

Update to Expressive 1.1.
Updating config/autoload/middleware-pipeline.global.php:

Add the following services to the dependencies:

ErrorHandler
ErrorResponseGenerator
NotFoundHandler


Remove the middleware_pipeline configuration.


Updating config/autoload/zend-expressive.global.php:

Add 'programmatic_pipeline' => true under the zend-expressive key.


Update all src/ExpressiveInstaller/Resources/config/routes-\*.php files:

Remove the routes configuration


Update src/ExpressiveInstaller/Resources/config/error-handler-whoops.php:

modify the final handler factory to instead reference the
ErrorResponseGenerator and new WhoopsErrorResponseGenerator factory.


Update public/index.php to use the pipeline as noted in the "Proposed
default pipeline for new users" section.

Documentation

We will need to update the documentation in the Expressive repository as
follows:

Detail the programmatic pipeline.
In the "configuration-driven" section, detail that:

You can make this the default by disabling the programmatic-pipeline
flag. Additionally indicate that for pre-1.1 versions, this was already
true.
Detail using the new ApplicationUtils class within a programmatic pipeline
in order to slurp in the pipeline and/or routes.


This information can be released as part of the Expressive 1.1 release, or
simply pushed to the master branch of the Expressive repository once the
Expressive Skeleton 1.1 release is ready.
Timeline

These changes can begin as soon as Stratigility 1.3 is tagged. From there, the
Expressive 1.1. changes must be completed first, followed by the skeleton
changes and documentation.
Looking forward to version 2.0

The above changes are made in order to be backwards compatible. However, we
should also provide a release that enables Stratigility 2.0, as that version is
highly optimized for http-interop middleware. Such a release would provide the
changes detailed below.
Middleware piping

First, we'd remove support for piping error middleware entirely.
Second, we'd update MarshalMiddlewareTrait::marshalLazyMiddlewareService() to
return a closure using the http-interop signature instead of the double-pass
signature. This would then use the composed respone prototype, or a new response
instance, when invoking double-pass middleware services; additionally, it would
create a closure around the provided delegate if it is not already callable.
Application factory

The application factory would no longer configure the pipeline or routing.
Additionally, we'd inject a response prototype during initialization.
Timeline

While we could do this immediately, my inclination is to wait until PSR-15 is
ratified before moving ahead on the 2.0 version. Alternately, we could wait a
specific period of time — e.g. 3 months — whereby if PSR-15 is not yet accepted,
we would issue a 2.0 release. We could then release a minor release later that
builds on a new Stratigility version that allows PSR-15 middleware, assuming
compatible signatures.