Skip to content

Instantly share code, notes, and snippets.

@rmccue
Last active April 9, 2017 17:12
Show Gist options
  • Star 11 You must be signed in to star a gist
  • Fork 2 You must be signed in to fork a gist
  • Save rmccue/722769379f4fc4148b1f to your computer and use it in GitHub Desktop.
Save rmccue/722769379f4fc4148b1f to your computer and use it in GitHub Desktop.

Integrating WP API into Core

Plan for integrating WP API into core

This document describes the process of integrating the WordPress JSON REST API ("API project") into the WordPress core codebase ("core").

Motivation

Accessing data remotely is a common goal for many WordPress users and developers. The ability to access and update data remotely is used by desktop publishing software, mobile applications, and in-browser applications. These users need to be able to access WordPress data in a standardised, interoperable manner.

Currently, core contains the XML-RPC API as a solution to this problem. The XML-RPC API contains issues which can cause problems for clients:

  • XML is a notoriously difficult format to parse. Implementation issues, plus the complexity of data representation can cause issues in clients; the WordPress iOS client at one point dedicated 40% of its codebase to cleaning up XML.

  • Server implementations have contained issues. XML is also a difficult format to generate, and numerous issues have appeared in the server-side implementation of XML-RPC in core. These include security issues, such as XXE, as well as encoding issues.

  • The RPC commands are a mixture of standardised and WordPress-specific APIs. This means that clients may be forced to learn multiple specifications in order to extract all data from the API.

Other solutions have been created to solve the shortcomings of the XML-RPC API, including the Jetpack JSON API, Thermal API and the [JSON API plugin][].

The API project provides the following benefits over the existing solutions:

  • Data serialization uses the JSON standard, which maps more directly to common data structures in most programming languages. JSON is standardized as ECMA-404 and RFC4627.

  • Communication with the API uses the HTTP 1.1 standard, available in almost every programming language. Representational State Transfer ("REST") semantics are used for interaction, which are available in the majority of HTTP clients, and compatibility is provided for those without this ability. HTTP is standardized as RFC2616.

  • The data serialization layer is independent of the data manipulation and accessing layer, enabling the JSON or HTTP semantics to be replaced with another protocol, such as MessagePack or Cap'n Proto.

  • The internal structure of the API allows use by regular PHP code. This enables using the data layer of the API for other uses, such as a future version of WP-CLI or internal form handlers.

  • The API is designed from the ground up for integration into core. This includes using core coding standards, liberal use of actions and filters, and full translatability.

  • Core data (that is, data types included with standard installations) is comprehensively covered by the API. This includes built-in post types and related metadata, as well as providing simple extensibility for the plugin and theme ecosystems.

  • The API relies on no external services. It includes methods for automatically locating the API given only a WordPress site address, as well as programatically assessing the availability of parts of the API.

Components

The API project is split up into roughly three parts:

  • The core API: This is the bulk of code and effort, and is the backbone for all future work on the API. This encompasses the code responsible for responding to API requests, as well as helpers around the code.
  • Reference clients: The API project also includes several reference clients. These include a Javascript client, a PHP client, and a WP-CLI client based on the general PHP client. These clients represent our idea of how users should interact with the API, but are naturally separate.
  • Authentication schemes: In addition to the core API, several authentication schemes are included as part of the API project. These are the OAuth authentication scheme, and the Basic authentication scheme. These are general WordPress plugins, and are not dependent on the core API (and vice versa).

Considered for Integration

Of these sections, the following components are designed for and aimed for integration with core:

  • The core API: The core API represents our effort thusfar at creating a best-of-class API for data managed by core. Whilst this effort is far from offering a "complete" coverage of core data, it is a comprehensive representation of the majority of data. It is expected over time that this will continue to grow to encompass all the data in core.

    The core API is the basis of the project, and other components depend on the core API to be able to do useful work.

    The core API itself has two parts, both to be considered for integration:

    • API infrastructure: This is the code responsible for managing API requests and mapping them to internal handlers, as well as responding to requests. This code handles the "JSON REST" part of the API, and forms the backbone of all other core API code. All semantics for handling HTTP requests and JSON data lives in the infrastructure.

    • Core endpoints: This is the code responsible for turning requests into responses by fetching and generating data. This code maps to internal WordPress data structures, and generally consists of normal PHP functions that can be called with parameters, and return data.

  • Javascript reference client: The Javscript client project is one of the API reference clients, and is designed for integration with core. The Javascript client is designed specifically for plugins and themes, and uses the Backbone and Underscore libraries included with WordPress.

    The Javascript client is explicitly designed to simplify integration with the API for Javascript running on the same server (that is, plugins and themes). It is hoped that this client will offer a simple yet useful API encompassing all data available from the core API.

    Without the Javascript client included in core, integrating with the API will be more difficult for developers. Developing with the API will require having knowledge of the API, rather than the ability to learn an abstraction in the existing language.

    However, no other code considered for integration depends on the Javascript client. The client is merely an addition on top of the core API.

  • OAuth authentication scheme: The OAuth authentication scheme is one of the authentication schemes, and is designed for integration with core. This scheme enables external clients (that is, those not running on the server) to interact with the API in a safe, secure way.

    The OAuth 1.0a protocol is an authorization protocol that allows an external client to make requests on behalf of a user. It uses client-specific tokens that ensure the integrity and security of user credentials, and discourage clients from requiring plaintext credentials. The protocol is standardised as RFC5849 by the IETF, and is maintained by the OAuth Working Group.

    The OAuth authentication scheme is independent of the API, and relies only on core's ability to add custom authentication handlers, introduced in #26706 and released with WordPress 3.9. The scheme contains extra code to integrate with the core API, however this is implemented through hooks and filters, and the scheme can operate independently.

Excluded from Consideration

The following projects are maintained by the API team, however are not aimed for integration with core:

  • PHP reference client
  • WP-CLI client
  • HTTP Basic Authentication authentication scheme

Rationale

Core API

The following points summarize the design of the core API:

  • The basic unit in the API is a PHP function. That is, requests to the API map to a function rather than an object or class.

  • Integrating with the API is done through use of hooks and filters.

  • Regular expressions are used to describe "routes" (the path relative to the API root) and map to "endpoints" (the handler function). This is in contrast to some API systems that use domain-specific syntax, such as Rails-style routing syntax:

    :controller(/:action(/:id))
    

    Routing syntax is instead regular regular expressions, with named capture groups used for specific variables:

    /posts/(?P<id>\d+)
    
  • Request parameters directly map to endpoint parameters. That is to say, GET or POST parameters, plus capture groups from the URL are used to provide the function arguments.

    For example, a request to /posts/4?context=edit would match the following route:

    /posts/(?P<id>\d+)
    

    This would give the following data, represented as a PHP array:

    array(
        'id' => 4,
        'context' => 'edit'
    )
    

    This data could then be sent to a function which takes those parameters:

    function get_post( $id, $context = 'view' )
    

    This would result in the equivalent of the following function call:

    $result = get_post( 4, 'edit' );
    

    This direct mapping format was chosen over other formats as it ensures endpoints are inherently usable internally. It encourages the use of readable endpoints (and the avoidance of allowing generic arguments), which also helps to generate cleaner requests.

    Directly mapping formats also allows using PHP's native constructs for default parameter values, and specifying parameters as required by excluding this default. This again encourages internal reusability.

  • Endpoints are inherently composable, due to the nature of mapping parameters to function arguments. As an example, the edit post endpoint internally calls the post retrieval endpoint rather than recreating the code:

    function edit_post( $id, $data, $_headers = array() ) {
    
        // Snipped for readability
        // [...]
    
        $retval = $this->insert_post( $data );
        if ( is_wp_error( $retval ) ) {
            return $retval;
        }
    
        return $this->get_post( $id );
    }

Javascript reference client

The following points summarize the design of the Javascript client:

  • The client should act as a drop-in library for themes and plugin developers to use. It shouldn't be particularly opinionated in favour of a certain workflow or methodology.

  • Backbone and Underscore are used for the implementation of the client. These pieces only include the Backbone Model/Collection objects, not the Controller or View objects. This falls in line with being unopinionated, and doesn't require users to have a huge amount of knowledge of a framework (as Backbone is relatively simple).

  • The client uses WordPress's built-in cookie authentication, but requires a CSRF token in the form of a WP nonce. This is automatically handled when enqueuing the client script.

OAuth authentication scheme

The following points summarize the design of the OAuth authentication scheme:

  • The OAuth scheme uses the OAuth 1.0a protocol. While a newer protocol (OAuth 2) is available, its use currently requires SSL. While extensions to the protocol are in development for non-SSL use, these are not yet finalised and are not expected to be ready in time to ship with the API.

    In addition, the OAuth 1.0a protocol has implementations in many popular languages, due to its popularity and longevity. A newer specification would require time to mature to the point where this is the case.

    Requiring SSL for authentication is a no-go for core, as it would significantly reduce compatibility with WordPress installations.

  • OAuth does not require providing plaintext credentials to clients, but rather uses tokens for authorization. (Technically, OAuth acts as an authorization protocol, as the user authorizes the client to act on their behalf rather than authenticating as the user.)

    Plaintext credentials cannot be converted to OAuth tokens; a process used by applications such as Twitter with the name XAuth. This is an intentional design decision, as it ensures that users are never encouraged to enter their login details outside of their site. This reduces the likelihood of users having their details compromised via a phishing scheme. An XAuth scheme may be added in the future if user experience concerns outweigh this.

  • OAuth clients must register with the site before having access. Access tokens are local to the site, as are client identifiers (client key and secret key). While this removes the dependency on a central repository, it also means that unlike a centralised service, clients cannot currently be registered across all installations.

    It is expected in the future that a central client registry will be created as another layer on top of this existing OAuth capability. The current scheme does not hinder this functionality, nor does it help.

Integration Plan

It is expected that integration with core will take place over the time allotted for the 4.0 merge window. Each component can be integrated independently, and thus each has separate plans.

Core API

  1. Merge the API lib/ directory into wp-includes/api/
  2. Copy the API plugin.php file to wp-includes/api.php
  3. Remove plugin-specific implementation details (activation/deactivation hooks)
  4. Move hooking calls (add_action/add_filter) to wp-includes/default-filters.php
  5. Change generic utility functions (get_timezone/get_avatar) to global functions in the general WP code
  6. Merge the API tests/ directory into the core testing directory
  7. Rename all instances of WP_JSON to WP_API
  8. Rename the API root from /wp-json to /wp-api

Javascript reference client

  1. Merge the Javascript client source into wp-includes/api.js
  2. Register the client Javascript code in wp-includes/script-loader.php, including the nonce object code

OAuth authentication scheme

  1. Copy the OAuth lib/ directory into wp-includes/oauth/
  2. Copy the OAuth oauth-server.php file to wp-includes/oauth-server.php
  3. Remove plugin-specific implementation details (activation/deactivation hooks)
  4. Move hooking calls (add_action/add_filter) to wp-includes/default-filters.php

Open Issues

There are still outstanding issues with all parts of the API, as it is still under continuous development:

  • The core API does not cover every piece of data available in core. This is a long-term issue, and is only related to the amount of time required to implement this. Currently unimplemented data includes menus, widgets, options, and further multisite data. This can be shipped in a future release.

  • The core API does not have 100% code coverage in unit tests. The 1.1 release of the API will heavily feature this with a testing sprint.

  • The Javascript client does not currently cover the entire API. Work is progressing quickly in this area with recent efforts, and this should be completely ready well before the release candidate period for 4.0.

  • The OAuth server includes minimal UI, and does not include UI for registering new clients. This is due to a lack of frontend work thusfar, and still requires work.

Backwards Compatibility

As these components are additions to the existing core feature set, no major backwards compatibility issues are expected. Renaming the API code during the merge window will ensure no conflicts between the feature-as-a-plugin and the core code.

While it's possible that existing sites could have /wp-api registered as a custom route, this is regarded as unlikely, as it begins with the conventionally-reserved wp- prefix.

Upon merge, the feature-as-a-plugin will be stripped down to a compatibility layer, ensuring that code written for the feature-as-a-plugin continues to work whilst also freeing core from any backwards compatibility concerns around it.

Forwards Compatibility

The API is expected to naturally evolve over time. While the current core API does not make significant provisions towards this, it is expected that future concerns can be handled by using HTTP headers and will not require a significant redesign of the internal structure of the API.

@mattisbusycom
Copy link

Hey, I can help with this if any of the WP Core team wants someone new on board -- I'm heavily experienced in large scalel API Design and Refactoring apps :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment