Skip to content

Instantly share code, notes, and snippets.

@wycats
Last active February 24, 2020 06:29
Show Gist options
  • Save wycats/cf73dd4c974352fcb767 to your computer and use it in GitHub Desktop.
Save wycats/cf73dd4c974352fcb767 to your computer and use it in GitHub Desktop.
Fetch API

These are just some thoughts I sketched out to try to figure out what the lowest-level of the network stack could look like. I wanted to think about where the actual security boundary was, and build an API right above it. Some outstanding issues involve whether preflight should be exposed and how streams should deal with metadata. It's extremely rough so don't judge me :)

Scope

This is the lowest-level API in the network fetching stack.

Specifically:

  • If you want to make a request that requires a CORS preflight, you need to populate the preflight cache manually first
  • The Request object doesn't automatically populate the Origin or Referer headers, but does restrict their values
  • There is no special support for deserializing JSON or XML. Instead, it provides a byte stream that you can pipe into deserializers.
  • You need to add credentials (cookies, basic auth) yourself. Support for opaque credentials is available for cross-origin, credentialed requests.
  • There is no redirect support. If you want to follow redirects, you need to follow redirects yourself. This makes the semantics of cross-origin redirection clear.
  • It supports cross-origin requests both with and without CORS. If you don't enable CORS, the response you get will be opaque, but can be piped into stream sinks like an image or script tag (within the constraints of CSP).

There should be a higher-level API that makes it easy to perform a network fetch with normal semantics and options.

Dependencies

Request

class Request {
  string get url();
  set url(ToString value);

  Headers get headers();

  set method(ToString value);
  string get method();

  string get origin();

  boolean get isCrossOrigin();
  boolean get requiresPreflight();
}

Headers

class Headers {
  mixin MapUtils;

  string get(ToString header);
  set(ToString header, ToString value);
}

Response

class StreamPromise<DOMOutStream> extends DOMPromise<DOMOutStream> {
  // act like a stream, with `pipe`, etc.
  // this stream behaves like the resolved stream
  mixin DOMOutStream;
}

class Response {
  mixin DOMOutStream;

  number get status();
  string get statusMessage();
  string get location();
  Headers get headers();

  boolean get isSuccess();
  boolean get isRedirect();
}

Fetch

module "web/network/fetch" {
  options fetchOptions = { Boolean cors=false };

  export StreamPromise<Response> function fetch(ToString url, fetchOptions options);
  export Promise<Response> function fetch(Request request, fetchOptions options);

  export default fetch;

  export DOMPromise<OpaqueResponse> function preflight(ToString url);
  export DOMPromise<OpaqueResponse> function preflight(Request url);
}

Open Questions

Streams and Promises

Streams and Promises both handle asynchronous operations, and users will want to treat a response as a single asynchronous object.

In the case of the Response, object, you will want the ability to plug the response stream into a stream sink immediately, and not have to wait for the response promise to resolve first. However, other aspects of the response, like the status, really do need to be part of a resolved promise.

There are a few approaches that can solve this problem:

1. The response promise has a stream hung off of it.

var promise = fetch(url);
promise.stream.pipe(imageElement);
promise.then(function(response) {
  // look at response.status
});
  • Pros: simple semantics, can be implemented with a vanilla promise
  • Cons: privileges promise over stream, taxes the piping case

2. The response is a self-resolving promise

var response = fetch(url);

response.pipe(imageElement);
response.status // undefined or throws

response.then(function(response) {
  // look at status
});
  • Pros: can use the response either as a promise or a stream, as needed
  • Cons: requires a self-resolving promise, response semantics pre-resolution of the promise

3. The response promise is a stream

It delegates its pipe to the eventually resolved stream.

var promise = fetch(url);

promise.pipe(imageElement);

promise.then(function(response) {
  // look at status
});
  • Pros: can use the response either as a promise or a stream
  • Cons: Requires a Promise subclass, more complex implementation

Suck it up and tell people to pipe in the callback

fetch(url).then(function(response) {
  response.pipe(imageElement);
  // look at status
});
  • Pros: Clearest semantics
  • Cons: Piping must be deferred until promise resolution, input stream isn't aware that a stream will eventually be connected

Option 4 looks okay at first glance, but it makes composition trickier, and makes it harder to use the return value of fetch in generic abstractions that expect Stream=>Stream connections.

Ideally, you could treat the return value from fetch as a stream when a stream was needed and a promise for a response when that was required.

Option 3 looks like the best mix of tradeoffs to me. You can think of it as sugar for the common case of Option 4. The JSIDL above and the examples below assume Option 3.

CSP

Some parts of CSP work by disallowing network requests that come from specific triggers.

With this factoring, the code that issues the request would refuse to issue the fetch request if banned by CSP. For example, an <img> tag could be seen as issuing a fetch in its element's readyCallback. If the src in an <img> tag was not allowed by the img-src directive in CSP, the readyCallback would reject the request.

The only CSP directive directly relevant to fetch is default-src. Any request blocked by default-src would be blocked at the fetch level.

If all network requests are seen as bottoming-out in fetch, this factoring allows components of the system to handle their own CSP semantics, while also ensuring the default-src works globally.

HTTP Cache

This document currently assumes that the operation of the HTTP cache is transparent to fetch.

At the very least, it would be helpful to be able to learn whether a response was fetched from the cache. For example, this would allow a site to avoid re-rendering UI whose conceptual "model" has not changed.

It might also be useful to have more direct control over the cache flow. For example, the primitive might simply resolve the promise with a 304 response, and offer HTTP caching primitives that the user of fetch could use.

Similarly, this document assumes that the If-Not-Modified and If-Modified-Since headers will be automatically added by the caching layer. It might make sense to expose the caching information as a primitive and require that the requester add it manually at this level of the abstraction.

That said, in both cases, opaque responses for cross-domain requests complicate things, and may suggest that the HTTP cache should be considered transparent, with an extra flag indicating that the cache was used.

Building Up

This section includes some examples of how to build up higher level abstractions from the low-level fetch API.

Cross-Origin JSON Fetches

import { preflight, fetch } from "web/network/fetch";
import { parseStream } from "web/parsers/json";

var url = "http://example.com/articles.json";

preflight(url).then(function() {
  var response = fetch(url, { cors: true });
  return parseStream(response);
}).then(function(json) {
  // use JSON
});

An abstraction:

module "web/network/cors_fetch" {
  import { preflight, fetch } from "web/network/fetch";

  export function fetch(request) {
    var responseStream;

    return preflight(url).then(function) {
      return fetch(url, { cors: true });
    });
  }
}

module "web/parsers/json" {
  export function parseStream(stream) {
    Promise.when(stream, function() {
      // lazy implementation, reads until EOF and then parses
      return stream.read();
    }).then(function(blob) {
      return JSON.parse(blob.toString());
    });
  }
}

module "web/network/fetch_json" {
  import { fetch } from "web/network/cors_fetch";
  import { parseStream as parseJSON } from "web/parsers/json";

  export default function(url) {
    return parseJSON(fetch(url).stream);
  }
}

It would also be straight-forward to implement the current XHR API, or a high-level fetch API.

Credentials

import { cookieForURL } from "web/network/cookies";

var url = "http://example.com/articles.json";

var request = new Request(url, {
  method: "GET"
});

// This may be an OpaqueCookie
request.headers.set("Set-Cookie", cookieForURL(url));

Redirection

import fetch from "web/network/fetch";

fetch(url).then(function(response) {
  if (response.isRedirect) {
    return fetch(response.headers.get('location'));
  }
  return response;
}).then(function(response) {
  // use the response
});

You could also implement something that followed N redirects:

module "web/network/redirect_following_fetch" {
  import rawFetch from "web/network/fetch";

  export default function fetch(request, options) {
    return rawFetch(url).then(function(response) {
      if (response.isRedirect) {
        return fetch(response.headers.get('Location'));
      } else {
        return response;
      }
    });
  }
}

CORS-less Cross-Domain Requests

import fetch from "web/network/fetch";

// HTMLImageElement is an InStream that passes streams to
// imageBitmapStream, which is a primitive that can convert
// a byte stream into an ImageBitmapStream. See below.
fetch("http://other.example.com/image.png").pipe(imageElement);

Simple Image Tag

import { imageBitmapStream, HTMLCanvasElement } from "web/html/canvas";

class CustomImage extends HTMLCanvasElement {
  readyCallback() {
    // 1. HTMLCanvasElement implements InStream
    // 2. imageBitmapStream is an internal primitive
    //    that can accept an OpaqueStream and produce
    //    an OpaqueBitmapStream
    fetch(this.src).pipe(imageBitmapStream).pipe(this);
  }
}

document.register('custom-image', CustomImage);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment