These are just some thoughts I sketched out to try to figure out what the lowest-level of the network stack could look like. I wanted to think about where the actual security boundary was, and build an API right above it. Some outstanding issues involve whether preflight should be exposed and how streams should deal with metadata. It's extremely rough so don't judge me :)
This is the lowest-level API in the network fetching stack.
Specifically:
- If you want to make a request that requires a CORS preflight, you need to populate the preflight cache manually first
- The Request object doesn't automatically populate the
OriginorRefererheaders, but does restrict their values - There is no special support for deserializing JSON or XML. Instead, it provides a byte stream that you can pipe into deserializers.
- You need to add credentials (cookies, basic auth) yourself. Support for opaque credentials is available for cross-origin, credentialed requests.
- There is no redirect support. If you want to follow redirects, you need to follow redirects yourself. This makes the semantics of cross-origin redirection clear.
- It supports cross-origin requests both with and without CORS. If you don't enable CORS, the response you get will be opaque, but can be piped into stream sinks like an
imageorscripttag (within the constraints of CSP).
There should be a higher-level API that makes it easy to perform a network fetch with normal semantics and options.
class Request {
string get url();
set url(ToString value);
Headers get headers();
set method(ToString value);
string get method();
string get origin();
boolean get isCrossOrigin();
boolean get requiresPreflight();
}class Headers {
mixin MapUtils;
string get(ToString header);
set(ToString header, ToString value);
}class StreamPromise<DOMOutStream> extends DOMPromise<DOMOutStream> {
// act like a stream, with `pipe`, etc.
// this stream behaves like the resolved stream
mixin DOMOutStream;
}
class Response {
mixin DOMOutStream;
number get status();
string get statusMessage();
string get location();
Headers get headers();
boolean get isSuccess();
boolean get isRedirect();
}module "web/network/fetch" {
options fetchOptions = { Boolean cors=false };
export StreamPromise<Response> function fetch(ToString url, fetchOptions options);
export Promise<Response> function fetch(Request request, fetchOptions options);
export default fetch;
export DOMPromise<OpaqueResponse> function preflight(ToString url);
export DOMPromise<OpaqueResponse> function preflight(Request url);
}Streams and Promises both handle asynchronous operations, and users will want to treat a response as a single asynchronous object.
In the case of the Response, object, you will want the ability to plug the response stream into a stream sink immediately, and not have to wait for the response promise to resolve first. However, other aspects of the response, like the status, really do need to be part of a resolved promise.
There are a few approaches that can solve this problem:
var promise = fetch(url);
promise.stream.pipe(imageElement);
promise.then(function(response) {
// look at response.status
});- Pros: simple semantics, can be implemented with a vanilla promise
- Cons: privileges promise over stream, taxes the piping case
var response = fetch(url);
response.pipe(imageElement);
response.status // undefined or throws
response.then(function(response) {
// look at status
});- Pros: can use the response either as a promise or a stream, as needed
- Cons: requires a self-resolving promise, response semantics pre-resolution of the promise
It delegates its pipe to the eventually resolved stream.
var promise = fetch(url);
promise.pipe(imageElement);
promise.then(function(response) {
// look at status
});- Pros: can use the response either as a promise or a stream
- Cons: Requires a Promise subclass, more complex implementation
fetch(url).then(function(response) {
response.pipe(imageElement);
// look at status
});- Pros: Clearest semantics
- Cons: Piping must be deferred until promise resolution, input stream isn't aware that a stream will eventually be connected
Option 4 looks okay at first glance, but it makes composition trickier, and makes it harder to use the return value of fetch in generic abstractions that expect Stream=>Stream connections.
Ideally, you could treat the return value from fetch as a stream when a stream was needed and a promise for a response when that was required.
Option 3 looks like the best mix of tradeoffs to me. You can think of it as sugar for the common case of Option 4. The JSIDL above and the examples below assume Option 3.
Some parts of CSP work by disallowing network requests that come from specific triggers.
With this factoring, the code that issues the request would refuse to issue the fetch request if banned by CSP. For example, an <img> tag could be seen as issuing a fetch in its element's readyCallback. If the src in an <img> tag was not allowed by the img-src directive in CSP, the readyCallback would reject the request.
The only CSP directive directly relevant to fetch is default-src. Any request blocked by default-src would be blocked at the fetch level.
If all network requests are seen as bottoming-out in fetch, this factoring allows components of the system to handle their own CSP semantics, while also ensuring the default-src works globally.
This document currently assumes that the operation of the HTTP cache is transparent to fetch.
At the very least, it would be helpful to be able to learn whether a response was fetched from the cache. For example, this would allow a site to avoid re-rendering UI whose conceptual "model" has not changed.
It might also be useful to have more direct control over the cache flow. For example, the primitive might simply resolve the promise with a 304 response, and offer HTTP caching primitives that the user of fetch could use.
Similarly, this document assumes that the If-Not-Modified and If-Modified-Since headers will be automatically added by the caching layer. It might make sense to expose the caching information as a primitive and require that the requester add it manually at this level of the abstraction.
That said, in both cases, opaque responses for cross-domain requests complicate things, and may suggest that the HTTP cache should be considered transparent, with an extra flag indicating that the cache was used.
This section includes some examples of how to build up higher level abstractions from the low-level fetch API.
import { preflight, fetch } from "web/network/fetch";
import { parseStream } from "web/parsers/json";
var url = "http://example.com/articles.json";
preflight(url).then(function() {
var response = fetch(url, { cors: true });
return parseStream(response);
}).then(function(json) {
// use JSON
});An abstraction:
module "web/network/cors_fetch" {
import { preflight, fetch } from "web/network/fetch";
export function fetch(request) {
var responseStream;
return preflight(url).then(function) {
return fetch(url, { cors: true });
});
}
}
module "web/parsers/json" {
export function parseStream(stream) {
Promise.when(stream, function() {
// lazy implementation, reads until EOF and then parses
return stream.read();
}).then(function(blob) {
return JSON.parse(blob.toString());
});
}
}
module "web/network/fetch_json" {
import { fetch } from "web/network/cors_fetch";
import { parseStream as parseJSON } from "web/parsers/json";
export default function(url) {
return parseJSON(fetch(url).stream);
}
}It would also be straight-forward to implement the current XHR API, or a high-level fetch API.
import { cookieForURL } from "web/network/cookies";
var url = "http://example.com/articles.json";
var request = new Request(url, {
method: "GET"
});
// This may be an OpaqueCookie
request.headers.set("Set-Cookie", cookieForURL(url));import fetch from "web/network/fetch";
fetch(url).then(function(response) {
if (response.isRedirect) {
return fetch(response.headers.get('location'));
}
return response;
}).then(function(response) {
// use the response
});You could also implement something that followed N redirects:
module "web/network/redirect_following_fetch" {
import rawFetch from "web/network/fetch";
export default function fetch(request, options) {
return rawFetch(url).then(function(response) {
if (response.isRedirect) {
return fetch(response.headers.get('Location'));
} else {
return response;
}
});
}
}import fetch from "web/network/fetch";
// HTMLImageElement is an InStream that passes streams to
// imageBitmapStream, which is a primitive that can convert
// a byte stream into an ImageBitmapStream. See below.
fetch("http://other.example.com/image.png").pipe(imageElement);import { imageBitmapStream, HTMLCanvasElement } from "web/html/canvas";
class CustomImage extends HTMLCanvasElement {
readyCallback() {
// 1. HTMLCanvasElement implements InStream
// 2. imageBitmapStream is an internal primitive
// that can accept an OpaqueStream and produce
// an OpaqueBitmapStream
fetch(this.src).pipe(imageBitmapStream).pipe(this);
}
}
document.register('custom-image', CustomImage);