pfrazee/web-data-protocols-proposal.md

## web-data-protocols-proposal.md

      
    Raw
  

              web-data-protocols-proposal.md
            
          
    Web Data Protocols ("Protos") Proposal


If you're in a rush: Skip the background material and see the proposal

I recently published a minimal Dat Identity Spec which establishes how we plan to move forward with user identities in Beaker. It establishes:

Identities are dat Websites (similar to Indie-Web)
Identifiers are domain names
User data is files on the dat sites
The browser will provide UIs and APIs to help manage access to the identities by apps. This will include a "sign in" metaphor which provides read & write access to the data.

Overall I'm happy with the spec and its reception! This gist is about a next step: data semantics.
The data semantics challenge

The spec mentions the need for high-level semantics for userdata in order to give the user a clear sense of what applications are doing. For example, users need to be told "This app wants to manage your Fritter contacts" rather than "This app wants write-access to /data/fritter/contacts."
This is important! Users need to understand what their applications are doing to their data and they need to be able to make informed decisions about permissions. Beakerverse apps also need to be able to share data between themselves, and that coordination is quite important. An app needs to be able to look in a dat-identity and say, "Yes I understand this data, let's go!" or "No, I'm going to be starting from scratch here." Ideally this will happen with minimal kludge or bugginess.
What's wrong with files?

Files are very powerful and simple, but without metadata they aren't user-friendly. They include very little information about what they contain or what purpose they serve. They have no way to describe possible actions except at the "file" or "blob" level; you can describe access in terms of reading or writing chunks of the file, but you can't describe access in terms of modifying the objects or object-relationships it contains. For that, you really need higher level semantics.
Coordination between apps is also a big challenge. When two apps are trying to interact with the same data, you get complicated questions about trust and correctness. You have to ask:

Do the apps totally understand each others' data?
Is it possible a misunderstanding could create bugs?
How could the shared ownership of the spec be gamed in order to attack the user?
How do we coordinate changes to the schemas which naturally occur as each app matures at their own pace?

Historically, these kinds of questions have been answered by using standards, but standards processes can be extremely slow and irritating to developers. We want to build apps, not committees! So what's the strategy for solving cross-app coordination?
Could the browser do it?

In winter of 2017/18, we proposed adding high-level data semantics to the browser itself. We would create a set of standard data formats, schemas, and APIs which everybody shares. This was somewhat reminiscent of Schema.org, in that it would try to create "one entology to rule them all."
Access would be mediated by Web APIs that wrap the identity-dat's filesystem. For instance:
var user = (await UserSession.fetch()).profile

await user.feed.list()
await user.feed.post({text: 'Hello world!'})

await user.friends.follow('dat://bob.com')

await user.photoAlbums.list()
await user.photoAlbums.create()
// etc
Because the browser managed the semantics, this solution made it very easy to explain to the user what apps are trying to do. If the app wanted to add photos to your photo album, it had to use the photoAlbums API, and the browser knew exactly what permission to ask from the user.
Perhaps unsurprisingly, this proposal got a very bad reception. Developers rightly pointed out that this would politically centralize the development of apps, because the schemas & formats would all go through the browser. It would effectively politicize everything, and put the standards bodies (led by the browsers) at the helm. We thought the convenience of the builtin APIs and a focus on completeness would offset that concern, but were resoundingly told no.
After that a bad reception, I started to think about how we could maintain the good parts while solving the political centralization. What if we could describe the data semantics and permissioning in userland? That approach has ultimately led to this proposal.
Proposal: Web data protocols ("Protos")

A "Web data protocol" (or "Proto") is a set of machine-readable specs which describe a data space. They are effectively a data model description. Protos provide the browser with instructions to manage user data with fine-grained permissioning.
As an example, let's suppose we wanted to create a "Proto" for the Fritter app. Fritter's API is described here.
We'll publish our Proto at dat://fritter.com. The files will look like this:
/dat.json    - standard metadata about this site
/proto.json  - information about the data protocol
/api.js      - an api module

Our consuming application will indicate the Proto it wants during signin, as well as the permissions it'll want from the Proto:
await session.requestSignin({
  protos: [{
    url: 'fritter.com',
    permissions: ['feed:read', 'feed:create-posts', 'profile:read']
  }]
})
When the signin flow occurs, the browser downloads dat://fritter.com/proto.json and loads the definitions. It will find a document which describes each of the permissions as file operations, as well as in terms that the user can understand.
The proto.json file would look something like this:
{
  "actions": {
    "feed:read": {
      "title": "Read Feed",
      "description": "Read posts from your feed.",
      "files": [{
        "match": "/posts",
        "ops": ["stat", "readdir"]
      }, {
        "match": "/posts/*.json",
        "ops": ["stat", "readFile"]
      }]
    },
    "feed:create-posts": {
      "title": "Create Posts",
      "description": "Publish new posts under your identity.",
      "requires": ["feed:read"],
      "files": [{
        "match": "/posts/*.json",
        "ops": ["writeFile"],
        "constraints": ["create-only"]
      }]
    },
    "profile:read": {
      "title": "Read Profile",
      "description": "Read your profile data and friends list.",
      "files": [{
        "match": "/profile.json",
        "ops": ["readFile"]
      }]
    },
    "profile:write": {
      "title": "Update Profile",
      "description": "Update your profile data and friends list.",
      "requires": ["profile:read"],
      "files": [{
        "match": "/profile.json",
        "ops": ["writeFile"]
      }]
    }
  }
}
Based on the permissions requested above in requestSignin() and based on this proto.json definition, the user would be prompted to okay the following permissions:

Read Feed. Read posts from your feed.
Create Posts. Publish new posts under your identity.
Read Profile. Read your profile data and friends list.

This is because the app requested 'feed:read', 'feed:create-posts', and 'profile:read' permissions. The browser mapped those permissions to the defined actions and used their labels to prompt the user.
If accepted by the user, the browser will record the file-access permissions described for each action. It will also create a new folder for the Fritter Proto if it doesn't already exist (/data/fritter.com).
After being granted the permissions, the app will only be able to run file operations which match the described operations. That means, for instance, it'll be possible to readdir('/data/fritter.com/posts'), but it won't be possible to writeFile('/data/fritter.com/profile.json') because that operation doesn't match the granted actions' filters. ('profile:write' wasn't requested.)
At this point, as a final step, the application will import the API module. This provides high level APIs which wrap the file actions.
import * as LibFritter from 'dat://fritter.com/api.js'

var fritter = new LibFritter()
fritter.setUser(session.profile)
await fritter.social.listFriends(session.profile)
This final step is important for providing the semantics of the data formats. The api.js is designed to translate high level operations to file operations. Any app which consumes the API module will use the exact same semantics, and so they'll be compatible with each others' usages!
(Behind the scenes, the imported LibFritter API will just be reading and writing files. Technically it's not necessary, but it's advisable because it helps ensure the apps are following the same rules.)
Other instructions could be included in a Proto. For instance, a Proto could include instructions to help with version migrations, or metadata for helping the browser describe the files' contents to the user in other interfaces. This proposal focuses on describing permissions because we know it's a key requirement for Protos.
Mechanics

Protos are effectively bundles of global business logic that apps share. By managing a folder on the user-dat, a Proto takes ownership of that folder's semantics and provides trustable management of the user data.
Protos are published at dat:// URLs and are identified by their domain name. Applications use Protos by pointing to the Proto's URL. This directs the browser to download the Proto dat and read its instructions for managing the user's data.
Once loaded and activated, the browser creates a dedicated directory for the Proto's data in the active userdat. For example, a Proto at dat://fooproto.com might be given the /data/fooproto.com folder in the userdat.
Protos describe their behaviors in the /protos.json file. It exports a set of actions which include:

User-friendly descriptions
File operation filters

Optionally (but recommended) a Proto can include javascript modules for consuming apps to import, providing high level APIs to complement the Proto.

FAQ

How do Protos solve the issue of trust?

Protos are motivated by complexities around trust and authority. Questions like:

What do the user's files contain?
Which applications can be trusted to describe them?
Who do we trust to describe permissions around accessing that data?
etc.

To help clarify this challenge, let's consider a hypothetical:

If app A tells you that /foo.json contains a cake recipe, and app B tells you it contains car schematic, which is telling the truth?

Obviously you could open /foo.json manually and try to suss out the content yourself, but this isn't always so easy for a user to do. Sometimes files are jumbles of structured data. The disputes can also be much more subtle: it might be a dispute about the right way to encode information, or about which attributes are required. It's very hard for the browser/network/system to answer this dispute. We need a way to ultimately rule in favor of app A or app B.
A natural solution might be to give priority to the creating app. For instance, if app A was the original author of /foo.json, then surely it's the authority on the file! The problem with that solution is, there's no correlation between "being first" and "being right" in a shared data space. Consider what might happen around common file-paths such as /profile.json. Every other app will want to write to that file! Which one is right? (In a way, they all are, because we have no strong definition of "right" in this setup.)
This challenge extends into security as well. Not only do we need a trustworthy description of the data -- we need a trustworthy description of the permissions for accessing the data! Since we need permissions to provide security, it's obviously important to be able to trust the permission descriptors.
Protos solve all of these challenges by forming an authority model. A Proto is an authority over its own data space. Consuming apps have to follow the Proto's decisions. Therefore there's much less potential for disputes; within a Proto's data space, the Proto has the last word.
Why use static definitions instead of just callable JS workers?

Q: The /proto.json file will need to describe possible access patterns in permissions, and there's a limit to what can be described this way. Wouldn't it be easier to have the Proto run a ServiceWorker or something similar? Consuming applications could then simply call to the worker via an exported RPC API!
A: This is a good question that I'm still debating myself. Let's call this "Static" vs "Live," where "Static" uses description JSON files (as described in this proposal) and "Live" uses JS provided by the Proto to act as a service.

Live is more flexible and requires much less to be specced by the browser.
Static is lower overhead. No processes need to be run.
Static gives the browser more opportunity to get involved (e.g. to provide the perms prompts).
Static lets us focus on specific use-cases, whereas Live becomes a challenge of process-management and APIs for the Proto scripts to handle things like perms.
Static might be easier for people to develop because it's more structured.

My current thinking is that we should start with Static and let Live become part of the solution if we feel it's needed. It doesn't hurt us to combine the approaches, and it's always an option to deprecate the Static solutions if we find they're not effective.