Hypermedia question

hypermedia.md

People have been talking about including hypermedia with REST Resources, but there are very few public examples that implement all of it. One common technique is to add *_url attributes:

{ "id": 1
, "self_url": "/issues/1"
, "comments_url": "/issues/1/comments"
}

This only gives you room to put a URL. Because of this, I've been leaning towards HAL:

{ "id": 1
, "_links":
  { "self": {"href": "/issues/1"}
  , "comments": {"href": "/issues/1/comments"}
  , "close": {"href": "/issues/1/close", "method": "post"}
  }
}

Should hypermedia tell you when you have permissions to access those resources?

// admin permissions
{ "id": 1
, "_links":
  { "self": {"href": "/issues/1", "method": "get,patch,delete"}
  , "close": {"href": "/issues/1/close", "method": "post"}
  }
}

// read-only permissions, no access to close/update issues
{ "id": 1
, "_links":
  { "self": {"href": "/issues/1", "method": "get"}
  }
}

The HAL spec doesn't mention method properties at all. But, I think letting a client know the difference between an Issue I can edit or close is very useful. Is it any better if I have a separate relation for each action?

{ "id": 1
, "_links":
  { "self": {"href": "/issues/1", "method": "get"}
    "edit": {"href": "/issues/1", "method": "patch" }
    "delete": {"href": "/issues/1", "method": "delete" }
  , "close": {"href": "/issues/1/close", "method": "post"}
  }
}

Finally, is the "close" relation even appropriate? Or should I assume clients know they can close Issues by setting "state" to "closed" (which is how the GitHub Issues API works)?

I actually had a very similar conversation with Mike Kelly about permissions several months ago. His suggestion was to have a separate resource that would explain the role/permissions and then just always include the links. Part of the problem with varying links based on permissions is it complicates caching if that is a concern.

Regarding method, I don't mind the explicitness of including a method attribute, especially when using something like HAL. Of course, pedantically, I don't think it really qualifies as HAL any longer, but a derivation/new media type. We also went so far as to add a data attribute that described parameters you could post to an url. I'm not sure I'd recommend that though.

The system we built with this never got too far off the ground for unrelated reasons so I can't really speak to the effectiveness of those choices...

@wickedshimmy "self-descriptive" should include the media type definition and the rel definitions. If the self rel is defined as requiring a GET, it is self-descriptive. Not everything needs to be described in the document itself.

Yeah, my phasing was a bit backwards :/ - I'll buy self as being defined as part of the media type, but that doesn't apply to the other (resource-specific) methods unless they map 1-1, IMO.

I'm a bit surprised this isn't addressed in HAL, though I'm supposing it's because Mike wants to keep it decoupled from HTTP? That would lead me down the path of a more complicated href value (where that varies with transport), either "href" : { url: "/issues/1", method: "POST" }, or just "href" : "POST /issues/1" or something like that. But that's a big break that's almost certainly not worth the trouble.

@aaronjensen: Hmm, caching is a very valid concern. They would mess up the nice new ETags that some of our endpoints return.

As for URL parameters, I've been experimenting with the latest URI templates. The endpoint for repository contributors could look like:

{ "_links":
  { "contributors": "/repos/foo/bar/contributors{?anon}"
  }
}

I built out an internal small internal API with the original proposal above (comma separate method names). It worked well over HTTP and ZeroMQ, using sawyer as the client, and faraday-zeromq as the ZeroMQ adapter. I didn't find anything about HAL (or my modified version) that was HTTP specific.

I like, but I wonder if (possibly) conflating resources and state transfers might end up being a bad idea. The comments on a ticket are a related resource, but closing the ticket is an action you can take related to the current resource. For relations, discovering acceptable methods via OPTIONS seems totally reasonable - but an action seems like it ought to be tied to a given method.

Another approach might be to model the actions as either resources or properties: You either close a resource by changing the "closed" property to true, or you go the attached ticket_state resource or something. Does... does any of that make sense or am I way off base?

I like where you're going with this. You're definitely on the right track.

Should hypermedia tell you when you have permissions to access those resources?

To reiterate @steveklabnik, absolutely.

The HAL spec doesn't mention method properties at all. But, I think letting a client know the difference between an Issue I can edit or close is very useful. Is it any better if I have a separate relation for each action?

I think where things are starting to get awkward is outbound links (LO), templated queries (LT), non-idempotent update (LN), and idempotent updates (LI) all use the same _links control. HTML gets away with it by having 2 separate controls: A for LO links, FORM for LT and LN links, and it doesn't have LI links. Collection+JSON separates these actions into links, queries, and templates.

To write this in HTML, I think you'd know exactly how to represent these actions. If you had access to edit an Issue, you'd find a form on the page containing a few inputs to fill in or change. HTML defines how a client should submit this back to the server.

<form method="post">
  <label>Name:<input name="name" type="text" value="" /></label>
  <input type="submit" />
</form>

Likewise, if you has permissions to close an Issue you'd find a button on the page which is simply another form with a hidden input.

<form method="post">
  <input name="close" type="hidden" value="true" />
  <input type="submit" value="Close" />
</form>

If the permissions to edit and close are one in the same, there's no reason why these two forms can't be combined by adding a checkbox to close or re-open the Issue.

This is oversimplified but it helps me to step back and think of this in terms of a media type with which I'm intimately familiar.

We used HAL and added a link for each action that could be taken. Each link had a different Rel and may or may not have the same Href. The Rel pointed to Metadata that a developer could use to create the Request. The Metadata document described what would happen on the server, what HttpMethod to use, what resources to put in the request, and what resource would be returned. We controlled what the user could and couldn't do by adding of not adding the Hypermedia. This changed based on security and context of the application.

I like @agnoster idea of putting the status of the ticket as a separate resource. Then you could have a rel and a link right to that status.

HTH

Should hypermedia tell you when you have permissions to access those resources?

I believe it should. The client should be able to discover, at runtime, which state transitions are possible from the current state. So, if the user is not allowed to perform certain actions, hypermedia should not be present.

Depending on how you end up defining links, including methods could be redundant:

{ "id": 1
, "_links":
  { "self": {"href": "/issues/1", "method": "get"}
    "edit": {"href": "/issues/1", "method": "patch" }
    "delete": {"href": "/issues/1", "method": "delete" }
  , "close": {"href": "/issues/1/close", "method": "post"}
  }
}

could probably just be expressed as:

{ "_links":
  { "self": {"href": "/issues/1"}
    "edit": {"href": "/issues/1" }
    "delete": {"href": "/issues/1"}
  , "close": {"href": "/issues/1/close"}
  }
}

considering client and server agree on application semantics and the media type is consistent. That is, the client knows that for an "edit" link it should "PUT", for "self" it should GET and so on.

Also, if the client will have all possible state transitions represented on links or controls, there's probably not need to include the id in the representation.

Whoah, I'm not getting notifications on this. Sorry! Responding to people as I read. :)

@technoweenie

What about a 'close' relation?

The close relation definition would say "Yo, send a POST to this relation to close stuff."

The one thing I don't like about OPTIONS is it's a completely separate request.

I don't really like OPTIONS, but I felt you should know it's an option. ;) I agree with your criticism here.

@wickedshimmy

inferring other stuff from the name of the relation doesn't seem like a particularly self-descriptive protocol,

I'm saying that if you're only ever gonna support one method, it should be in the link relation definition. If not, then you don't have one either; you support multiple methods.

@aaronjenson is saying what I'm saying, here.

@technoweenie

I got my idea for the 'method' parameter from watching Jon Moore's talk about their HTML API.

tags have a method attribute...

In the context of HTML, I think it makes a lot of sense! A <form> is a certain kind of affordance, one that's very generic. If your argument is "I want to add the affordance so that even more general processors that don't understand my relations can process the document, well, maybe that's a good idea. Or, thinking about you and some conversations I've had with @pengwynn, "I want to reduce my support burden because I think people will find this to be confusing" is another good reason. Mostly, it's not that it's wrong, it just seems redundant to me. With a good reason, that's no longer true. :)

@aaronjensen

We also went so far as to add a data attribute that described parameters you could post to an url. I'm not sure I'd recommend that though.

Oh no, don't tell Mike, he hates forms. ;)

@aaronjensen @steveklabnik

Requiring a "forms-like" JSON-based representation was one of the motivations for creating actions in the Siren hypermedia type. See an example here: https://github.com/kevinswiber/siren#example

HAL aims to put more information in the link relation documentation. Siren includes some of this inline, as to accomodate change. For instance, the method used or input field types could change on the fly. A client may need to be aware of an action by name and any input fields ahead of time, but that's not a requirement.

I find it helpful to break up hypermedia design into two parts: format and interaction. The media type defines the format. Additional documentation can provide the interaction hints: link relations, action names, input field names, etc. Atom has done this successfully.

@agnoster

I like, but I wonder if (possibly) conflating resources and state transfers might end up being a bad idea. The comments on a ticket are a related resource, but closing the ticket is an action you can take related to the current resource.

In RESTful/Hypermedia parlance, they're all resources, actually.

Oh, and @technoweenie, don't forget, link relations that aren't registered are required to be URIs, so those would be https://api.github.com/rels/close, not close, for example. This way they're namespaced per application. I can dig up the relevant part of RFC5988 if you care.

@steveklabnik: Ah yea, I guess I didn't get that far into the RFC. I really like HAL's simplicity. Apparently the same href can have multiple relations though:

Link: <http://example.org/>;
             rel="start http://example.net/relation/other"

@technoweenie: Siren solves for multiple link relations, as well, by making the link rel an array. It can be more verbose than HAL, but that's all part of the flexibility/usability sliding scale.

Thanks for the comments everyone. I think I'm settling on something like this:

{ "id": 1
, "_links":
  { "self": {"href": "/issues/1", "method": "get"}
    "edit": {"href": "/issues/1", "method": "patch" }
    "delete": {"href": "/issues/1", "method": "delete" }
  , "close": {"href": "/issues/1/close", "method": "post"}
  }
}

I can easily omit "method" on known relations (self, edit, etc), and add URIs describing the relations as desired. I guess if I have problems, I can version the media type and transition to a new hypermedia format :)

The important questions I see here are :

1. Can it be useful to specify a method on a link? I would say yes.
2. Is a relation a composite of a resource identifier and a method? I would say no.

Consider the following:

GET /order/123/item/34

{
    some: "data",
    _links: {
        "self" : { href: "/order/123/item/34", method: "GET" },
        "update" : { href: "/order/123/item/34", method: "PUT" },
        "delete" : { href: "/order/123/item/34", method: "DELETE" },
        "order" : { href: "/order/123" }
    }
}

Shouldn't those resource relationships for create, update, etc all be rel="self"? Wouldn't it make more sense to write :

GET /order/123/item/34

{
    some: "data",
    _links: {
        "self" : { href: "/order/123/item/34", methods: [ "GET", "PUT", "DELETE" ] },
        "order" : { href: "/order/123", methods: ["GET", "DELETE"] }
    }
}

In answer to your question, I would say that the 'close' relation and url are not appropriate except as a GET to a resource returning a representation containing a form which could be used to issue an unsafe request to some other resource such as the issue itself or a subordinate resource such as status. So as a shortcut, status makes more sense:

GET /issue/123

{
    some: "data",
    _links: {
        "self" : { href: "/issue/123" },
        "status" : { href: "/issue/123/status", methods: ["PUT"], type: "text/plain", options: ["open", "closed"] }
    }
}

Which might elicit a client request such as:

> PUT /issue/123/status
> content-type: text/plain
> 
> closed

@technoweenie personally, I think if you suspect some clients may need to act dynamically based on their permissions for a particular resource, it is better to expose those permissions as an additional related resource and to couch it in "domain terms" rather than raw http. e.g.

{ "id": 1
, "_links":
  { "self": {"href": "/issues/1" }
  , "permissions": { "href": "/issues/1/permissions" }
  , "edit": {"href": "/issues/1" }
  , "delete": {"href": "/issues/1" }
  , "close": {"href": "/issues/1/close"}
  }
}

where /issues/1/permissions is a resource that is only privately cacheable, and varies based on who is making the request. Its state should tell a given client what permissions it has on the issue at a given point in time.

This approach isolates the non-cacheable state in a separate resource that will only be accessed by the clients that require it, thus extending the cache potential TTL for the issue resource itself.

fiww, if a methods property on a link is considered a design approach that many people are keen on I am happy to look at including it in the specification.

so far I have been deliberately stingy with adding features in order to keep the complexity of the type down to the bear minimum - however most of us consider certain features a basic need in a hypermedia API then lets look at adding them. The easiest way to do that is to start a thread on the discussion group, so if you can spare the time please do.

@technoweenie

Apparently the same href can have multiple relations though:

Yep!

@technoweenie - I really like what you've settled one, it would be my choice. And @mikekelly's suggestion for a private permissions resource makes a lot of sense too.

@mikekelly, good idea to be stingy, but +1 for adding "method."

@mikekelly: A separate permissions resource is not a great option. Think of a mobile app rendering a list of resources. Without per-resource hints, the app has to either call that URL for each resource, or assume you have access to edit each resource. This is a real problem on the GitHub Android app, actually. But, I also understand how this complicates caching.

@kevburnsjr: I like the methods array. That was my original inclination (though I opted for comma separated values vs a JSON array).

@technoweenie - The method array looses the different named actions. I think there's more value in having the explicit named actions and decoupling the link objects; the client can always aggregate them if needed.

The downside of using the key for the rel is you can't represent collections with a similar rel. For example imagine a rel of "query" where I could have multiple queries. I prefer having a collection of links with REL and HREF explicitly specified. Or alternatively, not using rel at all and having first class elements within my media type. The values for those elements could be arrays if multiple were supported as in the Query case.

A few other thoughts.

• Embedding the method. Why not use options? The disadvantage is that the list may only accurate for a point in time, i.e. my permissions may have changed. I guess I can see the simplicity argument as a consumer can just read the payload and know what to use.
• If the key is REL it should probably still be qualified with a namespace so as not to conflict with other RELs that is unless it is something globally registered like "EDIT".
• Having granular links like "Edit", "Delete" I think is fine. One could arge if you have that you don't need to embed method as the media type docs could say what method you use.

@technoweenie ok if that is a concern you have a couple of options:

• have the app assume lowest level of permissions until the relevant resource permissions are fetched.
• make the permissions an _embedded resource. You can avoid this impacting caching by using a mechanism like ESI. I've been planning on adding this to hal+json. Something like:

200 OK
X-Edge-Include: true

{
  ...
  "_embedded": {
    "permissions": {
       "href": "/issues/1/permissions",
       "edge-include": true
    }
  }
  ...
}

which should be caught by an edge proxy and replaced with the contents of /issues/1/permissions.

If neither of those are attractive to you and you still want to trade-away cacheability, then you might still want to favour keeping the links consistent and expressing the permissions as part of the issue representation (varying by user), i.e:

{ "id": 1
, "_links":
  { "self": {"href": "/issues/1" }
  , "edit": {"href": "/issues/1" }
  , "delete": {"href": "/issues/1" }
  , "close": {"href": "/issues/1/close"}
  }
, "permissions": {
    "can": ["commit"]
  , "cannot": ["administer"]
  }
}

I read through the comments, and I didn't see anyone touch on the freshness of the permission to use method M (see CORS). Sorry if I missed it.

Oct 3 - I make a request and get back information that I can edit resource R
Oct 4 (after some time window, or after many other requests that changed the state of resources) - I try to edit R and fail

If you do the same, but use OPTIONS instead:
Oct 3 - same, just that you only get a link to the resource R - what you can do with it remains unknown
Oct 4 - OPTIONS R gives you back GET (no edit), and thus you cannot edit R, but you can read R

I personally like the granularity that OPTIONS gives you. I can refresh the permission for a resource R1, without refreshing the representation of another resource R2 that links to resource R1.

I agree with @andreineculau, though I'm not as convinced that OPTIONS is a necessity (it's an option, har har)

I think this needs to be handled using a smarter client. After all, given REST's preference for stateless server communication, you'd want to have a background poll ask for OPTIONS and update the display or perform another action (save a draft?) before a user clicks a button and tries to edit. I'm reminded of the traffic-less session sharing specified in OpenID Connect, where you can timeout a session on another domain in another tab by having support for smarter clients polling cross-domain cookies.

That all said, what's wrong with trying and failing to Edit, if a client is graceful enough to recover from errors without data/intention loss? Or, it's a pain, but if permissions are given as part of a session object, then cached as a session credential or OAuth key would be, well, it's not the responsibility of the object you want to edit to tell you whether or not you can edit it some time indeterminately in the future.

Perhaps the OAuth key is revoked or the session expires? Or perhaps you load something when not signed in, then sign in and want to perform actions against that object without reloading it? It seems easier to leave permissions granting and cues for such to the application to request separately as it needs to as part of maintaining session state or OAuth.

It could be a smarter layer of application- or domain-object inferred functionality (to hide or show certain actions) on top of a much simpler stateless structure (where all actions are offered regardless of state).

A question I would pose, with no right or wrong answer, is what about an API that's primarily read-only but to which admins have write access? Should read-only clients see actions they cannot perform as of moment? Despite potential confusion, I would argue yes, as we often see actions on the web which we are not allowed (e.g. commenting) that only prompt us for authentication after we show interest. In addition, it's likely better for understanding an API to reveal all actions at all times, as despite the added complexity, you can call the API under any circumstance and understand the totality of the API at any given moment, without having multiple accounts with differing permissions to view edge case actions.

As to some of the JSON I've seen in this GIST, I really feel like the more we go down this route and the uglier the JSON gets, the more I wonder if we should just embed simple JSON objects with an XML wrapper for navigation/context and be done with this nonsense. Play to the strengths of each, and put simple data in JSON and more complex data in XML. Though at that point you have to parse both, so maybe it's not as ideal either. Hmm...