Discovery Metadata is Just Data

Another comment on Eran's discovery mechanisms list (which is itself great). I'm gradually reaching the conclusion that he's right that content negotiation isn't the best idea, but for the wrong reasons. The question is, are the alternatives any better?

Quoth Eran:

HTTP Content Negotiation - using the 'Accept' request header, the consumer informs the server it is interested in the metadata and not the resource itself, to which the server responds with the metadata document or its location. In XRDS, the consumer sends an HTTP GET (or HEAD) request to the resource URL with an 'Accept' header and content-type 'application/xrds+xml'. This informs the server of the consumer's discovery interest, which in turn may reply with the discovery document itself, redirect to it, or return its location via the 'X-XRDS-Location' (or 'Link') response header.

[-] Resource Declaration - does not address as it focuses on the consumer declaring its intentions.
[+] Direct Metadata Access - provides a simple method for directly requesting the metadata document.
[-] Web Compliant - while some argue that the metadata can be considered another representation of the resource, it is very much external to it. Using the 'Accept' header to request a separate resource (as opposed to a different representation of the same resource) violates the HTTP protocol. It also prevents using the discovery content-type as a valid (self-standing) web resource having its own metadata.
[-] Scale Agnostic - requires access to HTTP request and response headers, as well as the registration of multiple handlers for the same resource URL based on the 'Accept' header. In addition, improper use or implementation of the 'Vary' header in conjunction with the 'Accept' header will cause proxies to serve the metadata document instead of the resource itself - a great concern to large providers with frequently visited front-pages.
[-] Extendable - limited to a single content-type for metadata, and does not allow any existing schemas (with well known content-type).

Minimum roundtrips to retrieve metadata: 1

All of the points above are addressable with minor tweaks, turning this into a usable BigCo-scale solution. Specifically, I'd argue that it's perfectly web compliant to regard a resource's 'metadata' as a variant representation of the resource itself. As an example, consider an image resource that can be requested in several variants: image/gif, image/jpeg, or application/image-meta+xml. The last format gives you the EXIF metadata about the image but in a more convenient XML format. Format A gives you image bits; format B gives you images bits and metadata; formact C gives you just the metadata. But what's data and what's metadata just depends on your point of view.

The other argument against this is that buggy proxy caches may cache the wrong representation of, say, http://yahoo.com. This is something of a strawman in that this would need to be a cache between an RP and an OP. In any case, returning an uncacheable redirect (303?) to a metadata resource would avoid problems in practice.

All of this said, configuring something like http://yahoo.com/ (or, ahem, http://google.com/) to do content negotiation to enable discovery is a tough sell. Whatever technology is used to serve (and cache...) that page needs to be reconfigured to do the Right Thing with regard to content negotiation, with a big downside if something goes wrong and, so far, a small upside if things go well. Not a great sales pitch.

So I think Eran is right in that this isn't a great solution, but not because of web design purity; because of practical deployment issues. If there's a good alternative we should look at it and weigh the pros and cons, which is what I plan to do in the next post.

Discovery Mechanisms and Scale

After Eran's sessions at IIW today on discovery, I have some random thoughts which I figure might as well be captured (inflicted?) as blog posts. One comment I have is that "Scale Agnostic" is a little misleading in his matrix:



None of these solutions, by itself, spans the scale from a hosted site owner with Dreamweaver up to yahoo.com, so this column should really be all red. That's okay, because everyone agrees that at least a couple of solutions are needed to span both ends of this scale -- so the overall solution needs at least one solution at each end of the scale. Proposed: Replace "Scale Agnostic" with two columns, "Long Tail Scale" and "BigCo Scale", and make sure that the overall solution includes green in both columns.

XRDS-Simple (Yadis) + URI Templates

In writing up the OpenSocial 0.8 RESTful API spec, I ran up against a discovery gap. Clients know what social network container they want to query (e.g., orkut.com) and they know what kind of data they want to get (e.g., list of friends). An easy way to spec this is to mandate that every site has to support the same URI paths, like /friends/{userid}, and then just concatenate to get a final URI -- http://orkut.com/friends/{userid}. This turns out to be difficult in general; there are existing path conflicts, different server technologies make certain paths hard, etc. It turns out that you need to let containers provide different paths. So, there's a real discovery problem.

Enter XRDS-Simple (aka Yadis), the discovery mechanism used by OpenID and OAuth. It lets a client query the URI http://orkut.com to find out what URI to use to retrieve friends. Perfect!

Except, that /friends/{userid} isn't a URI -- it's a recipe, or template, for a URI. So XRDS-Simple can't actually technically hand that to you, because it only hands out URIs. It could hand you a list of every possible URI, starting with user aaab and ending with user zziee, but that's not too scalable.

So I invented the URI-Template parameter, which uses a URI Template:

<os:URI-Template>http://api.example.org/people/{guid}</os:URI-Template>

The idea is that the discovery document can use URI-Template wherever it could use URI, letting it specify an entire class of URIs in one fell swoop. Of course the client has to know about the template parameter names and how to use the endpoint, but that's the same no matter what: The clients have to know how to deal with the URIs once they have them, based on the spec. The template parameters are baked into the spec, the exact URI format isn't.

Note that this also means a site could outsource its API to an entirely different site. And if you have a personalized discovery document for a given user, you can still use concrete URIs and everything works as it does in normal XRDS discovery.

Updated at to clarify that the new thing is the URI-Template element, not URI templates in general.