Skip to main content

RESTful partial updates: PATCH+Ranges

Over the past couple of months, there's been a lot of discussion aboutthe problem of partial updates in REST-over-HTTP[1][2][3][4][5].  The problemis harder than it appears at first glance.  The canonical scenario isthat you've justretrieved a complicated resource, like an address book entry, and youdecide you want to update just one small part, like a phone number. The canonical way to do this is to update yourrepresentation of the resource and then PUT the whole thing back,including all of the parts you didn't change.  If you want to avoid thelost update problem,you send back the ETag you got from the GETwith your PUT inside an If-Match: header, so that you know that you'renot overwriting somebody else's change.

This works, but it doesn't scale well to large resources or highupdate rates, where "large" and "high" are relative to your budget forbandwidth and tolerance for latency.  It also means that you can'tsimply and safely say "change field X, overwriting whatever is there,but leave everything else as-is".

I've seen the same thought process recapitulated a few times now on howtosolve this problem in a RESTful way.  The first thing that springs tomind is to ask if PUT can be used to send just the part you want tochange.  This can be made to work but has some major problemsthat make it a poor general choice. 
  • A PUT to a resourcegenerally means "replace", not "update", so it's semanticallysurprising.
  • In theory it could break write-through caches.  (This is probablyequivalent to endangering unicorns.)
  • It doesn'twork for deleting optional fields or updating flexible lists such asAtomcategories.
The next idea is generally to simply use POST to update the resource. This does work in many cases, but conflicts with the use of POST to adda resource to a collection.  That is, if you POST to a collection, areyou trying to add an element to the collection, or perform some otherupdate to the collection's metadata?  It's possible disambiguate usingMIMEtypes but it feels fragile.  It also doesn't capture the fact that theoperation is retryable; POST in general is not retryable.

A good solution to the partial update problem would be efficient,address the canonical scenarioabove, be applicable to a wide range of cases, not conflict with HTTP,extend basic HTTP as little as possible, deal with optimisticconcurrency control, and deal with the lost update problem. The methodshould be discoverable (clients should be able to tell if a serversupports the method before trying it). It would also be nice if thesolution would let us treat data symmetrically, both getting andputting sub-parts of resources as needed and using the same syntax.

There are three contenders for a general solution pattern:

Expose Parts as Resources.  PUT to a sub-resourcerepresents aresources' sub-elements with their own URIs.   This is in spirit whatWeb3Sdoes.  However, it pushes the complexity elsewhere:  Intodiscovering the URIs of sub-elements, and into how ETags work acrosstwo resources that are internally related.  Web3S appears to handleonlyhierarchical sub-resources, not slicing or arbitrary selections.

Accept Ranges on PUTs.  Ranged PUT leverages andextends theexisting HTTP Content-Range:header to allow a client tospecify a sub-part of a resource, not necessarily just byte ranges buteven things like XPath expressions. Ranges are well understood in thecase of GET but were rejected as problematic for PUT a while back bytheHTTP working group.  The biggest concern was that it adds a problematicmust-understand requirement.  If a server or intermediary accepts a PUTbut doesn'tunderstand that it's just for a sub-range of the target resource, itcould destroy data.   But, thisdoes allow for symmetry in reading andwriting.  As an aside, the HTTP spec appears to contradict itselfabout whether range headers are extensible or are restricted to justbyte ranges.  This method works fine with ETags; additional methods fordiscovery need to be specified but could be done easily.

Use PATCHPATCH is a method that's beentalked about for awhilebut is the subject of some controversy. James Snell has revived LisaDusseault's draft PATCH RFC[6] and updated it, and he's looking forcomments on the new version.  I think this is a pretty good approachwith a few caveats.  The PATCH method may not be supported byintermediaries, but if it fails it does fail safely.  It requires a newverb, which is slightly painful.  It allows for variety of patchingmethods via MIME types.  It's unfortunately asymmetric in that it doesnot address the retrieval ofsub-resources.  It works fine with ETags.  It's discoverable via HTTPheaders (OPTIONS and Allow: PATCH).

The biggest issue with PATCH is the new verb.  It's possible thatintermediaries may fail to support it, or actively block it.  This isnot too bad, since PATCH is just an optimization -- if you can't useit, you can fall back to PUT.  Or use https, which effectively tunnelsthrough most intermediaries.

On balance, I like PATCH.  The controversy over the alternatives seemto justify the new verb.  It solves the problem and I'd be happy withit.  I would like there to be a couple of default delta formats definedwith the RFC. 

The only thing missing is symmetricalretrieval/update.  But, there's an interesting coda:  PATCH is definedso that Content-Range is must-understand on PATCH[6]:
The server MUST NOT ignore any Content-* (e.g.  Content-Range) 
headers that it does not understand or implement and MUST return
a 501 (Not Implemented) response in such cases.
So let's say aserver wanted to be symmetric; it could advertise support forXPath-based ranges on bothGET and PATCH. A client would use PATCH with a range to send backexactly the same data structure it retrievedearlier with GET.  An example:
GET /abook.xml
Range: xpath=/contacts/contact[name="Joe"]/work_phone
which retrieves the XML:
<contacts><contact><work_phone>650-555-1212</work_phone>
</contact></contacts>

Updating the phone number is very symmetrical with PATCH+Ranges:
PATCH /abook.xml
Content-Range: xpath=/contacts/contact[name="Joe"]/work_phone
<contacts><contact><work_phone>408-555-1591</work_phone>
</contact></contacts>
The nice thing about this is that no new MIME types need to beinvented; the Content-Range header alerts the server that the stuffyou're sending is just a fragment; intermediaries will eitherunderstand this or fail cleanly; and the retrievalsand updates are symmetrical. 

[1]http://www.snellspace.com/wp/?p=683
[2]http://www.25hoursaday.com/weblog/2007/06/09/WhyGDataAPPFailsAsAGeneralPurposeEditingProtocolForTheWeb.aspx
[3]http://www.dehora.net/journal/2007/06/app_on_the_web_has_failed_miserably_utterly_and_completely.html
[4]http://tech.groups.yahoo.com/group/rest-discuss/message/8412
[5]http://tech.groups.yahoo.com/group/rest-discuss/message/9118
[6]http://www.ietf.org/internet-drafts/draft-dusseault-http-patch-08.txt

Popular posts from this blog

Personal Web Discovery (aka Webfinger)

There's a particular discovery problem for open and distributed protocols such as OpenID, OAuth, Portable Contacts, Activity Streams, and OpenSocial.  It seems like a trivial problem, but it's one of the stumbling blocks that slows mass adoption.  We need to fix it.  So first, I'm going to name it:

The Personal Web Discovery Problem:  Given a person, how do I find out what services that person uses?
This does sound trivial, doesn't it?  And it is easy as long as you're service-centric; if you're building on top of social network X, there is no discovery problem, or at least only a trivial one that can be solved with proprietary APIs.  But what if you want to build on top of X,Y, and Z?  Well, you write code to make the user log in to each one so you can call those proprietary APIs... which means the user has to tell you their identity (and probably password) on each one... and the user has already clicked the Back button because this is complicated and annoying.

XAuth is a Lot Like Democracy

XAuth is a lot like democracy:  The worst form of user identity prefs, except for all those others that have been tried (apologies to Churchill).  I've just read Eran's rather overblown "XAuth - a Terrible, Horrible, No Good, Very Bad Idea", and I see that the same objections are being tossed around; I'm going to rebut them here to save time in the future.

Let's take this from the top.  XAuth is a proposal to let browsers remember that sites have registered themselves as a user's identity provider and let other sites know if the user has a session at that site.  In other words, it has the same information as proprietary solutions that already exist, except that it works across multiple identity providers.  It means that when you go to a new website, it doesn't have to ask you what your preferred services are, it can just look them up.  Note that this only tells the site that you have an account with Google or Yahoo or Facebook or Twitter, not what the…
Twister is interesting.  It's a decentralized "microblogging" system based on putting together existing protocols:  Bitcoin, distributed hash tables, and Bittorrent.  The most interesting part for me is using Bitcoin for user registration and spam control.  Federated systems handle this with federated trust, which is at least conceptually simple.  The Twister/Bitcoin mechanism looks intriguing though I don't know enough about Bitcoin to really comment.  Need to read further.