Google is breaking the web!
I've been fighting (discussing) with some people about "response tailoring" REST APIs. That is, depending on the request or particular device making the request, you return a response that is tailored for that device (web, mobile: S60, iPhone, etc.). A few options for doing this (that I could think of):
- Support a well-known, defined set of
User-Agents - Support a well-know, defined set of MIME/
Content-Types - Support accessing sub-resources or partial resources through a URL tree, i.e.
places/1234, places/1234/addresses, places/1234/comments - Allow for dynamically tailored partial resources through matrix or query params, i.e.
places/1234?fields=place(name,address)
There are also multiple things to consider when dealing with mobile devices (non-exhaustive of course):
- Bandwidth: Overall bandwidth is a big deal as our success is measured on how efficiently and inexpensively we can deliver content to the consumer's device
- Latency: How long does the request take to the server (or intermediary cache) and the response to come back to the device
- Processing time: Mobile devices are inherently less powerful (particularly lower-end Symbian devices) and processing big JSON blobs is not always reasonable
The obvious is to take the first choice and vary the response on the User-Agent header. The Vary header can then clearly be used and normal HTTP caches can handle this just fine. Mobile clients can get trimmed down data when calling the root resource (places/1234) and make subsequent requests to sub-resources for their "partial" content (like addresses, contact details, etc.).
The response I get from the client guys however is, "That's too many requests" or "It's not flexible enough. Then we have to come talk to a backend dev every time we need a custom response." While I agree that this means more requests and that it's something you have to communicate about, the solution that Google proposes (Option 4 above) to this problem means that caches need to be either really damn smart and able to decompose responses already in the cache, or they don't have HTTP caching at all. The latter I am very doubtful of though. Their statement that this increases cacheability is only partly true in the case that the data that is retransmitted from a full response is not changing while the data from the partial responses is updated frequently.
So what's a guy to do? It's difficult to test the effects of any decision on the world without having a population to do so, distributed around the world with local caches. One choice might be faster where a local cache has the request you want, while the other may be faster and cheaper when you have a cache miss. What, in the end is the most effective thing for the web? Which is best for the consumer? Which is the best choice? Is it, break the web and it's cacheability, or provide the ultimate response tailoring that devices and client devs are asking for?
May 13th, 2010 - 13:36
I’m not sure that what google is doing is “breaking the web”, it’s the mature REST approach that Richardson spoke about. http://www.crummy.com/writing/speaking/2008-QCon/act3.html
As for too many requests, or how to cache all this around the world – no one said it would be easy.
May 13th, 2010 - 16:46
Of course, “breaking” is overkill, but the point is that allowing arbitrary query parameters to shape the response are incredibly difficult to make cache well and don’t always make sense either. Consider the following two URLs:
GET http://blah.com/entries?fields=entry(title,gd:rating,@gd:*) GET http://blah.com/entries?fields=entry(title)
The second returns a sub-set of the first response, but both will end up as different cache objects. It’s not a big deal if both requests are coming from beside the US datacentre where the responses originate, however Australia to the US is a big deal. I would prefer to hit a cache in Australia and get slightly more data than I need vs going all the way back to the origin server just to shave a few bytes. The difference in latency can be enormous, particularly at global scale. Not to mention the extra load that this incurs on the origin servers.
Anyways, I don’t think Richardson has anything to say about this specifically. I would imagine that a more appropriate approach to partial responses would be to break up into sub-resources and allow for sub-resources to be combined into bigger responses, not the other way around (and of course providing hyperlinks to combine sub-resources). This allows the server-side to constrain the total number of representations that can be fetched while in the Google case there can be hundreds of representations of the same data, just depending on the order of the query parameters (since most HTTP caches cache based on a simple hash of the URL).
But, I still have no great solution for mobile clients
We’ll decide something soon I hope.
May 14th, 2010 - 11:26
The caching issue is a tough one, and I wish I had a magic bullet for you…but alas. Keep me posted though, I’m interested in your direction on this one. Good luck.