Web Services and KISS

Adam Bosworth argues for the 'worse is better' philosophy of web services eloquently in his ISCOC talk and blog entry. I have a lot of sympathy for this point of view.  I'm alsoskeptical about the benefits of the WS-* paradigm.  They seem tome to be well designed to sell development tools and enterpriseconsulting services.

Why Aggregation Matters

Sometimes, I feel like I'm banging my head against a wall trying to describe just why feed syndication and aggregation is important.  In an earlier post,I tried to expand the universe of discourse by throwing out as manypossible uses as I could dream up.  Joshua Porter has written areally good article about why aggregation is a big deal, even justconsidering its impact on web site design: Home Alone? How Content Aggregators Change Navigation and Control of Content. 

Prediction is Difficult, Especially the Future

Mysecond hat at AOL is development manager for the AOL Polls system. This means I've had the pleasure of watching the conventions anddebates in real time while sitting on conference calls watching theperformance of our instant polling systems. Which had some potentialissues, but which, after a lot of work, seem to be just fine now. Anyway: The interesting thing about the instant polling during thedebates was how different the results were from the conventionalinstant phone polls. For example, after the final debate the AOLInstapoll respondents gave the debate win to Kerry by something like60% to 40%. The ABC news poll was more like 50%/50%. Frankly, I don'tbelieve any of these polls. However, I'll throw this thought out: Theonline insta polls are taken by a self selected group of people who areinterested in the election and care about making their opinions known. Hmmm... much like the polls being conducted tomorrow.
I'llgo out on a limb and make a prediction based on the various pollresults and on a lot of guesswork: Kerry will win the popular vote by asignificant margin. And, he'll win at least half of the "battleground"states by a margin larger than the last polls show. But, I make nopredictions about what hijinks might ensue in the Electoral College.

Update 11/11: Well, maybe not...

Random Note: DNA's Dark Matter

Scientific American's The Hidden Genetic Program of Complex Organismsgrabbed my attention last week.  This could be the biologicalequivalent of the discovery of dark matter.  Basically, the 'junk'or intron DNA that forms a majority of our genome may not be junk atall, but rather control code that regulates the expression of othergenes. 

The programming analogy would be, I think, that the protein-codingparts of the genome would be the firmware or opcodes while the controlDNA is the source code that controls when and how the opcodes areexecuted.  Aside from the sheer coolness of understanding how lifeactually works, there's a huge potential here for doing useful geneticmanipulation.  It's got to be easier to tweak control code than totry to edit firmware... (Free link on same subject: The Unseen Genome.)

Things in Need of a Feed

Syndicated feeds are much bigger than blogs and news stories; they're aplatform.  A bunch of use cases, several of which actually exist in some form, others just things I'd like to see:

• Blog entries for blogs I'm interested in
• Feed of all comments on entries I've authored
• News stories matching a custom filter I've set up
• Traffic conditions on my customary route(s)
• Fedex shipping feed giving status and history for all of my packages
• Customer support feed giving status and history for all my issues (any company)
• Product safety/recall information for everything I buy
  
• Amazon feed of new books matching my preferences
• All new material by a specific author (on any blog or online source)
• Feed of new feeds, of various types:
•   Just my friends
•   Authored by people whose blogs I already subscribe to
•   Filtered on personal profile/interests
• House for sale listings
• Newly discovered prime numbers (okay, a niche audience)
• Airport flight status alerts
• Movies in my Netflix queue and recommendations
  
• Audio / video content pushed onto my iPod (Podcasting)
• Auction information

Addendum 11/11:

• Multiplayer game results feed
  
• New government publications feed
• New computer virus alerts feed (with metadata giving virus signatures)
• Book queue
  

Niche Markets

Niche markets are where it's at: Chris Anderson's The Long Tailis exactly right. The Internet not only eliminates the overhead ofphysical space but also, more importantly, reduces the overhead offinding what you want to near-zero. When your computer tracks yourpreferences and auto-discovers new content that you actually want, it enables new markets that couldn't otherwise exist.

Update 10/11: Joi Ito's take.

Network Protocols and Vectorization

Doing things in parallel is one of the older performance tricks.  Vector SIMD machines -- like the Cray supercomputers -- attack problems that benefit from doing the same thing to lotsof different pieces of data simultaneously.  It's just a performancetrick, but it drove the design and even the physical shape of thosemachines because the problems they're trying to tackle -- airflowsimulation, weather prediction, nuclear explosion simulation, etc. --are both important and difficult to scale up.  (More recently, we'reseeing massively parallel machines built out of individual commodityPCs; conceptually the same, but limited mostly by networklatency/bandwidth.)

So what does this have to do with network protocols?  Just as the problems of doing things like a matrix-vector multiply very, very fast drove the designs of supercomputers, the problems of moving data from one place to another very quickly, on demanddrive the designs of today's network services.  The designs of networkAPIs (whether REST, SOAP, XML-RPC, or whatever) need to take thesedemands into account.

In particular, transferring lots of small pieces of data in serialfashion over a network can be a big problem.  Lots of protocols thatare perfectly fine when run locally or over a LAN fail miserably whenexpected to deal with 100-200ms latencies on a WAN or the Internet. HTTP does a decent job of balancing out performance/latency issues forretrieving human readable pages -- a page comes down as a medium-sizedchunk of data, followed by, if necessary, associated resources such asscripts, style sheets, and binary images, which can all be retrieved inparallel/behind the scenes.  Note, that this is achieved only throughlots of work on the client side and deep knowledge of the interactionsbetween HTML, HTTP, and the final UI.  The tradeoff is complexity ofprotocol and implementation.

How does this apply to network protocols in general?  One idea is tocarefully scrutinize protocol requests that transfer a single smallpiece of data.  Often a single small piece of data isn't very useful onits own.  Are there common use cases where a system will do this in aloop, perhaps serially, to get enough data to process or present to auser?  If so, perhaps it would be a good idea to think of "vectorizing"that part of the protocol.  Instead of returning a single piece ofdata, for example, return a variable-length collection of those piecesof data.  The semantics of the request may change only slightly -- from"I return an X" to "I return a set of X".  Ideally, the length shouldbe dynamic and the client should be able to ask for "no more than N" oneach request.

For example, imagine a protocol that requires a client to firstretrieve a set of handles (say, mailboxes for a user) then query eachone in turn to get some data (say, the number of unread messages).  Ifthis is something that happens often -- for example, automaticallyevery two minutes -- there are going to be a lot of packets hittingservers.  If multiple mailboxes are on one server, it would be fairlytrivial to vectorize the second call and effectively combine the twoqueries into one -- call it "get mailbox state(s)".  This would let aclient retrieve the state for all mailboxes on a given server, withbetter latency and far less bandwidth than the first option.  Of coursethere's no free lunch; if a client is dealing with multiple servers, itnow has to group the mailboxes for each server for purposes ofretrieving state.  But conceptually, it's not too huge of a leap.

There are other trade-offs.  If the "extra" data is large -- like abinary image -- it might well be better to download it separately,perhaps in parallel with other things.  If it's cacheable, but the maindata isn't, it may again be better to separate it out so you can takeadvantage of things like HTTP caching. 

To summarize, one might want to vectorize part of a network protocol if:

• Performance is important, and network latency is high and/or variable;
  
• The data to be vectorized are always or often needed together in common use cases;
  
• It doesn't over-complexify the protocol;
• There's no other way to achieve similar performance in other ways (parallel requests, caching, etc.)

Of course, this applies to the Atom API. There's a fair amount of vectorization in the Atom API from the start,since it's designed to deal with feeds as collections of entries.  Ithink there's a strong use case for being able to deal with collectionsof feeds as part of the Atom API as well, for all the reasons givenabove.  Said collections of feeds might be feeds I publish (so I wantto know about things like recent comments...) or perhaps feeds I'mtracking (so I want to be able to quickly determine which feeds havesomething interesting, before downloading all of the most recentdata).  It would be interesting to model this information as a synthetic feed, since of course that's already nicely vectorized.  But there are plenty of other ways to achieve the same result.