(At BADCamp, several people were asking me to explain what the heck the Web Services core initiative was trying to do, so I got to practice my elevator pitch. This is essentially that pitch in written form.)
Drupal today is very page-oriented. Every request that comes in is responded to with a full HTML page. It is possible to return something else, and finally in Drupal 7 there is, sort of, native support to do so with delivery callbacks, but by and large any non-page response is an after thought at best and a hack at worst. The entire system is built around the assumption that we're returning an HTML page. Why else would we still load the theme system and form system for an auto-complete callback?
In the past, that hasn't been a major issue. The web was a series of pages, in practice, and Drupal is one of if not the most flexible page-generating machine on the web today. Drupal 7 is, arguably, the pinnacle of this page-oriented world.
Just in time for that world to be fading fast.
In case you've been living under a rock, the web is in the midst of a massive upheval. HTML and HTTP, tools built for linking static pages together, are transmogrifying (somewhat painfully and with lots of torn shirts along the way) into a layer of the Internet atop the Internet. Everything is not a page, destined for a known browser. Everything is a response to a request. That is all that can be assumed.
Responsive design and mobile-friendly sites may get all the press, but that's the least of our worries. That's "just" theming (he says as the themers get pitchforks). The web is changing in even more fundamental ways.
Consider a site using Varnish. That can do more than just serve a full page from a RAM cache. Edge-Side Includes (ESI) let it cache most of the page but call on the server to render just certain portions of the page (say, those that are customized to a given user) quickly and stitch them together on the fly. That's next year's high-volume web sites.
Consider FaceBook. They switched to parallel rendering of the page through sub-requests over a year ago. They call it Big Pipe. The general idea, though, is breaking the page up to generate and deliver in segments, in parallel. This is last year's high-volume web site.
Consider Backbone.js, which moves nearly all logic into the browser and treats the server as simply a dumb data store. Panels in the client, anyone?
Now, try to combine those. No pages, just HTML fragments, JSON strings, raw persistent data sockets, etc. In practice, the server may almost never generate an entire page from html tag to html tag.
That's the future. That's also not something Drupal today can handle.
Supporting that sort of next-generation workflow requires changing the way we approach page handling. It requires treating an incoming request as just that, an abstract request, which may be returning a page, or a fragment, or JSON, or establishing a persistent connection, or any number of things we can't even imagine yet. It requires leveraging all of the HTTP protocol, not just the tiny subset that browser HTML lets us access.
Fortunately, work is already under way. We also can get outside help. One of the reasons I am so excited that we've begun adopting Symfony2 components in core is that the Symfony2 framework is built on exactly those principles. By leveraging the work of others, in good open source fashion, we can save time both in coding and in architecting basic components. I expect we will pull in more Symfony2 components as time goes on, as reinventing the wheel in these critical areas would accomplish nothing but ensure that we don't get to the real work.
Today, Drupal loads nearly all of its code base for every request. We need to allow most of the system to lazy-load on demand instead. Fortunately, we can now do that by converting code to PSR-0 compatible classes in core.
Today, Drupal figures out what function will handle a page based purely on the path alone. We need to enable it to look at the full HTTP request and then some to do the complex routing we need. Or perhaps this is another area we could punt to Symfony2 components to save time and debugging.
Today, Drupal figures out what sort of response it is going to return after it's already done the work. We need to make determining what the response will be before we've build a single render array element. That depends on not just the request itself, but on the Drupal-derived information from it. That is, the Context of the request.
Today, we lay out a page as a blob of content with smaller blobs of content arrayed around it. We need to move to a model where one block on a page is no more special than any other, and we approach the page outside in, not inside out. Fortunately we have the model of Panels to follow.
Today, when laying out a page in Panels we have to deal with the Panels UI, which is widely recognized to be not up to the task. Fortunately some smart people are already trying to think through how that could be re-imagined to make it more approachable for site builders and content editors alike. It will take a lot more directed thought and planning,though, to re-invision the UI.
Web sockets may not be straightforward to implement in PHP, but we're already trying to work out how we could enable them from Drupal. (If you've done any development with Web Sockets, please share your experience!)
See something you could help with? We'll see you in the issue queues.