At DrupalCon Chicago, Dries announced that the development process for Drupal 8 would be a bit different. Rather than a vast dog pile of efforts to improve Drupal in ways big and small, Drupal 8 will feature a number of major "core initiatives". These initiatives highlight major areas of work that represent not just a patch or three but major changes to Drupal's plumbing. Each initiative will have one or two initiative leads who have the ability to coordinate and make decisions relating to that initiative while working closely with Dries. In a large sense, it is a way for Dries to scale; Rather than Dries having to keep track of 1000 ongoing conversations himself, initiative owners can coordinate related changes while Dries coordinates the initiative owners. It also gives a clear indication of what work is happening and what to expect out of Drupal 8.
The first initiative for Drupal 8 has already been announced; Greg Dunlap will be leading the charge to overhaul Drupal's configuration system to provide more robust, performant, and deployable configuration and change management. That will be critical for Drupal's future as we push further into the corporate and enterprise sphere, as well as enabling more robust and unified configuration handling in the first place.
Today, I am pleased to announce Drupal's second core initiative: The Web Services and Context Core Initiative (WSCCI).
Web services are an area that is exploding in recent years, and will only continue to become more important. What does that mean? In a nutshell, a web service is a web site responding to a request not with an HTML page but with data intended for another program to consume. That could be as mundane as an RSS feed, as complex as a headless SOAP application server, or anything in between.
While Drupal has had web services support for years via the Services module, it has been a bolt-on to the core system rather than a piece of core functionality. Drupal today, at its core, is a monolithic HTML-page-based CMS framework. Most parts of the system assume it will be delivering a blog-like HTML page, with a main content area and secondary content areas, to a human viewer on a desktop or laptop computer. Anything else is, architecturally, an afterthought.
Those afterthoughts, though, are rapidly becoming the norm. More and more pages now require asynchronous requests back to the server to request tiny bits of data or to submit forms without refreshing the page. Mobile devices 4" and smaller with Internet connections will soon outnumber desktops and laptops with big, wide monitors. Tablet computers are the fastest growing segment of the market, but have a wide variety of form factors and sizes. Extracting raw data from a site in order to analyze or mash it up is already a mandatory feature for many sites. Soon, HTML pages will be the least of Drupal's worries.
For Drupal to truly embrace the future web, we need to fundamentally rethink how Drupal response to an incoming HTTP request. We need to treat HTML pages as what they are: A particularly common form of REST response, but only one among many. To that end, Drupal needs to evolve, and quickly, from a first-class web CMS into a first-class REST server that includes a first-class web CMS.
Fortunately, we already have a roadmap to do just that.
Drupal derives a lot of information out of the HTTP request. There is a lot of information there, too, and from that information we extract all sorts of information: What node is currently being viewed, what page callback to use, who the current user is, what the language of the site should be, and dozens of other important facts. However, there is no consistent way to do so. Every system and every module must invent their own way of doing so, usually with some global function or global variable. In order to build a more robust REST server within Drupal, we need to unify that contextual information in to a coherent and integrated system.
The solution here is to wrap the HTTP request into a single, extensible context object. That context object will act as a central gateway to information coming from the client (who may not be a user at all) as well as information we derive from it, such as the current node, current user, and so on.
By providing an integrated, extensible system for contextual information, we can build a more coherent, robust, flexible response system on top of it.
Drupal is, by design, a very extensible system. Hooks galore allow additional code to tie into the system and add functionality in a myriad of places to control Drupal's actions and output.
What hooks do not do, however, is offer a way to exchange functionality, or change one implementation out for another. That is the realm of plugins, an area where Drupal has no existing standard. Core itself handles swappable, configurable components in a half dozen different ways. Various contrib modules have invented their own systems, with the most widely used being the Chaos Tools Suite but many other one-off implementations still in the wild.
The solution here is to develop a robust plugin system into core, and leverage it in core. Not only will that make core itself cleaner, more robust, and easier to learn and extend, it will provide a mechanism that contrib modules can leverage as well, just as they do with hooks. That in turn makes Drupal easier to develop for, easier to extend, and more consistent. And a single, good framework is easier to optimize and make faster than a dozen incompatible systems.
This is a well-understood problem space, so we will be trying to leverage existing work both in Drupal's own ctools suite and in other, similar open source projects.
Both of those tools, context and plugins, are in a sense detours. They do not directly provide Drupal with web service capabilities. Rather, they are building blocks from which we will be able to build a new, robust underpinning for Drupal to support web services.
In the current approach, Drupal examines an incoming request URL and routes it to the appropriate callback function, then themes it as a page. In a REST server approach, Drupal should be able to examine the full HTTP request. A pluggable logic routine can then route the request to a pluggable response handler, which may be configured to respond with an HTML page but could just as easily respond with a JSON object, an Atom feed, or submitting a form and then firing off a redirect. All of those options become "first class citizens". Then, Drupal doesn't just support web services: Drupal is a web service.
That doesn't mean that our HTML page generating system can't improve. In fact, those same underlying tools should make it possible to build a new, plugin-based, context-aware page rendering system as simply one response controller among many. We already have a solid conceptual model to follow here: Drupal's own Panels module, which has shown itself to be a very robust and effective way to build much more complex pages than Drupal natively supports. With a more context aware front-end system, we can also more easily tie into other important technologies such as ESI, or even implement them ourselves within Drupal.
If all of that sounds familiar, it should. It is essentially the Butler project renamed, an informal initiative I have been kicking around and trying to drum up support for since DrupalCon San Francisco and before, and for which we laid out a battle plan at DrupalCon Copenhagen. For that reason, I am happy, humbled, and more than a little scared to announce that Dries has asked me to head up the Web Services and Context initiative for Drupal 8.
All of that sounds like a lot of work, and it is. It's a ton of work. Fortunately we already have a head start, as many people have been working on designing these solutions over the past year. Work has already begun on the unified context system, for now still called "Butler", as a Drupal 7 module that we can forward-port to Drupal 8. Design work is already under way for the plugin system, with several volunteers researching 3rd party systems to see how they handle pluggable systems and the Drupal ctools team bringing their extensive experience to bear on building a robust, forward-looking solution. For phase 3, we have the experience and institutional knowledge of the Services module to draw on.
We also have the Configuration core initiative, which will be instrumental in enabling us to have access to the early-bootstrap configuration data we will need to do all of the above in a performant way.
Does this sound interesting? Want to be part of Drupal: The Next Generation? Read up on the architecture that has already been developed, join the conversation, and join us in the issue queues. Also watch this space for future updates as work progresses. I'll be posting a more complete tech reference in the g.d.o group in the next few days.
I'll see you in the issue queues.