Announcing the Web Services and Context Core Initiative

Submitted by Larry on 11 April 2011 - 11:37am

At DrupalCon Chicago, Dries announced that the development process for Drupal 8 would be a bit different. Rather than a vast dog pile of efforts to improve Drupal in ways big and small, Drupal 8 will feature a number of major "core initiatives". These initiatives highlight major areas of work that represent not just a patch or three but major changes to Drupal's plumbing. Each initiative will have one or two initiative leads who have the ability to coordinate and make decisions relating to that initiative while working closely with Dries. In a large sense, it is a way for Dries to scale; Rather than Dries having to keep track of 1000 ongoing conversations himself, initiative owners can coordinate related changes while Dries coordinates the initiative owners. It also gives a clear indication of what work is happening and what to expect out of Drupal 8.

The first initiative for Drupal 8 has already been announced; Greg Dunlap will be leading the charge to overhaul Drupal's configuration system to provide more robust, performant, and deployable configuration and change management. That will be critical for Drupal's future as we push further into the corporate and enterprise sphere, as well as enabling more robust and unified configuration handling in the first place.

Today, I am pleased to announce Drupal's second core initiative: The Web Services and Context Core Initiative (WSCCI).

What web services means

Web services are an area that is exploding in recent years, and will only continue to become more important. What does that mean? In a nutshell, a web service is a web site responding to a request not with an HTML page but with data intended for another program to consume. That could be as mundane as an RSS feed, as complex as a headless SOAP application server, or anything in between.

While Drupal has had web services support for years via the Services module, it has been a bolt-on to the core system rather than a piece of core functionality. Drupal today, at its core, is a monolithic HTML-page-based CMS framework. Most parts of the system assume it will be delivering a blog-like HTML page, with a main content area and secondary content areas, to a human viewer on a desktop or laptop computer. Anything else is, architecturally, an afterthought.

Those afterthoughts, though, are rapidly becoming the norm. More and more pages now require asynchronous requests back to the server to request tiny bits of data or to submit forms without refreshing the page. Mobile devices 4" and smaller with Internet connections will soon outnumber desktops and laptops with big, wide monitors. Tablet computers are the fastest growing segment of the market, but have a wide variety of form factors and sizes. Extracting raw data from a site in order to analyze or mash it up is already a mandatory feature for many sites. Soon, HTML pages will be the least of Drupal's worries.

For Drupal to truly embrace the future web, we need to fundamentally rethink how Drupal response to an incoming HTTP request. We need to treat HTML pages as what they are: A particularly common form of REST response, but only one among many. To that end, Drupal needs to evolve, and quickly, from a first-class web CMS into a first-class REST server that includes a first-class web CMS.

Fortunately, we already have a roadmap to do just that.

The plan

Phase 1: Context

Drupal derives a lot of information out of the HTTP request. There is a lot of information there, too, and from that information we extract all sorts of information: What node is currently being viewed, what page callback to use, who the current user is, what the language of the site should be, and dozens of other important facts. However, there is no consistent way to do so. Every system and every module must invent their own way of doing so, usually with some global function or global variable. In order to build a more robust REST server within Drupal, we need to unify that contextual information in to a coherent and integrated system.

The solution here is to wrap the HTTP request into a single, extensible context object. That context object will act as a central gateway to information coming from the client (who may not be a user at all) as well as information we derive from it, such as the current node, current user, and so on.

By providing an integrated, extensible system for contextual information, we can build a more coherent, robust, flexible response system on top of it.

Phase 2: Plugins

Drupal is, by design, a very extensible system. Hooks galore allow additional code to tie into the system and add functionality in a myriad of places to control Drupal's actions and output.

What hooks do not do, however, is offer a way to exchange functionality, or change one implementation out for another. That is the realm of plugins, an area where Drupal has no existing standard. Core itself handles swappable, configurable components in a half dozen different ways. Various contrib modules have invented their own systems, with the most widely used being the Chaos Tools Suite but many other one-off implementations still in the wild.

The solution here is to develop a robust plugin system into core, and leverage it in core. Not only will that make core itself cleaner, more robust, and easier to learn and extend, it will provide a mechanism that contrib modules can leverage as well, just as they do with hooks. That in turn makes Drupal easier to develop for, easier to extend, and more consistent. And a single, good framework is easier to optimize and make faster than a dozen incompatible systems.

This is a well-understood problem space, so we will be trying to leverage existing work both in Drupal's own ctools suite and in other, similar open source projects.

Phase 3: REST Services

Both of those tools, context and plugins, are in a sense detours. They do not directly provide Drupal with web service capabilities. Rather, they are building blocks from which we will be able to build a new, robust underpinning for Drupal to support web services.

In the current approach, Drupal examines an incoming request URL and routes it to the appropriate callback function, then themes it as a page. In a REST server approach, Drupal should be able to examine the full HTTP request. A pluggable logic routine can then route the request to a pluggable response handler, which may be configured to respond with an HTML page but could just as easily respond with a JSON object, an Atom feed, or submitting a form and then firing off a redirect. All of those options become "first class citizens". Then, Drupal doesn't just support web services: Drupal is a web service.

Phase 4: Layouts

That doesn't mean that our HTML page generating system can't improve. In fact, those same underlying tools should make it possible to build a new, plugin-based, context-aware page rendering system as simply one response controller among many. We already have a solid conceptual model to follow here: Drupal's own Panels module, which has shown itself to be a very robust and effective way to build much more complex pages than Drupal natively supports. With a more context aware front-end system, we can also more easily tie into other important technologies such as ESI, or even implement them ourselves within Drupal.

The butler did it

If all of that sounds familiar, it should. It is essentially the Butler project renamed, an informal initiative I have been kicking around and trying to drum up support for since DrupalCon San Francisco and before, and for which we laid out a battle plan at DrupalCon Copenhagen. For that reason, I am happy, humbled, and more than a little scared to announce that Dries has asked me to head up the Web Services and Context initiative for Drupal 8.

The road ahead

All of that sounds like a lot of work, and it is. It's a ton of work. Fortunately we already have a head start, as many people have been working on designing these solutions over the past year. Work has already begun on the unified context system, for now still called "Butler", as a Drupal 7 module that we can forward-port to Drupal 8. Design work is already under way for the plugin system, with several volunteers researching 3rd party systems to see how they handle pluggable systems and the Drupal ctools team bringing their extensive experience to bear on building a robust, forward-looking solution. For phase 3, we have the experience and institutional knowledge of the Services module to draw on.

We also have the Configuration core initiative, which will be instrumental in enabling us to have access to the early-bootstrap configuration data we will need to do all of the above in a performant way.

Does this sound interesting? Want to be part of Drupal: The Next Generation? Read up on the architecture that has already been developed, join the conversation, and join us in the issue queues. Also watch this space for future updates as work progresses. I'll be posting a more complete tech reference in the g.d.o group in the next few days.

I'll see you in the issue queues.

Congratulations Larry! It's very exciting to have you leading the official initiative! :D

Sorry Larry. Now I'll never get time to talk to you again. :(

Great to hear! I would love to get more involved, but I am still not sure we have a general direction whether UX work would be part of the project and when this comes into play - it be great if we can get clarification on this.

Additionally how do you manage your time on this? It's a fairly important question for collaboration, when to schedule meetings and how much to expect.

There will be some UX needed later on, primarily in the realm of the new rendering system (phase 4). Since it is essentially Panels but baked in at a much lower level, we'll need to figure out a way to make that work that is easier to use and comprehend than the current Panels UI. I don't yet know what that is going to look like. Phase 3 may end up needing some as well if we end up needing to have some junior version of Page Manager in core as well. I'm not sure yet. Those would be the key places to think about UX, though.

I haven't worked out a schedule yet. I am talking with the owners at Palantir and will probably be able to take some time on-hours to work on WSCCI, but I'm not sure yet how much. I want to try and setup a regular schedule, though, once I have an idea of when that will be. For now it's still evening hours when I'm not still busy unpacking. :-)

I'm really excited that this is a priority for drupal-8 and that your going to focus on this.
We've been doing flash and mobile work with drupal for the past 4 years and have extensive services experience and have reached the limitations that you will ammend, one issue to think about is - if this is even php based?
One of the cool stuff we've started to do is to create hyper fast web services using node.js, mongoose and mongo.
Who says Drupal should only listen on one port ;) , We could easily spawn a node.js (small footprint) service and have it read information directly from mongo, we could write this schemas using drupal's growing support for the db's like mongo.
So optimally the world I imagine is one in which they would be a master (rw), slave(r) paradigm in which drupal would be the master but could serve content through non-php, asynchonyus services like node.
We'll set the default of the service on 443 or 8080 to be more fw friendly (for the net/security sceptics).
We've done this before with jquery on the client.. maybe it's time to think of it on the server...
Just my 2 cents
Lior

At this point we're concerned with Drupal itself, which is, and will remain, a predominantly PHP app. However, I believe there is a node.js bridge module available: http://drupal.org/project/nodejs I'm not entirely sure what it does, but it exists. :-)

Our primary environment is still, and will remain, a medium-sized to large PHP/SQL server setup. That is the de facto standard server for the web these days, and we have to run on it. That said, being able to scale up to larger systems or down to smaller systems is definitely a goal.

While I cannot see Drupal shipping with node.js and Mongo as the standard configuration, we already do support Mongo as a backend for most content and I see no reason why someone shouldn't be able to slap node.js next to it and hit the same database for super-fast web service data, if appropriate.

I totally understand the "out of scope" notion but because mongo is accessible and because the footprint is rather low and because of the community intrest (hey maybe we can get devseed back on board ;) ), so I think that it's an interesting road to explore.
On the otherhand you are committed to this and have an extraordinary track record of core work and I (and my team) have flirted with contrib - I know there are other things in place as well.
Anyways all i was saying is that it's not the first time we've added some js magic and accelerated drupal's growth.
Drupal 5 + jquery is the base to the slick interfaces we use these days and alot of ui goodness.
I'll checkout the bridge modules and try to write some case studies and maybe promote this through contrib - See you in Drupal 9 :)
Have a great release and thanks for all the contributions till now and all of the contributions coming up ahead...
Lior

heyrocker (not verified)

12 April 2011 - 4:47am

This is fantastic. Can't wait to work together. Also, congratudolences!

Rob (not verified)

12 April 2011 - 9:52am

This sounds great. I am wondering though, will the Context initiative be building off the current Context module, http://drupal.org/project/context , or will this be an entirely new start or based of some other module such as Panels?

We're writing new code, but it is drawing ideas from both Panels definition of context and the Context definition of context. It is, sadly, an over-used noun. :-( For now I'd suggest treating Butler context as its own "thing", which will, we hope, eventually supplant both existing context-y systems in Drupal.

Contracts Larry!

What do you think about AJAX in the Form API. It sits deeply there and perfect works with HTML. That is why it could be a problem. We can create abstraction layer for it or move AJAX from module to theme. What do you think is better or is there any other way to solve this issue?

There are lot of pieces that we will need to pry apart. FAPI is a particularly ugly one. Ideally, the post back to the server would trigger some sort of "FAPI AHAH update" response handler that initializes only those parts of the system it needs, rebuilds just the form, and responds with just the fragment it's supposed to. How that would work internally, I have no idea yet. That's why I'm coordinating this effort, not writing all the pieces of it. :-)

I think your concern, Vadim, was more of an issue with D6's AHAH system than it is with the D7 AJAX system. The D7 AJAX and FAPI systems are minimally coupled at this point, and I believe we can fully decouple them in D8. It will still be the case that a use-case where an AJAX interaction modifies the form (e.g., an "Add another item" button), that both systems must be used, but that doesn't make the systems coupled in a way that conflicts with the roadmap described in this blog post.

AJAX does provide an important consideration for Phases 3 and 4 of this work: that some responses must consist of a fragment in one format wrapped by another format (for example, an HTML fragment wrapped in a JSON object). How that affects themes and modules is being discussed in that issue, so if you have thoughts on it, please join the discussion there. I expect that issue to be in discussion mode for quite a bit longer, with a specific implementation not being finalized until phases 1 and 2 from the above roadmap are more complete.

I also think that we can use this web services initiative to enable high performance lightweight AJAX. Currently, a full Drupal bootstrap is needed for every AJAX request (unless you roll your own index.php replacement), and if a form is being modified, then a full drupal_rebuild_form() is needed, even if only a tiny piece of the form is being modified. The pluggable response handler system in phase 3 should open up some possibilities for creating optimized response handlers for much faster processing of some AJAX requests, though as Larry points out, we'll need to refactor some parts of FAPI to really do this right. That's one of the things that's so great about this initiative. It provides an overarching direction with some clear high level goals, and opportunities to make many important improvements to many of Drupal's subsystems, but in a focused rather than scattered way.

Jeff Burnz (not verified)

18 April 2011 - 2:56pm

Congrats and thank-you for heading this up! Awesome.

As a front end dev Phase 4 is what interests me the most, so I really wonder how much of that is in scope for D8 and what sorts of changes we might see, especially in terms of changes to how we build themes, for example perhaps there are no changes at all, aka like we can use Panels with almost any theme right now, or are we thinking more extreme as in Panels Everywhere type of thing, which really depends a different approach (which personally I like). Would we be able to include something like layout or style plugins in our themes like we can with Panels and Skinr?

For Bartik and Seven we're planning on working towards more adaptive/responsive concepts in D8 (mobile device support) and thinking about a new designs for core (a new theme) it would be cool to have some guidance on what might happen here and possibly even when (both hard/impossible to answer I assume, but best guess is better than no guess).

All in all though very exciting stuff.

Larry

24 April 2011 - 2:28pm

In reply to by Jeff Burnz (not verified)

In a sense, where this whole process started for me was with "what would it take to do Panels Everywhere in core, conceptually"? The answer turned out to be "rewrite everything else first", and being the stubborn git that I am we started looking into ways to do that. Fast forward a year and here we are. That is still, at the vague hand-wavy level, part of my goal.

In my pie-in-the-sky vision, which may not have any bearing on reality so take with the appropriate amount of salt, a typical page response boils down to recursively loading a block object, passing it context and configuration, rendering it, and placing the result into a layout template. Since blocks can contain blocks (is, Regions become a special case of block), you can nest however deep makes sense.

A theme becomes a collection of treatments for blocks, and the layout (as template files and CSS) to lay them out together. Hypothetically, depending on what we do to the theme system we could make it possible to use a different theme for each block, but I don't know how practical that is.

One concept I discussed with Jen Simmons back in Copenhagen is that each block would naturally correspond to either a section or article tag in HTML5 (or a div if you were so inclined). Which one should be which is something that would be decided by an implementer. Where that decision is made, though, (vis, in a Panels-like UI or in a hard-coded template file) is still a very open question. It would actually be great if you could help weigh in on that question over in the g.d.o group, as it has no easy answer.

One of the nice things about moving so much logic to blocks is that they become the natural place for all kinds of things. An AHAH response is a Block. We do render caching at the Block level. If you're doing a fixed-grid design, a block could be easily styled to line up to your grid columns. You could, I suppose, specify at the Block-level what blocks should disappear in given screen widths for adaptive designs.

Again, I don't know how realistic any of that is, but that's the rainbows-and-ponies vision I have for phase 4. Of course, no battle plan ever survives first contact with the enemy but that's at least my thinking at the moment. :-)

effulgentsia (not verified)

23 April 2011 - 6:26pm

Well, I think we all expected this to be one of the initiatives and for you to be heading it up: I'm glad it's now official. Your work on the D7 database layer was absolutely brilliant. I'm looking forward to collaborating with you on this. I expect it to also be one of the initiatives that attracts lots of new core contributors. Which would be awesome!