Refocusing WSCCI

Submitted by Larry on 20 February 2012 - 9:18pm

As Dries has already reported, we held a summit meeting at the Acquia offices in Boston last week. It was a good sprint for a couple of reasons. For one, a large number of leading core developers got more clearly on the same page about the direction of Drupal core. For another, we were able to break the "too big to swallow" logjam that has been plaguing the Web Services and Context Core Initiative (WSCCI).

Let's quickly look back at WSCCI's original charter. There were four parts to the original plan: 1) Centralize our request/context information; 2) Introduce a universal plugin system; 3) Expose real, modern, solid web services from Drupal; 4) Use those tools to build a "blocks everywhere" multi-layout rendering system.

Yeah, that's big. :-)

Conclusions

The first important thing we decided in Boston was that in order to achieve those goals, we had to split the initiative in half. Moving forward, WSCCI will be focusing on parts 1 and 3: That is, "make Drupal a first-class REST server." Parts 2 and 4 are being split off to a separate effort. (Stay tuned.)

The second key point is that the WSCCI rethink using Symfony components got a green light, and will be moving forward.

Why Symfony2? Well, quite simply, its low-level components are already solid REST-based systems, exactly what we've been trying to build. We should not under-estimate the work that goes into getting that "right"; HTTP is not a trivial spec. Open Source is all about sharing, collaborating, and not reinventing the wheel. With Symfony2 Components, we get access to a really solid set of code as well as a really solid community behind it.

The latter point should not be under-stated, either. For instance, at the sprint we ran across a bug in Symfony's Controller Resolver system that Fabien fixed, on the spot, in 20 minutes flat. Several Symfony developers have reached out to us for feedback on revising their session handling system to make it more useful for Drupal, especially their "flash message" system (a junior version of drupal_set_message(), basically), which is being updated as we speak. They've even asked for use cases from Drupal to help justify a particular change so that we can leverage that directly. That's the sort of relationship that can only be beneficial to both projects. (Read: Go help make Symfony 2.1 better! That can only help Drupal.)

Next steps

The new WSCCI roadmap, then, is roughly as follows:

  1. Get the basic request object in place. This begins to allow us to untangling our (ab)use of globals and super globals, and lays groundwork for a full request/response workflow. That is only a stop-gap, however, until...
  2. Wrap our current menu/routing logic with Symfony2's HttpKernelInterface and related components. This is mostly to get things broken up and lay the groundwork for limited non-GET-html handling, but leads directly to:
  3. Replace our routing resolution system (currently over-coupled with the menu system) with Symfony's HttpKernel implementation, including the EventDispatcher component, and potentially the Symfony2 routing library. (EventDispatcher is the Symfony equivalent of hooks, essentially, and let us leverage not just Symfony's general workflow but it's complete, mature request handling layer.) This is what gets us the real power, and ability to respond to all kinds of HTTP requests with rich HTTP responses. We may retain some of the underlying elements, like the menu_router table. We'll work that out as we go and see what works best. This part may get merged with the previous step, depending on how difficult it is to separate.
  4. With that new flexibility, begin cleaning up our existing non-GET-html logic (form submission, Ajax commands, etc.) and adding new functionality (standard libraries for exposing entities as REST, etc.). Details of this part will be worked out later when we get there.

Some of those steps may get compressed, depending on what short-cuts we're able to take in implementation that make it easier to combine steps than not.

There are lots of related topics we should be exploring that we would be able to leverage with this new under-the-hood power. Most notable are Web Sockets and Server Sent Events and Standard Export Formats, both of which should be made possible by these efforts. If you want to help, those are good places to start.

For module developers, we'll need to stop thinking in terms of pages and start thinking in terms of requests and responses. What we now call page callbacks will mutate into "body-like blocks", which will be on the same level as response controllers for JSON, SVG, SOAP, or whatever else we feel like sending. HTML is a just special case of REST.

That in turn means routing will need to become more robust, which likely means heavy changes to hook_menu(). It may even cease to be a hook and turn into configuration objects from the CMI initiative. We'll work that out as we go.

Some low-level operations, such as HTTP error handling and fancier menu tricks, will also shift to a more Symfony-based approach. That should only affect a small number of module developers, however.

Global state variables, such as $user and $language, will be eliminated in favor of the Symfony2 request object. The super-globals will also all be deprecated in favor of the request object, as that is exactly what it is designed to do. That in turn allows us to make portions of code that can run anywhere, even in fake internal HTTP requests, Edge Side Include-generated requests, or pure unit tests (which are several orders of magnitude faster than integration test).

Things like hooks, Fields, Entities, and so forth should be largely or completely unaffected, save perhaps having a request object or request-derived information passed to them rather than using a global.

Other beverages

Something else that became apparent at the sprint is that some people were seeing "WSCCI" as even bigger than it already was. There are a lot of related "framework-ification" tasks going around, many of which have the same people involved in them as WSCCI, but are not directly related to WSCCI. That doesn't make them unimportant; in fact, many of them are, I believe, critical to Drupal's success, which is why many of the people involved in WSCCI have been involved in them as well.

Among those are the discussions around class loaders, namespaces, converting core code in /includes to PSR-0-compatible classes, dependency injection, and so forth. Those are important, and time permitting I and others will continue to work on those, but they are not part of WSCCI proper. Only issues tagged with "WSCCI" or in the WSCCI sandbox are "part of" WSCCI.

There are also ongoing discussions of other parts of Symfony2 that we could leverage in core. Most of that is not directly related to WSCCI either, but this is a great opportunity to reduce the amount of code we have to maintain. If you're interested in reducing Drupal's Not Invented Here syndrome, that's a great place to jump in.

We have only 9 months left to get most of this work done. I think we can do it, if we commit ourselves to it. Now is the time to do. Let's do.

That's an impressive todo list, especially considering the 8 month time frame. It would be awesome though, especially if you can do this and maintain an upgrade path for modules. And themes.

How do you plan to test your code and functionality?

I can propose to check out my project - Codeception. It provides writing tests in more natural and simple manner.

http://codeception.com/01-20-2012/starting-acceptance.html
http://codeception.com/02-15-2012/unit-testing.html

A test of overall module functionality can be easily checked in terms of Codeception functional tests:

I set up module
I save settings
I see response contains what I expected

chx (not verified)

21 February 2012 - 3:50am

  • Replace our routing resolution system
  • EventDispatcher is the Symfony equivalent of hooks
  • cleaning up our existing non-GET-html logic (form submission

So you replaced menu, hooks and form API. Wouldn't it be easier to just write a new system from ground up and then produce a migrate documentation and not bother with patches?

Hey Chx,

while I'm no expert on TIFKAWSCCI (the initiative formerly known as wscci), I suspect you may be right insofar as we can expect a few (very) large patches, instead of the usual drupal style of lots and lots of smaller patches.

So in this sense, what happens may be very close to what you suggest anyway.

Is this good or bad? Personally, I think good although it certainly won't be easy or comfortable. Boldness and technical improvement are, I think, key to the long-term health of a technology project. Yes, we can keep Drupal as what it is - an excellent CMS for serving HTML pages. But in the longer-term doing so will severely limit Drupal and discourage good developers from using it/getting involved.

What I call a professionalisation of the web (and PHP) has taken place in the last few years, and I think there is an increasing tension between accessibility for the beginner and Drupal's strength and viability as a product. We can either keep Drupal where it is and lose some great developers and zeitgeist, or change it but possibility make it less accessible to non-developers/programmers (although making it more accessible to existing and non-drupal developers).

From a more pragmatic perspective, if we look at where most of the core developers come from (Acquia, and the big shops like Palantir, Lullabot, etc.) it seems inevitable that Drupal will head in this direction - they are, for the most-part not weekend-warriors (even if they started out this way) and their commercial interests are increasingly high-end - hence the need to incorporate REST, etc. that much smaller shops are less likely to find value in. I think the idea is that WP can have the brochure sites and blogs; the future of the web lies in HTTP and multiple devices (not HTML and just pcs).

ps. I'm obviously not connected to the high-end decision making at all, and it may be that Dries, Larry et al disagree with my reading, so obviously the above is just what I think.

I think there's something you get wrong. Being able to quickly create a backend with Drupal to manage content and entry points to web services (entry points to nodes, users, other data), and to do it only clicking your way through the interface, will be *extremely* appealing to small businesses and freelances (as I am), because it would leverage a not-that-hard-tool to manage data, and then focus on creating mobile apps, other websites with other technologies, Flash front-ends, etc.

To be honest, this is the single most important thing I want from WSCCI. Give me a Drupal so I can quickly architecture and present my data to the rest of the world, so I can play with javascript or whatever and build awesome apps, mobile or not, but specifically mobile. Even freelancers or small companies will able to do it. And it will help Drupal compete with all the backends written specifically in Node.js/Ruby/PHP for each app. This is a BIG win, and it will be accessible to lot of people, small business included.

Another point related to small business against big shops is that these big shops have the resources to manage complex and high traffic websites. Why ? Because Drupal is slow, especially for authenticated users. So the "Drupal rockstars" they have mainly exist because Drupal created the performance problems, and the big shops needed highly skilled people to solve them. I mean, for a small business, it's really tough, almost impossible, to find one guy who can install and do normal Drupal stuff + install and configure memcache (or similar in-memory cache) + install and configure Apache SolR to replace the very slow Drupal search (I agree it's mainly related to MySQL not being a search engine) + install and configure Varnish or any other reverse proxy strategy that exist to avoid boostrapping Drupal as much as possible + install and configure MongoDB to store field data in a faster persistence storage + install and configure stuff like Reddis/RabbitMQ/whatever to store sessions, watchdog and the likes. All of them are necessary when you want to create a complex Drupal website, but none of them address the business logic of your Drupal website. No small business can afford that.

Now with the new WSCCI architecture, Drupal will simply perform better, and I hope far better, so even moderately complex/high traffic/both Drupal websites will only need a simple Drupal developer to add value to your website, not solve Drupal performance. This is another BIG win, accessible again to many people.

WSCCI for Drupal 8, or nothing !

The fundamentals of Drupal was written in a different time. And it has served very well. Drupal is now one of the top CMS'es. But now is a different time with an evolved PHP and differences in practices. While there is a ton to be grateful about with the code that today runs Drupal, it is not Drupal. Drupal is also it's functionality. And after the fact of functionaity it should be a good thing to rethink how the code aligns with it. It should be a good thing to look at how the code aligns with the greater programming community to reach out. And Drupal is also its community. The community is not tied to a particular way of writing code. Everybody want change now and then.

Hey Chx,

yes, sorry if my post came across like that. The first few para's were in response to your point (to suggest that while I suspect an attempt will be made to use the existing Drupal process (of patches), it will be in practice massive patches which are not really 'patches' at all, but big replacements of core code. So my answer to your question is basically that a re-write is basically what is going to happen.

The rest was just me sharing my thoughts on what is going to be a massive set of changes to Drupal and was not meant to be targeted at you so apologies if it comes across in that way.

To be clear, at least at the moment we're looking to use EventDispatcher only for those things that tie directly to Symfony components that use it. There are still some issues preventing us from entirely replacing hooks with EventDispatcher, mostly relating to hook/event registration. WSCCI will not be working on that problem; if others choose to do so, however, that's fine, or more likely we'll end up with both for a while and contrib developers can try to solve that problem for us. :-)

Similarly, WSCCI is not looking to replace all of FAPI. However, we currently process forms by partially rebuilding a page as if it were a GET request, getting through the form, then processing the submission and redirecting. That's horribly wasteful. With the ability to give each form its own unique REST URI at which to send a POST, we can bypass that ugly process and just go straight to the form processing. Once the request information has gotten to FAPI, however, we're not looking to make any radical changes there as part of WSCCI. Again, others may want to look into that (there's even a Symfony Form component, although I have never looked at it), but that is not a WSCCI task.

As for why to not just build an entirely new system and migrate? Well, that's Symfony CMF. :-) If we wanted to do that, we'd just drop Drupal entirely and go work on that. I do not. There's more to Drupal than just code, and by doing an "in-place revamp" we can retain those things that make Drupal Drupal; which does not include hooks, FAPI, or the menu_router table.

Thinker (not verified)

14 May 2012 - 8:32am

To-do list is totally correct, but if you are a newbie in this stuff, you shouldn't cut (or try to find any short ways) it. Otherwise you can spoil everything.