Drupal's audience priorities

Submitted by Larry on 23 November 2010 - 3:04am

I've spoken a great deal recently about architectural priorities. In short, we as software developers cannot eat our cake and have it too. Improving flexibility in one area may hurt performance, while improving usability one another area may hinder flexibility. These trade-offs are not necessarily right or wrong, except in the context of the goals and purpose of the project with respect to its target audience.

But what is Drupal's target audience, and how does that impact our architectural decisions?

DrupalCon Copenhagen was a watershed event in terms of understanding how to conceptualize that question, in my view, based on conversations I had with the likes of Mark Boulton, Jen Simmons, and Sam Boyer. In his (excellent) keynote, Jeremy Keith noted that the HTML5 Working Group had a specific, defined set of priorities:

In case of conflict, consider users over authors over implementers over specifiers over theoretical purity.

That may be a good priority order; it may be bad. That depends on your point of view and your goals. It lays out the order in which different stakeholders should be listened to, and if you come to a decision where you have to screw one group over to benefit another how that decision should be made. Having such a clear understanding of your constituent priority is critical to making good, consistent architectural decisions.

So what are Drupal's priorities?

Constituencies of Drupal

One of the problems Drupal faces is that we don't have a clear, explicit understanding of how we prioritize our constituencies. As a result, we're struggling with "we're really trying to make it easy, really!" while people are still screaming "I can't !*%! use this thing, Drupal sucks!" Both statements can be true if they are about different constituencies.

If I could be so bold as to describe the Drupal community's implicit, ill-defined design priorities, I would define them like this:

In case of conflict, consider site builders over content editors over programmers over theoretical purity over CSS designers over HTML designers.

That may be a good order or a bad order, but it is the order that, in practice, we seem to follow in my experience. But let's consider what each of those means:

Site builders
By site builder, I mean the click-together site assembler. This is the person who creates node types by pushing buttons in the CCK/Fields UI, then clicks through the Views UI, then possibly spends time giving the Panels UI some mouse-clicking love.
Content editors
The content editors are the site administrators responsible for keeping tabs on content. Not content types, but content. Creating and editing a site's primary nodes. Moderating comments. Etc.
By this I mean anyone who is writing PHP or Javascript code. Pretty self-explanatory, with one caveat: Editing template.php counts as a programmer task. I don't care that it's front-end stuff, it's programming.
Theoretical purity
This is something of a catch-all for architectural cleanliness, security, performance, non-repeating code, code that non-Drupal-fanboys can comprehend, etc. (For architecture geeks like me, having this be so low on the list is frequently painful.)
CSS designers
Also self-explanatory. If you spend your days in *.css files, this is you.
HTML designers
For Drupal, this means people who edit template files. It also includes those whose primary skill is HTML in general.

Notably absent from this list is that weird Drupal-specific role we call "themer". There is, really, no such thing. It's a weird amalgam of template.php-dwelling programmers, CSS ninjas, and HTML gurus rolled uncomfortably into one. That is a fallacy that, I believe, hurts us in our understanding of our own system.

Sorry, designers

Also of particular note is the dead-last placement of the HTML designer. That is actually a deliberate decision, and a consequence of putting site-builders first and foremost. HTML is incidental to Drupal. We treat it as little more than an API.

Drupal treats HTML not as a user base in itself, but as an API between PHP and CSS.

Again, that could be good or bad depending on your point of view. However, consider that artistic web designers, those that actually figure out how the page should look; If they know anything about implementation are going to know HTML and some CSS. They're regular readers of A List Apart, which is all about the new hotness in HTML and CSS. For them, the way Drupal treats HTML as simply a means and not an ends is like nails on a chalkboard. They respond just as violently to it as Drupal's PHP developers do to, say, WordPress's "just dump everything into the template file, even if that means SQL" design. (Note: I'm sure WordPress developers will defend their architectural decisions; I am just noting that Drupal developers are horribly turned off by it, much as HTML designers are horribly turned off by Drupal's template design.)

Of course, there is no way to not have messy HTML while still retaining the preeminence of the click-together site builder. CCK, Views, and Panels cannot let a button-pusher build awesome sites "without writing code" without having ridiculously generic HTML output with every possible CSS hook you might need. And pristine, semantic, just-so HTML cannot be generated by a general purpose tool. Modules like Semantic Views help, but they're really just a short-hand way for HTML designers to hack selected templates. They don't change the architectural design.

Is that a good trade-off? If you're a site-builder, it's a great decision. If you're an HTML-savvy designer, it seriously sucks.

Casual architecture

Similarly, even within the programming ranks the same conflict exists. Drupal tries, culturally, to cater to the "casual, self-taught PHP developer", the "weekend warrior" who knows just enough to be dangerous. We want them to be able to throw up community sites over a holiday weekend to aid in the democratization of the Internet. That's an explicit goal.

The flipside, of course, is that we routinely turn off professionally trained software engineers (which are not the same as programmers), software architects, and anyone who has experience with more traditional architectures. We are the last serious PHP framework to still be struggling with the idea of using OOP. We have been doing a hacked-up Aspect-Oriented Programming approach for years, but didn't even know what we were talking about so haven't always leveraged it intelligently. We collectively don't have a clue about performance. We architect for what sounds like a clever hack without considering the systemic implications... if we bother to "architect" at all. We routinely integrate systems that have no architectural reason for being related in the name of site-builder usability, ignoring the performance and bug-hiding risks that come with that.

We are a weird amalgam of a professional CMS built by amateurs and an amateur CMS built by professionals, with all of the ugliness that comes with that.

Is that a good trade-off? If we want to appeal to the thousands of casual self-taught PHP hackers in the world, yes. If we want to appeal to the thousands of professional software engineers in the world, no.

The real target audience

Of course, as with any open source project the only constituencies that matter are those that do work, and the more work they do the more they matter. Who is Drupal's target market? The people building Drupal.

The practical reality is that Drupal's primary target audience is Drupal consulting shops.

Why? Because most of the leading core and contrib developers work for Drupal consulting shops or are freelance consultants. They're site builders and programmers, and their paying clients are content editors. They are not HTML/CSS ninjas nor designers. It doesn't matter how much we insist that we are the everyman CMS, or how much we care about end-user experience, or how designer-friendly we say we want to be.

When you get right down to it, our target market is us.

Tough choices

Of course, all of that begs the question... is that the right priority for us to have? Should we really favor site builders at the expense of architectural quality and HTML designers? If we want to make Drupal easier for HTML designers, are we willing to sacrifice button-pushing power to do so? (We would have to.)

Is making the admin easier for content editors (as the D7UX project aimed to do) at the expense of programmer frustration a good trade-off? (Maybe?)

As we begin to adopt HTML5 for Drupal 8... who will get to decide where to use what new-and-cool HTML tag? The site builder? The programmer? The HTML designer? I don't know. But I do know that we need to not delude ourselves into thinking that we can let all three do so with equal ease. We have some hard decisions to make, but they must be made explicitly, not implicitly, if we are to survive the next wave in our evolution.

Very well said. I don't have any answers about audience priority, but I've thought for a while that drupal's direction is being driven by money. That is, those drupal shops which are able and willing to fund employees to work on core effectively shape its architecture. The more patches – and the bigger those patches – you can write, the more influence you're likely to have over the architectural direction.

That's not necessarily a bad thing, just a fact of life.

So in answer to "who will get to decide where to use what new-and-cool HTML tag?" I'd say: whoever has the time to write, promote and pursue the system wide patch it will probably require :)

It's not just a question of a shop funding development, though. It's also one's experience. 99% of my own core work is on my own time, not Palantir's time. However, what I work on is informed by and driven by two factors: 1) What I find cool and 2) What will make my life easier at the day job.

Perhaps even more because I'm not getting paid for it, what I choose to work on is driven by my needs, not by the needs of some other constituency that's not giving me money. I suspect that's true for the vast majority of core contributors.

Great write-up, Larry. And great you try to be not too opinionistic.
Everyone has to think about if these things are good or bad.

Thinking in personas helps a lot IMHO to untangle a lot of jammed discussions. What is good to one target audience might be bad for the other and we need to keep this in mind.

If there is a decision to approach a new audience, basically it needs consensus. In quite some cases in the last three years there was no consensus, so the fighting was to begin...

As with other discussions, the tug-of-war between click-it-together projects and large-scale ones is a serious consideration. The needs of "Content Editors" on a local news site are very different than those on Verizon's corporate intranet, and those in turn are very different than the editors on a social collaboration site (effectively, all the active users). This doesn't mean that we can't work to improve the experience for all of them, but it should definitely cause us to think carefully about the audience inside of the world of "content editors" who will benefit.

Today, Drupal's editing experience is schema based: node forms, overview screens, user management tools, and so on are essentially windows into the content storage model. That's a very comfortable approach for developers and site builders, and in a lot of ways it's similar to the automatic administrative screens that systems like Rails and django can assemble for developers automatically. A number of tools also allow us to pretty up those screens with role-based disclosure, conditional hiding of unnecessary information, and so on. At the end of the day though we're still offering "windows to the schema."

The real long-term solution to privileging content editors isn't incremental polishing of our CRUD screens; it's the development of task specific tools and interfaces that are tailored to the needs of those who do certain kinds of work. Pieces like Actions and Views Bulk Operations allow us to wire together some of those easily, but it's an approach that is very unfamiliar for many in the Drupal community.

At the end of the day, that's the secret sauce of WordPress' success: relentless focus on the small number of tasks necessary to create and maintain a single or several-user blog. As it grows into areas of beefier functionality, it risks losing that focus.

If you can't create a site, then you don't have a site to write content for. I would say this is what we should shoot for:

In case of conflict, consider designers over site builders over content creators over developers over theoretical purity.

Why? Because if you solve correctly for the designer, most likely you've also solved for the content creator. The reverse is not true.

Solid write up Larry. It's so true, the community is definitely building for the community.

Jeff, if we're serious about building for designers over everyone else, here's what that would require:

- Kill the giant arrays of doom.
- Render API goes away, template preprocess functions go away, hell most of the theme layer gets ripped out.
- Template-first development.
- Views become something you setup via a long function call in a template file directly. (Say good bye to alternate template engines.)
- Completely reverse the entire page life cycle. We start by loading page.tpl.php and rendering it. Oh, you want a different layout? That's another page-foo.tpl.php.

Sure some of that could be exposed via the UI as forms, but there's considerable challenges (including security concerns) to editing templates via the UI, unless you build everything off of Panels.

(Of course, the Butler project aims to do just that, make the Panels pull-based model the default rather than our current inverted, inside-out process, in a large part in order to better target designers. There's still a few thousand hours of work yet to get there, and we haven't all agreed on the implementation details.)

Changing our priority list in a non-trivial way is going to be painful. Very painful. As in, "makes the D7UX fights seem like a fun date" painful. We may need to do it, we may not, but it *will* be painful.

That's certainly a defensible position, Dries. However, as noted above as long as we prioritize button-pushers over HTML jockies Drupal will *always* be seen as designer-unfriendly.

Putting developers that low is also therefore putting our primary engaged-community-member constituency at the end of the line. We're also at a size and complexity where we can no longer afford to ignore theoretical purity. "Theoretical purity" is not a negative thing; academic understanding of software architecture is based on decades of experience, and for us to ignore that is the height of hubris. Ignoring the entire rest of our industry is how we end up with the utter trainwreck that is system.module, and the horrific circular dependencies that result. And that causes bugs that affect everyone.

There is no easy answer, unfortunately. :-)

Before starting to use Drupal circa 2003 I was used to hand crafting CSS and HTML, and I find a lot of the current brouhaha with designers claiming unfriendliness and moaning about markup semantics echo'd my own thinking back then 'omg look at this markup'.

After upgrading a few sites through several major drupal versions I've developed a more pragmatic view. Maintainability is at least as important as good semantics to me now.

I worry that if the HTML jockies come too high up the priority list they'll just run into the upgrade wall instead 'omg it's too hard to upgrade! half my custom markup makes no sense anymore!'. Or worse they'll build beautifully crafted sites for clients and then walk leaving the client with an upgrade headache and drupal with a bad rep.

It worries me that in this debate over HTML purity and supporting designers no one ever mentions maintainable markup. We need to set a good balance between semantic purity and maintainability.

Note: Please remove above

Although this is the 50th time we discuss this, I still do not feel we have much clarity on the matter. The whole priotisation discussion, still focuses almost soley from a technologist perspective. If we want to turn this a good understanding we are missing the business and user axes.

The Real target audience
If you are talking about audiences, and you outline 6 of which 4 are programmers in some way - to me this shows how biased this analysis is of stakeholders in Drupal.

We can definitely not ignore the fact that we are big and thus many conflicting priorities exists. However the reality today is that passionate people work on the stuff they find intresting, although D7 has seen a large influx of UX work, if you put in perspectice its probally equal or less effort compared to the DX work. However UX work is far more easy to be questioned.

To me it's a silly idea of having a "real" target audience, it dilutes the fact that Drupal has grown big and therefore now has many diffrent stakeholders and thus audiences involved. Although having one audience, is admirable I think it often unrealistic for a mature product - unless that audience is "a human".

Consider this very hypothetical usecase, for every developer - who builds 1 website, there is 1 end-user updating and creating content. Assume you build 10 websites in your liftetime - that is 10 end-users, per developer! This is ofcourse a silly example, but in terms of actual numbers end-users by-far outnumber developers. My only point here, is that your audience analysis is only covering a very small part of the spectrum.

I might be sounding a bit harsh, but I think this is analysis is very obviously flawed. It states a fact, most contributions to core are for us, thus we should be building *it* for us? Even when the product Drupal itself has many dimensions of users interacting with it.

Opensource love
I think essentially it comes down to a core value. That of opening up technology like Drupal for more people to use. It sounds a bit like an opensource mantra, but I feel its admirable goal - and never one to lose out of sight, making new silo's to me holds little admirable value.

For some reason, there is a false assumption that Drupal UX is very usable for technologists, however most of the issues we find in usability tests equally effect both end-users as programmers. Are our differences more important then our similarities?

I think this is a circular discussion, we have to take care of several stakeholders anyway.

I deliberately left out "users" as a constituency, because like "Themer" it is a grossly misleading term. Which set of users? Site builders *are* users. Content editors *are* users. The D7UX effort dealt with those groups.

True, there are millions of "visitors" to Drupal sites that fall into none of the above categories... but as far as architectural decisions go they should not be able to tell. The ins and outs of the theme system are, or at least should be, completely irrelevant to the visitor constituency. Catering to them is the job of the site builder and the rest of the site development team.

Visitors aren't unimportant, they're just mostly irrelevant to questions of, say, render API or database abstraction or plugin architectures.

Also, I think you missed my point entirely. I very much did not say "most contributions to core are for us, thus we should be building *it* for us". On the contrary, I pointed out that despite our protestations and efforts to the contrary the primary constituency for Drupal, and almost any other open source project, is the people working on it. That's not a good or bad, nor is it a "should". It is simply an observation.

Focusing on a different constituency instead is possible, but very hard.

I also agree completely we need to take care of several stakeholders. My point is that to think we can cater perfectly to all of them equally is the height of hubris. We cannot, because often their needs are at odds with each other. What makes life easier for a content editor may make life hell for a site builder. What makes an HTML designer happier may make a site builder pull his hair out.

These priorities do exist, and we pretend otherwise at our own peril. I'm not saying our current priority order is good or bad, simply that it is. And we have to understand that, explicitly, and come to terms with it, if we want to change it, and figure out what we want to change it to.

You are correct a majority of architectural decisions will not effect end-users. Although I agree that it is the job of "site builders" and the rest of the site development team to cater to them, right now that is either not or happening only a little. Unless Drupal shops step up and contribute significantly to this, it will still come down to the community to solve these problems.

You can never cater to everyone perfectly, to be quite frank you can probably not cater to even one perfectly (unless you are designing something for one person). However I think your black and white argument of the content editor vs. the site builder is meaningless, they are both important stakeholders we have to cater to. And although occasionally at odds, they largely live in different areas of the interface.

In terms of usability and ux? You are probably overestimating the differences between audiences. Especially for future references, you might want to create more clarity between audiences and their requirements - rather than how it impacts the UI.

Although this is the 50th time we discuss this, I still do not feel we have any clarity on the matter - primarily because this is prioritisation in many ways from a programmers perspective. Or rather from a technology perspective, if we want this to make a good understanding of priority we are missing two other axes that of business and users.

I mean the list includes 6, of which 4 programmers in some way (or at least technology at the for front, not their actual main activity). This is not a very objective stakeholder analysis.

Even then, we cannot ignore the fact that we are big and thus many conflicting priorities exists - the reality however still is that we solve the problems we find interesting. If you look at D7, you could say a lot of focus has been on UX - but if you put it in perspective on the full amount of changes its probably equal (or maybe even less) to allot of other DX's. However the differences are obviously far more perceivable.

I think when talking about the "real" target audiences - and not even mentioning who ends up using that Drupal site, shows all the false assumptions made in the reasoning. I think its very natural to assume that you yourself are the target audience because 1) Designing for yourself makes sense (you know what you want, ideally), 2) Preaching to the Choir(comments on this blog are 95% of technologists). The reality again, is that we have grown considerable - there are many stakeholders of which both end-users and programmers are considerable stakeholders.

Very nice article. I agree about the current audience order and real target audience. I also like how you parsed out the themer role.

We collectively don't have a clue about performance. We architect for what sounds like a clever hack without considering the systemic implications... if we bother to "architect" at all.

This is way too harsh. A bunch of us run drupal core through xhprof and cachegrind over and over and over. We submit patches and write reviews and so on. Effort and talent here is on par with major subsystems like Content, UX, etc. And the results speak for themselves. Drupal 7 core is fast, and scalable. It gets slow only when you gorge at the module buffet.

I've long been frustrated by Drupal's insistence on serving multiple masters (small sites on shared hosts and 50 person web site teams). IMO the smallcore ideas gives us a way to target both audiences efficiently. smallcore is dormant, but not dead. I still think it will prevail.

For the record, I don't consider #smallcore dormant at all. The "name" is, but the goals have been shaping quite a few key patches, in-progress contrib projects for D7, and so on. That's where the rubber meets the road. ;-)

Agreed regarding the amount of work that goes into optimizing Drupal's performance and scalability. I think one of the frustrations that's shared by a number of developers is the sense that performance-versus-features is a tug of war rather than a tradeoff: one group of devs will often push for a new feature/API/etc., and a different group has to crack the performance whip. Often, they arrive on the scene after the performance damage has been done, and are forced into cleanup mode.

That's inevitable on any sufficiently large project, so we can't beat ourselves up about it too much, but it's definitely part of the dynamic.

Often, they arrive on the scene after the performance damage has been done, and are forced into cleanup mode.

Exactly. We don't consider performance (either CPU time or memory usage) to be a feature. We consider scalability a critical feature, we consider "ease of hacking into something in unholy ways" a critical feature, but we don't consider performance a critical feature. We spend inordinate amounts of time making sure we can always add another alter hook for flexibility; how much time to we spend making sure Drupal will deliver an uncached page in under 100 ms and 16 MB of RAM (the PHP default)? Frankly, almost none.

There are people in Drupal who spend a lot of time with benchmarking tools, but as Matt said that's not what I'm talking about. That's not taking performance seriously.

When we start seeing big-O notation analysis of algorithmic complexity as a standard part of our patches, then we're talking performance seriously. Until then we're just making a token gesture.

Eaton (not verified)

23 November 2010 - 11:36am

In reply to by Larry

"When we start seeing big-O notation analysis of algorithmic complexity as a standard part of our patches, then we're talking performance seriously. Until then we're just making a token gesture."

I don't think it's fair to call the work that's done a token gesture -- there is a lot of really serious work being done by people who care about it and push real improvements through the patch and commit process. But as you point out, today scalability is treated as a higher goal than performance in a lot of cases. There's definitely a case to be made for that: the odds that a memory-constrained site on $5 shared hosting will ALSO be grappling with the challenges of massive growth are slim. Sites feeling massive performance crunches generally have some way of scaling OUT, and that's an approach that is generally more sustainable than forcing Drupal to rely on beefier and beefier single-server setups as it grows.

However, when that emphasis on scaling is treated as a solution for performance planning (not performance optimization, but performance planning), it can force smaller and smaller sites to begin the hard jump to 'scaling out' earlier than they would have otherwise. Basically, we're lowering the barrier for scaling out but we're also forcing people to do it sooner.

True, I don't want to belittle the work done by those who spend 100+ hours in XDebug profiling Drupal 7. We are far better off for their work. My point is that is still too little, too late if we really want to get serious about performance.

As far as scalability vs. performance goes, you're exactly right. But consider also that the most scalable system is the one that doesn't need to scale; Making it easy to add 4 more servers is great, but not needing to add them in the first place is even better.

And that's an architectural question that cuts to the core (pun intended).

I agree that developers/engineers should pay more attention to performance; Personally, I think performance is more important than scalability. I blame that Drupal has low barrier of entry (ok, it's higher than WordPress, but still very low, since it doesn't require people understanding asymptotic analysis). and we lack of a top-down policing approach to reinforce performance.

As someone who spent 100+ hours in xdebug and xhprof against Drupal 7, there's several problems we have with performance:

* Almost no-one developing for core or contrib does either profiling or even basic benchmarks for regressions while developing, certainly not as part of their normal workflow. I include myself in this when I'm not specifically working on performance issues.

* Only a slightly larger group keep performance or scalability in mind while developing. Security, E_ALL - these at least get a passing thought or at their best are built in to coding practices for many of us. However I routinely (very routinely) find major, major issues whenever I profile a module or site for the first time. Contrib modules setting variable_set() on every page (including very popular ones), filters preventing text format caching, full file directory scans on every request, quite simple queries that don't use indexes. At various times in the Drupal 7 lifecycle I've found issues as bad or worse than these in core and almost always after they'd been reviewed and committed without a passing thought.

* To my knowledge I was the first person who put Drupal 7 through a profiler in any depth, this was about a year into code thaw, but that time the damage was already done, so it was always going to be playing catchup. Fortunately a few people joined in during the process, and I'm hopeful Drupal 8, despite or hopefully because of the various performance regressions in Drupal 7, will start from a better point.

* We definitely don't architect for performance, and in my experience any patch which is focused primarily on performance and is not fixing a regression that was introduced with a feature, DX or cleanup patch has a very hard time getting in. Like eaton said, performance isn't considered a feature. The burden of proof on those patches is significantly higher, the barriers to review are higher - not as bad as postgresql or OpenID, but those are niche areas, whereas performance is a different category of problem.

I think this comment is missing the point though:

When we start seeing big-O notation analysis of algorithmic complexity as a standard part of our patches, then we're talking performance seriously. Until then we're just making a token gesture.

The biggest problem with performance in Drupal is not that people do not discuss the theoretical impact of performance patches enough, it's that this is all that happens in a lot of cases. What is ignored is analysis of actual usage, and data to back up changes (especially for patches that aren't primarily concerned with performance).

I've seen a lot of 'performance' patches that tried to optimize for loops over 10-20 items in admin page callbacks, that's all very well but it's not really helpful when you have patches get in that break page caching, memory leaks of several mb in some of the most popular contrib modules, etc. etc. mainly because people aren't looking.

I have hopes that we can make architectural improvements for performance in Drupal 8, particularly sllmming down the bootstrap, hopefully improving the rendering pipeline etc. but there also needs to be a cultural change, equivalent to the introduction of simpletest in D7, in terms of how performance is treated relative to other concerns, because there is more than enough work for dozens of people just to deal with the damage control at the moment.

And yes, a cleaner split between "architecture and framework" and "application" is a non-trivial task, but is one way to make the pain experienced by those lower on the totem pole less. It doesn't remove the tradeoffs but it can change the costs associated with them.

There's a reason I'm big on architecture these days. :-)

Asher Webb (not verified)

23 November 2010 - 8:47am

Way to go with an article that gets into the innards of a complex (and sometimes convoluted) CMS. As someone relatively new to Drupal, who comes from a CSS and HTML background, I can relate to some of the issues. First of all I definitely agree HTML and CSS designers should be close to last on the list of the target audience. One of the biggest boons of drupal is the complexity of a site you can build without having to hardcode php and JS. Now of course in the back-end, people developing core, have done this in some great ways to save the site builders the pain of having to do this. This makes drupal highly extensible which in my opinion is really where it shines.

But the point-and-click site builders will get fed up with that method of developing drupal rather quickly if they wish to build anything more than the simplest of sites. I mean sure this in a necessary evil (or good) depending on who you ask (and their familiarity with the command line). In my opinion drupal should target site builders who use drush, who understand the basics of server-side technology and the command line. These people can make complex and flexible sites relatively quickly. They can incorporate the css and html designers along with the hardcore programmers into the mix to really drive innovation with the CMS and it's ecosystem. I think the biggest mistake for drupal is to try to appeal to your average mom-and-pop site builder who doesn't have a good understanding of web technology, trends and best practices. Let WordPress have this market because Drupal is more suitable for higher expectations!


23 November 2010 - 11:36am

In reply to by Asher Webb (not verified)

Let WordPress have this market because Drupal is more suitable for higher expectations!

That's a very dangerous position to take. "Cede the low end" is exactly how platforms fail, because whatever takes over the low end is then well-placed to move upscale and push out the old dinosaurs.

Minicomputers did it to Mainframes. PCs did it to Minicomputers. The web is doing it to desktop applications. Linux did it to Unix on the server. Palm did it to Windows, and for years survived only because it dominated the low-end of the market. PHP did it to JSP.

Drupal has done that to proprietary CMSes. If we cede the "I just want to put up pictures of my cat" market to WordPress, then when those people decide to put up a website for CAT instead they will stick with WordPress, not go to Drupal.

I was just on a plane coming back from BADCamp last week next to a woman whose company just standardized on WordPress for all of their sites. Non-trivial sites, non-trivial company. (Green building firm; something that would normally be a great target market for Drupal.) That's not just a dozen sites not running Drupal, it's dozens and dozens of people at that company now becoming WordPress experts instead of Drupal experts.

Cede the low-end at your own peril.

Eaton (not verified)

23 November 2010 - 11:46am

In reply to by Larry

I think one of the primary points of confusion when discussing the CMS market boils down to a fundamental lack of creativity. Complexity isn't a continuum from Blogging to CMS to Enterprise CMS -- it's a continuum from task focused tools to multipurpose tools to frameworks.

Task focused tools will always, always beat multipurpose tools when users are trying to perform that particular task. The rise in blogging propelled WordPress to success because it was a better blogging tool than other blogging tools, not because it was a better general purpose CMS than multipurpose tools. I think a lot of people are disheartened at the idea of "fighting" WordPress, in large part because it is very good at what it does. We don't have to turn Drupal into the tightest, simplest, easiest blogging tool to remain competitive in the "low end" however. Especially with distros, we have the ability to create excellent turnkey brochureware sites, personal portfolio tools, community decision making tools, and so on faster than dedicated tools can be built from the ground up.

There is more to the low end than blogging, and we would do well to remember that. Chasing other products' success is easier (from a creative standpoint) and harder (from an implementation standpoint) than finding needs that are not well served by existing focused tools.

Asher Webb (not verified)

23 November 2010 - 11:55am

In reply to by Larry

This is a good point. I guess I didn't mean to say that Drupal should cede the market for simple sites to Wordpress. I just think that Drupal's core development doesn't really need to target this market. If Drupal core gives site builders the tools to create frameworks and distros that can accomplish the simple tasks extra-ordinarily well than this is a great option. Then we will really gain traction, not for creating simple websites (which can more easily be accomplished by Wordpress) but by giving site builders, programmers and designers the tools to make frameworks for any kind of website (simple to complex)

I just think Drupal is great because it can give non-programmers the tools to build web applications, and power to everybody who is building distros and web apps that allow end-users to build focused, effective websites.

Drupal doesn't need to focus on being simple.

Vingborg (not verified)

23 November 2010 - 6:57pm

In reply to by Larry

How is this dangerous? I know what you are getting at, but I fail to comprehend how it represents a danger? Competition, perhaps, but certainly not a danger ... not if Drupal is worth a damn in its own right. Let us not forget that seemingly trivial detail. The "low-end" and the "high-end" contenders can be sources of inspiration, but only if we recognize their respective strong points ... and the weak points of Drupal.

For all its glory, Drupal is not always the answer, nor can it ever be.

That being said, I find your analysis supremely relevant.

The danger is in the idea to leave a market to a competitor because there is a bigger market to focus on. This at least is the understanding of disruptive innovation, many cases show how good technologies flee upmarket to get pushed-out at the upper end. It is good to prioritize as been discussed in this post, but an expression as:

Let WordPress have this market because Drupal is more suitable for higher expectations!

Can indeed be dangerous even if in the end the prioritize leads to the same outcome. So it is the expression and the reasoning behind it that is dangerous.

you have to pick your battles, trying to take on WP is futile. I think it's true that joomla is a better competitor than WP for Drupal. We don't have to cater to all markets. I think we have a more robust eco-system than WP, with a more open-source feel. We have to take risks to succeed, so maybe this line of thought is dangerous, humans have to innovate when faced with danger. You know views and CCK are great, although perhaps the low-end of the market would rather not be faced with such "complicated" things. Put it this way, does Apple cater to the low-end of the market, NO, and they are being heavily rewarded for their focus and marketing--and moreso for their innovation. Lets keep the core Drupal focused on innovation--built in RDF is great part of D7 core and really promotes semantic web, and the HTML 5 group is great as well.

As far as catering to themers/html jockies/css freaks why??? People who love drupal make sweaver and skinr module and Fusion themes, so why try to please this camp in core. Just make it to where css isn't broken. As far as focusing on site editors I recommend 3 things--2 for drupal 8 and 1 for now--1.built in wysiwyg editor. 2. Better, simpler, more elegant content editor's dashboard. 3. BETTER DOCUMENTATION.

Focusing Drupal to be built for the Drupal tribe is the way to go. And then Drupal tribe can work magic with Drupal and make money of its being incorporated into the mainstream.

I don't know that it's accurate to compare Wordpress to Drupal in that way, though. Wordpress is the easy one to put up next to Drupal because Wordpress has the widest range (and largest number) of users - but there is a huge chasm between what Wordpress is able to do and Drupal does (core + contrib), as well as the type of users.

It's important to remember why people use Drupal, how they use it, and who they are. Because of that, I think it's much more accurate to compare Drupal to something like Joomla - what does their build team offer that Drupal doesn't? How similar is their community to ours? Do they prioritize the experience for the largest demographic of users?

I like this. I use Drupal because it's the wweb's equivalent of a big box of Lego, and my mantra is 'never hack the core'. So please, put the needs of site builders first, then editors, and so on. People who know how to hack, please contribute to Drupal.

I would consider content editors + site builders to be on the same ground. I am amazed at how quickly I can now throw together a full-featured website (minus any special/custom functionality) in a weekend, without touching one line of code (PHP or HTML—usually a little CSS).

For any large project, there are bound to be a lot of customizations, tweaks, etc... but I think where Drupal really shines, and makes some great inroads, is in its ability to be a quick 'put this together and get it done' tool. You can build out some great functionality using just Views and D7 core now. I think that's a great thing.

Having mostly a design background, I remember early in my Drupal days being worried about the HTML and 'purity.' But after finishing up about 60 vastly different Drupal sites (just did my first D7 site this weekend), I am worried less and less about the code, and more and more about getting things done and helping people. There's no way I could be this efficient with another CMS.


This is one of the best analysis I have written about Drupal and the path it is taking.

I believe your timing is perfect to initiate a discussion about these issues, which all the great comments also confirms.

I have only been a member of the Drupal community for just over two years, but in that time I have seen the massive growth it has done, and also the acceptance and inroads it has done on the enterprise.

I have been thinking a lot about what is going on and where the project is going. My feeling is that things maybe have been going a little to fast in some areas, which have resulted in a bit of tunnel-vision. With that I mean, it has been easy to get the feeling Drupal is on an unstoppable train.

Your post have already proven that we need to have a deeper discussion about what is good or bad, but also how we better can plan for the future as you give many good examples about areas where Drupal still needs to improve a lot.

I have no doubt about that the community is capable of having this discussion and come up with ideas and plans on how to best proceed from here to be able to cope with the growth pace we are in.

Thanks again for initiating this.


Larry, you have a better breakdown of audiences than what I've seen before. The ranking of priorities (site builders > content editors > programmers, etc.) probably suffices for this discussion, although I would try a heuristic that addresses things a bit differently. More like weighting the audiences rather than a strict hierarchy. (EG. give each group $100 to bid on the features they want--although this might be hard to implement). If HTML designers are at the bottom of the food chain, chances are they won't eat--they'll only be eaten.

The whole idea is a bit theoretical, but worth thinking about. But the reality is that some of the groups are not as well represented in core development (nor in the development of major contrib modules). A group's priority doesn't have much meaning if their voice is not heard.

It has been pointed out that the low-end is important, but it is the low-end users that don't have much community involvement and consequently, not much of a voice. The strength of Drupal is the community and we must make sure the community is representative of every group and that each group has their voice.

As a postscript, I'm with you on the theoretical purity...

Alex UA (not verified)

23 November 2010 - 3:37pm

I think it's very misleading to say that "the practical reality is that Drupal's primary target audience is Drupal consulting shops." Drupal shops exist because of business/organizational needs, and it is those needs that drive development (we try to do 20% time for non-development work, but rarely are able to take that much time). What does it matter if Sony pays Earl to work, for several years, on Views/Panels/CTools etc, or if they hired a shop to do the work? Outsourcing doesn't change the nature of the demand, which is driven by client needs, not by any of those people you list (who all work for someone else). As an owner of a Drupal shop, I technically work for myself. But, given the regrettable fact that I was not born rich, I have to work for "the man/woman," and those clients (and our attempts to server them better and better as time goes on) are just as important (if not more important) drivers of the platform as are the developers, IMO.

Yes, C.R.E.A.M. is as true as it ever was, but thank god that cash is paying for something awesome that anyone (who can figure it out) can work with for free.

Now, back to the salt mines!

Those companies like Sony, Examiner etc. that have in-house developers who do significant core or contrib work are 'Drupal shops' even if they're not consulting shops.

Also in terms of what drives development I think things go both ways. Sony was paying Earl to work on Views/Panels because it was in their interests to have those site building tools available to build a large number of sites without doing custom development for each one. However Earl didn't start working on those modules because of Sony, and doesn't work on them for Sony now. I'm sure there are plenty of organisations around that have a similar need for dozens of sites, that are not investing time and money in generic tools like views and panels (or patches for them) - despite having the same overall business need.

There is a massive layer (and sometimes wall) between business needs for the people with the budget, and nice contributable code, we know the arguments for them but they are not necessarily followed or properly understood. Also the 'Drupal consulting shop' isn't that common an entity compared to other communities (based on the summaries I've seen) - companies based around Joomla are selling actual code/themes to site builders rather than site-building services, Automattic is based around wordpress.com (so massively invested in a strictly controlled workflow). A lot of web development shops or departments are using a bunch of different systems, both new and legacy, and aren't committed to any particular one.

So while people are only getting paid to work on things that someone wants, how they do that varies massively, and there's a massive filter between those two things - particularly with Drupal where we thrive on conceptual abstraction and indirection, so that often when actually working on code, it's a couple of levels removed from the feature requested - since the actual feature will be implemented by clicking around.

Vingborg (not verified)

23 November 2010 - 7:51pm

I mostly agree with your assesment that the primary target audience of Drupal is, first and foremost, the Drupal community itself. And since the community is, arguably, one of the strongest selling points of Drupal, that is an important point to make. But at the same time, that very community is so many things to so many people, that it, IMHO, transcends easy definitions.

What you ask, between the lines, is an impossible question: who belongs to the community?

That being said, I firmly believe that it is possible to create a framework that would cater reasonably to all of your constituent roles. But it probably wouldn't resemble anything we currently recognize as Drupal. It might not even be written in PHP.

Grant (not verified)

23 November 2010 - 8:17pm

Aside from all the programmers here (I'm one too) agreeing with a viewpoint very much in alignment with their own, I thought I'd present a different perspective on the finding, "The practical reality is that Drupal's primary target audience is Drupal consulting shops."

I'll start with what I always bring up, that the Drupal community really took off after the programmers were no longer the only ones involved. Talk is silver and code is gold and all that, an assumption that the contributions all still come from coders. That's a crock. What about those who run DUGs, Camps, Summits and Cons? What about those who document, those who evangelize, those who test, those who suggest enhancements, those who review, those who give help in IRC and in forums, etc, etc. I say, talk is silver and contributions are gold. Yes, the coders still dominate because they control the core of it all and are still the biggest group, but there are always PMs, designers, sales folks, muddlers, hobbyists, etc at every Drupal gathering and maybe fewer than we'd like because we coders tend to dominate the conversations and make it less useful/fun for them.

Further, while your conclusion sure does apply to a large number of Drupalers out there, it's a generalization that I feel falls well short. I've seen too many users pick Drupal for themselves after doing their homework (Drupal's awards and results in certain surveys count for a lot). In some cases Drupal got picked after seeing demos of several CMS. Word of mouth is also one of the biggest factors, quite possibly the biggest. Nonprofits share info between themselves all the time and seem to pick their CMS themselves a great deal of the time, and then find a vendor who can give them their existing choice. I've seen Drupal picked by people at tech conferences, after going to a Drupal table or panel.

Your comment also excludes the large number of Drupal sites built in-house by tech teams picking Drupal in similar ways to the above and further excludes the large number of designers I know who use Drupal and/or Wordpress and/or other CMS to give more ownership of sites to their clients and really don't much care about the technical end (and the designers pick Drupal because they can figure it out, and this is more true with every new version). The comment also leaves out popular new markets with big futures like those filled by Drupal Gardens and Buzzr. Distros are also driving some Drupal adoption.

So your constituents may cheer, but that does not mean that you've nailed this. I'd argue that it's really from D5 onward that we've seen huge growth, and what happened more and more with each version? We made UX inroads and made the tool easier for non-coders to use. Yes, the coders still hold the high ground and have the most influence, but a lot of adoption has been driven by different sectors and following a developer-driven focus sounds pretty short-sighted to me.

I am not advocating a developer-centric approach. I'm not actually advocating anything in particular. :-) I'm providing context and a lens for how Drupal exists in now, and asking hard questions that we need to be considering.

I am certainly not denying the significance of non-programmer contributions to Drupal at all. We need more of them.

I've been a Drupal themer and site builder for just over two years. In that time I have learned to love Drupal: blocks, nodes, menus, content types, views, etc. It's an absolute dream for a non-programmer to be able to build almost any website imaginable with virtually no PHP ability.

But the reality, as I see it, is that Drupal is already really losing that low-end market of people like me (designers, themers and site builders). I built a project in Wordpress 3.0 last weekend just to try out and it was a pleasure to site build and theme.

Last, the most incriminating part, when I show content creators (the true users) the Wordpress admin interface they 'oooh' and 'ahhh', but when I show them Drupal's (even D7's) they tend to furrow their eyebrows. There's no way to sugar-coat this - it is sad, but it's still true.

Wordpress has done an exceptional job on their UI and with keeping site builders and content creators at the fore. And, although I'm suspect Wordpress 4 won't be a CMS in the traditional sense, I'm positive it will fulfill the needs of most design shops, freelancers, and site builders - the 'low-end'.

The low end is people making websites with 2-3 content types, 1-2 menus, a jquery carousel and a few static pages. The next Wordpress will likely make this very, very easy. In these cases, it's already easier to avoid using Drupal.

fndtn357 (not verified)

24 November 2010 - 10:20am

I have been thinking a lot about this post since I read it and I would like to add one possible direction to this conversation. It is not off-topic but it is stretching into a specific vulnerability of Drupal as opposed to other popular CMS/frameworks.

I have collected numerous clients that need a Drupal webmaster for "adding content", "manipulating blocks", "adding regions", and the dreaded "updating". My clients do not have a dev environment to apply the security and/or development updates and make sure the site remains functional before they upload or make the changes to their production server. My clients frequently have been left with a website that is not developed for them to add content in an effective way. Also, clients are left in the lurch about how to administer their website once they receive ownership of it. Drupal does not allow a website owner to update painlessly, does not support a non-programming model, and if they need more functionality they cannot "enable" that or even approach it without a complete redesign of their site.

What is the client base we want to assist? Are these the clients we are developing to? How well do we train them to use the system we have developed for them? If Drupal systems can't support some changes along the way, do we think the client will pay another few thousand to create a new website that will be obsolete in two years? Perhaps this is where many clients jump ship and look seriously at other CMSs.

Maybe this is where an entire specialized group of support, and revenue stream, can be developed. Create a category of Drupalists that offer the best in support and updates.

In my own development as a newer Drupal developer I am constantly being thrown by trying to stay updated (security patches, improvements, and optimization of well developed modules). Maybe there is a fundamental problem with my workflow and I am open to hearing about it.

ldweeks (not verified)

24 November 2010 - 11:46am

Thanks very much for your helpful article, Larry. These discussions within the Drupal community always set off lots of debate, with many comparisons to other site-building software (Wordpress in particular). And I'm always mystified when people (Dries, in this case), makes statements like:

'In the case of conflict, consider content editors over site builders over designers over developers over theoretical purity.'

My question boils down to this: how do we privelege content editors without turning Drupal into a Wordpress clone? That is to say, without turning it into a very task-specific tool?

Now, I'm pretty sure that I can safely assume that Dries is smarter than I am, and that he understands the software and the market better than I do. And so I must be misunderstanding something, because I simply can't imagine why he'd want to engineer drupal core in such a way that it priveleges the content editors over drupal experts. (I use the term "drupal experts" instead of "drupal shops" to include individuals and teams, programmers and non-programmers). My reasoning is that Drupal puts power into the hands of experienced site-developers because it is an awesome combination of framework and vibrant community. But to take advantage of all that goodness, you need a *lot* of drupal knowledge in your head. Gargantuan amounts, IMHO. You need to decide whether you'll be using contexts, panels or both, and why. You need to know that you should use wysiwyg, wysiwyg_bridge, imce, imagecache, better_formats... etc, etc in order to enable full wysiwyg functionality. And on and on.

In my experience, Drupal does nothing but cause misery and pain for content editors (or for web designers, for that matter) who attempt to spin up a simple site on their own without any previous knowledge of Drupal. They'd be much better off using Wordpress.

Things have certainly gotten *much* better with the excellent work done on Drupal 7. I just checked out text formats in drupal 7, and they are *awesome* (thanks, for whoever did that). But Drupal 7 is still a multipurpose tool, and, as Eaton points out, "Task focused tools will always, always beat multipurpose tools when users are trying to perform that particular task." If you're using a general purpose tool to create something more specific, it seems like the job of the site-builders to privilege the content-editors, and not the job for the folks engineering the general purpose tool itself.

So what am I missing?

Form follows function, however, if your are going to build houses and skyscrapers with the same framing material, the houses will have to be built with steel frames.

But that's my point: nobody pays for Drupal. Rather, people pay guys like you and me to build things with Drupal. So, again, it seems to me that it's my job to privilege the content editor, and not the job of those building Drupal in the first place.

If you modify it slighty to say that its Drupals job to *help* me privilege the content editor, then I'm on board (and adding vertical_tabs, which I use on every site I build, to core is a perfect example).

There are two ways we need to respect the content editor (assuming the are the client). #1 we have to give them a great admin experience and #2 we have to be good stewards of their money. If front-end development costs twice as much because we don't support front-end developers, there needs to be a comparable gain somewhere else. Making front-end development easier would reduce costs, which ultimately benefits the content editor.

Further, people hire guys like you and me - and that guy that uses Wordpress because Drupal is just too hard. I want Drupal (a better codebase in my opinion) to be chosen over Wordpress all the time, even for simple blogs.

There are two ways we need to respect the content editor (assuming the are the client). #1 we have to give them a great admin experience and #2 we have to be good stewards of their money. If front-end development costs twice as much because we don't support front-end developers, there needs to be a comparable gain somewhere else. Making front-end development easier would reduce costs, which ultimately benefits the content editor.

Further, people hire guys like you and me - and that guy that uses Wordpress because Drupal is just too hard. I want Drupal (a better codebase in my opinion) to be chosen over Wordpress all the time, even for simple blogs.

In my mind one of the biggest problems is that there is simply too much flexibility in the theming layer - and by extension too many decisions that need to be made (and remade) every single time you build a site. There a number of standard subthemes, most notably Zen & Fusion, but even within these categories, you can change regions and otherwise completely reinvent the HTML. If we, as a community, solicited the right kind of advice (say from Jeffery Zeldman) on what HTML we should standardize on it would go a long way to keeping HTML designers from getting so frustrated. We have very strict coding standards for the php that gets committed as contributed modules, and yet the HTML is garbage. It's no wonder that HTML designers appear to be the bottom of the list - but they don't need to be.

We do have a standard for the HTML we produce already. It's a very strict standard, too. "When in doubt, add a div with lots of classes so CSS gurus can do anything they want." That is the Drupal standard right now, and if you're a CSS guru or a site builder it's an awesome standard. The new "Stark" theme in Drupal 7 also serves as a sort of standard, and is derived largely from Zen, I believe.

Designers are very picky people with their own personal tastes (just like programmers). The "standard" HTML that would appeal to Morten.dk would probably alienate Mark Boulton, both of which would make Jeffrey Zeldman cry. (No offense to any of those people, mind you.) And none of those would actually work for the "What's this HMTL [sic] thing?" casual site builder who just wants to push a few buttons and have a blog.

That's the point, really. Drupal's HTML is not garbage. It's actually extremely well designed... if your goal is to push buttons to build a site, then slap CSS on top of it. If your goal is to write pristine HTML and then throw PHP behind it to generate *that* HTML, the Drupal itself is horribly designed and you do not want to allow people to just push buttons and get a blog.

(I separated CSS and HTML designers for a reason.)

So what is our priority? Button pushers or HTML gurus? We have to accept that we cannot optimize Drupal for both groups at the same time. We have to prioritize one or the other at various times. Right now, we prioritize the button-pushing site builder. If we want to prioritize the HTML guru, that is going to, inherently, harm the button pusher.

Are we willing to do that?

The vast majority of people who use CSS are also tied to HTML. Infact I personally don't know anyone who just uses CSS. Building on that, a bunch of div tags isn't a standard for most front-end designers/developers. Most people in the HTML/CSS care about semantic mark up; they want to separate the form and function in their code. There's currently very little support for this in exisiting drupal themes and modules resulting in ugly div-soup.

That said, I think Drupal can and should be more friendly in the HTML it outputs. I think the way to get there is a return to the old style Drupal 4 event modules- but this time built on Views and CCK/Field modules. It would make development much faster and allow for better HTML that people feel like they need to rewrite. Also Semantic Views should be included in core.

mrktlr (not verified)

27 November 2010 - 4:52am

In reply to by Larry

Wordpress has done an exceptional job on their UI and with keeping site builders and content creators at the fore. And, although I'm suspect Wordpress 4 won't be a CMS in the traditional sense, I'm positive it will fulfill the needs of most design shops, freelancers, and site builders - the 'low-end'.
Mark Taylor
testking 642-974 certified

I think the build your own module approach with the tools you mentioned (CCK/Views/Panels) is good, but there should be pre-fabricated versions that are ready to go out of the box. With some definition about what a particular content type actually is (event, blog post, etc) the task of setting up, administering and theming would be much easier. You could always extend it, modify it or roll your own, but we shouldn't pretend that history isn't repeating itself.

That's what the Features module aims to be, and at a larger scale distributions. There are, of course, considerable technical challenges there. :-) Help overcoming those challenges would certainly be welcome.

donquixote (not verified)

17 December 2010 - 9:52am

There are different participants involved in building and maintaining a Drupal site, feeding it with content, and consuming the content, and using it to communicate.

It is not sufficient to trade-off the interests of each of these roles. Instead, we need to look at how these people collaborate and interact throughout the project lifetime, how they can split up their work, how one has to clean up the other's mess.

Between these participants, there are contracts, deadlines, power structures, communication structures, or a lack thereof.
The participants might have a team routine, or they might not know each other at all.
The work is probably done in an iterative process, where the client is giving constant feedback on the technical progress.

The project itself might have a test install, a staging install, and a production install - which brings additional requirements for the workflow and the platform. As the developers / site builders implement new features on the test site, the client and end users continue to produce content on production, which will have to be merged in some way with the changes on the .

As for the roles outlined in the blog post: These might be typical, but it often happens that one person fills more than one of these roles. Such as, site building + code + html/css + thinking about seo, usability, accessibility, a nice editor's backend, and talking to the client.

One missing role is the graphic desginer, who often does not speak any html or css, and sometimes is not even involved in the iterative process.

The consequence?
A focus on single person roles is not enough.
Instead of personas, we should be writing project and team workflow scenarios. The real tradeoff is between these project workflow types, not between participants.
Wordpress is a great choice for a one-person project, where the content editor will be left alone with the site most of the time, and the developer, if there is one, has a limited budget. The content editor should be able to install new themes, and do some changes here and there, without buying developer time.
How does this look for Drupal? There are casual sites built entirely by click-click and a bit of half-assed design and theming. The maintainers probably chose Drupal because of some non-standard functionality they don't get with WP. Or simply because they thought it is cool. The site might look a bit ugly, but it does work. The site builders are probably identical with the content writers, but a bit more geeky - but still not real experts.
Starting from there, we find a lot of different types of project, with different workflows, more or less end user interactivity, etc.

I could be writing more, but: As I have not done any research on this, I should rather shut up. I hope I could still make my point.

Allie (not verified)

28 December 2010 - 2:37am

Thanks for sharing. .. After reading your article I had to rethink about the architecture in my Drupal site.

Allie @ Web Development