On Drupal's Leadership

Submitted by Larry on 25 October 2014 - 9:20pm

My DrupalCon Amsterdam Core Conversation on Managing Complexity has generated quite a bit of follow-up discussion. That's good; it's a conversation we as a community really need to be having.

There are a few points, though, that I feel bear clarification and further explanation as I fear the point of the talk has gotten lost in the details.

Before continuing, if you haven't yet I urge you to watch the session video as well as the background resources linked from the session page. This is not a new conversation; it's the latest chapter in a very long-running discussion that is larger than the Drupal project, and it behooves us all to be aware of the history and context around it.

First of all, a number of people have reacted very negatively to my comment that Initiative Leads and, to a lesser extent, subsystem maintainers are "glorified secretaries" (despite the fact that I was referring to myself, derisively or not). There are probably some fascinating sociological reasons why that word carries far more intense negativity for some, and certainly far more than I intended, but that's a discussion for another time.

To the point at hand, I was specifically referring to the official, in-writing job descriptions of Initiative Leads. The only written, referenceable description of what Initiative Leads do is, again:

Initiative Leads have the following responsibilities:

  • Coordinate work on the initiative.
  • Communicate the plans, progress, and needs of the initiative to the community and the branch maintainers.

"Coordinate" and "communicate". Those are both fundamentally clerical tasks (hence "secretary", a clerical position). That's not to say unimportant; Good communication and coordination is critical for any medium to large scale endeavor, software or otherwise. (Having worked on projects without a project manager before, I will never do so again.) But "just" communicating does not in any way imply "leadership".

When I asked in my session who the Technical Lead was for Drupal 8, someone jokingly called out that I was. In all modesty it's true that I have had an enormous impact on Drupal 8's architecture and direction, some of it under the aegis of WSCCI, some not. But none of that came from being an initiative lead; it came from being an outspoken, persuasive, stubborn git.

My fellow initiative lead, Gábor Hojtsy, has responded to the discussion by discussing the benefits of "soft power": The ability to persuade rather than coerce, advocate rather than force, etc. To be sure, "soft power" has its benefits, and it's well established that "people skills" are just as important as applied skills in whatever the task at hand is (technical or otherwise). I also don't want to diminish Gábor's unquestionable skills in those areas. However, Gábor contrasts it with "hard power" by saying that "hard power" is about ordering people what to do and throwing around one's "authority", a concept that doesn't fit well with volunteer organizations.

That is, quite simply, a straw man.

Soft power, for all its benefits, is ultimately a form of informal structure. Informal structure, as outlined by Jo Freeman and as I quoted extensively in my session, is by its nature very fluid and opaque. It is inherently complex, because different responsibilities and relationships overlay on top of each other in unpredictable and undocumented ways. Informal structure is inherently not transparent. Anyone who values transparency should be wary of informal structure (and, by extension, over-reliance on "soft power") as it runs directly contradictory to that goal.

A good formal structure offers something that an informal structure cannot: Clarity. If a new contributor wants to ask someone in a "leadership" position if what they're proposing is consistent with the project direction, goals, or software design, who should they ask? Me? Gábor? catch? xjm? Daniel Wehner? Tim Plunkett? Angie Byron? Alex Bronstein? chx? Dries himself? Depending on the question the answer could be very different; even among those "leaders" I doubt there's consensus on which of them to ask, much less what the answer should be. That's actively hostile to new contributors, because then they don't know if what they're suggesting (be it big or small) would even be accepted until it's committed (and occasionally not even then). That is, it makes most contributions spec work, even for those in de facto leadership positions.

It also contributes to the belief that I've heard from numerous Drupal-savvy but not core-involved people that contributing to core involves posting a patch; waiting a few months; getting 300 comments; someone new rewriting the patch; refiling as a duplicate of another issue two or three times; reopening the issue; and then a patch sitting at "needs review" forever because no one can make a decision. (Yes, I have heard that story given by people that could be contributing to core, but aren't because the process barrier to entry is too high.)

With all that informality, it's quite easy to simply ignore the informal structure case-by-case when it's convenient. That's very disrespectful to those who have earned positions within the informal hierarchy through whatever means, and it prevents people from being able to "step back gracefully" as our Community Code of Conduct calls for. If you don't know if you've stepped up, how do you know to step down? As I noted in the session, Paris Liakos and I completely rewrote the logging system for Drupal 8, without the blessing or even awareness of the maintainer of the two logging modules in core. Of course, he wasn't there to tell us no so, and no one else suggested involving him. So did we do it right or were we backstabbing him? In all honesty I don't know.

I do know, however, that elbowing out de facto leaders who don't want to be elbowed out is a very painful and ugly process, one that has hurt the Drupal community and several initiatives over the Drupal 8 cycle; it was worse in 2011 and 2012 than recently but still, it is harmful to people. That process drove Jacine Luisi (HTML5 Initiative Lead) out of Drupal. Yet that is the primary way that de facto informal soft power transitions.

Drupal's extensive documentation and coding standards, for all the seemingly unnecessary anality, do have the distinct advantage of of being objective and transparent. There's very little in Drupal you can say that about.

Another failing of soft power is that it privileges those with an excess of time or an excess of persuasiveness, neither of which necessarily correlate in any way to an excess of skill or expertise at the task at hand. You want people in positions of authority who know what they're talking about, who understand the problem space, are connected with the various threads of work involved, and, yes, have the people skills to persuade and coordinate. But they also need to have the backing to make hard decisions if for whatever reason "consensus" doesn't work (as it often does not). They need to be understood as the domain expert in whatever the domain in question is, and that understanding needs to be transparent. If it's based entirely on soft power and who remembers that so-and-so wrote a particular piece of code then it is anything but transparent.

Fears of "hard power" people just throwing their weight around are also exaggerated. It has been my experience (in three different organizations, Drupal being the 3rd) that when you give someone with soft-power a more formal position with responsibility and the authority to be able to follow through on it one of two things happens:

  1. They become mad with power and throw their weight around, trampling all over those that get in their way.
  2. They adopt a broader perspective and become more collaborative, more open to discussion, and more amenable to reasonable disagreement than they ever were before because their position in the informal de facto pecking order and ability to influence direction is not under constant (real or perceived) threat. This is often an entirely unconscious distraction before hand that is neatly removed through formal recognition.

That is, if you are in an informal soft de facto position of leadership, part of your attention is always taken up by whether or not you still are at any given time. (Not necessarily in a defensive way, but that can happen too even without anyone being "power hungry".) But that's the last person you want distracted by court politics.

Naturally you want to have a lot of people from group 2 and avoid people in group 1. But it is self-defeating to let the fear of group 1 blind you to the benefits of group 2. Yet the Drupal project has largely let its fear of group 1 and it's zeal for the "one hour, one vote" of Do-ocracy hamstring our ability to leverage and capitalize on the benefits that formal structure can provide (transparency, clarity, efficiency, accountability, etc.). Again, quoting Jo Freeman:

If the movement continues deliberately to not select who shall exercise power, it does not thereby abolish power. All it does is abdicate the right to demand that those who do exercise power and influence be responsible for it. If the movement continues to keep power as diffuse as possible because it knows it cannot demand responsibility from those who have it... it insures that the movement is as ineffective as possible.

Formal structure can certainly be taken too far as well; requiring all patches to be filled out in triplicate with a cover letter attached to the TPS report would be lunacy. But as software passes Norris numbers and communities pass Dunbar numbers formalities need to be put into place to allow both to scale. It's a "right tool for the job" question, and the larger the problem space the more right a formal structure becomes.

Just as procedural code is fine up to a certain level of complexity and then needs the increased rigor of more formal methods like OOP, dependency injection, functional programming, data types, and so on in order to be sustainable so too can a community rely solely on "rough consensus and working code" only up to a point, and then needs to build in more formal structures and processes to allow it to grow.

Ultimately, this debate should not be about hard power vs. soft power. It's not an either-or proposition. A good, sustainable structure happens at the intersection of hard power and soft power; at the healthy combination of formal and informal structures, in a way that is appropriate to the scale of the organization or community.

Right now, Drupal leans far, far too much on soft power and informal structure, and that is harming us, as a community and as a code base. We need to bring that into balance in a way that recognizes just how big Drupal has become.

What sort of formal structure should we add? That's open for healthy debate. Sam Boyer has proposed one model. I've proposed one, taking inspiration from the Linux project. Lee Rowlands has proposed one, taking inspiration from a number of other projects. These are the discussions we should be having.

We should also look outward and dig into how other "super-large" Open Source projects handle these issues. There are only 8 others besides Drupal; it shouldn't take that long to research.

chx (not verified)

26 October 2014 - 3:21am

> Of course, he wasn't there to tell us no so, and no one else suggested involving him. So did we do it right or were we backstabbing him

And even if the maintainer is there and tells you no, forcefully, repeatedly, ad nauseam you still don't give a damn so don't be so sanctimonious. In fact, your tweet where you compared the necessary power transition to the transition from procedural to OOP tells us everything we need to know.

We are in a unique position here - a testament to the businesses who've gone out and spread the Drupal word however a continuing conundrum for those who create the product itself due to the increased responsibility of building, supporting and maintaining a system which powers a shedload of the internet.

I am more surprised that you haven't got the power your blog discusses - for me free software is just as affected by the forces as proprietary software so I have no qualms in increasing/describing the 'powers' initiative leads have, in fact totally agree this will help more than hinder.

In terms of acceptance, I briefly blogged on this subject a few years back but I do feel strongly that if we work to create a 'standard' we move further forward. I used Drupal because coding websites wasn't exciting and Drupal gave me the tools to build quickly what people needed - not that it's necessarily the best technology but it gets the job done. The FSF have a call on for their LibrePlanet 2015 conference on "Free Software Everywhere" - after attending back in 2006 and being inspired by a talk on how TVs wouldn't exist if we'd patented the electronic spectrum and seeing "The Future of Programming" http://vimeo.com/71278954 I'm working on a talk which tries to encapsulate this in current terms, with Drupal as the centre - we so need to get away from the Drupal vs WP vs Joomla stuff if we're going to do what I believe Dries' vision dictates.

http://purkiss.com/blog/steve-purkiss/2011/12/13/drupal-time

Larry, let's take some glorified secretaries (your word) who you adore: Lawrence Lessig, Cory Doctorow and Aral Balkan. What they do in their respective fields is they coordinate work and communicate the plans, progress and needs of their initiatives to the community. (Also the definition of Drupal 8 initiative leads). Nobody gave them authority.

What if president Obama, the president of the EU and other high functioning leaders would say these are good folks and people should listen to them (but ignore them in policymaking and don't give them any position because that would let them direct money/resources)? How would that change their position/abilities? (They would get more recognition but no authority).

Now realize you are the Lawrence Lessig, Cory Doctorow or Aral Balkan of Drupal web services, blessed by Dries Buytaert. That's who you are. You are supposed to draw up your vision, mobilise people, grab every opportunity to spread your ideas and get people to tag along your journey. The people with the resources may not agree with you and you don't control the resources, so you need to be coherent, clear and convincing. What's jaw-dropping is your lack of appreciation for leadership traits other than authority, that you consider this a glorified secretary position. Hope that definition fits with the mentioned gentlemen too.

----

The question of who is Drupal's architect is akin to asking who is the architect of the street where you live. Well, nobody and several people at the same time. Is that very disturbing? Like the same answer to Drupal? Your house was probably designed by one or more architects, possibly involving several more professions if its more recent (eg. energy engineers). When the plan looked all good it went to an office for acceptance. The upfront investment to do all the planning is significant and yet it may be refused if the angle of your roof will not fit into the natural environment for example (even if it would be ideal for energy reasons). So you are back to the drawing board and return with a new plan. There are objective rules for that plan such as the gates that Drupal 8 attempted to introduce and then so spectacularly failed because people did not want the additional bureaucracy and pain to meet for example performance constraints. There are also subjective requirements such as that your building fits into the cultural environment it is going to be placed, which is unless it is an exact replica of the neighbour house will be down to the feeling of the official (AKA core committer). The architect is not allowed to approve their own plans exactly so that there is some oversight among the different architects and the neighbourhood makes sense as a whole.

The elegance of core gates is that there is no fixed human to be involved to be the bottleneck to check or approve your work (and be in a bad mood), you can do it yourself. I agree both the community structure and the rules are better written down. The core gates and initiatives are attempts at these two in a way that don't overlap. We should investigate why the Drupal 8 core gates did not work because they are way more flexible ways to distribute power and resolving problems while retaining the reviewer for your plans that the real life city architecture system retained for centuries to this day.

But we cannot run to the office to approve each little change we make!? Right. If you want to put a bench in your garden or separate rooms by raising a new drywall in your house, who cares? Do it. If you want to tear down your outer walls and expand your kitchen, then you need to get a permit. This is not much different in Drupal either.

Also decisions are distributed when planning a house. Exterior and interior designers may be involved. An energy engineer may work with the architect to change significant pieces. Its not the permits office telling people to distribute that power. Its people hiring others who know better. Eventually all of their work is reviewed altogether in the office and approved or rejected. "But I already approved the energy engineer's designs myself" does not work in this review process. Drupal 8 was already structured (and communicated to be structured) into initiatives. If you did not structure your initiative and communicated that structure that is within your responsibility.

Being an architect and decision maker in one is not just problematic, it also does not address most of what leadership means, and that is a much bigger gap.

-----

The drupal.org teams are very actively organized into content, software and infrastructure groups. See https://www.drupal.org/governance/drupalorg-working-groups These groups are then broken down to subgroups, for example the software group is broken down to developer tools, community tools and translation tools. They have commitment lengths, clear authority and conflict resolution defined, for example the translations team: https://www.drupal.org/node/2297441. Sounds like what you are looking for Drupal the software?

When I was asked to set up the translations team, my first question was, what do we get in return. I mean those who already work on these tools will not magically have more time. They will not magically have more power. Hopefully our powers are better documented. If you heard about the groups that is. That's nice. People now know where to go, who to talk to, right? But that will not make anything happen. The drupal.org team considers the translation tools area being in "support community" capacity, so we get great help setting up testing environments and such but not in actual implementation of anything. So we can make decisions but that will not make things move. We would also need to dramatically improve availability of people and there is no budget involved. We only got authority.

I admit I am not following the groups very closely, but my feeling is the groups that got budget work pretty well while the rest don't work any better or worse than they did before they were officially recognized. It is also probably too early to draw conclusions.

-----

In short I think there is a lot to learn from Lawrence, Cory and Aral to look broader as well as core gates and the drupal.org working groups to look at our own backyard. I agree better documented structures and especially better documented guidelines would be great. They were in a large part already your responsibility and they are also not the silver bullet.

"The architect is not allowed to approve their own plans"

That is true, but I don't believe anyone is asking for no oversight. The problem with the analogy is that you are describing a process with well defined roles and processes, which is not what Drupal is at all. I agree with you that authority does not make a leader, but not every initiative in Drupal is going to be some sort of grassroots movement, neither will consensus always be reached. I don't feel authority has to fall in a person, it could fall on the process. We could implement majority rules by votes, or any other process to keep moving forward when communication and persuasion fails, but that is still a clear authority on how things get done. One person, a committee of people, a process, it does not matter as long as we have something to start with and iterate upon.

effulgentsia (not verified)

29 October 2014 - 11:50pm

Hi Larry,

Thanks for writing up these thoughts as you're continuing to refine them. I agree with some of your points and disagree with others, but regardless, I want to commend you for communicating them.

A good, sustainable structure happens at the intersection of hard power and soft power; at the healthy combination of formal and informal structures, in a way that is appropriate to the scale of the organization or community. Right now, Drupal leans far, far too much on soft power and informal structure, and that is harming us, as a community and as a code base. We need to bring that into balance in a way that recognizes just how big Drupal has become.

I mostly agree with that, except I want to take a moment to acknowledge the work that's already happened in the last few years to formalize and scale Drupal governance:

  • Excluding Dries, Drupal 8 has 4 people with authority to commit code to core compared to Drupal 7's 1 person while it was in development. Furthermore, 1 of those 4 people is limited to committing only documentation changes, which is the first time that Dries has extended such targeted commit authority, and given the success of that experiment, is a model that I think we can extend to more targeted areas in the future.
  • Within the last year or two or so (depending on from when you start counting), Drupal has evolved formal working groups for some governance issues, including resolving certain kinds of conflicts when informal processes fail.
  • We have 3x (or more?) as many component/module maintainers as in Drupal 7. While these people don't have commit authority, they are transparently (in contrast to your concerns about informal structures being opaque) documented as people who can be contacted when their advice is wanted. There's also transparent documentation about how to become one.
  • We've incorporated an order of magnitude more PHP and JS libraries from other projects into Drupal 8 core than we have in Drupal 7 core. Each one of these is a decoupled zone of authority. And potentially, at some point, we can experiment with moving some \Drupal\Component components into separate projects with their own autonomous project leads.
  • We have Drupal core mentoring, which is a phenomenal example of successful leadership without formal authority. The last point in Jo Freeman's article that you reference is the importance of equal access to resources (at least as an ideal, even if not perfectly attained). One of the scarcest resources in the Drupal community is issue/patch reviewers, and core mentoring is an example of Drupal's "elites" going out of their way to make that resource available to everyone.
  • And we have the Drupal 8 Initiatives, which among other benefits, help overcome the problems of a community scaling past Dunbar's number. I'm glad that you, Gábor, Greg, and other initiative leads are sharing your perspectives on what has worked well and what has worked poorly with respect to initiative structures and operations.

If a new contributor wants to ask someone in a "leadership" position if what they're proposing is consistent with the project direction, goals, or software design, who should they ask?

Consider the same question in a formal hierarchy, such as a corporation. You can ask your manager, and they might give you one answer. But if your question is about something that's in the scope of another department, you might need to ask that department's manager. Either one of those managers might give you an answer that is overruled by the company's VP or CEO. You could try to ask the CEO directly, but in most large companies, that's not an easily available option, because CEOs tend to be busy. The situation in Drupal is similar: if your question pertains to an initiative, ask the initiative lead. They should be able to answer the question or help you find someone who can. If your question doesn't pertain to an initiative but pertains to a component, ask one of the component maintainers. If you don't have luck getting a response from those people, or don't know what initiative or component your question relates to, ask a core mentor for help. Now, there's always the risk that a core mentor or an initiative lead or a component maintainer will give you an answer that is overruled by a branch maintainer, just as a company manager can give you an answer that is overruled by someone higher in the org chart. Humans are fallible, and formal authority doesn't solve that. In Drupal, you can always try to reach out to a branch maintainer directly, but recognize that they're busy, so use some judgment.

That's actively hostile to new contributors, because then they don't know if what they're suggesting (be it big or small) would even be accepted until it's committed (and occasionally not even then).

"Will it be accepted?" is a different question than your earlier example of "Is it consistent with the project direction and goals?" To be accepted, the idea needs to be implemented to sufficient quality, be understandable and maintainable by more than one person (since the original author might move on to other interests and commitments), be completed at the appropriate time of the development cycle, and not introduce problems elsewhere (or if it does, then for a judgment call to be made about the relative benefits and costs). To me, an analogy is the difference between asking a lawyer, "do I have a case?", versus wanting clarity about "will I win this case?". An answer to the first does not fully predict the answer to the second. And even if you had access to ask the judge presiding over the trial, she wouldn't be able to answer you until you've completed the trial. And if the trial is one that is decided by a jury rather than a judge, you have even less predictability. And even after a case is decided, it can be appealed and reversed. I mention the judicial analogy, because that's an example of a highly formal structure that offers very little predictability of result. But obviously, judicial structures are designed to handle far more serious matters than what code gets committed to Drupal, so if we want to, we can choose to design Drupal structures differently.

Paris Liakos and I completely rewrote the logging system for Drupal 8, without the blessing or even awareness of the maintainer of the two logging modules in core. Of course, he wasn't there to tell us no so, and no one else suggested involving him. So did we do it right or were we backstabbing him? In all honesty I don't know.

In my opinion, you did nothing wrong. If component maintainers want to oversee their components, they need to be proactive in doing so. If they're not monitoring the component's queue, they're choosing to not be involved (which is an okay choice, but that shouldn't put all progress to a halt). Furthermore, branch maintainers have the explicit authority to override component maintainers. So, if a branch maintainer committed your patch without consulting the component maintainer, that's their prerogative. If you'd like to suggest that we should change our structure to require component maintainer input for certain kinds of issues, then that's a topic for a separate, dedicated blog post.

That process drove Jacine Luisi (HTML5 Initiative Lead) out of Drupal.

I very much liked and still like Jacine and was very sad to see her struggle with those challenges. According to her post, what drove her out of leading the initiative was that the role put her in a spotlight and made her and her suggestions the target of heated criticism. But I think that formal authority tends to increase that problem. The transparency of formal authority can be good for the group, but it comes at a personal cost to the people vested with it. But perhaps where formal authority would have helped was if it was vested with someone else, or with a group, who could have at least partially protected her from that. And in fact, the Community working group was created partially to fill that role (not just for Jacine, but for everyone).

In summary, I think we're in agreement that at a minimum, more clarity and transparency on what our formal and informal structures currently are, and what our pain points are, would be very helpful, and will take many more insightful posts and conversations to flesh out. And once we have some of that clarity, we can start experimenting with where and how additional formality can be added. In the meantime, I'm grateful to be part of a community where even though elites exist, more often than not, they use their soft power to empower others, and because of that, we've attracted enough contributors to be one of the largest developer communities in the world and having the problems we have.

Hi Alex.

I agree, we have made some progress within the Drupal 8 cycle. As I said in my presentation, Drupal 8 is the least coupled version of Drupal to date despite the constant structural pressure to become less decoupled. (I.E., to mirror our communication and decision making process.) That doesn't mean we don't still have a long way to go because that pressure is increasing and I can see the coupling creeping back in at times.

Which is actually another point that I think may have gotten lost in the "power structure" question: Like anything else, Drupal the software mirrors the organization that created it. We've all had the clients who we have to tell repeatedly "your org chart is not a good model for your IA". That's been researched and found to be a natural tendency, too. Improving Drupal the software in many cases requires improving Drupal the development process/community.

Just adding global-level committers helps with time and resourcing, but not necessarily decision making. Different committers soft-specialize in different areas, but that's not written down anywhere. Eg, performance related stuff *usually* gets deferred to catch, but not always, and if you aren't fully clued-in to day to day core development you probably don't know that. Jen as documentation maintainer is the key, and very positive, exception; you know that, where documentation is concerned, Jen Hodgdon is the voice of authority and the gate keeper to speak to and convince, and I don't think I've ever seen Dries over-rule her. (Even if I disagree with her about something, it's still clear who I need to work with.) That is a big step forward, as well, and fits nicely with Sam's suggestion of "simplicity guardians" who focus on code quality, not features.

I also don't want to present formal structures as a silver bullet; certainly one can have horribly complex and convoluted formal structures, which can be even worse than just an informal structure. The key is properly balancing formal and informal structures, and making both as explicit as possible. Right now, our balance is still off. Even knowing, explicitly and in writing, the situations in which the regional manager can be or is likely to be overruled by the CEO would help reduce uncertainty and bikeshedding and improve stress levels for everyone (especially that regional manager!)

As far as Jacine, I don't want to go into too much personal detail there but her blog post was considerably more oblique than the conversations she and I had as initiative leads. Suffice to say she ran into the same "Someone is throwing code at me and as initiative lead I don't have the ability to say 'no, slow down, we're doing this way'" problem that Greg Dunlap describes toward the end of his keynote blog post. She just decided to cut her losses and leave before Greg or I did. (Perhaps she's the wisest of us all.)

Dries had been asked to help clarify that relationship and did not do so, at least not to any meaningful degree. Instead of adding structure, we've added process. That's progress, but we still have a lot of work to do.

> despite the constant structural pressure to become less decoupled.

So, making Drupal more secure is now a "constant structural pressure" ( https://www.drupal.org/node/2322809 ) . Nice.

Let me condense this article and the last three years for you: what Larry says is right. Larry should have authority to enforce this. That's all.