Backward compatible APIs

Submitted by Larry on 2 January 2012 - 12:05am

As we begin a new year, it seems appropriate that the discussion of backward compatibility has come up yet again in Drupal. It's a perennial question, and you can tell when a new Drupal core version is ready for prime time when people start complaining about lack of backward compatibility. It's like clockwork.

However, most of these discussions don't actually get at the root issue: Drupal is architecturally incapable of backward compatibility. Backward incompatibility is baked into the way Drupal is designed. That's not a deliberate decision, but rather an implication of other design decisions that have been made.

Drupal developers could not, even if they wanted to, decide to support backward compatibility or "cleanup only" type changes in Drupal 8. It is possible to do so in Drupal 9. If we want to do that, however, then we need to decide, now, in Drupal 8, to rearchitect in ways that support backward compatibility. Backward compatibility is a feature you have to design for.

What is backward compatibility?

First, we need to understand what backward compatibility even means. As some comments in the post linked above note, backward compatibility is not the same thing as easy upgradeability. It is also not the same thing as easy to-relearn. Drupal 7's UI is completely not backward compatible with Drupal 6, for instance, but it is easier to learn.

For now, we are speaking strictly of API backward compatibility. It is related to, but not the same thing as, easy upgradeability of modules (but unrelated to data; more on that some other time). The fewer APIs change from one version to the next, the easier it is to upgrade. But you can also have API changes that are easy to upgrade to. More than one module developer, when porting a module from Drupal 6 to Drupal 7, noted that they were able to throw out (and therefore not have to spend time upgrading) hundreds of lines of code by switching to the Drupal 7 query builders. That's an API change (and thus lack of backward compatibility) that made upgrading easier, as well as code better. As Dries has noted before, if we don't let ourselves do that then we will never advance, and will just carry around dead weight baggage forever. That's how Windows Me happens, and no one wants that.

Backward compatibility is also a factor in how long major modules take to be upgraded, but not as much as many people think. Views, Panels, and Context are often cited as examples of "Drupal not being ready yet" because those modules don't have stable releases. That is a specious argument, however. In the case of Views and Panels, the initial ports to Drupal 7 were done over a year ago. The lack of an official release was because the developers decided to make other overhauls and changes and feature additions at the same time, and those took a while to stabilize. In the case of Context, it's because the development team behind it changed when DevSeed moved on and the new team has been swamped. (Also, Context didn't have a stable D6 release for most of its existence, either.)

So if that's what backward compatibility is not, then what is it?

Backward compatibility of a code component is a measure of how many changes (good or bad) code that depends on it needs to make in order to move from one version to the next version.

The fewer changes need to be made, the more backward compatible code is, obviously with an upper bound of "no changes needed." Note that says nothing about the quality of the code or the value of those changes; often a backward incompatible change is absolutely necessary for certain improvements.

Defining dependency

What does it mean for one code component to depend on another as far as compatibility goes? It means that those two code components touch. There are two ways that systems could "touch":

  1. One component calls a function or method of another.
  2. One component directly addresses data structures from another.

Of those two, the second is a tighter coupling because it means dealing with implementation details.

There is also the general question of how much one component touches another. In general, the more two components interact the larger their shared "surface area".

The larger the surface area, the tighter the coupling. The tighter the coupling, the more likely a change in one component is going to necessitate a change in another component.

Reducing surface area

Backward compatibility happens when the touch points between two components do not change. Anything else can change, but those touch points do not. As a result, the smaller the surface area between two components the more you can change without breaking compatibility.

As noted in the Wikipedia article linked above, accessing raw data structures is always tight coupling. Raw data structures are an implementation detail. When implementation details get shared, that is a classic Code Smell known as Inappropriate Intimacy.

Inappropriate intimacy is a massive amount of surface area. It covers the entire data structure. That is, any change to the data structure whatsoever is a potential, if not actual, backward compatibility break. The reason is that you do not know what parts of that data structure some other component may care about. You have thrown the doors (and your proverbial pants) wide open, and said "have at it, world!" Once you do that, every change, no matter how slight, could break backward compatibility because you simply do not know who is doing what with your private data structures. (You should be squirming about now.)

Rather, the first step in making backward compatibility possible is to put your pants back on, protect your data structures, and take control of your surface area.

Interface-driven development

By far the easiest way to take control of, and reduce, your surface area is to define it explicitly. That is the essence of an Application Programming Interface (API): It is the explicit definition of the surface area of your component.

That could take the form of function calls or method calls on an object. Both could suffice as an API. The latter, however, has the added bonus of a language structure called Interface, which explicitly and in code defines the surface area of an object. By design, it defines that surface area independently of implementation details.

What will not suffice, however, regardless of whether one uses classes and objects or not, is defining ways by which one will get complete access to a components internal implementation details and raw data structures.

Access to raw data structures does not constitute an API. It constitutes avoiding the responsibility of defining an API.

Now, sometimes there are good reasons to do that. Sometimes. Not often.. But when that's done, it must be understood that backward compatibility is made effectively impossible.

With a clearly defined interface, we know what we can change and what we cannot, if we want to preserve backward compatibility. We also can more easily document where things are going to break, and offer documentation or automation to make it easier to migrate.

Data hiding

One of the popular arguments in favor of object-oriented code is "data hiding". That is, you can explicitly demand, in code, that certain data is kept hidden from certain other components, and is not part of your surface area. That can only be done in procedural code by convention.

That convention has been used before, of course. In a previous life I developed for Palm OS, which used an entirely procedural C-based API. It passed around a lot of raw data structures, because in C that's all you can do. However, it was extremely bad form to ever touch them directly. Rather, there were copious amounts of functions that were, in any practical sense, methods, just called inside out. So you'd call Form_Add_Element(form, …) rather than form.AddElement(...). Doing anything with form directly, while it would compile, was not guaranteed to continue working even in minor bugfix releases of the OS. Here be dragons.

That's a viable option, but doesn't really change the amount of work that has to be done to define an API. Unless you say, either by convention or code, that implementation details and data structures are off limits and not guaranteed, you do not have a controlled surface area and therefore you do not have an API.

Concrete example

In many recent presentations I have used this example from the Drupal 7 database layer. A "raw data structure procedural" implementation of the new select builder would look like this:

<?php
$fields = array('n.nid', 'n.title', 'u.name');
$tables = array(
 'n' => array(
    'type' => NULL,
    'table' => 'node',
    'alias' => 'n',
    'condition' => array(),
    'arguments' => NULL,
    'all fields' => FALSE,
  ),
 'u' => array(
    'type' => 'INNER JOIN',
    'table' => 'user',
    'alias' => 'u',
    'condition' => 'u.uid = n.nid',
    'arguments' => array(),
    'all_fields' => FALSE,
  ),
);
$where = array(
 array(
    'field' => 'u.status',
    'value' => 1,
    'operator' => '=',
 ),
 array(
    'field' => 'n.created',
    'value' => REQUEST_TIME - 3600,
    'operator' => '>',
 ),
);
$order_by = array(
 'n.title' => 'ASC',
);

db_select($tables, $fields, $where, NULL, $order_by, array(), NULL, array(0, 5));
?>

Aside from the obvious DX problems that has, mostly in terms of not being self-documenting, it makes the entire implementation public. Do we use "INNER JOIN" or just "INNER" to specify the join type? And if we change it, does that not break every single query in the system? Yes it does.

However, we did in fact change from INNER JOIN to INNER at some point during Drupal 7's development cycle, and it was not an API change. That's because the query builders use an object-oriented, interface-driven, non-raw-data API:

<?php
$select = db_select('node', 'n');
$select->join('user', 'u', 'u.uid = n.uid');
$select
 ->fields('n', array('nid', 'title'))
 ->fields('u', array('name'))
 ->condition('u.status', 1)
 ->condition('n.created', REQUEST_TIME - 3600, '>')
 ->orderBy('n.title', $direction)
 ->execute();
?>

We're separating the internal data structure from the join() method. As long as that method doesn't change, the internal implementation could change today, mid-Drupal 7's lifetime, without breaking an API. That is possible only because it eschews raw data structures in favor of a well-thought-out, abstracted, interface-driven API.

That is what it takes to be backward compatible.

Backward-compatible API changes

Of course, sometimes we need to change the API. That happens, often for very good reason. If the system has been designed properly, however, it may still be possible to retain backward compatibility, at least for a time. KDE is a good example. Most KDE 3 apps work under KDE 4, albeit not as cool as they would as KDE 4 apps, and without integrating with the awesome new plumbing that KDE 4 offers. As a result, nearly all applications have been rewritten for KDE 4 by now as it's in their interest to do so.

It is possible to support multiple versions of an API at the same time, but there is a cost to doing so. First and foremost, that must be planned for in advance. There are likely others beyond what I am listing here, and I would appreciate feedback in the comments on other good approaches.

Microsoft Direct X took the approach of "version everything and keep it". When requesting a new Draw object, for instance, you specify a version of it. So if you code to Direct X 6, you ask for the DrawV6 object. If a user has Direct X 7, the DrawV6 object is still in there, just as it was, and still works. On the upside, this means code almost never breaks. On the downside, it's a lot of baggage to carry around indefinitely, and that baggage only increases every version. It also requires that you have APIs that are broken up into very clear discrete objects (not function calls), and that your version-request mechanism is baked in from the start.

If you can keep your general class structure the same, then you can also simply expand your API. As long as the existing interface doesn't change, adding more operations to it braks no existing code. in the simplest case, this is simply adding optional parameters to the end of a function signature. My very first Drupal patch did exactly that, in fact. When you have a lot of parameters, though, or a more complex case, it's easier to add more methods to a language interface and object.

If you would need to change the way a given method behaves, or change its signature, then it's also possible to simply add a new method that does the new thing instead, and leave the old one in place. Perhaps the old one could be reimplemented internally to use the new one, but calling code doesn't care because the contents of its surface area hasn't changed. The old method could then be explicitly marked deprecated, and give developers time to migrate over to the new one before it is removed. The downside of course is that you need a new name for the new method, which is one of the two hardest problems in computer science. (There are only two hard things in Computer Science: cache invalidation, naming things, and off-by-one errors.)

Another alternative is to simply fold both versions into a single call. As a trivial example, consider everyone's favorite PHP WTF, implode(). At some point in the past, it took $pieces and then $glue to turn an array into a string. That was inconsistent with explode(), which was a major DX problem. To resolve that, implode() was enhanced to take $pieces and $glue in either order. That provided both backward compatibility and a more consistent API moving forward... except that the old version was never removed, and is still not even marked as deprecated, so there's still plenty of old code out there using the old parameter order making it impossible to remove the old and inconsistent baggage.

The moral of the story

Backward compatibility, then, requires a number of things:

  1. Clearly, explicitly defined APIs. Raw data structures are not APIs. Good APIs are designed semi-independently of their implementations, making the implementation changeable. (They cannot be completely independent in practice, as nice as it would be.)
  2. No exposed raw data structures, or else clear documentation that such data structures are subject to change without notice.
  3. A plan, from the beginning, for how additions or modifications will be handled, either through API versioning, add-but-don't-remove strategy, or some other clearly and explicitly defined plan.
  4. A sunset plan for how to remove support for old APIs. They cannot be held around forever or you turn into Windows Me... or PHP itself.
  5. A decision criteria for when it's OK to backward compatibility anyway, other than "meh, whatever".

That's why, no matter how much users want it and no matter how much Drupal developers may want to do so, Drupal 8 will not be, cannot be, API backward compatible with Drupal 7. Drupal today is based on passing around raw data structures rather than having clear APIs. (Render API, Form API, etc. thus do not technically qualify as APIs by this definition.) When we have APIs, they're generally not designed with future extensions in mind (except in so far as conventions for adding more stuff to a raw data structure). We have no long term strategy for future development or how we're going to maintain compatibility between versions, even through legacy add-ons.

Again, quoting Dries:

But what to do if many of your users slowly force you to change one of your core values? It seems inevitable that sooner than later, we will have to be a lot more careful about breaking peoples' code. And when that happens, I fear that this will be the end of Drupal as we have come to know it.

Dries wrote that around the release of Drupal 4.7, another very hard release. But really, it already is the end of Drupal as we knew it then. Drupal as we knew it then does not exist, and the Drupal of today is quite different, both in terms of APIs and in ways entirely unrelated to code. Thinking about API compatibility is not a death-knell for Drupal.

But, if we want increased API compatibility and stability, we need to take steps, now, to ensure that there is a structure to support that. That means, first and foremost, designing APIs, not raw data structures, and not just as an outgrowth of a particular implementation. That's a cultural shift as much as a technical one, but a technical one as well. It means changing the way we think about software design. It means, quite simply, interface-driven development.

Fortunately, such thought has already been happening in both the WSCCI and Multilingual initiatives at least. Interface-driven development, complete with real language interfaces, is where Drupal is headed anyway. We should embrace it, fully, and allow ourselves to be open to the potential for improving compatibility between Drupal versions that will result... if we take it.

Yups.. changing an API is evolution, and be not daring to change things you'll end up with a product like windows. Microsoft did not dare to change things, because theoretically it would break backwards compatibility with existing DOS applications. In real life, people who's daily bread depended on that DOS application, did not want to upgrade to newer versions of windows because they where afraid that it would collapse.

Switch this to drupal,, in the near future I can think of no serious reason to upgrade my D6/D7 projects to D8. So being backwards compatible is not as important as one might think. And you for some reason need or want to upgrade, consider doing a data migrate instead. One of the weakest links in the drupal chain, especially with some contrib, is that to much logic is in the database. Making it almost impossible doing a code only upgrade. That's why I advise builders to migrate their data into a new drupal site. So they can have the advantage of new contrib, changed strategies instead of just having the newest version with the same old logic.

Switch that to having clear API's, i really think that Fabian's Symfony2 team hit the nail right on the head with the @API tag in code. They first designed the API, then molded the code to fit that API and finally froze the API on key components for the entire life-cycle of the specific version. Meaning that in the future you still could checkout symfony 2.x, and rely on the same API. I do think that if we had a strong DB API to start with, VIEWS would not have become the beast it now evolved into. Yet everybody seems to depend on views to build a website, while in the long run, you might actually might be better of and learn some PHP and write a few queries yourself using a strong DB self explaining DB API.

I do believe that for drupal 8 this is the only way to go! Redesign API's, integrate even more with the symfony2 platform and concentrate on where drupal really shines like a diamond. Being a more user friendly versatile platform to build not just sites with, but web-applications. The future clearly is not HTML, it's HTTP!

So yeah, you got my blessings for what they are worth, to change the API's on D8! And for those who are afraid of change i say Poor you... Hope you like living on island called Prehistoricia ;)

Yes, the way Symfony handles that is fairly good. @api means "you can rely on this method", @deprecated means "you'd better not rely on this method". :-)

However, as noted that assumes your class structure is fixed. There's work right now to change the way Symfony's session handling works, because it's not very robust right now. Most of the proposed changes will necessitate breaking BC, because they're significant enough that you have so. So better documentation and planning helps, but is not the entire picture.

DjebbZ (not verified)

2 January 2012 - 4:51am

Keep up the good work. Keep evangelizing about good software development practices. Drupal needs this change, and I'm sure the community will embrace them if you (well, not you alone, hopefully) keep pushing things through education. Or was it called political maneuver :) ?

Dig1 (not verified)

2 January 2012 - 5:05am

Yes, it seems that the gist of what you suggest is that in the past Drupal successfully evolved with a philosophy of 'lets learn and implement'. This was good in a startup phase because it got traction and results.

Growth in the user and developer base now means that the 'ship is bigger' and the journey needs to be planned. I suppose strong apis are probably the foundation of successful route planning because they allow changes of course without rocking the boat too much.

My vote is strong apis are a necessity and good backwards compatibility is a bonus (not necessity) where possible. Ideally good backwards compatibility could mean developers spend less time recoding their own modules and more time coding core.

Cheers

valthebald (not verified)

2 January 2012 - 1:04pm

I am really happy that backwards compatibility issue is discussed without negative connotation by a core dev (for the first time ever, I guess?). Probably, Dries's post from 2006 has established prejudice against even discussing compatibility issues, just to not "loose Drupal as we know it".
That makes your statement (more precisely, ascertaining) about today's Drupal being already something very different from 2006's Drupal, very important.
There's almost nothing to add to such a thorough (as usual) analysis.
Just one question: why do you think, that DirectX-like approach can't be used to provide compatibility of D8 with D7?

Say someone changes the structure of the node form, that will immediately break backwards compatibility for hook_form_alter().

Drupal can't provide two different versions of the same array, then make one form out of them, that's one of the central issues this post is dealing with.

There are some API changes in Drupal core that it would be possible to provide legacy wrappers around (for example we occasionally keep backwards compatibility layers around temporarily when big patches get committed, to allow refactoring of other subsystems to be done over time rather than all at once), but there are not a great number of places this would make sense to do, and it introduces a whole set of issues around which changes require their addition vs. not, whether anyone will spend time fixing bugs in the legacy implementations (if they're not 100% wrappers) etc.

Just one question: why do you think, that DirectX-like approach can't be used to provide compatibility of D8 with D7?

A couple of reasons. One, what catch said. :-)

Two, and more importantly, it's a lot of code to carry around. If it's lazy-loaded classes then it's not quite as bad, but if you mix old and new code together then you will have both in memory at the same time, eating up memory.

Also, the priorities are different. Direct X, like most Microsoft software, was written on the assumption that code has to remain binary compatible for a decade. In that case, dragging around old code is a necessity, even though it leads to piles of old dead code you cannot fix for fear of breaking something.

Drupal currently provides no backward compatibility. If we were instead to offer only some, it would likely only be for one version back, and only on some APIs. In that case, it's simply not worth it to always have to instantiate NodeV7, NodeV8, NodeV9 instead of just Node, which in V9 has a couple more methods added to it that older code just ignores.

Anonymous (not verified)

2 January 2012 - 3:35pm

In the d.o thread on Symfony-based context you have mentioned a couple of times your dislike for the Form API. So, to take your thoughts and run with them... could we reasonably rewrite FAPI to work more like the Database API, and would it be useful to do so? Using the textfield example from the docs, something like the following would get me a Title field...

<?php
$form = create_form();
$form->element->create('title', 'textfield', 'Subject');
?>

which I could later alter using

<?php
$form = get_form($some_form_id);
$form->element->size('title', '100');
?>

Larry

2 January 2012 - 10:27pm

In reply to by Anonymous (not verified)

That's been tried a few times over the years, by me and others. The big question is performance. Function calls are more expensive than direct property access, and if we start mixing in magic methods or ArrayAccess then it gets even more of an issue. How much of an issue it would be in practice I don't know, since that couldn't really be tested without doing the entire conversion and trying it out, and possibly throwing it away. No one has been willing to do that. :-)

Also, for all its faults FAPI is extremely flexible, and hook_form_alter is extremely powerful. It would be hard to replicate that in another form.

That said, if someone else wanted to experiment with an OO FAPI or with using the Symfony Form system to see if it would work for us, I won't stop them. It's just not an area I'm focused on right now.

xtfer (not verified)

2 January 2012 - 10:55pm

In reply to by Larry

I suspect the limitation is that you'd still want FAPIs render arrays running in the background anyway, due to their inherent flexibility.

I am not planning to do anything with the theme until I port to Drupal 7, and then either use Bartik or some fully responsive theme. Not sure when I'll have time to do that. :-) (There's still a module or two missing. Which, naturally, I maintain...)

"Backwards compatibility" eventually becomes "future limitations". If those limitations are too severe then Drupal becomes the CMS that was good in 2011. Remember the websites that were good in 1999? :) People can still use the old code as long as they want, they just need to take a bigger step to get the new features.

Stable interfaces would come a lot closer to compatibility but they still aren't perfect. The dependencies could still be built on undocumented effects, unsupported parameters, and implementation bugs. A function with a fixed parameter list can still have a near-infinite number of values for those parameters, especially if they include arrays with structured data.

It might be possible to have interfaces that look like data structures though. I haven't been thinking about how to re-architect the themeing/render/form system lately but something like that could be a bit harder to represent in a function-based interface (and have performance implications like you said). Maybe it needs to stay as a data structure with the consistency of a function interface. Or maybe you just need to change the order in which elements are defined and it could be done with functions.

The best thing to do is to build a forward-looking core and then give module developers time to understand it and use it. Any change can take years to be supported and used (almost always longer than we expect).

Taran (not verified)

4 January 2012 - 10:54pm

It's a difficult topic and I appreciate the candor. Still, I'm going to get down to brass tacks in a competitive market. Drupal has grown considerably over the years (I, for one, remember the 4.7 upgrade) and it's gotten enough mindshare now where a thought of direction should also include the user base.

One of the things that suffers most with every new version is the issue of contrib modules - contrib modules that Drupal is actively marketed as having. Core and contrib need to work better together to assure that contrib modules are able to upgrade quickly. This is a key reason why Drupal 6 is still being run. When Drupal 6 is no longer supported (read: soon), organizations - particularly SMEs that don't have the budgets - will choose between the costs of upgrading to Drupal or moving laterally to another CMS/framework. The choice to be able to do so, even as Drupal 8 discussion begins in earnest, is largely about budget and direction of the overall project (Drupal) and the direction of implementations (specific sites) as well as a need for perceived stability. This is why there are temporal 'forks' in the Drupal implementations.

The decision to break new ground is very attractive to developers. As a developer with 2 decades behind me, I almost always would rather work on something new instead of having to maintain something. The issue is that users don't want to have to pay for implementation of a website every time someone decides to break the backward compatibility when there's an upgrade. In one particular interview, Dries said that this was necessary to compete with the enterprise level proprietary CMS's out there... but I have yet to consider that a very large market; it's most certainly a lucrative market for the relative few implementations.

If this is the direction of the project, issues related to the SMEs that use Drupal will have to be addressed or that user base may move. They have options. If and when they move, they most certainly will be paying less (or they wouldn't do it) but to get paid, developers will need to support their choice. When those developers support the non-Drupal choice, the Drupal community loses.

The API decisions as related to backwards compatibility have larger implications for the project as a whole. While I don't expect those that are being fed by the changes to agree, I imagine that it would be in the interests of the project to at least entertain the thought and consider it beyond what (we) developers think.

Client budgets, developer mindshare, project direction. Backward compatibility plays a part there. The standard, "Core VS. Contrib" issue that comes up needs to become "Core AND Contrib".

xtfer (not verified)

4 January 2012 - 11:00pm

In reply to by Taran (not verified)

When Drupal 6 is no longer supported (read: soon), organizations - particularly SMEs that don't have the budgets - will choose between the costs of upgrading to Drupal or moving laterally to another CMS/framework.

Or not upgrade at all. Arguably, for a lot of SMEs, the cost of upgrading their Drupal site two versions will be quite high and not necessarily seen as having any benefit beyond a set of technical reasons which are generally quite abstract to those who have to sign the cheques.

Hi Larry,

this is unrelated, but have you got my emails about interviewing you for my podcast? I've used the contact form on the site, but it might have been killed by spam.

Anyway, if you're interested my email is friendlydrupal.. at.. gmail.com

Thanks for yet another great post helping me to better understand these things.

I have been advocating for quite some time that we should simply ditch the upgrade support between major versions. You post is just giving a lot more reasons why that would be a good idea. It has also been discussed in an Drupal core issue posted by @catch over a year ago.

Since Upgrading actually is a core feature, I fully understand why so many read it as working like it does for operating systems and client applications. Just killing that feature and replace it with something like the Migrate module would change things I'm sure.

That's a possible approach, yes. However, there are other requirements for that. Namely, we'd absolutely have to have a standard and known export/import mechanism that worked, always, and was tested, and complete independence between *all* CRUD operations and forms.

That's something else we need anyway, besides cleaner API separation, but it's also something we don't have yet. Either way, we have a lot of work ahead of us.