Drupal 7 gets introspective code registry

At DrupalCon Sunnyvale 2007, Rasmus Lerdorf chided Drupal on spending over half of its request time on just the bootstrap process. As a GHOP Task , Cornil did a performance analysis of Drupal and found its two largest performance drains were the bootstrap process and the theming layer. Quite simply, Drupal spends too much time including code.

Drupal 6 has the beginnings of a solution. Page handlers, the most unused code in Drupal, can now be split out into conditional include files and the menu system is able to conditionally load just the file it needs for a given page request. Based on earlier benchmarks, just that code shuffling netted Drupal 6 a 20% performance boost. The downside, however, is that it does require the module author to explicitly specify file to be included, and the syntax for it is just a little bit annoying what with the file name and file path being separate keys on the menu handler.

Fortunately, Drupal 7's self-learning code registry system has just landed, which should obliterate most of the wasted bootstrap cost.

I started toying with the registry concept last fall, but it didn't really get useful until February when chx visited Chicago for two weeks as part of the Data API Design Sprint. It's gone through several iterations in the past three months, but the final version is really slick. All a module author need do is specify in the .info file what loadable code files the module provides, including the .module file itself. The rest is automatic.

The heart of the registry is a token_get_all() call. That's a PHP function that parses a file of PHP code into a huge array. From that, it's trivial to build an index of all functions and classes in the system and save that to the database, along with information about, for instance, what function maps to what hook, and therefore what hook implementations exist. That then eliminates the need to load code and use function_exists(), since we can now dip into the database to determine if a function or class exists and if so where. Add some caching to taste. Now no code needs to be loaded until it's actually needed. Nifty.

So what do you need to do in order to make your module Drupal 7-ready? First, list all of your loadable code files in your .info file, in the files[] array. After that, it's just a matter of cuting and pasting code for optimal loading and refreshing the cache on the admin/settings/performance page.

Any class or any function that is called indirectly (via module_invoke(), module_invoke_all(), drupal_get_form(), theme(), etc.) can live anywhere; functions (but not classes) that are called directly need to either live in the .module file or else be lazy-loaded first, like so:

<?php
if (drupal_function_exists('myfunction')) {
 
myfunction($foo, $bar);
}
?>

If a hook supports optional dynamic forms, such as hook_form_alter(), make use of the dynamic version (e.g., hook_form_$form_id_alter()). That way, you can load just the alters you need. If you make use of classes, those will lazy-load from anywhere thanks to PHP 5's autoload capabilties.

So what makes optimal organization? Well, that will require some experimentation. Registry-style hooks like hook_menu() and hook_theme() are loaded only once in a blue moon, so those can easily be moved to an example.registry.inc file. Form alters for rarely-used forms, such as admin forms, can also be moved out to their own files, or more likely the example.pages.inc and example.admin.inc files that most modules already have as of Drupal 6. Node pseudo-hooks can easily be placed in a file per node type. (Yes, that means .node files are now a reality! Or can be if you want.) Any form that appears in the body of the page can be moved out as well.

Beyond that, we will need to do some benchmarking and profiling. While the intent is to reduce the amount of code that needs to be parsed on each page load, we need to avoid ballooning the number of files that are loaded, too. File IO is expensive. If no one else looks into it I will try to do some investigation into some good guidelines, although it will vary a bit by module. (Feel free to post your own investigation results, though!) chx and Morbus have said they will be working on leveraging the registry to clean up the bootstrap process and possibly even include hook weights. (Oooo...)

Huge thanks to Justin Randell, Peter Wolanin, Moshe, and everyone else who joined the issue queue and helped make the registry really rock.

Comments

Impact if everything is in RAM

Larry, first thanks for taking the time to write this up. I really appreciate being able to keep up on core development with your posts.

I know that one of Java's innovations was dynamic class loading, which was supposed to offer over the network just in time code delivery, and smaller resource foot print. In practice, a generation of static loading programmers couldn't understand why the wrong classes were being loaded from multiple class loaders. The solution turned out to be to get Java application servers to just load everything into memory and then only load new files when the server was prompted to do so. This reduced loading time, since resource requirements were simply increased to accommodate this performance improvement.

Two questions. How will the registry prevent the wrong functions from being loaded? Will this make a difference for sites that just spent $200 to buy more RAM and rely on Memcache and RAMdisks for loading files.

DRY

Yes, having multiple functions/classes with the same name will break the registry. The solution is simple: Don't have two classes or functions with the same name. That is already a non-issue for 99% of Drupal code because of module/theme namespace policies. The only place it is an issue is for pluggable systems that initialize after the registry and do not have a manual initialization step. The standard solution in that case is a factory class/object, which was the subject of a previous article that was targeted specifically at this issue.

Are you considering the

Are you considering the performance with opcode caches like APC or eAccelerator? They do have problems to cache dynamic/autoload includes.

Answer

Yes, it is problem. So better don't use dynamic includes.

Not anymore

I think that current versions of the common opcode caches do not have (basic) problems with dynamic includes anymore.

Gallery 2 is using dynamic includes since it exists, which is about 3 years if you consider the alpha phase as well. We used to declare incompatibility with APC for exactly this reason but APC and eaccelerator have become much more robust since these days and we highly recommend opcode caches today.

BTW: Gallery 2's dynamic include is mostly based on a registry to filter out inactive modules, the factory pattern to load only the required MVC controller / view and a facade for the rich core API where function definitions are grouped into several files which are included on request. Using a facade class for the API allowed to do such auto-loading without PHP 5's built-in autoloader and without explicit if/else blocks all over the code base.

Already the case

First, that's a very poor answer to the problem at hand. From an opcode cache perspective, Drupal is already doing dynamic includes. What files it includes are based on database state (specifically the system table), which can vary at any time. Opcode caches that can't handle non-trivial include structure already hate Drupal; we're not making anything worse in that regard.

Secondly, the vast majority of Drupal sites run on shared hosts that don't have opcode caches. If we want to optimize for the common case, the common case is $10/month over-sold shared servers. Reducing Drupal's footprint in that environment is a big win, both for Drupal and for hosts. And since we're already doing dynamic includes and can't break opcode caches anymore than they already are in that regard, at worst this will have no effect on opcode-cache-using sites.

But there are still opcode cache right ?

IO time can be widely reduced using an opcode cache such as APC. This is not a general solution for most PHP5* code, but in PHP 6 APC will be directly included in the Zend core engine.

So ok, it's great to do such optimisations, and I'm sure the final goal is not only to reduce IO, but at the moment PHP 6 will be released, chasing IO produced by require_* include_* will just be deprecated.

Ok, did not read the

Ok, did not read the previous comments, but the fact is still there, is this really usefull to cut the code into smaller pieces when some great framework like Zend spend most of it's time doing the same thing as Drupal, i.e. load files. OPcode cache seems a pretty universal solution to this problem.

Shared hosting

Only a tiny fraction of the PHP using world has an Opcode cache available to them. For the other 95% of the world, lazy-loading of code is the most straightforward way to reduce bootstrap time.