The Drupal development list has been sharing Drupal development server setup secrets lately (try saying that five times fast), so I figured I'd toss mine out there. And since knowing me it wouldn't be short, I figured I'd blog it instead of just posting it in email. :-) Here, then, is the development environment we've set up for Palantir.net.
Several people have noted that each developer on a project has their own complete copy of a site, including database. We actually take the opposite approach. Everyone has their own install, but the database and files directory are shared specifically so that we don't need to worry about synchronization issues.
We have a single development server in-house, on which everyone has an account. They also have a sandbox, with a subdirectory corresponding to each project we have active. Each user/project is dynamically mapped to its own domain name, so we don't have subdirectory pathing issues to worry about.
We also have a Subversion repository with our in-house Drupal 5 install profile. It's a stock Drupal 5 (now 5.2) download, an install profile with a few extra key modules (CCK, Views, and a half-dozen others), a sites-dist directory instead of sites, a slightly modified Zen theme in sites-dist/default/themes, and a stub for a site-specific module in sites-dist/default/modules.
When we setup a new project (say fooinc), we have a script I wrote that does the following:
- Create a new repository fooinc.
- Export from the main Drupal5 repository to garfield/sandbox/fooinc.
- svn add all of the new files to the fooinc repository.
- Create a blank database fooinc.
- Move sites-dist/ to sites/.
- Create a shared files directory /shared/files/fooinc and symlink it to sites/default/files.
- Set permissions on settings.php and sites/default/files as needed.
I (or whoever is setting up the project) can then just run the Drupal install script and select our install profile, which does the extra setup we need, and then hit svn commit. yes, we even commit the settings.php file, because the database is shared and we don't need to set a base path.
Anyone else joining the project, then, just runs another script which:
- Checks out from the fooinc repository to lastname/sandbox/fooinc.
- Symlink sites/default/files to the shared files directory for that project.
- There is no step 3.
The advantage of this setup is that we can have a new project created and ready to go in, usually, under a minute, and new people can join the project in under 30 seconds (give or take checkout time if we have large modules). The shared database generally doesn't cause us a problem, either. Since so much of modern Drupal site setup isn't coding but configuration (CCK fields, Views, node settings, comment settings, Actions, etc.), having to merge the databases would be a nightmare. This way we have a single shared system. In practice it's very rare that we break anything on someone else's dev environment by changing a site config in the database. Far less common than a conflict when checking in a file that two people have been working on (which happens sometimes even with the best commit practices).
The other advantage is ease of upgrading. When Drupal 5.2 was released, we simply had to upgrade our base repository. Then one person on each project runs another script (yay for automation!) that just exports from that base repository over the files in that project. Because all site-specific code is kept to the sites/ directory (this is why Drupal people always tell you do to that!), and the base repository has a sites-dist instead, we don't lose anything. Simply run the update.php script, commit, and you're done. (For 5.2 we also needed to merge the settings.php changes, but that's rare.) Everyone else just does an update as usual. It works for updating modules in the install profile, too.
We have some tweaks planned in the future for this setup, mostly in the install profile itself. I keep wondering if I should be using svn vendor branches and other advanced magic, but so far I haven't needed to and this has worked well enough for our needs.
Hopefully this entry will prove helpful to someone else setting up their own multi-developer environment. And if you have suggestions on how it could be improved, I'm interested in those, too!
out of office
do you expose your shared DB server to the outside world? do you use a VPN? i guess your team doe snot do much offlie development. thats the only flaw in your setup. i agree that the conveniences of shared DB are substantial. i worked at a large non drupal web site that used their production DB for development too. that eliminates all data sync issues which is nice. almost all DB access went through well tested sprocs so it wasn't as crazy as it sounds.
Our dev server is local behind a firewall, and we have a VPN for external access when working remotely. (We're not a virtual company, though, and rarely telecommute.) Yes we can't do much work on an airplane this way, but that doesn't come up often for us. :-)
Automation of upgrades and shared databases
Thanks for sharing, Larry. It's really interesting to see how hardened Drupalers have evolved their development environments.
So each project has its own entire Drupal codebase (exported from Drupal 5.2) and always uses sites/default/ ? That's an interesting way round to do it: what made you do that (and shift the automation from subversion and sites/domain.name/ to hand-rolled scripts)? It gives you more freedom to make client-specific changes to core, which is a plus, but from the looks of it you don't do that.
I may have to rethink aiming towards the shared/separate database, as the general opinion seems to be it's too much faff and not enough reward to separate them (you hardly ever get cross-developer conflicts). One big consideration for us is that we have multiple offices, so the single development server isn't always the most efficient way for developers to work on the code, at which point you're looking at installing multiple development code/databases anyway.
.../files sharing isn't something I've looked at yet, so thanks for the heads-up on that.
Yes, every site uses sites/default/ for site-specific stuff. Actually we have an internal standard that any contrib modules we use verbatim go in sites/all, and code we custom built or contribs we have to modify go in sites/default/.
The main reason I set it up that way was that each of our personal sandboxes is its own domain (projectname.garfield.devserver.net). That means we can't share the same settings.php and site-specific modules directories unless we use sites/default for everything. This way, we can even share settings.php and let Drupal figure out which domain its own.
Remote locations would be a complication, as would offline development as moshe noted. We don't do much of either, and when we do we just VPN into the network. Pretty much everything is on the dev server, including the sandboxes, not on the desktops. That's good because if you can VPN in and mount your home directory share, you get pretty much the entire dev environment without any synchronization issues. That's bad because if you can't VPN in, you can't really do anything.
Personally I rather like not being able to work when not in the office, but YMMV. :-)
One True Drupaling Style
It seems there's no canonical right way to do distributed Drupal, but a number of off-the-shelf ways of doing various bits of it. Including discouraging suggestions that you should work when out of the office!
Push to Prod?
Thanks for the great writeup. Our own little company will base our development practices off of this.
You've got me wondering about production deployment. When you are ready to push version 1 to the prod server, it's probably just a matter of letting your .profile handle the configuration of setup in the DB. But what about version 1.2? How do you handle keeping your Staging and Production environments in sync?
Actually our usual development routine involves building the site in our dev environment, then just uploading the files and a database dump to the live server. Client reviews, we make changes locally, and re-upload as needed. Usually we don't let the client near the site until we're done with it.
Once the live site is up (whether launched or not), we consider the dev copy canonical for code/theme and the live site canonical for content. Periodically we then dump the live database back to our dev server to resync it. That way we avoid most syncronization issues. It's not perfect, but sometimes the simple solution is "good enough".
Resyncing the database from prod to dev for content is the same thoughts I had. How do you handle configuration changes? For example you need to make a change in development to support something new. Do you just record the steps execute them by hand on the production site?
By the time the "live" database is database-definitive, we rarely are making drastic config changes. If so, then yes we do double-enter them. For Views, we push all Views to code so that we can get proper versioning on them, and can also just re-upload a code file, refresh the views cache, and be done with it. That's usually the most complex "config change" we have to deal with on most sites.
Thanks for your
Thanks for your suggestions.
There's one point in the development cycle where shared db won't work: when developing new modules.
Say I'm creating a module that hasn't been checked in yet, or has been checked in but is in alpha stage and should not be activated by anyone but me. I activate the module, then someone else in the dev teams visits the site with the same db. Two things can happen: 1) they don't have the module so it gets automatically deactivated by Drupal; 2) they now have an activated alpha stage module that might interfere with their work.
And when the database isn't shared, the /files shouldn't be shared either, is how I understand it.
Then again, when the modules have been created and are hardened out somewhat, a shared db is definitely the way to go.
What about d6
Is the process mentioned similar for your d6 sites?
It was really interesting to
It was really interesting to read your post man. I have been working in software development for many years and I still work there. I have to admit that Drupal is a hobby for me and I am a new to Drupal. But I have plans related with Drupal in the future. I think that this sphere really could be very successful. But it is only my plans. Anyway thanks a lot for the interesting article and I will be waiting for other great ones from you in the future.
Peter Wilson from custom software development
Multi-developer support is
Multi-developer support is always trickiest due to the multiple access point. Although, it looks like you solved many of the synchronization issues that generally come about from it.