Keeping your sysadmin happy

As Drupal gets bigger and bigger in the marketplace, it is moving into areas where system administrators still hold sway. Dedicated servers or server farms have a different set of needs than a shared host when it comes to monitoring and performance.

That's not even Drupal specific. For any high-end web app, it's useful to be able to interact with it for administrative purposes through standard system tools. On Windows, that's the Windows Administrative Tools or IIS. On LAMP, that could be a unified web app like webmin or a KDE control panel plugin or a Gnome applet. Getting a web app into certain organizations requires offering existing sysadmins a way to integrate it into their existing management workflow.

But what pieces of the app do sysadmins want in their existing admin tools? Calling all sysadmins, what do you want from us? :-)

This isn't entirely a hypothetical question. There is some work already in Drupal that can improve sysadmin workflow integration. Drupal 6, for instance, ships with a module that routes logs to the syslog logging system on a Unix-style server. There's also a module now that can route log messages to Growl for OS X. But that's only the beginning, and there's far more that we can do.

So, I'll put the question out to sysadmins, developers, and anyone else who wants to get to their CMS (Drupal or otherwise) from their system tools. What do you want?

  • Directly access the CMS logs from a system tool, or have the CMS route its logs to a common logging utility a la syslog?
  • Expose tracking information to a system tool, or have the CMS save its tracking logs to a 3rd party system (could be Google Analytics, or an in-house tool, or some system-level utility)?
  • Manage CMS users directly from a system tool?
  • Integrate CMS users with the server user base (via LDAP, Active Directory, or even PAM)?
  • Expose selected configuration options through your system tool?
  • Something else?

I'm not a sysadmin, so I don't know what you need from us. We can write it, but first you need to tell us what you need.

Comments

Wishlist

In addition to what you suggested, an event system that exposes it self via:

  • a Nagios plugin for alerts
  • snmp (e.g. for resource planning with cacti)
  • customizable RSS feeds

Btw, Growl sounds like a good idea -- can you add a link to the plugin?

Included

The link in the article goes to the Drupal Growl plugin. I figure it's only a matter of time before someone writes a KNotify equivalent for KDE and a Win-whatever for Windows.

Some tools

There are some tools already that exist for this.

For example, using syslog for the watchdog is already in core since Drupal 6.x. This allows messages to be sent over the network to a central server that holds logging for more than one Drupal server, and perhaps other applications too.

Enterprise tools exist to parse the logs for certain error levels and/or strings and pass those on per priority to various means (screen, pager, ...etc.)

The logging and alerts project also contains tools to route certain levels to different email addresses. Lately, I added the ability of logging to the web server log too to that project.

I want to make Drupal a MIB (management information block), so it can be monitored via SNMP too. This would be very useful for a large enterprise, and integrates into many tools out there.

URL?

Got a link to this logging and alerts project?

I don't really know from SNMP, but if it has an API already that we could plug into that would be totally cool.

logging and alerts project

The logging and alerts modules can be found here:

http://drupal.org/project/logging_alerts

LDAP and AD

I believe most large organization that already have LDAP or AD already integrated with their enterprise software would expect their CMS to also utilize LDAP or AD. In fact, I've known those above me at my workplace that will dismiss information systems on the basis that it isn't AD compatible. We can argue against the validity of such policies, but if Drupal is to break the enterprise ceiling then it needs to at least be able to handshake with our Windows systems.

I've had great luck with using the contributed LDAP integration module with my Drupal sites behind the firewall. However, increasingly the desire at work is to have our Web applications use Active Directory instead of LDAP. What a joy it would be to use the user accounts and groups (for Drupal roles) managed in Active Directory for Drupal sites.

System admin needs

Logs:
a module that would be able to grab the log files in specified area given by user. ALA http://www.Splunk.com

SNMP:
a module be built that would display these results and possibly be graphed with RRDTOOL ALA Cacti

Network Graphing:
simple module that will ping an ip address and graph the latency. RRDTOOL tie in.

Xen Virtualization:
Xen module-that allows admin to issue calls to xen console or listening ip-address

VNC Plugin:
module that allows a pre-defined screen size or pop-up vncviewer console to connect to ip addresses.

Pear channel

Drupal is a very nice CMS.

The main problem for a sysadmin hosting Drupal is doing upgrades. Upgrading from normal Drupal is IMHO a pain compared to some other CMSes(see disclaimer).

I would like to see Drupal add support for installing Drupal as a PEAR package. This could also be done to the normal distribution - the src directories could just be parts of a private PEAR installation. Then upgrading Drupal would just be a question of doing pear upgrade

Disclaimer: I have been (and sometimes is) a developer working on the Midgard CMS. There we use pear upgrades with good results.

Modules?

How exactly would that work vis a vis the way Drupal handles modules? A Drupal site structurally has Drupal itself as the top-level "piece" of a web site, and them modules (both custom and existing) get added to a modules subdirectory deeper in the directory hierarchy as needed. How would a PEAR channel handle that?

The direction the Drupal community seems to be moving is actually using Drush, a Drupal-centric command line utility that offers much of the same sort of functionality using Drupal.org as the "mothership" rather than relying on disparate 3rd party repositories. Would a more developed Drush be sufficient?

Re: Modules?

I am not an expert at the inner workings of modules, but I would use multiple PEAR packages for different modules and themes.

A PEAR package may contain files that are in directories bellow other packages, so this is not a problem. For an example, see the Structures_Datagrid package (first one I came up with).

IMHO it is not a good idea to develop Drush to handle updates when you already got a ready made, tested and widely used packaging system in PEAR.

You could however use a tool like Drush to wrap PEAR for your specific needs. AFAIK this is what symfony does for its plugin system.

My main requirement (or feature request) is quick and easy upgrades. I just happen to think PEAR is the easiest way to do this :)

And the answer is...

As you are probably starting to figure out, what sysadmins want depends quite a bit on the environment they happen to be working in, their methodologies for collecting information, their way of interacting with systems, performing upgrades, monitoring, etc. Some sysadmins will be happy to monitor the web server and database server without requiring that drupal actually have its own MIB. Some will want integration with Windows logging, others with syslog on *nix. Some will want GUI access, others not-so-much. Some do upgrades with PEAR, others have a formal rollout process that makes it irrelevant what Drupal does to aid in upgrades. Some will want the MIB, and others (myself included) wonder what value could be added by creating a MIB for drupal over just monitoring the web server, database server, host OS, etc (using MIBs for SNMP or other tools).

If you're a Drupal developer with sysadmin experience, awesome! But if you're just someone who thinks Drupal is cool and would be useful at work, you'll need to ask around and see what the requirements are for any CMS to work in your environment. A good sysadmin should be able to develop a document stating the requirements for consideration, and justify those requirements. Some of them can do it in a language you'll even understand, which is a bonus :)

Research

Yeah, I expected to have a variety of answers. :-)

I'm asking from the perspective of a PHP developer generally and Drupal developer specifically, who does not do sysadmin work but wants the stuff I do to be sysadmin friendly; that is, to be something that people running server farms will find friendly enough to their workflow that they're willing to install it. I'm sure it's a long and heterogeneous checklist, which is the goal here: To collect a checklist so that we (collectively) can work on addressing it.

Pluggability

Drupal ain't bad for sysadmins. I reckon it is DBAs (OK that might be the same person at a lot of sites), that probably get more pain from Drupal :)

What I mean by that is that the codebase side of things is nicely scriptable and deployable by things like drush or even by hand is pretty easy. Its the DB merging/staging type stuff that is harder. I have to admit not being entirely up with the play regarding the D7 DB layer though - maybe improvements in that area are already under way.

Maybe if form submissions could be automatically turned into hook_update_n() functions? A bit like devel macros but with the hassles of drupal_execute() and more limited functionality available to update hooks.

Anyway, back ontopic to sysadmin stuff...

I think the key is to just make as much of Drupals subsystems as possible pluggable rather than coding up specific solutions as each site can be quite different in the way they operate. As long as the API is flexible enough, the alternative implementations can come from contrib.

Logging and caching is much more pluggable now which is a good thing - eg the syslog module. These were two of the big things on my wish list :)

One thing that would be useful from the LDAP/AD side would be if the user account database could be partially or completely pluggable. eg allowing the possibility of LDAP providing the user account / profile / role membership data (presumably with some Drupal caching of the LDAP queries for performance).

I'm not so sure about SNMP stuff personally (especially controlling anything from SNMP). And I'm also not quite sure what Drupal specific stuff could be built for eg Nagios that isn't already handled by more generic web site monitoring tools. But hey, I'm probably just lacking imagination.

Although I am a sysadmin, I don't deal with huge sites or enterprise environments so my perspective might be a bit limited :)

--
Cheers
Anton