readfile() not considered harmful

Posted: 4 May 2012 - Reading time: 9 minutes

• PHP
• Drupal

If you're like me, you've probably read a dozen or two articles about PHP performance in your career. Many of them are quite good, but some are simply flat out wrong, or misinformed.

One of the old truisms that has been repeated for as long as I can recall is "don't use readfile() if you have big files, because it reads the whole file into memory and your server will explode." The usual advice is to manually stream a file, like so:

$fp = fopen('bigfile.tar', 'rb');
while (!feof($fp)) {
 print fread($fp, 1024);
}
fclose($fp);

There's just one problem with that age-old truism: It's not true.

Recently I was running some benchmarks to compare the performance of various different ways to read big files off disk in PHP. PHP has no shortage of tools for shoveling data from disk to browser. I count at least four possible approaches, including the manual approach above:

$src = fopen($file, 'rb');
$dest = fopen('php://output', 'a');
stream_copy_to_stream($src, $dest);
fclose($src);
fclose($dest);

$fp = fopen($file, 'r');
fpassthru($fp);
fclose($fp);

readfile($file);

So I setup some test scripts to bang away at, delivering a 20 MB file. I figured I'd see some sort of variation between the different approaches, with some CPU vs. memory trade-off. Instead, averaged over 50 runs this is what I found:

(Peak memory numbers are as reported by memory_get_peak_usage(TRUE); memory_get_peak_usage() was slightly lower and varied a bit more, but not by a statistically significant amount.)

The "real" memory usage of all four methods was identical. The CPU time was nearly the same for fpassthru(), readfile(), and stream_copy_to_stream(), but about 50% higher with fread(). I even tried cranking my PHP memory limit way down, below the 20 MB file size I was using, to cause it to crash. Nada.

As it turns out, readfile(), fpassthru(), and stream_copy_to_stream() are nearly identical internally. All use the PHP streams API internally, and in fact the same internal operation, php_stream_passthru(). Depending on your OS, it will either use mmap or do its own chunked iteration using 8 KB chunks. That is, in the worst case, it will do exactly the same thing you would have done manually, but all done in C code so that it's faster. If your OS supports it, it will use mmap(), which is an OS level operation that allows contents on disk to be read as if they were in RAM; the OS takes care of paging the file in and out of physical memory as needed.

So why the stigma around readfile()? There's a couple of possible reasons that I see.

1. top. The top command, available on virtually any *nix, reports among other things the memory used by various running processes. It's a very old tool, however, and when using memory-mapped files (mmap) doesn't know how to differentiate between actual memory usage and mmaped files. That is, it may over-report the memory the file is using.
2. Output buffering. PHP optionally supports buffering of output sent by a script. There are various good and bad reasons to do so, but one side effect is that data is not sent until the program says to. If you're sending a large file, then that file will get buffered and not sent like any other output. That is, it's not PHP's normal working memory that overflows but the output buffer. Since it's not reported separately, though, that can lead to people misinterpreting the root cause.
3. "It's common knowledge." Let's face it, most PHP tutorials online either date from the Bill Clinton presidency or are derived from other tutorials that old. Some of the PHP introductory material online is truly god-awful. (If you ever find a tutorial that suggests to use the mysql_*() functions, close it immediately.) It's possible (although I did not dig through ancient C code to confirm it) that somewhere in PHP's dark past readfile() was less smart than it is now, or perhaps operating systems were less capable. So warnings about readfile() may have been valid at some point, but they have not been since PHP 5.0, at the very least.

Or maybe none of those. I'm not sure. In any event, readfile() is not the enemy. In fact, if you're sending a file to the browser then any of the available automated functions are perfectly fine... except for looping on fread() yourself, which is the slowest option available. Even if you need to do some processing on the data as you go, you're better off implementing a stream filter.

If you're feeling more adventurous, there's also mod_xsendfile for Apache (also built into lighthttpd), which should be even faster. I haven't worked with it, though, so caveat programmer.