The perils of under-engineering
Posted:
Conventional wisdom in software development is to avoid overengineering things. The YAGNI principle (You Ain't Gonna Need It) speaks to that, as do dozens of articles and blogs and conference talks from Big Name developers.
What is less often discussed, though, is the cost of under-engineering. A cost I recently paid rather severely.
A typical YAGNI-inspired approach says to start with the smallest thing that will solve the immediate problem and don't build anything else until you know you actually need to. To some extent that's valid advice, but far too often I see it taken to an extreme and have done so myself. The basis of YAGNI is "you don't know what you'll need tomorrow, so don't bother building it until you know."
Which is all well and good but... what if you could actually know what you're going to need tomorrow? Like, planning and forethought? Those are still useful.
There's another principle that I have found to ring true, and have been bitten by when not following it:
Weeks of coding can save you hours of planning.
That is part of why MiDy took two years to write. The biggest factor was having a kid, but early on I actually had plenty of spare time to work on MiDy's core engine. And because I was approaching it as a noodling experimental project rather than a serious project, I wanted to Keep It Simple, Stupid (KISS). It turns out, simple is highly relative.
As a result, MiDy got rewritten about 8 times.
Version 1: Basic file system
The first thing I tried doing was just mapping files 1:1 from disk to URL. That was easy enough, and arguably the core routing setup was a bit more than needed. (Did I really need a PSR-15 middleware stack? Arguably not, but it worked.)
Of course, that fell down almost immediately when I wanted to allow file name prefixes to control sort order, because I couldn't just look for a file matching the URL. I had to deal with wonky prefix and postfix wildcards. And then the sorting itself meant a full directory scan every time.
Version 2: Webmozart/Glob
For flexibility, I wanted to try using a custom file wrapper for the routes path. Unfortunately, that was, er, hard, and also, PHP's glob() function doesn't work on non-file streams. (It just delegates to the OS glob(), which means the OS has to be able to see the disk.)
So, let's try a user-space glob alternative, Webmozart/Glob. Its API was much nicer, and it worked on other streams!
However, digging into it further revealed that "worked" was a bit of a stretch. On non-file streams, it was simply doing a full directory scan itself. So the API was better than doing it by hand, but the performance impact was the same.
Version 3: Flystream
OK, so maybe try Flystream? It allows wiring up a Flysystem object to a PHP file stream wrapper. That resulted in a reasonably nice API, but... The same performance issue at the end of the day. It's always a full directory scan to do anything.
Le sigh. At least I was able to submit some cleanup and bug fix PRs to Flystream in the process.
Version 4: OK, but no database
I'd hit an impasse. I wanted to keep it simple and avoid all the complexity that caching brings, but everything I tried resulted in a complex and unperformant mess that was going to thrash the hell out of the file system.
So I paused and went to see what Grav did. Grav is another file-on-disk CMS that I had tried out but didn't like how it did things. I dug into their code and found... they scanned the entire disk and cached everything. The code to do it was so complicated I couldn't follow it to see exactly how it worked, but I could determine that much.
So if even Grav couldn't do it without building its own cache, I guess I couldn't, either. Damn. So I threw out all the stream-based code and started over, this time doing an incremental cache build (to avoid a mandatory build step) to files on disk, using PHP's native serialization tools (which are, in fact, extremely fast).
The result was more involved than I liked, but it worked, was considerably less complex than the stream approach, and avoided a full disk scan. However... it wasn't searchable. See, I didn't just want to show pages; like any good CMS, I wanted MiDy to allow you to search pages by at least some criteria, like path or tag. And that meant I needed not just a cache, but an index.
I could try building my own indexes of the fields I felt like supporting, but... at this point I'm building my own small database. I don't want to build a database. I just want to use a database.
Version 5: Fine, SQLite, but just PDO
OK, PHP, I give up. You win. I'll use a database to store the cache information so that I can query it with SQL. But I'm only going to use SQLite, and just raw PDO. I don't want the added dependency and weight of something like Doctrine.
So that meant learning SQLite's paltry excuse for a type system and how I could model that into typed objects. Fortunately, I have only two real "models" (Folder and Page), so hand-coding that mapping was straightforward enough. Property hooks in PHP 8.4 (which was released only after I started this whole saga) made that much nicer, but more on that another time.
This approach actually worked fairly well and got me pretty far. I could scan, cache, read, and load pages with great performance, even without any DB indexes yet. And most of the queries were pretty basic, so hand-writing them was fine. I even had a nice setup for creating and reusing prepared statements with nice ergonomics.
And then I tried to write the search logic. That is complicated with a query that has different fields depending on the submitted query. Because of course it is. Why did I ever think it wouldn't be? I needed a query builder, something PDO doesn't provide.
Version 6: Fine, but no Doctrine, just YiiDB
I was still trying to avoid the weight of Doctrine, so I instead tried to use the Yii Database library. It took a while to learn all its quirks and silly bits, but I was able to get it working and had a working query system that I could build on top of. Huzzah! Finally, at long last!
This got me pretty far, in fact, but there was one big drawback: Yii-DB was throwing dozens of deprecation notices in my tests, because it hadn't been cleaned up for PHP 8.4 yet. This wasn't a fatal issue; deprecations aren't errors. It still bugged me, though, and there were parts of Yii I didn't care for, like how it needed its own cache subsystem that wasn't fully pluggable.
Version 7: Maybe Symfony?
Around this time, I also started questioning if building from scratch was even the right approach. Would it be possible to port the interesting bits of MiDy over to Symfony to gain the benefits of that whole ecosystem? After talking with Nicolas Grekas, I gave it a try.
Of course, that meant porting the database logic to Doctrine DBAL, because it made no sense at all to pull in Yii-DB to Symfony. Fortunately, the database logic was all contained to a single repository class. (Hooray for good object model design!) The biggest challenge was Doctrine handling conditionals differently than Yii-DB, and having completely different weird and silly quirks. It took a while, but I was able to get a Doctrine version of the repository working.
I eventually decided I didn't want to use Symfony for this after all, but... now I had a working Doctrine version of the repository, which didn't have any deprecation warnings and orders of magnitude more users to encourage future contributors.
Le sigh.
(In fairness to Yii, they have since released a new major version of Yii-DB that is, I presume, deprecation-free under recent versions.)
Version 8: I give in
So... I give up. Doctrine DBAL it is, but I drew the line at Doctrine ORM (which is far too overengineered for this use case). So after months of faffing about and throwing away hundreds of lines of code, multiple times, I ended up... with Doctrine DBAL and a database. Exactly what I had spent all that time trying to avoid. That's what shipped in MiDy beta last week.
In hindsight, it should have been obvious. Had I sat down and actually planned out my feature set and what it would require, I would have realized far earlier that an indexable cache was mandatory (and that SQLite was the obvious choice), and that a query builder could not be avoided. Every attempt to Keep It Simple made it more complex.
Because, Yes, I Was Gonna Need It.
Months of coding saved me hours of planning. That's what I get for not listening to my own advice and instead following YAGNI/XP recommendations. Lesson learned, the hard way.
May this tale serve as a warning, so you can learn this lesson from it the easy way.
Go forth and plan out your projects. Don't overengineer them, sure... but don't under-engineer them either.