• Download PHPStorm for Linux from JetBrains website.
• Now extract the tarball and run the bin/phpstorm.sh script.
• PHPStorm should start up.

The next step is to configure your license.  In my case, I was using a corporate license server.  The issue with this is that you need to log into JetBrains' website using a special link to activate the license.  Unfortunately:

• By default, WSL doesn't have a browser installed.
• Firefox can't be installed because the default build uses a snap image, and WSL apparently doesn't support snap.
• PHPStorm doesn't appear to be able to properly deal with activating via a Windows browser (I tried pointing it to the Windows Chrome executable and got an error page that points to a port on localhost).

So how do we get around this?  Well, we need to install a browser in WSL and configure PHPStorm to use it.  So here's what we do:

• Skip the registration for now by starting a trial license.
• Download the Vivaldi for Linux DEB package from Vivaldi's website.  You could use a different browser, but I like Vivaldi and it offers a convenient DEB package, so I used that.
• Install the Vivaldi DEB.  WSL will be missing some packages, so you have to run apt install --fix-broken after installing it.
• Go into the PHPStorm settings and configure your web browsers to include Vivaldi and set it as the default browser.
• Go back to the registration dialog and try again.  This time, PHPStorm should start up Vivaldi and direct you to the appropriate link.
• Log into your JetBrains account and follow the instructions.  The web-based portion should succeed and registration should complete when you click "activate" in PHPStorm again.

There we go - PHPStorm is registered and works.  Mildly annoying setup, but not actually that bad.

Anyway, I noticed a rather annoying issue where Nextcloud was leaving hundreds of stale auth tokens in the database.  Apparently, I'm not the only person this has happened to.

While Nextcloud has a menu item to revoke and remove stale sessions on their settings page, it's on a per-item basis.  So if you have hundreds of stale sessions, the only way to remove them is to go through, one by one, and click the menu and select the "revoke" option.  Needless to say, this is terrible.

The less annoying solution is to just go straight into the database and delete them there.  You can just run something like:
DELETE FROM oc_authtoken WHERE last_activity < <whatever_timestamp>;
That might be ugly, but at least it doesn't take forever.

It's important to note that, in addition to being annoying, this is evidently also a performance problem.  From what I've read, it's the reason that authenticating to my Nextcloud instance had gotten absurdly slow.  The app responded fine once I was logged in, but the login process itself took forever.  It also seems to be the reason why my hosting provider's control panel has been showing I'm way over my allotted MySQL execution time.  After deleting all those stale sessions, not only is login nice and snappy again, but my MySQL usage dropped off a ledge.  Just look at this graph:

As you can see, January is a sea of red, and then it drops off to be comfortably under the limit after I deleted the old sessions.  The Nextcloud team really needs to fix this issue.

The thing is, I actually remember when NextCloud split from ownCloud.  At the time, I was working on a (now-defunct) product that involved ownCloud.  Basically, my company's core business at the time was data backup, so we had a lot of servers with big disks and were looking for a way to monetize that extra space.  The idea at the time was to do that by integrating a "file sync and share" product into our offerings, and that product was a rebranded ownCloud Enterprise.  Of course, the "file sync and share" space was already pretty crowded, so that product never gained much traction, but it did help me get more into ownCloud and the company even paid to send me to their user conference in Berlin, where I got to meet their team (who, at the time, seemed not-very-impressed with the whole "NextCloud" thing) and see some sites.  So it was actually a great experience, even if the product didn't pan out.

Anyway, despite my affection for ownCloud, my motivation for this change was actually pretty simple and prosaic - I was upgrading my home server (that'll be another post), and I didn't want to downgrade shit.  See, I actually run two ownCloud instances - one on my local network for accessing various media files, and another in my web hosting, for caldav/carddav and files that I want to be highly available.  For my home instance, I was doing a fresh install of the latest Ubuntu MATE on  a brand-new box.  This shouldn't be an issue, except that MATE comes with PHP 8.1, but for some reason, ownCloud only supports PHP 7.4.

Yes, you heard that right - 7.4.  That's the newest version that's officially supported.  The last 7.x release.  The one that's no longer actively supported and has less than six months of security updates left.  That one.  That's what they still expect me to use.

For my previous home box, I believe I'd actually hacked up the source a bit to make it work (since I don't think I depended on anything that didn't work in 8.x), but week I was sick and I just didn't feel like it.  Depending on a version that's about to lose security fixes is crazy anyway.  So I figured I'd "upgrade" to NextCloud, since they actually recommend PHP 8.1.

For my home server, I just did a fresh install, which is fairly straight-forward.  The only annoying part was the Apache configuration, and that was only annoying because I was running NextCloud on a non-standard port and forgot to add a "Listen" directive. 🤦‍♂️ For this instance, there was no real need to do any migration, because the only data I had in there was the (very small) list of users - the rest was just files, which can be trivially re-indexed.

Upgrading the instance on my web hosting was another story.  Since that had my carddav and caldav data, I really did need to migrate that.  I was also already several versions behind on my updates - it was running ownCloud 10.3, whereas 10.8 was current.  However, this turned out to be a blessing in disguise.

You see, NextCloud includes support for migrating from an ownCloud instance.  The thing is, they only support specific migrations.  In my case, the relevant case was that you can migrate from exactly ownCloud 10.5 to NextCloud 20.  Sadly, it took me a couple of tries to realize that the version migration matrix are exact, so there was no path to directly migrate from ownCloud 10.3 to NextCloud.  So I had to use the auto-updater to update ownCloud 10.3 to 10.4, and then manually update ownCloud 10.4 to 10.5 (because the auto-updater wanted to go all the way to 10.8).  Then I could follow the migration process and manually update to NextCloud 20.  From there, I was able to use the NextCloud auto-updater four times to upgrade to the current version.

So the upgrade process was...tedious.  Not really "hard", but definitely tedious.  The directions are pretty clear and simple, it's just a lot of steps to get to a current version of NextCloud.  But at least none of the steps were particularly complicated or prone to error.  As data migrations go, it could be much worse.  And the best part is that it maintained URLs and credentials, so I didn't even have to reconfigure my caldav/carddav clients.

As far as NextCloud itself goes, it seems...pretty much like ownCloud, but nicer.  They've made the UI prettier (both for the web interface and the client app), added a nice dashboard landing page, and made some other cosmetic improvements.  They also seem to have a wider range of installable apps, which is nice.  I haven't had all that long to play with it yet, but so far it seems like a distinct upgrade.

But one of the things that PHPStorm did automatically, which was quite handy, was to add PHPDoc comments to functions automatically.  This is kinda nice because, let's face it, unless you're writing a long description, most of a docblock is just typing.  You duplicate the parameters and return signature and, if the names and types are pretty obvious (which they should be), then there's not really much to say.  But having them is part of the coding standard, so you can't just skip them, even though they don't add much.

Fortunately, Vim has a plugin for that, known as PDV.  It will read the declaration of a function (or class, or a few other things) and auto-generate a docblock for you.  This is nice, but the extension was a little out of date - it hadn't been updated to support return type annotations.  There was a pending pull request to add that, but it hadn't been merged.  I'm not sure why - apparently that repo is dead.

So I decided to just create my own fork and merge the outstanding pull requests.  Now I have a version that supports modern type annotations, which is nice.  While I was at it, I also added an alternative set of templates for NaturalDocs doc comments.  I use NaturalDocs for LnBlog, so I figured it would be nice to be able to auto-generate my docblocks there too.  All I needed to do as add a line to my Sauce config to change the PDV template path.

Well, they're wrong.  PHP's documentation sucks.  And if you disagree, you're just plain wrong.

Actually, let me add some nuance to that.  It's not that the documentation sucks per se, it's that it sucks as documentation. 

You see, a lot of PHP's documentation is written with an eye to beginners.  It has lots of examples and it actually does a very good job of showing you what's available and giving you a general idea of how to use it.  So in terms of a tutorial on how to use the language, the documentation is actually quite good.

The problem is that, sometimes, you don't need a tutorial.  You need actual documentation.  By that, I mean that sometimes you care less about the generalities and more about the particulars.  For instance, you might want to know exactly what a function returns in specific circumstances, or exactly what the behavior is when you pass a particular argument.  Software is about details, and these details matter.  However, PHP frequently elides these details in favor of a more tutorial-like format.  And while that might pass muster for a rookie developer, it's decidedly not OK from the perspective of a seasoned professional.

Case in point: the socket_read() function.  I had to deal with this function the other day.  The documentation page is rather short and I was less than pleased with what I found on it. 

By way of context, I was trying to talk to the OpenVPN management console, which runs on a UNIX domain socket.  We had a small class (lifted from another project) that basically provided a nice facade over the socket communication functions.  I'd noticed that, for some reason, the socket communication was slow.  And I mean really slow.  Like, a couple of seconds per call slow.  Remember, this is not a network call - this is to a domain socket on the same box.  It might not be the fastest way to do IPC, but it should still be reasonably quick.

So I did some experimentation.  Nothing fancy - just injecting some microtime() and var_dump() calls to get a general idea of how long things were taking.  Turns out that's all I needed.  It quickly became obvious that each call to the method to read from the was taking about 1 second, which is completely absurd.

For context, the code in that method was doing something like this (simplified for illustration):

$timeoutTime = time() + 30;
$message = '';
while (time() < $timeoutTime) {
    $character = socket_read($this->socket, 1);
    if ($character === '' || $character === false) {
        break;  // We're done reading
    }
    $message .= $character;
}

Looks reasonable, right?  After all, the documentation says that socket_read() will return the number of characters requested (in this case one), or false on error, or the empty string if there's no more data.  So this seems like it should work just fine. 

Well...not so much.

The problem is with the last read.  It turns out that the documentation is wrong - socket_read() doesn't return the empty string when there's no more data.  In fact, I couldn't get it to return an empty string ever.  What actually happens is that it goes along happily until it exhausts the available data, and then it waits for more data.  So the last call just hangs until it reaches a timeout that's set on the connection (in our case, it was configured to 1 second) and then returns false.

So because we were relying on that "empty string on empty buffer" behavior to detect the end of input, calling that method always resulted in a one-second hang.  This was fairly easily fixed by just reading the data in much larger chunks and checking how much was actually returned to determine if we needed another read call.  But that's not the point.  The point is that we relied on what was in the documentation, and it was just totally wrong!

And it's not like this is the first time I've been bitten by the PHP docs.  Historically, PHP has been very bad about documenting edge cases.  For example, what happens if a particular parameter is null?  What's the exact behavior if the parameters do not match the expected preconditions?  Or what about that "flags" parameter that a bunch of functions take?  Sometimes the available flags are well documented, but sometimes it's just an opaque one-line description that doesn't really tell you what the flag actually does.  It's a crap shoot.

To be fair, the PHP documentation is not the worst I've ever seen.  Not even close.  And it really is very good about providing helpful examples.  It's just that it errs on the side of being light on details, and software is details.

I'm not entirely sure why.  But it's a pain in the butt.

This has come up a couple of times in working on one of our projects for work.  This particular one is an internal command-line application written in PHP and using the command-line components from the Symfony framework.  I won't get into the details, but the point is that it glues together the various steps required to configure and run certain things on Linux-based servers.  So to test it, I have to put the code on my test server and make sure it works there.

The problem that I ran into today was that I tried to add a new command class to the application and it barfed all over itself.  The Symfony DI container complained that it couldn't find a certain class name in a certain file. The PSR-4 autoloader requires that the class name and namespace match the filesystem path, so usually this indicates a typo in one of those.  But in this case, everything was fine.  The app worked fine on my laptop, and if I deleted the new command, it worked again.

Well, it turns out that running composer du -o fixed it.  I suspect, based on the composer documentation, that the issue was that the class map was being cached by the opcache.  The Symfony cache was empty, so that's about the only one left.  Unfortunately, this is pretty opaque when it comes to trouble-shooting.  I would have expected it to fall back to reading the filesystem, but apparently there's more to it than that.  Perhaps it's related to how Symfony collects the commands - I haven't put in the time to investigate it.

But in any case, that's something to look out for.  Gotta love those weird errors that give you absolutely no indication of the solution.

I was working for deviantART.com at the time.  That was a very interesting experience.  For one, it was y first time working 100% remote, and with a 100% distributed team, no less.  We had people in eastern and western Europe, and from the east coast to the west coast of the US.  It was also a big, high-traffic site.  I mean, not Google or Facebook, but I believe it was in the top 100 sites on the web at the time, according to Alexa or Quantcast (depending on how you count "top 100").

It also had a lot of custom tech.  My experience up to that point had mostly been with fairly vanilla stuff, stitching together a bunch of off-the-shelf components.  But deviantART was old enough and big enough that a lot of the off-the-shelf tools we would use today weren't wide-spread yet, so they had to roll their own.  For instance, they had a system to do traits in PHP before that was actually a feature of PHP (it involved code generation, in case you were wondering).

Today's post from the archives is about my first dealings with one such custom tool.  It nicely illustrates one of the pitfalls of custom tooling - it's usually not well documented, so spotting and resolving issues with it isn't always straight-forward.  This was a case of finding that out the hard way.  Enjoy!

Lesson learned this week: object serialization in PHP uses more space than you think.

I had a fun problem recently.  And by "fun", I mean "WTF?!?"  

I got assigned a data migration task at work last week.  It wasn't a particularly big deal - we had two locations where user's names were being stored and my task was to consolidate them.  I'd already updated the UI code to read and save the values, so it was just a matter of running a data migration job.

Now, at dA we have this handy tool that we call Distributor.  We use it mainly for data migration and database cleanup tasks.  Basically, it just crawls all the rows of a database table in chunks and passes the rows through a PHP function.  We have many tables that contain tens or hundreds of millions of rows, so it's important that data migrations can be done gradually - trying to do it all at once would hammer the database too hard and cause problems.  Distributor allows us to set how big each chunk is and configure the frequency and concurrency level of chunk processing.

There's three parts to Distributor that come into play here: the distributor runner (i.e. the tool itself), the "recipe" which determines what table to crawl, and the "job" function which performs the actual migration/cleanup/whatever.  We seldom have to think about the runner, since it's pretty stable, so I was concentrating on the recipe and the job.

Well, things were going well.  I wrote my recipe, which scanned the old_name table and returned rows containing the userid and the name itself.  And then I wrote my migration job, which updated the new_name table.  (I'm fabricating those table names, in case you didn't already figure that out.)  Distributor includes a "counter" feature that allows us to trigger logging messages in jobs and totals up the number of times they're triggered.  We typically make liberal use of these, logging all possible code paths, as it makes debugging easier.It seemed pretty straight-forward and it passed through code review without any complaints.

So I ran my distributor job.  The old_name table had about 22 million rows in it, so at a moderate chunk size of 200 rows, I figured it would take a couple of days.  When I checked back a day or two later, the distributor runner was reporting that the job was only 4% complete.  But when I looked at my logging counters, they reported that the job had processed 28 million rows.  WTF?!?  

Needless to say, head-scratching, testing, and debugging followed.  The short version is that the runner was restarting the job at random.  Of course, that doesn't reset the counters, so I'd actually processed 28 million rows, but most of them were repeats.  

So why was the runner resetting itself?  Well, I traced that back to a database column that's used by the runner.  It turns out that the current status of a job, including the last chunk of data returned by the crawler recipe, is stored in the database as a serialized string.  The reason the crawler was restarting was because PHP's unserialize() function was erroring out when trying to deserialize that string.  And it seems that the reason it was failing was that the string was being truncated - it was overflowing the database column!

The source of the problem appeared to be the fact that my crawler recipe was returning both a userid and the actual name.  You see, we typically write crawlers to just return a list of numeric IDs.  We can look up the other stuff st run-time.  Well, that extra data was just enough to overflow the column on certain records.  That's what I get for trying to save a database lookup!

This improvement is, I'm slightly sad to say, pretty boring.  It basically involves fixing a "bug" that's really an artifact of some very old design choices.  These choices led to the existing implementation behaving in unexpected ways when the workflow changed.

The Problem

Originally LnBlog was pretty basic and written almost entirely in HTML and PHP, i.e. there was no JavaScript to speak of.  You wrote posts either in raw HTML in a text area box, using "auto-markup", which just automatically linkified things, or using "LBCode", which is my own bastardized version of the BBCode markup that used to be popular on web forums.  I had implemented some plugins to support WYSIWYG post editors, but I didn't really use them and they didn't get much love.

Well, I eventually got tired of writing in LBCode and switched to composing all my posts using the TinyMCE plugin.  That is now the standard way to compose your posts in LnBlog.  The problem is that the existing workflow wasn't really designed for WYSIWYG composition.

In the old model, the idea was that you could compose your entire post on the entry editing page, hit "publish", and it would all be submitted to the server in one go.  There's also a "review" button which renders your post as it would appear when published and a "save draft" button to save your work for later.  These also assume that submitting the post is an all-or-nothing operation.  So if you got part way done with your post and decided you didn't like it, you could just leave the page and nothing would be saved to the server.

At this point it is also worth noting how LnBlog stores its data.  Everything is file-based and entries are self-contained.  That means that each entry has a directory and that directory contains all the post data, comments, and uploaded files that are belong to that entry.

What's the problem with this?  Well, to have meaningful WYSIWYG editing, you need to be able to do things like upload a file and then be able to see it in the post editor.  In the old workflow, you'd have to write your post, insert an image tag with the file name of your picture (which would not render), add your picture as an upload, save the entry (either by saving the draft or using the "preview", which would have trigger a save if you had uploads), and then go back to editing your post.  This was an unacceptably workflow clunky.

On top of this, there was a further problem.  Even after you previewed your post, it still wouldn't render correctly in the WYSIWYG editor.  That's because the relative URLs were inconsistent.  The uploaded files got stored in a special, segregated draft directory, but the post editor page itself was not relative to that directory, so TinyMCE didn't have the right path to render it.  And you can't use an absolute URL because the URL will change after the post is published.

So there were two semi-related tasks to fix this.  The first was to introduce a better upload mechanism.  The old one was just a regular <input type="file"> box, which worked but wasn't especially user-friendly.  The second one was to fix things such that TinyMCE could consistently render the correct URL for any files we uploaded.

The solution - Design

The actual solution to this problem was not so much in the code as it was in changing the design.  The first part was simple: fix the clunky old upload process by introducing a more modern JavaScript widget to do the uploads.  So after looking at some alternatives, I decided to implement Dropzone.js as the standard upload mechanism.

The second part involved changing the workflow for writing and publishing posts.  The result was a somewhat simpler and more consistent workflow that reduces the number of branches in the code.  In the old workflow, you had the following possible cases when submitting a post to the server:

1. New post being published (nothing saved yet).
2. New post being saved as a draft (nothing saved yet).
3. Existing draft post being published.
4. Existing draft post being saved.
5. New (not yet saved) post being previewed with attached files.
6. Existing draft post being previewed with attached files.

This is kind of a lot of cases.  Too many, in fact.  Publishing and saving were slightly different depending on whether or not the entry already existed, and then there were the preview cases.  These were necessary because extra processing was required when an entry was previewed with new attachments because, well, if you attached an image, you'd want to see it.  So this complexity was a minor problem in and of itself.

So the solution was to change the workflow such that all of these are no longer special cases.  I did this by simply issuing the decree that all draft entries shall always already exist.  In other words, just create a new draft when we first open the new post editor.  This does two things for us:

1. It allows us to solve the "relative URL" problem because now we can make the draft editing URL always relative to the draft storage directory.
2. It eliminates some of those special cases.  If the draft always exists, then "publish new post" and "publish existing draft" are effectively the same operation.  When combined with the modern upload widget, this also eliminates the need for the special "preview" cases.

The implementation - Results

I won't get into the actual implementation details of these tasks because, frankly, they're not very interesting.  There aren't any good lessons or generalizations to take from the code - it's mostly just adapting the ideosyncratic stuff that was already there.

The implementation was also small and went fairly smoothly.  The upload widget was actually the hard part - there were a bunch of minor issues in the process of integrating that.  There were some issues with the other part as well, but less serious.  Much of it was just integration issues that weren't necessarily expected and would have been hard to foresee.  You know, the kind of thing you expect from legacy code.  Here's some stats from Process Dashboard:

As you can see, my estimates here were not great.  The upload part involved more trial and error with Dropzone.js than I had expected and ended up with more bugs.  The draft workflow change went better, but I ended up spending more time on the design than I initially anticipated.  However, these tasks both had a lot of unknowns, so I didn't really expect the estimates to be that accurate.

Take Away

The interesting thing about this project was not so much what needed to be done but why it needed to be done. 

Editing posts is obvious a fundamental function of a blog, and it's one that I originally wrote way back in 2005.  It's worth remembering that the web was a very different place back then.  Internet Explorer was still the leading web browser; PHP 5 was still brand new; it wasn't yet considered "safe" to just use JavaScript for everything (because, hey, people might not have JavaScript enabled); internet speeds were still pretty slow; and browsing on mobile devices was just starting to become feasible.  In that world, a lot of the design decisions I made at the time seemed pretty reasonable.

But, of course, the web evolved.  The modern web makes it much easier for the file upload workflow to be asynchronous, which offers a much nicer user experience.  By ditching some of the biases and assumptions of the old post editor, I was more easily able to update the interface.

One of the interesting things to note here is that changing the post editing workflow was easier than the alternatives.  Keeping the old workflow was by no means impossible.  I kicked around several ideas that didn't involve changing it.  However, most of those had other limitations or complications and I eventually decided that they would ultimately be more work.  

This is something that comes up with some regularity when working with an older code-base.  It often happens that the assumptions baked into the architecture don't age well as the world around the application progresses.  Thus, when you need to finally "fix" that aspect of the app, you end up having to do a bit of cost-benefit analysis.  Is it better to re-vamp this part of the application?  Or should you shim in the new features in a kinda-hacky-but-it-works sort of way?

While as developers, our first instinct is usually to do the "real" fix and replace the old thing, the "correct" answer is seldom so straight-forward.  In this case, the "real" fix was relatively small and straight-forward.  But in other cases, the old assumptions are smeared through the entire application and trying to remove them becomes a nightmare.  It might take weeks or months to make a relatively simple change, and then weeks or months after that to deal with all the unforeseen fallout of that change.  Is that worth the effort?  It probably depends on what the "real" fix buys you.

I had a project at work once that was a great example of that.  On the surface, the request was a simple "I want to be able to update this field", where the field in question was data that was generally but not necessarily static. In most systems, this would be as simple as adding a UI to edit that field and having it update the datastore.  But in this case, that field was used internally as the unique identifier and was used that way across a number of different systems.  So this assumption was everywhere.  Everybody knew this was a terrible design, but it had been that way for a decade and was such a huge pain to fix that we had been putting it off for years.  When we finally bit the bullet and did it right, unraveling the baked-in assumptions about this piece of data took an entire team over a month.  At an extremely conservative estimate, that's well over $25,000 to fix "make this field updatable".  That's a pretty hefty price tag for something that seems so trivial.

The point is, old applications tend to have lots of weird, esoteric design decisions and implementation-specific issues that constrain them.  Sometimes removing these constraints is simple and straight-forward.  Sometimes it's not.  And without full context, it's often hard to tell when one it will be.  So whenever possible, try to have pity on the future maintenance programmer who will be working on your system and anticipate those kind of issues.  After all, that programmer might be you.

Feature Addition: Webmention

For this second step, I didn't so much refactor as add a feature.  This particular feature has been on my list for a while and I figured it was finally time to implement it.  That feature is webmention support.  This is the newer generation of blog notification, similar to Trackback (which I don't think anyone uses anymore) and Pingback.  So, basically, it's just a way of notifying another blog that you linked to them and vice versa.  LnBlog already supported the two older versions, so I thought it made sense to add the new one.

One of the nice things about Webmention is that it actually has a formal specification that's published as a W3C recommendation.  So unlike some of the older "standards" that were around when I first implemented LnBlog, this one is actually official, well structured, and well thought out.  So that makes things slightly easier.

Unlike the last post, I didn't follow any formal process or do any time tracking for this addition.  In retrospect I kind of wish I had, but this work was very in and out in terms of consistency and I didn't think about tracking until it was too late to matter.  Nevertheless, I'll try to break down some of my process and results.

Step 1: Analysis

The first step, naturally, was analyzing the work to be done, i.e. reading the spec.  The webmention protocol isn't particularly complicated, but like all specification documents it looks much more so when you put all the edge cases and optional portions together.  

I actually looked at the spec several times before deciding to actually implement it.  Since my time for this project is limited and only available sporadically, I was a little intimidated by the unexpected length of the spec.  When you have maybe an hour a day to work on a piece of code, it's difficult to get into the any kind of flow state, so large changes that require extended concentration are pretty much off the table.

So how do we address this?  How do you build something when you don't have enough time to get the whole thing in your head at once?

Step 2: Design

Answer: you document it.  You figure out a piece and write down what you figured out.  Then the next time you're able to work on it, you can read that and pick up where you left off.  Some people call this "design".

I ended up reading through the spec over several days and eventually putting together UML diagrams to help me understand the flow.  There were two flows, sending and receiving, so I made one diagram for each, which spelled out the various validations and error conditions that were described in the spec.

That was really all I needed as far as design for implementing the webmention protocol.  It's pretty straight-forward and I made the diagrams detailed enough that I could work directly from them.  The only real consideration left was where to fit the webmention implementation into the code.

My initial thought was to model a webmention as a new class, i.e. to have a Webmention class to complement the currently existing TrackBack and Pingback classes.  In fact, this seemed like the obvious implementation given the code I was working with.  However, when I started to look at it, it became clear that the only real difference between Pingbacks and Webmentions is the communication protocol.  It's the same data and roughly the same workflow and use-case.  It's just that Pingback goes over XML-RPC and Webmention uses plain-old HTTP form posting.  It didn't really make sense to have a different object class for what is essentially the same thing, so I ended up just re-using the existing Pingback class and just adding a "webmention" flag for reference.

Step 3: Implementation

One of the nice things about having a clear spec is that it makes it really easy to do test-driven development because the spec practically writes half your test cases for you.  Of course, there are always additional things to consider and test for, but it still makes things simpler.

The big challenge was really how to fit webmentions into the existing application structure.  As I mentioned above, I'd already reached the conclusion that creating a new domain object for the was a waste of time.  But what about the rest of it?  Where should the logic for sending them go?  Or receiving?  And how should sending webmentions play with sending pingbacks?

The first point of reference was the pingback implementation.  The old pingback implementation for sending pingbacks lived directly in the domain classes.  So a blog entry would scan itself for links, create a pingback object for each, and then ask the pingback if its URI supported pingbacks, and then the entry would sent the pingback request.  (Yes, this is confusing.  No, I don't remember why I wrote it that way.)  As for receiving pingbacks, that lived entirely in the XML-RPC endpoint.  Obviously none of this was a good example to imitate.

The most obvious solution here was to encapsulate this stuff in its own class, so I created a SocialWebClient class to do that.  Since pingback and webmention are so similar, it made sense to just have one class to handle both of them.  After all, the only real difference in sending them was the message protocol.  The SocialWebClient has a single method, sendReplies(), which takes an entry, scans its links and for each detects if the URI supports pingback or webmention and sends the appropriate one (or a webmention if it supports both).  Similarly, I created a SocialWebServer class for receiving webmentions with an addWebmention() method that is called by an endpoint to save incoming mentions.  I had originally hoped to roll the pingback implementation into that as well, but it was slightly inconvenient with the use of XML-RPC, so I ended up pushing that off until later.

Results

As I mentioned, I didn't track the amount of time I spent on this task.  However, I can retroactively calculate how much code was involved.  Here's the lines-of-code summary as reported by Process Dashboard:

For those who aren't familiar, the "base" value is the lines of code in the affected files before the changes, while the "total" it the total number of lines in affected files after the changes.  The magic number here is "Added & Modified", which is essentially the "new" code.  So all in all, I wrote about a thousand lines for a net increase 700 lines.

Most of this was in the new files, as reported by Process Dashboard below.  I'll spare you the 31 files that contained assorted lesser changes (many related to fixing unrelated issues) since none of them had more even 100 lines changed.  

It's helpful to note that of the 717 lines added here, slightly less than half (315 lines) is unit test code.  Since I was trying to do test-driven development, this is to be expected - the rule of thumb is "write at least as much test code as production code".  That leaves the meat of the implementation at around 400 lines.  And of those 400 lines, most of it is actually refactoring.

As I noted above, the Pingback and Webmention protocols are quite similar, differing mostly in the transport protocol.  The algorithms for sending and receiving them are practically identical.  So most of that work was in generalizing the existing implementation to work for both Pingback and Webmention.  This meant pulling things out into new classes and adjusting them to be easily testable.  Not exciting stuff, but more work than you might think.

So the main take-away from this project was: don't underestimate how hard it can be to work with legacy code.  Once I figured out that the implementation of Webmention would closely mirror what I already had for Pingback, this task should have been really short and simple.  But 700 lines isn't really that short or simple.  Bringing old code up to snuff can take a surprising amount of effort.  But if you've worked on a large, brown-field code-base, you probably already know that.

Apparently you can just do composer global init ; composer global require phpunit/phpunit and get PHPUnit installed in your home directory rather than in a project directory, where you can add it to your path and use it anywhere.  It works just like with installing to a project - the init creates a composer.json and the require adds packages to it.  On Linux, I believe this stuff gets stored under ~/.composer/, whereas on Windows, they end up under ~\AppData\Roaming\Composer\.

That's it.  Nothing earth-shattering here.  Just a handy little trick for things like code analyzers or other generic tools that you might not care about adding to your project's composer setup (maybe you only use them occasionally and have no need to integrate them into your CI build).  I didn't know about it, so I figured I'd pass it on.