LinLog

Author's note: It's officially the holiday season and I'm feeling lazy. So guess what - it's another episode of "From the Archives"! That's right, it's time for more of the series where I trot out something that's been sitting in my drafts folder for ten years because I can't muster the energy to write something new.

This article is from way back on March 18, 2007. It actually appears to be finished, so I'm not sure why I never published it. Perhaps I was filled with crippling self doubt that my analysis was actually stupid and people would make fun of me, so I never hit the publish button. That was actually a thing I did for many years. These days, I'm more casual about it. One of the benefits of getting older is that you're more experienced and able to have a more objective view of the quality of your work. Another is that you have a wider perspective on what really matters in life. Another way to put that is "you don't care as much what other people think."

Anyway, this article is about my attempts at implementing the various blogging APIs in LnBlog. This included the Blogger, MetaWeblog, and MoveableType APIs. I guess that seemed like a good idea at the time. In retrospect, it was a nice educational exercise, but not especially useful. I mean, the idea of a generic API that third-party clients can use to post to your blog is great. But in practice, it doesn't seem like that's a thing that many people actually need. I certainly never found a good use-case for it. Maybe that's why the APIs never really got fleshed out.

But that's enough out of present-day me. Let's hear from 2007 me. Enjoy!

Last month, I finally got around to doing some actual testing on the MetaWeblog and Blogger API implementations for LnBlog. By that, I mean that rather than testing it with my own code, I actually installed a few free blogging clients. I learned a few interesting lessons from this.

The Clients

I tested four blogging clients. The first is Deepest Sender 0.7.9, a Firefox extension that supports LiveJournal, Blogger, Wordpress, MSN Spaces, and generic MetaWeblog blogs. The second is KBlogger 0.6.2, a simple KDE panel applet that supports the MetaWeblog and Blogger APIs. Third is QTM 0.4.0, a C++/Qt4 application that support the Blogger, MetaWeblog, and MovableType APIs. Last is BloGTK 1.1, a Python/GTK+ application that also supports Blogger, MW, and MT.

My results were mixed. KBlogger worked basically as advertised. In fact, it's the only one of the clients that seemed to understand the APIs in the same way that I do. The only problem is that it's a bit short on features.

BloGTK seemed to work pretty well. However, it worked best when set to use the MoveableType API. When using the MetaWeblog API, I had problems editing posts. It also has a few weird little bugs, such as things getting messed up when switching accounts.

While it has a nice interface, QTM simply would not work with LnBlog. For the record, this is not my fault but rather due to the fact that this version of QTM did not correctly implement the APIs. When sending calls to the server, it sent blog IDs, post IDs, category IDs, etc. as integers, whereas the specification calls for them to be strings. While the difference may be academic for some servers, LnBlog really does use strings as IDs, so the requests raise errors in the XML-RPC library. (Note: this seems to have been corrected in CVS.)

And as for Deepest Sender, I just can't get it to work as advertised. I don't know why. It can post entries, but editing them results in a hung edit window, the "active blog" box is shrunk down to an unrecongizable control, and I have yet to even see a category selection box.

Server Problems

The first problem I encountered with LnBlog's implementation of the Blogger 1.0 and MetaWeblog APIs was my silly assumption that, just because they are two separate API specifications, I could implement them separately. And so, that's exactly what I did: I wrote one PHP script to implement the Blogger 1.0 API and a different one to implement the MetaWeblog API.

Oh, what a fool I was!

While that attitude made perfect sense when looking just at the specs, it just doesn't work that way in practice. Of the four clients I tested, KBlogger was the only one that worked when the MetaWeblog server didn't implement the Blogger 1.0 API at the same URI. The others all blithely assumed that the same URI would implement the Blogger, MetaWeblog, and MovableType APIs. I guess few people even stopped to consider that a server might have independent implementations of the different APIs. Or perhaps it's just that KBlogger is designed to support clients that only understand Blogger 1.0 while the others assume MetWeblog support. It's hard to tell.

Placing the blame

After going back to look at the specs, I believe much of the blame for this situation rests with the MetaWeblog API specification itself. The problem is that it's just a bad specification. In fact, it's really more a sketch of a specification than an actual spec. It's too vauge, too confusing, and leaves too much open to interpretation.

For instance, take the metaWeblog.getCategories method. According to the specification, this method returns a struct, with each member being a struct with the description, HTML URL, and RSS URL for each category.

For non-programmers, "struct" is short for "structure," and is simply set of key/value pairs. In this case, having member structs with keys for description and URLs makes perfect sense.

However, putting all of these in a struct doesn't make sense. The spec says that the category structs are to be returned in a struct, but says nothing about the key names of this container struct. But the entire point of having a struct is to associate key names with values. A struct with no particular key names is meaningless. It's like writing a book where the index lists words, but not page numbers - if you can't pick the word (key) you want and go straight to the corresponding page (value), then the entire exercise is pointless.

Another shortcoming of the API is that it does not clearly specify a way to identify blog posts. For example, the API includes the metaWeblog.editPost and metaWeblog.getPost methods, both of which take a post ID. It also includes a metaWeblog.getRecentPosts method to get an array of the most recent posts for a blog. You would think that you could call getRecentPosts, let the user pick a post, edit it, and then call editPost to commit the changes to the server. But you can't.

That's because the API does not specify how to get the post ID. The metaWeblog.getPost and metaWeblog.getRecentPosts methods return a struct and an array of structs respectively, and the spec states that the members of these post structs are the members of RSS items. But there is no mention of where the post ID comes in. RSS certainly has no concept of a post ID, so it's not clear which member should be used for that purpose. Presumably, this is why the MovableType extensions to MetaWeblog include a postId field in the post structs.

Of course, RSS does provide a GUID (Globally Uniquie Identifier) field, which seems a natural fit for the post ID. The problem is that the RSS spec does not require that this field be present. It could also be argued that the GUID has a meaning distinct from a blog post ID. But either way, if the MetaWeblog spec meant that the GUID should be the post ID, then it should have said so.

Judging from the MetaWeblog spec, the only place clients can count on getting a postID is from the return value of metaWeblog.newPost. That's fine if the client can assume it is making all posts to a blog, but it is insufficient if there is also, say, a web interface. If your blogging client can only edit posts it created, you've just cut its usefulness in half.

Missing Links

The MetaWeblog API depends heavily on the Blogger 1.0 API. By itself, it is missing too much to be truly useful for the development of rich blogging clients. If nothing else, this is clear from the absence of something resembling blogger.getUsersBlogs.

Actually, that's not entirely fair. There was an RFC to add the Blogger methods to MetaWeblog, so the spec has been amended to correct this shortcoming. Or has it? I actually only learned about this by reading the code for the WordPress MetaWeblog implementation. The "official" MetaWeblog spec doesn't actually mention this or contain a link to the new RFC. That seems rather odd considering that the spec does contain notes regarding other updates. So has the spec been ammended, superceded, or was this just a "Request For Comment" that was never actually adopted?

Bottom Line for Implementers

So what does all this mean for those enterprising individuals who want to try their hand at writing blogging software? It means you've got an up-hill battle.

Don't get me wrong - it's not that implementing the various specifications is difficult. The APIs are actually pretty simple. The problem is that you can't trust them.

For starters, if you want to write a server that is compatible with existing rich blogging clients, you will have to implement the Blogger 1.0, MetaWeblog, and MovableType APIs, and you will have to do it all at the same URI. This isn't really a problem, so much as an inconvenience, as you can't simply work from a single specification at a time, but have to jump back and forth between three of them just to get a workable MetaWeblog implementation.

If you're writing a client, things are just as annoying. As previously mentioned, there's the post ID problem to deal with. Handling that is not difficult, but you have to rely on the good will of the server to send you a sensible post struct, since it is not required to.

If you want to support MovableType, there's also their brain-damaged category handling to deal with. Rather than using MetaWeblog categories, MT has separate mt.getPostCategories and mt.setPostCategories methods, which deal with category IDs rather than textual categories. Again, this is not hard to deal with, but it means you have to implement category handling twice - once for MT, and once for servers that use MW categories. But on the up side, at least MT gives you an explicit postId field.

Conclusion

All in all, the old blogging APIs suck. They're imprecise, lacking in features, and tend not to be so platform-agnostic. I think they can be best described as "just barely adequate."

I have yet to look at the Atom API. I'm hoping it will turn out to be better, but I'm not going to hold out a lot of hope. At the very least, I suppose it can't be any worse than the old APIs.

Trivia entry for the week: Yesterday this blog apparently celebrated its 15th year of existence.  It looks like the first entry was February 18th, 2005.

I say "looks like" because the publication date actually says December 14th, 2006.  But that's almost certainly an artifact of the way the first versions of LnBlog stored date metadata - they didn't.  The date at which an entry was "published" depended on the file path.  Since storage is file-based, each entry goes in a folder like entries/2005/02/18_1200/, which corresponds to the date and time that I hit the "publish" button.  That part of the system actually hasn't changed in the last 15 years, but what has changed is that the publication date is now stored in the entry metadata, along with everything else.  At some point (presumably in 2006) when I added the date to the metadata, there was apparently a bug in the code such that it didn't properly fetch the publication date from the path.  So a bunch of early entries actually show the wrong publication date.  I should probably fix that at some point.

Despite the fact that I have few to no regular readers, I'm slightly proud of myself for sticking with this blog this long.  My level of posting consistency has varied radically over the years, but I've always come back to it, for at least a few posts a year.  There's something to be said for that, though I'm not sure exactly what.

I'm also slightly proud of myself for keeping this blog more-or-less working that long.  It's powered by LnBlog, which is the first real PHP program I wrote, starting in late 2004 to early 2005.  I wrote the entire thing myself, from scratch with no framework, starting in PHP 4 and now evolving into PHP 7.  And it's actually got a fairly decent feature set, so I consider it a minor accomplishment that it's still working.

For the last couple of months, I've been trying to do the weekly post thing, taking advantage of the "scheduled publication" feature to write things in advance and automatically spread out the publication.  This week is actually the first one in a couple of months that hasn't had a Saturday post (mostly because I was getting ready for vacation and didn't get around to writing anything).  We'll see how long we can keep that up.  But hopefully I'll be able to keep some level of consistency going for another 15 years.

It's time for the third, slightly shorter, installment of my ongoing series on refactoring my blogging software.  In the first part, I discussed reworking how post publication was done and in the second part I talked about reworking things to add Webmention support.  This time, we're going to talk about two mini-projects to improve the UI for editing posts.

This improvement is, I'm slightly sad to say, pretty boring.  It basically involves fixing a "bug" that's really an artifact of some very old design choices.  These choices led to the existing implementation behaving in unexpected ways when the workflow changed.

The Problem

Originally LnBlog was pretty basic and written almost entirely in HTML and PHP, i.e. there was no JavaScript to speak of.  You wrote posts either in raw HTML in a text area box, using "auto-markup", which just automatically linkified things, or using "LBCode", which is my own bastardized version of the BBCode markup that used to be popular on web forums.  I had implemented some plugins to support WYSIWYG post editors, but I didn't really use them and they didn't get much love.

Well, I eventually got tired of writing in LBCode and switched to composing all my posts using the TinyMCE plugin.  That is now the standard way to compose your posts in LnBlog.  The problem is that the existing workflow wasn't really designed for WYSIWYG composition.

In the old model, the idea was that you could compose your entire post on the entry editing page, hit "publish", and it would all be submitted to the server in one go.  There's also a "review" button which renders your post as it would appear when published and a "save draft" button to save your work for later.  These also assume that submitting the post is an all-or-nothing operation.  So if you got part way done with your post and decided you didn't like it, you could just leave the page and nothing would be saved to the server.

At this point it is also worth noting how LnBlog stores its data.  Everything is file-based and entries are self-contained.  That means that each entry has a directory and that directory contains all the post data, comments, and uploaded files that are belong to that entry.

What's the problem with this?  Well, to have meaningful WYSIWYG editing, you need to be able to do things like upload a file and then be able to see it in the post editor.  In the old workflow, you'd have to write your post, insert an image tag with the file name of your picture (which would not render), add your picture as an upload, save the entry (either by saving the draft or using the "preview", which would have trigger a save if you had uploads), and then go back to editing your post.  This was an unacceptably workflow clunky.

On top of this, there was a further problem.  Even after you previewed your post, it still wouldn't render correctly in the WYSIWYG editor.  That's because the relative URLs were inconsistent.  The uploaded files got stored in a special, segregated draft directory, but the post editor page itself was not relative to that directory, so TinyMCE didn't have the right path to render it.  And you can't use an absolute URL because the URL will change after the post is published.

So there were two semi-related tasks to fix this.  The first was to introduce a better upload mechanism.  The old one was just a regular <input type="file"> box, which worked but wasn't especially user-friendly.  The second one was to fix things such that TinyMCE could consistently render the correct URL for any files we uploaded.

The solution - Design

The actual solution to this problem was not so much in the code as it was in changing the design.  The first part was simple: fix the clunky old upload process by introducing a more modern JavaScript widget to do the uploads.  So after looking at some alternatives, I decided to implement Dropzone.js as the standard upload mechanism.

The second part involved changing the workflow for writing and publishing posts.  The result was a somewhat simpler and more consistent workflow that reduces the number of branches in the code.  In the old workflow, you had the following possible cases when submitting a post to the server:

1. New post being published (nothing saved yet).
2. New post being saved as a draft (nothing saved yet).
3. Existing draft post being published.
4. Existing draft post being saved.
5. New (not yet saved) post being previewed with attached files.
6. Existing draft post being previewed with attached files.

This is kind of a lot of cases.  Too many, in fact.  Publishing and saving were slightly different depending on whether or not the entry already existed, and then there were the preview cases.  These were necessary because extra processing was required when an entry was previewed with new attachments because, well, if you attached an image, you'd want to see it.  So this complexity was a minor problem in and of itself.

So the solution was to change the workflow such that all of these are no longer special cases.  I did this by simply issuing the decree that all draft entries shall always already exist.  In other words, just create a new draft when we first open the new post editor.  This does two things for us:

1. It allows us to solve the "relative URL" problem because now we can make the draft editing URL always relative to the draft storage directory.
2. It eliminates some of those special cases.  If the draft always exists, then "publish new post" and "publish existing draft" are effectively the same operation.  When combined with the modern upload widget, this also eliminates the need for the special "preview" cases.

The implementation - Results

I won't get into the actual implementation details of these tasks because, frankly, they're not very interesting.  There aren't any good lessons or generalizations to take from the code - it's mostly just adapting the ideosyncratic stuff that was already there.

The implementation was also small and went fairly smoothly.  The upload widget was actually the hard part - there were a bunch of minor issues in the process of integrating that.  There were some issues with the other part as well, but less serious.  Much of it was just integration issues that weren't necessarily expected and would have been hard to foresee.  You know, the kind of thing you expect from legacy code.  Here's some stats from Process Dashboard:

As you can see, my estimates here were not great.  The upload part involved more trial and error with Dropzone.js than I had expected and ended up with more bugs.  The draft workflow change went better, but I ended up spending more time on the design than I initially anticipated.  However, these tasks both had a lot of unknowns, so I didn't really expect the estimates to be that accurate.

Take Away

The interesting thing about this project was not so much what needed to be done but why it needed to be done. 

Editing posts is obvious a fundamental function of a blog, and it's one that I originally wrote way back in 2005.  It's worth remembering that the web was a very different place back then.  Internet Explorer was still the leading web browser; PHP 5 was still brand new; it wasn't yet considered "safe" to just use JavaScript for everything (because, hey, people might not have JavaScript enabled); internet speeds were still pretty slow; and browsing on mobile devices was just starting to become feasible.  In that world, a lot of the design decisions I made at the time seemed pretty reasonable.

But, of course, the web evolved.  The modern web makes it much easier for the file upload workflow to be asynchronous, which offers a much nicer user experience.  By ditching some of the biases and assumptions of the old post editor, I was more easily able to update the interface.

One of the interesting things to note here is that changing the post editing workflow was easier than the alternatives.  Keeping the old workflow was by no means impossible.  I kicked around several ideas that didn't involve changing it.  However, most of those had other limitations or complications and I eventually decided that they would ultimately be more work.  

This is something that comes up with some regularity when working with an older code-base.  It often happens that the assumptions baked into the architecture don't age well as the world around the application progresses.  Thus, when you need to finally "fix" that aspect of the app, you end up having to do a bit of cost-benefit analysis.  Is it better to re-vamp this part of the application?  Or should you shim in the new features in a kinda-hacky-but-it-works sort of way?

While as developers, our first instinct is usually to do the "real" fix and replace the old thing, the "correct" answer is seldom so straight-forward.  In this case, the "real" fix was relatively small and straight-forward.  But in other cases, the old assumptions are smeared through the entire application and trying to remove them becomes a nightmare.  It might take weeks or months to make a relatively simple change, and then weeks or months after that to deal with all the unforeseen fallout of that change.  Is that worth the effort?  It probably depends on what the "real" fix buys you.

I had a project at work once that was a great example of that.  On the surface, the request was a simple "I want to be able to update this field", where the field in question was data that was generally but not necessarily static. In most systems, this would be as simple as adding a UI to edit that field and having it update the datastore.  But in this case, that field was used internally as the unique identifier and was used that way across a number of different systems.  So this assumption was everywhere.  Everybody knew this was a terrible design, but it had been that way for a decade and was such a huge pain to fix that we had been putting it off for years.  When we finally bit the bullet and did it right, unraveling the baked-in assumptions about this piece of data took an entire team over a month.  At an extremely conservative estimate, that's well over $25,000 to fix "make this field updatable".  That's a pretty hefty price tag for something that seems so trivial.

The point is, old applications tend to have lots of weird, esoteric design decisions and implementation-specific issues that constrain them.  Sometimes removing these constraints is simple and straight-forward.  Sometimes it's not.  And without full context, it's often hard to tell when one it will be.  So whenever possible, try to have pity on the future maintenance programmer who will be working on your system and anticipate those kind of issues.  After all, that programmer might be you.

About a year and a half ago, I wrote an entry about the first step in my refactoring of LnBlog.  Well, that's still a thing that I work on from time to time, so I thought I might as well write a post on the latest round of changes.  As you've probably figured out, progress on this particular project is, of necessity, slow and extremely irregular, but that's an interesting challenge in and of itself.

Feature Addition: Webmention

For this second step, I didn't so much refactor as add a feature.  This particular feature has been on my list for a while and I figured it was finally time to implement it.  That feature is webmention support.  This is the newer generation of blog notification, similar to Trackback (which I don't think anyone uses anymore) and Pingback.  So, basically, it's just a way of notifying another blog that you linked to them and vice versa.  LnBlog already supported the two older versions, so I thought it made sense to add the new one.

One of the nice things about Webmention is that it actually has a formal specification that's published as a W3C recommendation.  So unlike some of the older "standards" that were around when I first implemented LnBlog, this one is actually official, well structured, and well thought out.  So that makes things slightly easier.

Unlike the last post, I didn't follow any formal process or do any time tracking for this addition.  In retrospect I kind of wish I had, but this work was very in and out in terms of consistency and I didn't think about tracking until it was too late to matter.  Nevertheless, I'll try to break down some of my process and results.

Step 1: Analysis

The first step, naturally, was analyzing the work to be done, i.e. reading the spec.  The webmention protocol isn't particularly complicated, but like all specification documents it looks much more so when you put all the edge cases and optional portions together.  

I actually looked at the spec several times before deciding to actually implement it.  Since my time for this project is limited and only available sporadically, I was a little intimidated by the unexpected length of the spec.  When you have maybe an hour a day to work on a piece of code, it's difficult to get into the any kind of flow state, so large changes that require extended concentration are pretty much off the table.

So how do we address this?  How do you build something when you don't have enough time to get the whole thing in your head at once?

Step 2: Design

Answer: you document it.  You figure out a piece and write down what you figured out.  Then the next time you're able to work on it, you can read that and pick up where you left off.  Some people call this "design".

I ended up reading through the spec over several days and eventually putting together UML diagrams to help me understand the flow.  There were two flows, sending and receiving, so I made one diagram for each, which spelled out the various validations and error conditions that were described in the spec.

That was really all I needed as far as design for implementing the webmention protocol.  It's pretty straight-forward and I made the diagrams detailed enough that I could work directly from them.  The only real consideration left was where to fit the webmention implementation into the code.

My initial thought was to model a webmention as a new class, i.e. to have a Webmention class to complement the currently existing TrackBack and Pingback classes.  In fact, this seemed like the obvious implementation given the code I was working with.  However, when I started to look at it, it became clear that the only real difference between Pingbacks and Webmentions is the communication protocol.  It's the same data and roughly the same workflow and use-case.  It's just that Pingback goes over XML-RPC and Webmention uses plain-old HTTP form posting.  It didn't really make sense to have a different object class for what is essentially the same thing, so I ended up just re-using the existing Pingback class and just adding a "webmention" flag for reference.

Step 3: Implementation

One of the nice things about having a clear spec is that it makes it really easy to do test-driven development because the spec practically writes half your test cases for you.  Of course, there are always additional things to consider and test for, but it still makes things simpler.

The big challenge was really how to fit webmentions into the existing application structure.  As I mentioned above, I'd already reached the conclusion that creating a new domain object for the was a waste of time.  But what about the rest of it?  Where should the logic for sending them go?  Or receiving?  And how should sending webmentions play with sending pingbacks?

The first point of reference was the pingback implementation.  The old pingback implementation for sending pingbacks lived directly in the domain classes.  So a blog entry would scan itself for links, create a pingback object for each, and then ask the pingback if its URI supported pingbacks, and then the entry would sent the pingback request.  (Yes, this is confusing.  No, I don't remember why I wrote it that way.)  As for receiving pingbacks, that lived entirely in the XML-RPC endpoint.  Obviously none of this was a good example to imitate.

The most obvious solution here was to encapsulate this stuff in its own class, so I created a SocialWebClient class to do that.  Since pingback and webmention are so similar, it made sense to just have one class to handle both of them.  After all, the only real difference in sending them was the message protocol.  The SocialWebClient has a single method, sendReplies(), which takes an entry, scans its links and for each detects if the URI supports pingback or webmention and sends the appropriate one (or a webmention if it supports both).  Similarly, I created a SocialWebServer class for receiving webmentions with an addWebmention() method that is called by an endpoint to save incoming mentions.  I had originally hoped to roll the pingback implementation into that as well, but it was slightly inconvenient with the use of XML-RPC, so I ended up pushing that off until later.

Results

As I mentioned, I didn't track the amount of time I spent on this task.  However, I can retroactively calculate how much code was involved.  Here's the lines-of-code summary as reported by Process Dashboard:

For those who aren't familiar, the "base" value is the lines of code in the affected files before the changes, while the "total" it the total number of lines in affected files after the changes.  The magic number here is "Added & Modified", which is essentially the "new" code.  So all in all, I wrote about a thousand lines for a net increase 700 lines.

Most of this was in the new files, as reported by Process Dashboard below.  I'll spare you the 31 files that contained assorted lesser changes (many related to fixing unrelated issues) since none of them had more even 100 lines changed.  

It's helpful to note that of the 717 lines added here, slightly less than half (315 lines) is unit test code.  Since I was trying to do test-driven development, this is to be expected - the rule of thumb is "write at least as much test code as production code".  That leaves the meat of the implementation at around 400 lines.  And of those 400 lines, most of it is actually refactoring.

As I noted above, the Pingback and Webmention protocols are quite similar, differing mostly in the transport protocol.  The algorithms for sending and receiving them are practically identical.  So most of that work was in generalizing the existing implementation to work for both Pingback and Webmention.  This meant pulling things out into new classes and adjusting them to be easily testable.  Not exciting stuff, but more work than you might think.

So the main take-away from this project was: don't underestimate how hard it can be to work with legacy code.  Once I figured out that the implementation of Webmention would closely mirror what I already had for Pingback, this task should have been really short and simple.  But 700 lines isn't really that short or simple.  Bringing old code up to snuff can take a surprising amount of effort.  But if you've worked on a large, brown-field code-base, you probably already know that.

The first part of my LnBlog refactoring is kind of a large one: changing the publication logic.  I'll start by giving an overview of the old design, discussing the new design, and then looking at some of the actual project data.

History

LnBlog is a very old system.  I started it back in 2005 and did most of the work on it over the next two years.  It was very much an educational project - it was my first PHP project (in PHP 4, no less) and only my third web-based project.  So I really didn't know what I was doing.

As you might expect, the original design was very simple.  In the first version, "publishing an entry" really just meant creating a directory and writing a few files to the disk.  As the system grew more features, more and more steps were added to that process.  The result was a tangle of code where all of the logic lived in the "controller" (originally it was a server-page) with some of the actual persistence logic encapsulated in the domain objects.  So, roughly speaking, the BlogEntry object knew how to write it's metadata file to the appropriate location, but it didn't know how to properly handle uploaded files, notifications, or really anything else.  

Originally, LnBlog used a server-page model, with each URL being a separate file and code being shared by including common configuration files and function libraries all over the place.  Shortly prior to starting this project, I had consolidated the logic from all those pages into a single, massive WebPages class, with one public method for each endpoint - a monolithic controller class, for all intents and purposes.  This still isn't great, but it gave me a common entry point to funnel requests through, which means I can better control common setup code, handle routing more easily, and generally not have the pain of dealing with a dozen or more endpoints.

Anyway, this WebPages class served up and edited entries by directly manipulating domain objects such as BlogEntry, Blog, BlogComment, etc.  The domain objects knew how to save themselves to disk, and a few other things, but the majority of the logic was right in the WebPages class.  This worked fine for the main use-case of creating an entry from the "edit entry" page, but it made things very awkward for the less-used locations, such as the Metaweblog API endpoint or scheduled publication of drafts.

Furthermore, the publication logic in the WebPages class was very hairy.  All types of publication flowed through a single endpoint and used a common code path.  So there were flags and conditions all over the place, and it was very difficult to tell what needed to be updated when making changes.  Bugs were rampant and since there was no test automation to speak of, testing any changes was extremely laborious and error prone.

The New Design

There were two main goals for this refactoring:

1. Create a new Publisher class to encapsulate the publication logic.  The idea was to have a single entity that is responsible for managing the publication state of blog entries.  Given a BlogEntry object, it would know how to publish it as a regular entry or a static article, unpublish it, handle updating or deleting it, etc.  This would give us a single entity that could own all the steps involved in publishing.
2. Create unit tests around the publication process.  The logic around the entire process was more complicated than you'd think and the old code was poorly structured, so it broke with disturbing regularity.  Hence I wanted some automated checking to reduce the risk of bugs.

So the design is actually pretty straight-forward: create a "publisher" class, give it operations for each of the things we do with blog entries (create, update, delete, etc.), copy the existing logic for those cases into the corresponding methods, update the endpoints to use this new class, and call it a day.  

So it was mostly just a reorganization - there wasn't any significant new logic that needed to be written.  Simple, right?  What could go wrong?

Results

While I was happy with the result, this project turned out to be a much larger undertaking than I'd assumed.  I knew it was going to be a relatively large task, but I was off by a factor of more than two.  Below is a table summarizing the project statistics and comparing them to the original estimates (from PSP data I captured using Process Dashboard). 

When I originally did the conceptual design and estimate, I had planned for relatively modest changes to the Article, BlogEntry, and WebPages classes and the creators.php file.  I also planned for new Publisher and BlogUpdater classes, as well as associated tests and some tests for the WebPages class.  This came to 29 hours and 946 new or changed lines of code across nine source files.  Definitely a big job when you consider I'm working in increments of two hours or less per day, whenever I get around to it.

In actuality, the changes were much larger in scope.  I ended up changing 27 additional files I hadn't considered, and ended up creating two other new utility classes (although I did ditch the BlogUpdater class - I no longer even remember what it was supposed to do).  The resulting 3138 lines of code took me 64 hours spread over five months.

Effects of Testing

I did test-driven development when working on this project.  I've found TDD to be useful in the past and it was very helpful to me here.  It was also very effective in meeting my second goal of building a test suite around the publication logic.  PHPUnit reports statement coverage for the Publisher tests at 97.52% and close to 100% coverage for the tested methods in the WebPages class (I only wrote tests for the endpoint that handles creating and publishing entries).

More importantly, using TDD also helped me to untangle the logic of the publication process.  And it turns out there was actually a lot to it.  I ended up generating about 2000 lines of test code over the course of this project.  It turns out that the design and unit test phase occupied 65% of the total project time - about 41 hours.  Having a comprehensive test suite was immensely helpful when I was rebuilding the publication logic across weeks and months.  It allowed me to have an easy check on my changes without having to load all of the code I worked on three weeks ago back into my brain.

Sadly, the code was not such that I could easily write tests against the existing code.  In fact, many of the additional changes came from having to break dependencies in the existing code to make it possible to unit test.  Luckily, most of them were not hard to break, e.g. using an existing file system abstraction layer, but the work still adds up.  It would have been very nice to have an existing test suite to prevent regressions in the rewrite.  Unfortunately, even integration tests would have been awkward, and even if I could have written them, it would have been very hard to get the level f coverage I'd need to be confident in the refactor.

Conclusion

In terms of the results, this project worked out pretty well.  It didn't really go according to plan, but I got what I was looking for in the end - a better publication design and a good test suite.  However, it was a long, slow slog.  Maybe that was too big a slice of work to do all at once.  Perhaps a more iterative approach could have kept things moving at a reasonable pace.  I'll have to try that on the next big project.

Today, we're going to talk a little about design and refactoring.  As a case-study, we're going to use a little blogging application called LnBlog.  You probably haven't heard of it - it's not very popular.  However, you have used it, at least peripherally, because it's running this site.  And you also have at least a passing familiarity with the author of that application, because it's me. 

Motivation

Software is an "interesting" field.  The cool new technologies, frameworks, and languages get all the press and they're what everybody wants to work with.  But let's be honest: it's generally not what makes the money.  I mean, how could it be?  It just came out last week!

No, if you have the good fortune to work on a grown-up, profitable product, it's almost certainly going to be the "old and busted" tech.  It might not be COBOL or Fortran, but it's almost certainly "legacy code".  It might be C++ or Java or, in our case, PHP, but it's probably old, poorly organized, lacking unit tests, and generally hard to work with.

I work on such a product for my day job.  It's a 10-year-old PHP codebase, written in an old-fashioned procedural style.  There are no unit tests for the old code, and you couldn't really write them even if you wanted to.  Sure, there's a newer part with proper design, tests, etc., but the old code is the kind of stuff that "works for the most part", but everybody is afraid to touch it because it's so brittle and tightly coupled that God alone knows what will break when you make a change.

This also applies to LnBlog.  It was my very first PHP application.  I started it way back in early 2005, in the bad old days of PHP 4.  Over the next two or three years, I managed to turn it into something that was relatively functional and full-featured.  And for the last ten years or so, I've managed to keep it working.

Of course, it hasn't gotten a whole lot of love in that time.  I've been busy and, for the most part, it worked and was "good enough".  However, I occasionally need to fix bugs or want to add features, and doing that is a truly painful process.  So I would very much like to alleviate that pain.

The Issue

Let me be honest: I didn't really know what I was doing when I wrote LnBlog.  I was about four years out of school and had only been coding for about six or seven years total.  And I was working mostly in Visual Basic 6 at the time, which just barely counts.  It was also only my third web-based project, and the first two were written in classic ASP and VBScript, which also just barely counts.

As a result, it contains a lot of questionable design decisions and overly-complicated algorithms.  The code is largely procedural, kind of buggy, and makes poor use of abstraction.  So, in short, it's not great.

But, in fairness to myself, I've seen worse.  In fact, I've seen a lot worse.  It does have a class hierarchy for the domain objects (though it's a naive design), an abstraction layer for data access (though it's inconsistently used), and a templating system for separating markup from domain logic (though the templates are an ungodly mess).  And it's not like I had a role model or mentor to guide me through this - I was figuring out what worked on my own.  So while it's not great, I think it's actually surprisingly good given the circumstances under which it was built.

The Goal - Make the Code "Good"

So I want to make LnBlog better.  I've thought about rewriting it, but decided that I wouldn't be doing myself any favors by going that route.  I also hold no illusions of a grand re-architecture that will fix all the problems and be a shining beacon of design perfection.  Rather, I have a relatively modest list of new features and bug fixes, and I just want to make the code good enough that I can make changes easily when I need to and be reasonably confident that I'm not breaking things.  In other words, I want to do a true refactoring.

If you haven't read Martin Fowler's book, the word "refactoring" is not a synonym for "changing code" or "rewriting code".  Rather, it has a very specific meaning: improving the internal design of code without changing the external behavior.  In other words, all you do is make the code easier to work with - you don't change what it does in any way.  This is why people like Bob Martin tell you that "refactor X" should never be an item in your Scrum backlog.  It is purely a design and "code cleanliness" activity, not a "feature" you can deliver.

So my goal with LnBlog is to gradually reshape it into what it should have been in the first place.  This is partially to make changing it easier in the future.  But more importantly, it's a professional development goal, an exercise in code craftsmanship.  As I mentioned above, I've worked professionally on many systems that are even more messed up than LnBlog.  So this is a study in how to approach refactoring a system.  

And So It Begins...

My intention is to write a number of articles describing this process.  I've already completed the first step, which is rewriting some of the persistence and publication logic.  I'm using the PSP to track my planned and actual performance, so I'll have some actual data to use in my discussion of that process.  Hint: so far, the two are very different.

With any luck, this project will enable me to draw some useful or interesting conclusions about good and bad ways to approach reworking legacy systems.  Maybe it will also enlighten some other people along the way.  And if nothing else, I should at least get a better codebase out of it.

This weekend I did something I've been meaning to do for a while: I redesigned my website. In fact, unless you're reading this in your RSS aggregator, you probably already noticed. It was about time, too - I'd had the same fractured, dated design for years. There are probably still some kinks to work out, but for the most part I think it looks much cleaner.

This time I decided to do a real site design. As in, not only did I update this blog, but also the other sections of this site and the landing page as well. In fact, this started as a re-do of my landing page and I ended up abstracting that design out and making it a theme for LnBlog. Then I just set this and my other LnBlog blogs (you know, the ones in the header that nobody reads) to use it. Update accomplished!

I figured that, what with being an experienced front-end web developer, it would probably be a good idea to make my site look decent. Lends to the credibility and all. The randomly differing styles of the old pages looked kinda crappy and the sharp edges,weird colors, and embellishments were not so great. Of course, I'm no graphic designer, but I think this looks a bit cleaner. And at the very least, it's consistent.

I have gamercise-induced writer's block. That's my term for being unable to think of anything good to blog about because I'm tired from playing high-impact video games.

Aside: Boy, "high-impact video game" is kind of a tortured phrase, isn't it? It sounds like the force-feedback controller is going to leave bruises and knock things off your shelves. Actually, it remindes me of one gamer I heard refer to "high-stakes games." Doesn't that give you images of the computer electrocuting you if you lose?

The reason I'm tired is because Sarah ordered herself a copy of Dance Dance Revolution for the PlayStation 2 the other week. I didn't have any interest in it, but she wanted something fun to do for exercise. It also had the side benefits of being relatively cheap and actually putting my PS2 to use.

However, when I watched Sarah actually playing, it looked like a lot of fun. I always thought it looked kind of silly, but when I tried it, it really was a lot of fun. And I can't even dance! In fact, not only am I not very good, Sarah tells me I look like a spastic chimpanzee while I'm playing. But I don't case, because it's still a lot of fun.

I'm kind of enjoying this whole physical video game thing. It's a lot of fun and it actually does get your pulse up. In fact, it would be nice to get the PS2 in a room other than our bedroom. That way, I could play Dance Dance Revolution for my morning exercise rather than just doing half an hour on the eliptical machine. I wouldn't be able to read at the same time, but it sure would be more exciting.

And while I'm at it, maybe I should get a Wii. Those look like they'd be great fun too....

As you may have noticed, I've severely stepped up my blog posting rate lately. I'm trying get into the habit of posting on a consistent schedule. I'm doing this in an effort to improve my communication skills and prevent brain atrophy.

Writing is an important skill to have, especially for a programmer. There are two reasons for this. First, it makes you look good by comparison, as many programmers are half-literate code junkies who can just barely put a coherent sentence toegether. Second, and more importantly, a lot of the communciation we engage in is over the internet, i.e. written. We thrive on mailing lists, comment boards, e-mail, IRC, and the like. This is especially true in the FOSS world, where you may easily find yourself working closely with someone you will never see face to face.

Given that, it is clearly important that we be able to express our thoughts clearly, succinctly, and without putting the reader to sleep. You develop that ability the same way you learn anything else: practice. And for me, the most convenient way to practice is to blog.

In addition to becoming a better writer, I'm hoping that using a blog as my practice medium will also make me a faster writer. You see, I'm a painfully slow writer. I'm not at all good at coming up with witty, insightful commentary off the top of my head. Instead, I overthink what I'm trying to say, rewrite the same paragraph a dozen times, and eventually spend so much time on a relatively small amount of text that I feel like I've wasted a lot of time and have to rush to get it done. By setting myself a regular schedule of 1 post per business day, I'm hoping that my brain will adapt and get a little quicker on the draw.

And last but not least, it's just nice to have place to hash out whatever I might be thinking along a computing vein. Writing your thoughts down forces you to process them a bit more, to shape them into a point that you can articulate. When it comes to technical topics, particularly related to software development, I don't have a lot of people around to discuss things with, so writing is the most readily available outlet.

I guess we'll see how that all turns out. With any luck, you'll notice this blog getting progressively better in the months to come. If not, then maybe I'll just have to accept that I'm not as smart as I always thought I was. But let's hope it doesn't come to that!

I've been trying to spruce up the blog the last few days. I'm trying to make things a little more reader-friendly and possibly increase traffic a little.

First, for the 11 people who actually subscribe to it, I've changed the RSS feed over to FeedBurner. That gets me subscriber statistics, a little extra exposure, and a bunch of other miscellaneous features. It's also completely painless for everybody else, since I just set a mod_rewrite rule to redirect everybody to the new external feed. I even did a little hacking to make the new feed URL integrate nicely with LnBlog.

I also set myself up with a Technorati account. This provides another handy tool for gaining wider exposure. It's a nice complement to TrackBack and Pingback in addition to actually being a nice service to use.

I'm also experimenting with my page layout. In particular, I'm reorganizing the sidebar. I don't really know what the optimal layout it, but I do know I wasn't particularly happy with the old layout. There's still plenty of stuff left to play with, so I guess I'll just try some different things and see what works.

Tomorrow I'll get into the motivation for the change. Right now, I need to go to bed. I was just barely able to muster the concentration to write this, so I'm certainly not up for a long explanation.

Last week, Jeff Atwood posted an anecdote demonstrating yet again that bloggers aren't real journalists. I know this meme has been floating around for some years, but I'm still surprised when people bring it up. In fact, I'm still surprised that it ever got any traction at all.

I'm going to let you in on a little "open secret" here: blogging in 2007 is no different than having a Geocities site in 1996. "Blogging" is really just a fancy word for having a news page on your website.

Oh, sure, we have fancy services and self-hosted blog servers (like this one); there's Pingback, TrackBack, and anti-comment spam services; everybody has RSS or Atom feeds, and support for them now built into browsers. But all that is just gravy. All you really need to have a blog is web hosting, an FTP client, and Windows Notepad.

That's the reason why bloggers in general are not, and never will be, journalists. A "blog" is just a website and, by extension, a "blogger" is just some guy with a web site. There's nothing special about it. A blogger doesn't need to study investigative techniques, learn a code of ethics, or practice dispassionate analysis of the facts. He just needs an internet connection.

That's not to say that a blogger can't practice journalism or that a journalist can't blog. Of course they can. It's just that there's no necessary relationship. A blogger might be doing legitimate journalism. But he could just as easily be engaging in speculation or rumor mongering. There's just no way to say which other than on a case-by-case basis.

Like everything else, blogging, social media, and all the other Web 2.0 hype is subject to Sturgeon's law. The more blogs there are out there total, the more low-quality blogs there are. And the lower the barrier to entry, the higher the lower the average quality is. And since blogs have gotten insanely easy to start, it should come as no surprise that every clueless Tom, Dick, and Harry has started one.

I think George Carlin put it best:

Just think of how stupid the average person is. Then realize that half of them are stupider than that!

Any average person can be a blogger. Thus the quality of those blogs will follow a standard distribution. For every Raymond Chen, Jeff Atwood, and Roger Johansson, there are a thousand angst-ridden teenagers sharing bad poetry and talking about not conforming in exactly the same way. They're definitely bloggers, but if we're going to compare them to journalists, then I think society is pretty much done for a blog.