LinLog

In the spirit of not freaking out and overdoing it for the holidays, for Christmas this year I've scheduled another "from the archives" post.  If you haven't been keeping track, that's my lazy posting series where I slap a preamble on something that's been sitting in my drafts folder forever and call it "new content".

This one is actually seasonal, though, as it's dated December 28, 2007.  It's also completely non-technical and yet was also tagged "Software Engineering" for some reason.  Maybe I intended to add something to it, or maybe I just fat-fingered the tag widget.  But that's neither here nor there.  The point is that my scheduled Saturday post falls on Christmas day, and this is related to Christmas.

Since this one is super-short, I'm going to add some inline commentary to it because, well, otherwise it's just boring.  But this is also purely biographical, which means that nobody else cares anyway.  So this is largely a "me" post.

The post is below the line with current-day comments in italics.  Hopefully it'll be at least modestly entertaining.  Enjoy and happy holidays!

Boy, it's been a long time since I did any blogging. (Now: Exactly a week, to be precise.)  I've been meaning to get back to it, but it seems like I just haven't been able to find the time or energy.  Hopefully that will change now that things are finally settling down.  (Now: This actually still applies, sadly.  Despite being part of the "zoomocracy" and hence managing the pandemic as well as can be expected, I've found the whole thing has been a constant slow sap on my energy.  I've tried to maintain healthy habits and schedules and have managed fairly well, but still....)

There's been a lot going on here in the last couple of months.  Of course, there was the usual holiday hubbub.  My wife always insists on making a big production for Thanksgiving and Christmas, even when she doesn't have time to.  (Now: One up-ish side of the pandemic is that this hasn't been an issue the last couple of years.  That's not precisely a "good" thing, it does put the stress of big holiday plans in perspective.  Although I do miss some of it.  For instance, I was just thinking the other day about how every year I would make and package a bunch of cookie dough for my grandmother at Christmas.  She passed in early 2020 at the age of 89, in the first wave of the pandemic.  She was never very good at making cookies - when they were kids, apparently when she mentioned to my uncles that she might make some cookies, they would tell here that maybe she should wait for my mom to come home and let her do it.  So every year I would freeze her a bunch of pre-rolled cookies that she could just pop in the oven.  She would take them to events at church and people would tell her what good cookies she made.)  We also found a buyer for our house in Corning and moved the last of stuff out.  I've been trying to unpack and organize things, but it's very slow going.  

In the moving process, we also acquired a piano.  It's actually my mom's piano, which she doesn't play and has been wanting to get out of the house for some time.  I, however, do play and I've been immensely enjoying trying to get back in practice.  (Now: This is also still relevant.  I don't play that much anymore, but I still find it relaxing to do every now and then.  The last few weeks I've broken out the old Christmas carol sheet music in honor of the holidays.  Even though things are better than last Christmas, we're still not really back to normal.  So in these weird times, I find playing the piano a nice way to decompress, put aside the worries, and think about better days.)

Author's note: This episode of From The Archives is the stub of an article I wrote on December 10, 2007.  At the time, I was working for eBaum's World (this was back around the time ebaum sold it, but before he was forced out).  It was kind of a weird time because it was my first experience working for an actual tech company.  (It was also a weird place to work, for other reasons, but that's a different story.)  Previously, I'd been stuck in the world of public-sector internal IT which is...not great, by comparison.

This particular post was expressing my annoyance over how our development environment was broken.  Our dev environment, at the time, was literally just a shared server that we all pushed code to. And yes, that means we had plenty of opportunity to step all over each other and things could easily break for non-obvious reasons.

This was obviously terrible, but it's what we had to work with.  We had a four-man development and system administration team with not a huge budget or set of internal resources.  And, of course, there was always something more important to do than improving our dev environment setup.

These days, I still have dev environment issues, but for completely different reasons.  My company has an entire infrastructure for spinning up development VMs, including dedicated host clusters, custom tooling to setup environments, and teams responsible for managing that stuff.  So now I have my own virtual instance of every server I need (which, for reference, is currently about eight VMs).  However, there are still some holes in that infrastructure.

Part of the issue is that we have a lot of teams that share or depend on the same infrastructure and/or services.  For instance, one of the services that my team maintains is also worked on by at least three other teams.  And we all share the same database schema, which doesn't automatically get updated in dev environments when changes are made in production, so when you pull in the develop branch, you frequently get breaking changes that you may not have ever heard of, usually in the form of config or database changes that aren't set in  your environment.  Sure, everything "just works" if you start fresh, but nobody ever starts fresh because it takes too long to spin up all the required pieces and set up the test data.  (However, we are moving toward a Docker/Kubernetes setup for the pieces that don't need test data, so things are moving in the right direction.)

Not that I can complain too much.  Even for my personal projects, which are much simpler, my dev environment is frequently in disarray.  Things get out of date, permissions or services don't get correctly configured after an update, or my setup is out of sync with the production environment.    In fact, I had that problem just the other day - I pushed some code for this blog that worked fine locally, but broke on the live server because it was running a different version of PHP.  Granted, the code in my dev environment was wrong (and I really should have caught it there), but that's not the point.

The point is that environment maintenance is hard and dev environment maintenance doubly so because it changes frequently and is of much lower priority than production.  That's something I hadn't really had to worry about in my internal IT job, because I was writing desktop apps and everybody was running identical Windows desktops.  It was a a simpler time....

This week's installment of "Pete's list of things that suck" features broken development environments.  I spent the last couple of days at work wrestling with one and it really, really sucks.

The thing is, our development environment is...really screwed.  Things went really bad during the production server upgrade a couple of months ago and we had to hack the hell out of the code just to keep the site running.  As a result, we had all kinds of ugly, server-specific hackery in production which just plain broke when we moved it back into devel.  And since, of course, we have a huge backlog of projects that management wants implemented, we've had no time to go back and reconfigure the development servers.

This week, I've been trying to test some changes to our user upload process.  However, due to some NFS misconfiguration and some code problems, uploads just plain don't work in our main development environment.  We do have a new, additional set of development servers (well, one box with a half-dozen virtual machines, but you get the idea), but I couldn't get those to work either.  Part of it was that the configuration on those servers was incomplete, and part of it was that Firefox sucks.  (Note from the present: I no longer remember why that was.)

Unlike most episodes of "From the Archives", this one is just going to be current-day commentary. This was supposed to be a "linkblogging" post from way back on February 22, 2006. (That was probably not long after I added draft support to LnBlog!) However, it only had three links and I no longer care about two of them.

The one that is still good is Douglas Crockford's account of The Expurgation of Maniac Mansion for the Nintendo Entertainment System. (The URL has changed since 2006, but but luckily Doug was considerate enough to keep a redirect. However, this publication date is a little misleading.) It's about the back-and-forth they had with Nintendo when trying to port Maniac Mansion to the NES.

I remember playing Maniac Mansion on the NES. I rented it from Video Entertainment, the video rental store in the next town over, several times. I never finished the game, but I remember liking it. I never played the PC version, but even the NES version was more than a little weird.

This article is Crockford's account of some of the odd things that Nintendo insisted they remove from the NES version. They range from silly to head-scratching. But if you've ever heard anything about Nintendo's approval process for NES games, you know that they were very strict about maintaining a certain image and so were very particular about what kind of content they'd allow. Either way, it's an entertaining read.

Author's note: Welcome to another edition of "From the Archives", where I post some crappy, half-finished thing that's been sitting in my drafts folder for the last 10 years!

This articles is one I started on September 14, 2008, and apparently made some edits to on October 4, 2016 (don't ask me what they were). It's about Amazon Music, or as it used to be called Amazon MP3. Nobody cares about MP3 or any other specific format anymore, though. In fact, at this point I think most people just stream music through some app that doesn't even tell them what format it's in.

But back in the 2000's, it was all about MP3s. The only other format going was AAC, and only because that's what iTunes used. People would download these files and copy them to dedicated music playing devices. Yes kids, that was a thing. I had a bunch of those devices. They were fine at the time, but it's important to realize that this was in the days before high-speed data was ubiquitous and phones had tens of gigabytes of usable storage available.

Anyway, Amazon MP3 has since become Amazon Music and now focuses more on streaming than downloading. Fortunately, you can still download MP3s from Amazon Music, you now just have to do it through their desktop app. It's not too bad, but I actually don't like it as much as the download experience for the old AmazonMP3 version of the service. The app isn't really focused on that and they keep changing the interface.
And yes, I do still care about downloading the music I buy - that's why I have a Plex instance. I like to feel like I have some measure of control over digital products I buy, even if much of it is an illusion these days.

But anyway, that's enough from now. Back to 2008. Enjoy!

I've really gotten to like Amazon's MP3 download service. I've bought a number of songs and albumns through it in the last couple of months, and it's quite nice. In fact, it's what a music download service should be.

The big win for Amazon, of course, is selection. They might not have everything, but they come damn close. Nearly every other download service I've seen over the years had a limited selection. Great if you're into discovering new artists in particular genres, but they never had mainstream stuff.

The other main selling point for Amazon is price. You can buy individual songs for $0.99 or $0.89 (just as cheap as iTunes) and entire albumns at a discount. No subscriptions or other commitments required.

Aside from those obvious issues, the service is actually very well designed. For starters, it's web-friendly, which already puts it ahead of iTunes in my book. The searching and browsing works well and they have the usual Amazon suggestions and reviews. There's a nice little Flash app for song previews and Amazon's trademark one-click purchasing. It even works well in Opera for Linux, which is notorious for questionable Flash support.

The one non-web-friendly thing about AmazonMP3 is the download app. Instead of an actual MP3, you download a .amz file, which is handed off to this download app. It queues up the files for download and drops them in appropriately organized folders. Apparently it can also import them into iTunes and WMP too. That's about it, though. It's invoked by the browser as the file handler for .amz files and, really, that's the only way you'd ever run it. I mean, other than download files, it really doesn't do anything.

On the up side, the download app is widely supported and failry inocuous. It's available for Windows, Mac, Debian, Ubuntu, Fedora, and Open SuSE, so Linux people aren't left out in the cold. It's a small program, too. The Ubuntu package is a grand total of 772KB uncompressed. Hardly the massive 60MB iTunes package.

Author's note:  Happy new year!  I thought I'd start the year off with another old post that's been sitting in my drafts folder since March 24, 2013.  This time, though, I'm going to provide more inline commentary.  As usual, the interjections will be italicized and in parentheses.

You see, this post is on LnBlog, and specifically what is (or was) wrong with it.  If you don't know, LnBlog is the software that runs this website - I wrote it as a "teach yourself PHP" project starting back around 2005.  I've been improving it, on and off, ever since.  So in this post, I'm going to show you what I thought of it back in 2013 and then discuss what I think now and what has changed.  Hopefully it will be somewhat enlightening.  Enjoy!

The year is (relatively) new and it's time for some reflection. In this case, reflection on past code - namely LnBlog, the software that runs this site.

I've come a long way from LnBlog, which as my first "teach yourself PHP" project. I've now been doing full-time professional PHP development since 2007 and can reasonably claim to have some expertise in it. And looking back, while the LnBlog codebase is surprisingly not horrifying for someone who had a whopping two months of web development experience going into it, it's still a mess. So it's time to start slowly refactoring it. And who knows? Blogging my thought process might be useful or interesting to others.

(Back to now: I actually did blog some of this stuff, but not until 2017 or so.  And I still agree with that initial assessment.  The code had plenty of problems then and it still does.  If I were starting fresh today, I'd probably do almost everything differently.  But on the other hand, I've seen much worse in much newer code.  And in the last three years or so I've been making slow and steady improvements.)

The Issues

There are lot of things about LnBlog that need changing. A few of them are functional, but it's mostly maintenance issues. By that I mean that the code is not amenable to change. It's not well organized, it's too hard to understand, and it's too difficult to make updates. So let's go over a few of the obvious difficulties.

1. The plugin system

I have to face it - the plugin system is an unholy mess. The entire design is poorly thought out. It's built on the premise that a "plugin" will be a single PHP file, which makes things...painful. Any plugin with significant functionality or a decent amount markup starts to get messy very quickly. The "single file" limitation makes adding styles and JavaScript ugly as well.

On the up side, the event-driven aspect works reasonably well. The code for it is a bit nasty, but it works. The main problem is that there aren't really enough extension points. It needs a bit more granularity, I think. Or perhaps it just needs to be better organized.

(Back to now: I still agree with most of this, except perhaps the thing about extension points.  So far, the only place where that's been a real problem is when it comes to inserting markup mid-page.  But yeah, the whole "a plugin is one file" thing was ill-conceived.  The good news is that it's totally fixable - I just need to figure out some design conventions around splitting things out, which hasn't been a priority so far.)

2. The templating system

This one is also an unholy mess. The idea isn't bad - allow any file in a theme to be over-ridden. However, I tried to abstract the template files too much. The files are too big and contain too much logic. Also, the simple template library I'm using is more a hindrance than a help. I'd be better off just ditching it.

I've also been thinking of getting rid of the translation support. Let's face it - I'm the only person using this software. And I'm only fluent in one language. Granted, the translation markers don't cause any harm, but they don't really do anything for me either, and accounting for them in JS is a bit of a pain.

(Back to now: The only thing I still agree with here is that the existing templates are a mess.  But that has nothing to do with the template system - I just did a bad job of implementing the template logic.  I'm working on fixing that - for instance, I added some Jinja-like block functionality to the template library.  I had considered re-writing the templates in Twig or something, but it quickly became obvious that that would be a huge amount of work, that it would be difficult to do in a piece-wise fashion, and it's not clear that the payoff would be worth it.  Likewise with the translation markers - taking them out would be a bunch of work for almost zero payoff and the JS thing isn't really that big a deal.  Besides, if I ever changed my mind again it's WAY more work to put them back in.)

3. The UI sucks

Yeah, my client-side skills have come a long way since I built LnBlog. The UI is very Web 1.0. The JavaScript is poorly written, the style sheets are a mess, the markup is badly done, and it's generally "serviceable" at best.

As I realized the other day, the style sheets and markup are probably the worst part. Trying to update them is difficult at best, which is exactly the opposite of what you want in a theme system. In retrospect, my idea to replace files wholesale rather than overriding seems misguided. They're too fragmented. When it comes to the style sheets and JavaScript, this also hurts performance, because there are a lot of files and everything is loaded in the page head.

(Back to now: This is pretty much still accurate.  I've been slowly improving the UI, but it's still not looking particularly "modern".  That's not such a big deal, but the templates and CSS are still a pain-point.  Really, what I need to do is rework the theme system so that I can easily make lighter-weight themes, i.e. I should be able to just create one override CSS file and call it good.  I have the framework for that in place, but I have yet to actually go through the existing themes and make that work.)

4. Too much compatibility

When I first started writing LnBlog, I had a really crappy shared web hosting account. And by "really crappy", I mean it offered no database server and had safe-mode and the various other half-baked PHP "security measures" enabled by default. So I actually built LnBlog to be maximally compatible with such an environment.

These days, you can get decent hosting pretty cheap. So unless you can't afford to pay anything, there's no need to settle for such crappy hosting. And again, let's be honest here - I don't even know anyone other than me who's using this software. So supporting such crappy, hypothetical configurations is a waste of my time.

In addition, I really put an absolutely ridiculous number of configuration settings into LnBlog. The main config file is extensively documented and comes to over 700 lines. That's completely nuts and a pain to deal with. It contains a lot of settings that are pointless and that hardly anyone would ever want to override. And most of those could be moved into a GUI rather than having to edit a file. There's absolutely no reason for many of those settings.

(Back to now: This is also still true.  I've been looking at redoing the config system, but that's another one of those things that is a big change because it has tendrils all through the code.  I have been moving some stuff out of the main blogconfig.php file, and I've been avoiding adding to it, but there's still a lot there.  For the most part, it's not a huge issue, since most of the things you would want to configure are through the UI, but still....)

5. No real controller structure

I knew nothing of MVC or design patterns when I first wrote LnBlog. As a result, the "glue" code is in the form of old-style procedural pages. They're messy, poorly organized, and hard to maintain. A more modern approach would make things much easier to deal with.

(Back to now: The old "pages" are dead in all but name.  A handful of them still exist, but they're three-liners that just delegate to a controller class.  The bad news is that it's pretty much just two monolithic controller classes with all the old logic dumped into them.  So that sucks.  But they have dependency injection and some unit test coverage, so this is still an improvement.  And I've at least got a little routing groundwork laid so that I could start breaking off pieces of functionality into other classes in the future.)

The Problem

While I'd like to fix all this stuff in one shot, there are three big problems here:

1. That's a lot of stuff, both in terms of the number of tasks and the amount of code involved.
2. I no longer have the kind of free time I did when I first wrote this.
3. I'm actually using this software.

Of course, those are two sides of the same coin.  LnBlog isn't huge, but it isn't tiny either - the codebase is upwards of 20,000 lines.  That wouldn't be a big deal if I were working on it as my full-time job, but this is a side-project and I can devote maybe a couple hours a day to it sometimes.  So major surgery is pretty much out.  And the third factor means that I need to be careful about breaking changes - not only do I not want to break my own website, but I also want to avoid having to do a lot of migration work because writing migration scripts is not my idea of a fun way to spend my free time.

(Back to now: This is always a problem with open-source and side projects.  Nothing has changed here except, perhaps, my development process.  After that year I spent learning about the Personal Software Process, I started using some of those methods for my personal projects.  The main change was that, when making any kind of a big change or feature addition, I actual do a semi-formal process with a requirements and design phase and review phases.  It sounds kind of silly for a personal project, but it's actually extremely useful.  The main benefit is just in having my thoughts documented.  Since I might be going a week or more between coding sessions on any particular feature, it's insanely helpful to have documentation to refer back to.  That way I don't have to remember or waste time figuring things out again.  And by having design- and code-review phases as part of my development process, I have a built-in reminder to go back and check that I actually implemented all those things I documented.  Having the whole thing written out just makes it much easier when you have long gaps in between work sessions.)

General commentary from the present: So as you can see from the above comments, I've fixed or am fixing a lot of the things that bothered me about LnBlog eight years ago.  In the last two or three years I've put a lot of work into this project again.  Part of it is because I actually use it and want it to be better, but part of it is also "sharpening the saw".  I've been using LnBlog as an exercise in building my development skills.  It's not just coding new features, like the flurry of development in the first two years or so that I worked on LnBlog, it's cleaning up my past messes, adding quality assurance (in the form of tests and static analysis), updating the documentation and figuring out how to balance responsible project management with limited resources).  It's an exercise in managing legacy code.

To me, this is a useful and important thing to practice.  As a professional developer, you will have to deal with legacy code.  In my day job, I've had to deal with code that was written by our CEO 10+ years ago when he started the company.  Software is a weird combination of things that live a week and things that live forever, and there's seldom any good way to tell which group the code will be in when you're writing it.  So while it's important to know how to write code correctly the first time, it's also important to know how to deal with the reality of the code you have.  And no, "let's rewrite it" is not dealing with reality.  And when you have a code-base that's 15 years old, that you're actively using, and that you originally wrote, it's a great opportunity to experiment and build your skills in terms of modernizing legacy code.

And that's just what I'm doing.  Slowly but surely, LnBlog is getting better.  I've implemented a bunch of new features, and in the process I've worked on my design and analysis skills, both at a product level and at a technical level.  I've fixed a bunch of bugs, which makes my life easier.  I've implemented additional tests and static analysis, which also makes my life easier by finding bugs faster and giving me more confidence in my code.  I've improved the design of the system, which again makes my life easier because I can now do more with less effort.  Sure, there's still plenty do to, but I've made lots of progress, and things are only getting better.

Author's note: It's officially the holiday season and I'm feeling lazy. So guess what - it's another episode of "From the Archives"! That's right, it's time for more of the series where I trot out something that's been sitting in my drafts folder for ten years because I can't muster the energy to write something new.

This article is from way back on March 18, 2007. It actually appears to be finished, so I'm not sure why I never published it. Perhaps I was filled with crippling self doubt that my analysis was actually stupid and people would make fun of me, so I never hit the publish button. That was actually a thing I did for many years. These days, I'm more casual about it. One of the benefits of getting older is that you're more experienced and able to have a more objective view of the quality of your work. Another is that you have a wider perspective on what really matters in life. Another way to put that is "you don't care as much what other people think."

Anyway, this article is about my attempts at implementing the various blogging APIs in LnBlog. This included the Blogger, MetaWeblog, and MoveableType APIs. I guess that seemed like a good idea at the time. In retrospect, it was a nice educational exercise, but not especially useful. I mean, the idea of a generic API that third-party clients can use to post to your blog is great. But in practice, it doesn't seem like that's a thing that many people actually need. I certainly never found a good use-case for it. Maybe that's why the APIs never really got fleshed out.

But that's enough out of present-day me. Let's hear from 2007 me. Enjoy!

Last month, I finally got around to doing some actual testing on the MetaWeblog and Blogger API implementations for LnBlog. By that, I mean that rather than testing it with my own code, I actually installed a few free blogging clients. I learned a few interesting lessons from this.

The Clients

I tested four blogging clients. The first is Deepest Sender 0.7.9, a Firefox extension that supports LiveJournal, Blogger, Wordpress, MSN Spaces, and generic MetaWeblog blogs. The second is KBlogger 0.6.2, a simple KDE panel applet that supports the MetaWeblog and Blogger APIs. Third is QTM 0.4.0, a C++/Qt4 application that support the Blogger, MetaWeblog, and MovableType APIs. Last is BloGTK 1.1, a Python/GTK+ application that also supports Blogger, MW, and MT.

My results were mixed. KBlogger worked basically as advertised. In fact, it's the only one of the clients that seemed to understand the APIs in the same way that I do. The only problem is that it's a bit short on features.

BloGTK seemed to work pretty well. However, it worked best when set to use the MoveableType API. When using the MetaWeblog API, I had problems editing posts. It also has a few weird little bugs, such as things getting messed up when switching accounts.

While it has a nice interface, QTM simply would not work with LnBlog. For the record, this is not my fault but rather due to the fact that this version of QTM did not correctly implement the APIs. When sending calls to the server, it sent blog IDs, post IDs, category IDs, etc. as integers, whereas the specification calls for them to be strings. While the difference may be academic for some servers, LnBlog really does use strings as IDs, so the requests raise errors in the XML-RPC library. (Note: this seems to have been corrected in CVS.)

And as for Deepest Sender, I just can't get it to work as advertised. I don't know why. It can post entries, but editing them results in a hung edit window, the "active blog" box is shrunk down to an unrecongizable control, and I have yet to even see a category selection box.

Server Problems

The first problem I encountered with LnBlog's implementation of the Blogger 1.0 and MetaWeblog APIs was my silly assumption that, just because they are two separate API specifications, I could implement them separately. And so, that's exactly what I did: I wrote one PHP script to implement the Blogger 1.0 API and a different one to implement the MetaWeblog API.

Oh, what a fool I was!

While that attitude made perfect sense when looking just at the specs, it just doesn't work that way in practice. Of the four clients I tested, KBlogger was the only one that worked when the MetaWeblog server didn't implement the Blogger 1.0 API at the same URI. The others all blithely assumed that the same URI would implement the Blogger, MetaWeblog, and MovableType APIs. I guess few people even stopped to consider that a server might have independent implementations of the different APIs. Or perhaps it's just that KBlogger is designed to support clients that only understand Blogger 1.0 while the others assume MetWeblog support. It's hard to tell.

Placing the blame

After going back to look at the specs, I believe much of the blame for this situation rests with the MetaWeblog API specification itself. The problem is that it's just a bad specification. In fact, it's really more a sketch of a specification than an actual spec. It's too vauge, too confusing, and leaves too much open to interpretation.

For instance, take the metaWeblog.getCategories method. According to the specification, this method returns a struct, with each member being a struct with the description, HTML URL, and RSS URL for each category.

For non-programmers, "struct" is short for "structure," and is simply set of key/value pairs. In this case, having member structs with keys for description and URLs makes perfect sense.

However, putting all of these in a struct doesn't make sense. The spec says that the category structs are to be returned in a struct, but says nothing about the key names of this container struct. But the entire point of having a struct is to associate key names with values. A struct with no particular key names is meaningless. It's like writing a book where the index lists words, but not page numbers - if you can't pick the word (key) you want and go straight to the corresponding page (value), then the entire exercise is pointless.

Another shortcoming of the API is that it does not clearly specify a way to identify blog posts. For example, the API includes the metaWeblog.editPost and metaWeblog.getPost methods, both of which take a post ID. It also includes a metaWeblog.getRecentPosts method to get an array of the most recent posts for a blog. You would think that you could call getRecentPosts, let the user pick a post, edit it, and then call editPost to commit the changes to the server. But you can't.

That's because the API does not specify how to get the post ID. The metaWeblog.getPost and metaWeblog.getRecentPosts methods return a struct and an array of structs respectively, and the spec states that the members of these post structs are the members of RSS items. But there is no mention of where the post ID comes in. RSS certainly has no concept of a post ID, so it's not clear which member should be used for that purpose. Presumably, this is why the MovableType extensions to MetaWeblog include a postId field in the post structs.

Of course, RSS does provide a GUID (Globally Uniquie Identifier) field, which seems a natural fit for the post ID. The problem is that the RSS spec does not require that this field be present. It could also be argued that the GUID has a meaning distinct from a blog post ID. But either way, if the MetaWeblog spec meant that the GUID should be the post ID, then it should have said so.

Judging from the MetaWeblog spec, the only place clients can count on getting a postID is from the return value of metaWeblog.newPost. That's fine if the client can assume it is making all posts to a blog, but it is insufficient if there is also, say, a web interface. If your blogging client can only edit posts it created, you've just cut its usefulness in half.

Missing Links

The MetaWeblog API depends heavily on the Blogger 1.0 API. By itself, it is missing too much to be truly useful for the development of rich blogging clients. If nothing else, this is clear from the absence of something resembling blogger.getUsersBlogs.

Actually, that's not entirely fair. There was an RFC to add the Blogger methods to MetaWeblog, so the spec has been amended to correct this shortcoming. Or has it? I actually only learned about this by reading the code for the WordPress MetaWeblog implementation. The "official" MetaWeblog spec doesn't actually mention this or contain a link to the new RFC. That seems rather odd considering that the spec does contain notes regarding other updates. So has the spec been ammended, superceded, or was this just a "Request For Comment" that was never actually adopted?

Bottom Line for Implementers

So what does all this mean for those enterprising individuals who want to try their hand at writing blogging software? It means you've got an up-hill battle.

Don't get me wrong - it's not that implementing the various specifications is difficult. The APIs are actually pretty simple. The problem is that you can't trust them.

For starters, if you want to write a server that is compatible with existing rich blogging clients, you will have to implement the Blogger 1.0, MetaWeblog, and MovableType APIs, and you will have to do it all at the same URI. This isn't really a problem, so much as an inconvenience, as you can't simply work from a single specification at a time, but have to jump back and forth between three of them just to get a workable MetaWeblog implementation.

If you're writing a client, things are just as annoying. As previously mentioned, there's the post ID problem to deal with. Handling that is not difficult, but you have to rely on the good will of the server to send you a sensible post struct, since it is not required to.

If you want to support MovableType, there's also their brain-damaged category handling to deal with. Rather than using MetaWeblog categories, MT has separate mt.getPostCategories and mt.setPostCategories methods, which deal with category IDs rather than textual categories. Again, this is not hard to deal with, but it means you have to implement category handling twice - once for MT, and once for servers that use MW categories. But on the up side, at least MT gives you an explicit postId field.

Conclusion

All in all, the old blogging APIs suck. They're imprecise, lacking in features, and tend not to be so platform-agnostic. I think they can be best described as "just barely adequate."

I have yet to look at the Atom API. I'm hoping it will turn out to be better, but I'm not going to hold out a lot of hope. At the very least, I suppose it can't be any worse than the old APIs.

Authors note: Here's yet another installment of "From the Archives".  Clearly I haven't really felt like coming up with new ideas lately.  I blame the pandemic.  Seriously - I'm not going to be back in the office until at least next summer.  Even though restrictions have eased (at least for the time being), lock-down fatigue has definitely long since set in.

At any rate, this is another one on using Komodo with Vim emulation.  This was written on April 16, 2014, just a few days after the last one on the same topic.  These days I'm using Vim all the time, so none of this is relevant to me anymore.  However, it is a nice example of how it's possible to extend a good IDE to customize your workflow.

In this case Komodo is (or was) customizable using JavaScript, which is nice - lots of people know JavaScript.  The down side is that, to do actually useful thing, you also it also used XUL and SciMoz, the Mozilla Scintilla binding.  These are less commonly known, to put it mildly.

To be fair, Vim isn't much better on this score.  While it supports multiple scripting languages, the primary one is, of course, VimScript, which is...not a great language.  However, it's also quite old, quite well documented, and there are lots of examples of how to use it.  The VimScript API is also pretty stable, as opposed to Komodo, which was in the process of moving away from the XUL-based stuff when I stopped using it.  And really, a little VimScript will actually take you farther than you think.

In any event, I guess the idea is that it's good to know how to customize your editor, at least a little.  You know, sharpening the saw, knowing your tools, and all that.  Good stuff.  Enjoy!

Since upgrading to Komodo IDE, I've been looking a little more at customizing my development environment.  This is actually made somewhat easier by Komodo's "sync" feature, which will synchronize things like color schemes, key bindings, etc. between IDE instances via ActiveState's cloud.

Anyway, as part of this I've also been looking more at the Vim keybindings.  I've been a casual Vim user for a very long time, but I was never hard-core enough to do things like stop using the arrow keys or give up using ctrl+C and ctrl+V for copy and paste.  So now I'm trying to do just that.

Of course, Komodo's VI emulation mode is a pale imitation of what's available in Vim.  However, even that pale imitation is actually pretty good.  In fact, its even better than Komodo's documentation would lead you to believe.  In addition to the basic modal editing stuff, Komodo supports a decent range of movement commands, variants of change and delete commands, etc.  Basically, it supports everything I already knew about plus a lot more.  So now I'm trying to get that extra stuff into my muscle memory.

In the course of looking at some Vim command guides, I naturally came across some handy looking commands that Komodo didn't support.  So I'm going to try to fix that.

The first one is the reg command.  Vim's registers were something I hadn't really worked with before, but it turns out that they're not only pretty cool, but that Komodo actually has some support for them.  I only know this because the documentation mentions the key binding for the "set register" command.  However, it doesn't implement the reg command, so you can't actually see what's in any of those registers.

So, long story short, I fixed that with a macro.  Just create a new macro named "reg" in your "Vi Commands" folder in your toolbox and add the following code (this requires another macro, executed at start-up, containing the "append_to_command_output_window" function lifted from here):

var viCommandDetails = Components.classes['@activestate.com/koViCommandDetail;1'].
                                getService(Components.interfaces.koIViCommandDetail);
var count = new Object();
var args = viCommandDetails.getArguments(count);

append_to_command_output_window('');
append_to_command_output_window("--- Registers ---");

for (item in gVimController._registers) {
    if (args.length > 0 && args.indexOf(item) < 0) {
        continue;
    }
    if (typeof gVimController._registers[item] !== 'undefined') {
        append_to_command_output_window('"' + item + '   ' + gVimController._registers[item].trimRight());
    }
}

This allows you to type ":reg" and get a list of the current registers in the "command output" window in the bottom pane.

Another good one:

var scimoz = ko.views.manager.currentView.scimoz;
if (scimoz.selText.length == 0) {
    ko.commands.doCommand('cmd_vim_cancel');
} else {
    ko.commands.doCommand('cmd_copy');
}

This can be bound to ctrl+C and allow you to keep the default "copy text" behavior when there is text selected, and still work for Vim's "back to normal mode" when nothing is selected.

Author's Note: We're back with another installment of "From the Archives", the blog show where I declare writing bankruptcy and just post an old, half-finished article that's been sitting in my drafts for years.  This entry is from April 9, 2014.  This was in the midst of my long stint as a Komodo IDE user. 

One of my favorite things about Komodo was that it had pretty good Vim emulation.  I started using that because a few years before I'd spent a lot of time going back and forth between a Windows PC and a Macbook Pro.  The Macbook keyboard had that weird Apple layout going and it routinely messed with me, so I eventually gave up and decided to use Vim-mode because that's the same on both platforms.

Of course, things have changed since then.  I've become a full-time Vim user, and have all the fancy faux-IDE stuff set up.  I actually like it so much that I stopped using PHPStorm for work and switched to doing all my development in Vim.  So this post is no longer relevant to me, but it at least has a few handy links, so enjoy!

I've been a Vim user more or less since I started using Linux.  Mind you, I was never really a hard core Vim user.  I still use the arrow keys, for instance, and manage to get by on maybe a couple dozen keybindings and commands.  I have no clue how Vim's scripting or configuration systems work.  All I know about ctags is that they're a thing that exists.  So really, I'm more of a dabbler.

The other part of this is that I like at least a small amount of IDE in my normal working-day editor.  I kind of like having some sort of "project view" of my files, a code hierarchy viewer, some form of Intellisense, etc.  And while you can get most of the stuff I like in Vim, they're not there out of the box.  And even if you can get them, you can't count on having an obvious graphical way to manipulate them.  Typically, you just have to read the documentation to find out what the key bindings are to trigger everything.

So the upshot of this is that I use Komodo IDE with the Vi emulation setting.  This essentially turns on a Vim emulation mode that makes the editor modal and enables a lot of the standard Vim keybindings as well as a small subset of common commands.  So I get Vim goodness with all the convenience of a full IDE.  I had never really looked closely at just how much of Vim Komodo would emulate, though - I just knew it supported everything I commonly used.

Well, I had some extra time after finishing all my bug fixes the other day, and since really learning Vim has been on my list of things to do for, er, over 10 years, I decided to look up some Vim reference sheets and see how many of the commands and keybindings actually worked in Komodo.  Turns out it was a pretty decent amount.  (Note from the future: I didn't write down the details at the time and don't care enough to catalog now that I no longer use Komodo.  Suffice it do say that Komodo's Vi emulation was actually pretty good.  Maybe not as good as IdeaVim, but pretty good.)

Author's note: Here's another old article that I mostly finished, but never published - I'm not sure why.  This one is from way back on August 23, 2013.

I was working for deviantART.com at the time.  That was a very interesting experience.  For one, it was y first time working 100% remote, and with a 100% distributed team, no less.  We had people in eastern and western Europe, and from the east coast to the west coast of the US.  It was also a big, high-traffic site.  I mean, not Google or Facebook, but I believe it was in the top 100 sites on the web at the time, according to Alexa or Quantcast (depending on how you count "top 100").

It also had a lot of custom tech.  My experience up to that point had mostly been with fairly vanilla stuff, stitching together a bunch of off-the-shelf components.  But deviantART was old enough and big enough that a lot of the off-the-shelf tools we would use today weren't wide-spread yet, so they had to roll their own.  For instance, they had a system to do traits in PHP before that was actually a feature of PHP (it involved code generation, in case you were wondering).

Today's post from the archives is about my first dealings with one such custom tool.  It nicely illustrates one of the pitfalls of custom tooling - it's usually not well documented, so spotting and resolving issues with it isn't always straight-forward.  This was a case of finding that out the hard way.  Enjoy!

Lesson learned this week: object serialization in PHP uses more space than you think.

I had a fun problem recently.  And by "fun", I mean "WTF?!?"  

I got assigned a data migration task at work last week.  It wasn't a particularly big deal - we had two locations where user's names were being stored and my task was to consolidate them.  I'd already updated the UI code to read and save the values, so it was just a matter of running a data migration job.

Now, at dA we have this handy tool that we call Distributor.  We use it mainly for data migration and database cleanup tasks.  Basically, it just crawls all the rows of a database table in chunks and passes the rows through a PHP function.  We have many tables that contain tens or hundreds of millions of rows, so it's important that data migrations can be done gradually - trying to do it all at once would hammer the database too hard and cause problems.  Distributor allows us to set how big each chunk is and configure the frequency and concurrency level of chunk processing.

There's three parts to Distributor that come into play here: the distributor runner (i.e. the tool itself), the "recipe" which determines what table to crawl, and the "job" function which performs the actual migration/cleanup/whatever.  We seldom have to think about the runner, since it's pretty stable, so I was concentrating on the recipe and the job.

Well, things were going well.  I wrote my recipe, which scanned the old_name table and returned rows containing the userid and the name itself.  And then I wrote my migration job, which updated the new_name table.  (I'm fabricating those table names, in case you didn't already figure that out.)  Distributor includes a "counter" feature that allows us to trigger logging messages in jobs and totals up the number of times they're triggered.  We typically make liberal use of these, logging all possible code paths, as it makes debugging easier.It seemed pretty straight-forward and it passed through code review without any complaints.

So I ran my distributor job.  The old_name table had about 22 million rows in it, so at a moderate chunk size of 200 rows, I figured it would take a couple of days.  When I checked back a day or two later, the distributor runner was reporting that the job was only 4% complete.  But when I looked at my logging counters, they reported that the job had processed 28 million rows.  WTF?!?  

Needless to say, head-scratching, testing, and debugging followed.  The short version is that the runner was restarting the job at random.  Of course, that doesn't reset the counters, so I'd actually processed 28 million rows, but most of them were repeats.  

So why was the runner resetting itself?  Well, I traced that back to a database column that's used by the runner.  It turns out that the current status of a job, including the last chunk of data returned by the crawler recipe, is stored in the database as a serialized string.  The reason the crawler was restarting was because PHP's unserialize() function was erroring out when trying to deserialize that string.  And it seems that the reason it was failing was that the string was being truncated - it was overflowing the database column!

The source of the problem appeared to be the fact that my crawler recipe was returning both a userid and the actual name.  You see, we typically write crawlers to just return a list of numeric IDs.  We can look up the other stuff st run-time.  Well, that extra data was just enough to overflow the column on certain records.  That's what I get for trying to save a database lookup!

Author's note: I started the draft of this article way back on July 1, 2013. Sadly, it's still pretty relevant.

So I'm built myself a nice little web-based bookmarking app. I wanted something that would both give me some insight into how I use my bookmarks and also save me from worrying about syncing bookmarks between multiple browsers on multiple devices. And since I've regained my distrust of "the Cloud" with the demise of Google Reader, I decided to write my own. (Note from the future: Yes, I know seven years ago, but I still don't really trust cloud services.) If you're interested, check out the GitHub page. Maybe one day I'll make a real, official release of it. I call in Lnto, with "ln" being the UNIX "link" command and a tie-in to "LnBlog", the software that runs this site. (For LnBlog, the tie-in was to "ln" for the natural logarithm, i.e. "natural bLOG". Get it? I swear I thought it was funny at the time.)

One must-have feature for such an app is a "bookmark this page" feature. With native browser bookmarks, this is built in. With a web app...not so much. So the solution is to either write an extension for every browser I want to support (which is a lot of work), or just write a bookmarklet - a little piece of JavaScript that you can bookmark and run with a single click. Since this is a personal project that I'm doing in my limited free time, the latter seemed like the obvious choice.

There's just one problem here - mobile browsers. In addition to my laptop and my desktop, I have an Android phone and a Kindle Fire that I want to support. And while the actual bookmarklet code works just fine on all of those devices, actually bookmarking it isn't quite so easy. Because they're mobile browsers, you can't just drag the link to the toolbar as you would on the desktop.

Until recently, Firefox Mobile handled this well. (Author's note: We're back to the current time now, not 2013.) It would allow you to bookmark a bookmarklet like a normal bookmark. You just had to press the link and select a "bookmark this link" item from the menu. Then you could just bring up the bookmark screen when you were on a page and it would run the bookmarklet. However, with the updates for Firefox Mobile 81, that doesn't work anymore - the javascript: URL scheme doesn't seem to get executed when you invoke the bookmark. And other browsers don't seem to support bookmarking the bookmarklet in the first place. This link suggests that it's possible using bookmark syncing, but I'm not sure if that still works and I don't really want to turn on syncing anyway.

What I eventually did was just create a page that I can paste a URL into and it will do the same thing as the bookmarklet. It's not great, but it's serviceable. At some point, maybe I'll get around to creating an Android app. Then I'll have some native integration options to work with.

Author's Note: This entry from my archives was written on March 18, 2007 and has been sitting in my drafts folder ever since.  Not sure why I didn't publish it at the time.  I think I was going to add more, but never got around to it.  At any rate, this was back then RSS feeds were a trendy, new-ish thing and this article was supposed to be a less technical discussion of what they are and why they're good.

These days, of course, RSS is passé, and when people refer to a "feed", it's usually coming from Facebook, Twitter, Instagram, or whatever vendor-locked service the kids are using this week.  I find this sad.  The idea of the open web was so promising, but not that much really came of it.  Instead of being spoon-fed our information and entertainment by big media companies via broadcast and print media, we're now spoon-fed our information and entertainment via the internet by big tech companies.  And this time, the content is not selected by gate-keeping editors, but by AI algorithms that are tuned to feed us whatever will keep us clicking, with little to no regard for whether it's true, useful, or even remotely good for us.

For the record, I still use RSS feeds all the time.  I use the Tiny Tiny RSS aggregator, which is quite nice, to read the various blogs and news that I'm interested in following.  I have accounts with a few of the big social media platforms, but I rarely ever read them and never post anything.  I find them to be a huge time-sink and not especially conducive to good mental health, and so better off avoided.  Of course, your mileage may vary, but just keep in mind that you don't need to look at these sites - if anything truly important happens, someone will tell you about it.  I mean, unless you're a shut-in with no friends or family.  In that case, maybe social media is a good thing for you.  

At any rate, these were my thoughts in 2007.  Perhaps they'll be interesting or enlightening.  Or perhaps entertaining in their naivete.  Enjoy!

If you frequent tech sites or weblogs, you've probably seen the RSS icon or the XML feed icon .  You may also have seen other icons or text links referring to XML feeds, RSS, or even podcasts.  In fact, if you're using a web browser other than Internet Explorer, you may have seen one of these icons pop up in the address bar or status bar.  In this article, I will try to explain, in layman's terms, what RSS is and why it is so useful and important to the future of the internet.

What is RSS?

RSS stands for Really Simple Syndication.  As this name suggests, it is a syndication format.  

By "syndication," we mean essentially the same thing as when we talk about syndicated television shows.  A syndicated TV show is one that is shown on multiple independent channels at the same time, as opposed to being exclusive to a single network.  So for example, a syndicated show in the United States might be broadcast by NBC, Fox, the Sci-Fi channel, and USA all at the same time.

RSS works the same way.  An RSS file, usually referred to as a feed, contains a list of recent updates to a site.  The site operators publish this file on the web site and allow other people to subscribe to it, which is really just a fancy way of saying they automatically download it on a regular basis.  These people can then "republish" the information, either by incorporating it into their own sites or simply reading it into a desktop application.  The idea is that if the site's operators update the RSS feed every time they update the site, anyone who subscribes to it will automatically get the next "episode" the next time he downloads the file.

But what would I do with it?

If you are not already familiar with RSS, you may be wondering why anyone would bother with this.  After all, if you just want to read the updates of a site, isn't it just as easy to read the home page?  

At this point, you may be thinking that this doesn't sound much different from just visiting the web site in question.  After all, why would you want to bother with this RSS thing when you can just go to the site's home page like you've been doing for years?  

You wouldn't be wrong to think that.  If you're talking about just one site, with one set of updates to track, then RSS doesn't make any sense.  It would just be a different way of doing the same thing.

The beauty of RSS is that, unlike a web page, you can easily write a program to break up the information and organize it in a useful way.  For example, you can have a script on your web site that takes RSS news feeds from CNN, the BBC, and others and puts them all together in a news ticker on your home page.  You can also have programs such as RSS aggregators, which pull together news items from multiple sites and display them together so that you can browse them quickly and easily.  

I will discuss some other uses of RSS, including the trendiest of them all, Podcasting, later in this article.  (Note from the future: I never actually did that.)  But before that, we need to cover why RSS is useful and separate the fact from the hype.

A brief technical digression

What makes RSS so useful and so widely applicable is that it is a standard format.  It is an application of XML, the eXtensible Markup Language, which is an industry standard markup language for use with structured information.  I won't bore you with a description of XML, but the upshot of this is that RSS files all contain a certain set of standard information which is always marked with the same standard tags.  This means that a program can easily go through the file and pick out particular pieces of information, like the title of a particular news item, without having to pay any attention to what the title actually says or how it is formatted for display.  And because RSS is based on XML, there is already a wide array of programming tools that can be used to create and manipulate the files.

This is in stark contrast to web pages.  Although HTML, the markup language used to build web pages, has a standard set of tags, there is no standard for how a page is structured.  So while there are fixed ways of defining lists, tables, and paragraphs in HTML, there is no agreed upon way to say, "This is a news item, this is its title, and this is the link to its page in the archives."  (Note from the future: With the advent of HTML 5 this is no longer technically true.  However, semantic markup is not universally or consistently applied, so it's still close enough.)  So while a human being can easily look at a page full of items and determine where one ends and the next begins, there is no simple and general way for a computer to do that.  Because everyone is free to pick how they want to denote those things, a program would have to analyze the content of the page and figure out what the author of each page was thinking.  Needless to say, this kind of mind-reading is not something computers are particularly good at.

Getting past the hype

You know what the computing industry is best at producing?  It's not software, hardware, or anything else you can buy in the local office supplies store.  It's hype.  And to really appreciate how good RSS is, you have to get past the hype generated breathless pundits who seem to think it will cure cancer, feed the starving, and bring peace to the world.  (Note from the future: Nobody gives two hoots about RSS or XML anymore.  Hype has a very limited life span.)

From a technical standpoint, there is absolutely nothing revolutionary about RSS.  It's just a standard way of formatting a text file.  You could even create an RSS file in Windows Notepad if you really wanted to.  

And when you think about it, using RSS feeds is just a fancy name for putting a text file on your web site and then letting people download it and mess around with the information it contains.  How is that revolutionary?  We could do that 20 years ago.  

However, it is important to remember that "revolutionary" and "innovative" are not the same as "good."  RSS is good because it provides a standard way of transmitting certain kinds of information in a way that puts all the power in the hands of the consumer, not the publisher.  It's not the technology itself that's revolutionary, but rather the way people have chosen to apply it.

The real reason RSS and the whole "Web 2.0" thing are important is not because of the technology, but because of the paradigm.  That paradigm is: the user is king.  In an era where Hollywood and the music industry are trying to tell you when and where you're allowed to use the music and movies you paid for, open standards put that power in your hands.  (Note from the future: People have forgotten about that aspect, so now "Web 2.0" means "the site uses AJAX."  Also, substitute "Hollywood" for "big tech companies.")

Author's note: Here's another article from the archives. This one was written way back on March 17th, 2007. At the time I was still working for the county government and using Linux on all my home computers. This rant came out of frustration at the interaction of not-fantastic bespoke software and annoying semi/anti-features in a commercial flash drive.

We don't have that in the Linux world. Most of our software is free or open-source and is installed from a single trusted distributor. We don't have to worry about whether a package is bundled with spyware, a crappy browser toolbar, or a useless systray icon.

This was brought home to me today when I had to deal with a user's flash drive at work. His department has a number of laptops that can't connect to our network, but this crappy application that the state foisted on us requires them to move data from back and forth between our file server and the laptops. The easy solution to this is just to use flash drives.

Well, this user's flash drive broke and the department didn't have any spares, so he went out and bought a 1GB Cruzer Micro. Nice enough drive, plenty of capacity for his purposes. The only problem was, it didn't work with The Crappy Software Foisted On Us By The State (TCSFOUBTS, pronounced 'ticks-fouts').

You see, TCSFOUBTS is one of those half-baked, government consultingware type of systems. It has a bazillion features and is highly customizable. On the down side, it's slow as all hell, has a terrible user interface, and is difficult to configure. For example, some customization options require manually editing MS Access database tables and quite a number involve hand-editing INI files or other text files.

Now, having planned ahead, the TCSFOUBTS people built in a feature to let users automatically copy their data onto a removable drive with just a couple of clicks. However, by "removable drive," what they really meant is "a path." For this feature to work, you have to configure TCSFOUBTS with the drive letter for your removable device.

By now, you probably see where I'm going with this. TCSFOUBTS needs the drive letter, which has to be configured by the system administrator. However, unlike the old flash drive, the Cruzer Micro has a bootable partition that displays as a CD-ROM drive. (Note from the present: this was before booting from USB was a common thing.) Of course, that's the first partition, and so get the drive letter where TCSFOUBTS wants to copy its files.

What was on that first partition? I don't remember. Some useless utility software that nobody asked for. One of those "value-adds" that they put on flash drives because they want to use the space for something. But it meant that the drive was stuck with an unusable partition that I didn't immediately have the ability to remove. So for the user to use this drive, they would have to hand-edit some INI file, and then edit it back when they got a different drive. And since the users for this system were all non-technical, that sounded less than appealing.

Eventually the user decided to just not bother and requisition an "official" drive. It was just a frustrating series of events. It goes to show how a few misplaced assumptions can completely mess up a user experience.

Author's note: This is based on some notes and links I started collecting in November of 2015.  The bulk of the commentary is actually relatively new - from last year, August 15th, 2019 - so it's not too out of line with my current thinking.

As might be clear from my review of Bertrand Meyer's Agile!: The Good, the Hype, and the Ugly, I've been rethinking the whole agile development craze that has swept the industry.

There are a number of good presentations online questioning the "agile" movement.  For a more provocative point of view, I recommend Erik Miejer's One Hacker Way talk.  Dave Thomas, one of the "pragmatic programmers", also has a good talk on the topic. There's also a good one by Fred George on the hidden assumptions of agile.

My current thought (note: we're back to 2019 now) is that "agile" has become pretty much a meaningless buzz word.  Pretty much everybody is "doing agile" now - or at least claiming they do.  It's come to mean "anything that's not waterfall".  And we all know that "waterfall" doesn't work, which is why everyone is doing "agile".  (Side note: Winston Royce, in his paper that initially described the waterfall process, actually says that it doesn't really work.  But, of course, that didn't stop people from trying.)

Not that agility is a bad concept.  It isn't - being flexible is a good thing.  Responding to change is almost a requirement in most shops.  The values in the Agile Manifesto are all good things to emphasize.  But none of that amounts to a process.  It's just a loose collection of principles and ideas that are useful, but aren't a road-map for how to build software.

That's why, in practice, most shops that "do agile" are using some variation on Scrum.  And while I have no problem with Scrum per se, it's hardly the be-all and end-all of development processes.  In fact, the main problem with Scrum is probably that it's not a software development process - it's more of a project management framework.  It doesn't have a whole lot to say about the details of how to code, how to test, how to manage defects and other quality issues, how to manage releases, etc.  It's up to each shop to figure that stuff out for themselves.

Of course, that's not bad.  Every business is different and you should expect that you'll have to adapt any process to a certain extent.  Scrum is useful in that it gives you a framework for tracking what needs to be done and creating a feedback loop to improve your process.  But you still have to actually use that feedback loop to improve your process, i.e. you have to do the hard work of self-improvement.  Simply going through the motions of what the "agile consultant" says you should do it's going to cut it.  As with everything else in life, there are no shortcuts.

Author's note: Welcome to another episode of "From the Archives". This is the stub of an article that I wrote twelve years ago, on April 25, 2008. At the time, KDE 4.x was freshly into stable release. I was a use and fan of KDE at the time, and there had been a lot of hype about how awesome version 4 was going to be. My initial reaction, however, was...not so great.

This is actually slightly relevant because I have resurrected the GUI on my "home desktop", by which I mean my "home server". This is the box sitting under my desk in the basement that runs Ubuntu 18.04 and runs various web and media server software. It does have a GUI installed, but I hadn't really used it in years - in part because GNOME didn't really work well on it. This isn't super surprising, since it's an old box with just the integrated graphics chip. But it's got more than enough memory and processing power for the workload I want it to do, so there's not really any point in upgrading.

Anyway, due to the current CONVID-19 pandemic I'm now working from home and sitting at that desk all day every day, so I decided to fix up my desktop. Part of my cleanup process was to axe GNOME and install the Trinity Desktop Environment (TDE). This is a project I just recently discovered and immediately fell in love with. It's essentially a fork and continuation of KDE 3.5. It's since evolved into its own thing, apparently, but it's still noticeably KDE-like, so I'm very comfortable in the UI. Just like the original KDE 3.5, TDE is powerful, intuitive, generally pleasant to use, and works quite well on my not-so-new system. It doesn't have the fancy graphical bells and whistles, but I never really cared much about that anyway. I would definitely recommend it to any old-school KDE 3.x fans.

Anyway, here are my thoughts from the time. To be fair, I haven't used "real" KDE to any extent since this, so I'm sure all of my complaints have been addressed. But then again, I don't really care. I'm happy with TDE. Enjoy!

Kubuntu 8.04 was released yesterday (note: again, this was twelve years ago). That means it's upgrade time for me again.

This time around, Kubuntu comes in 2 varieties: the "rock-solid" KDE 3 version, and the KDE 4 remix. I had been intending to get on the leading edge and install the KDE 4 version. However, just to be on the safe side, I decided to give the live CD a try first. And after messing around with it for half an hour or so, I'm glad I did.

Bottom line: I think I'm going to wait for KDE 4.1. Or maybe 4.2.

I just don't care for 4.0.3. It definitely looks different...but not better. I just didn't see any new features that looked even remotely interesting, let alone compelling. The splash screens were kind of nice, and the plasma widget effects on the desktop were pretty neat, but that's about it.

There seemed to be a lot more down sides. Of course, I'm not sure how many of these are KDE 4 issues and how many are Kubuntu issues, but I found them annoying either way. Here's my list:

1. The window style for the default desktop theme is unbearably ugly. It's too dark and too monochromatic. I guess somebody must like it, but it really bothers me.
2. Where's the control panel? I don't see it and the context-sensitive configuration panels don't have all the options in them.
3. In fact, where the heck are the other options?
4. What the heck happened to Amarok and the other applications? The UI is completely different and it feels like half the features are missing.

I could go on, but why bother? There's just no reason for me to upgrade at this point.

Author's Note: Well, folks, it's time for another episode of "From the Archives"! The show where I take those old draft blog posts that I never published, flesh them out and add this lame intro, and then publish them so that I don't have to bother coming up with new content!

Today's post is my review of my very first smart phone, a Palm Centro. Of course, it was only sort of smart, because I was too cheap to pay for a data plan, which was kind of pricey at the time. Still, I really liked that phone. On the next upgrade (because Verizon was still doing the "upgrade every two years for an absurdly low price" thing in those days), I actually ended up downgrading to a dumb-phone. In retrospect, that was a mistake. But it's OK, because two years later i finally got a real smart phone (I believe it was the Samsung Galaxy Nexus) and never looked back.

Anyway, I figured this might be an interesting bit of retrospective trivia. This post was written on December 1, 2008, so cellt phones have obviously changed a lot since them. Enjoy!

Well, early last month I had my cell phone upgrade day. I dropped into the Verizon Wireless store on election day and picked out new phones for Sarah and myself.

For the first time since we signed up with Verizon, I got 2 different phones. For Sarah, I eventually settled on the Samsung Sway. Her requirements were pretty basic (camera, text and picture messaging, downloadable games and ringtones) with the one exception that she didn't want a clamshell design. Our last 2 phones were clamshells and she wanted something more like our first phone - a stick phone. Of course, they don't really make stick phones anymore, so I figured the sliding design of the Sway would be close enough. So far, she seems to like it well enough. The only problem is that Verizon, in their infinite suckiness, seems to have disabled the ability to set MP3 ringtones from files stored on the microSD card. Typical customer-hostile behavior.

I, on the other hand, totally geeked out this time and got a basic smart phone. I'd been going back and forth for a while, but I ended up going with the Palm Centro. So far I absolutely love it.

The Centro may be the low-end model (as compared to, say, the Treo), but it gives me pretty much everything I've always wanted in a cell phone. The reason I chose a smart phone was that I was tired of being limited by what Verizon decided I should be able to do. With a feature phone, you are limited not just by the hardware, but also by your provider's firmware. If Verizon decides you shouldn't be able to create custom ringtones or copy files over OBEX, you're out of luck. But a smart phone is just a very small computer - that's the main selling point. You have enough control over the system to add features and do some customization.

Let me start with the things I don't like about the Centro. There aren't many.
1) The single most annoying thing is the battery cover. It feels a bit flimsy and is hard to get off. This wouldn't be a problem, except that you need to get to the battery relatively often.
2) Relating back to point 1, you need to open up the battery cover to get to the microSD port. At least you don't need to take the batter out, but it's still annoying. The fact that the Centro has a side door for the card which you can't open with the battery case on just adds insult to injury.
3) Again, relating back to the battery, is the fact that the phone will actually crash on occasion. I haven't had any problems with the bundled software so far, but a few add-on programs have caused the phone to lock. And, of course, the only fix for that is to reboot the phone by taking out the battery.
4) Three words: non-standard data port. Seriously, what's wrong with micro USB? Plus, my data cable has a habit of falling out if I move the phone around the desk. Maybe that's just me.

Those are my only real non-fixable complaints. There are some other annoyances, but many of those are actually fixable by installing additional software. I'll get into a few of those must-have toys and utilities in another post. (Author's note: it's now 12 years later, so yeah, that's not gonna happen.)

So what do I like about the Centro? Pretty much everything else. On the hardware end, it's fairly nice. The touch screen is very handy and the QWERTY keyboard, despite being extremely small, is actually surprisingly easy to use. Yet despite the decent screen size and keyboard, the Centro still isn't that much bigger than a feature phone. It also has something I've never seen before - a physical switch to set the phone to vibrate mode. Still not entirely sure whether I like that or not, but it's certainly different.

On the software side, you actually get a fairly decent set of base programs to work with. The included PIM software - calendar, address book, todo list, memo app - actually isn't too bad. They also throw in a copy of Documents To Go, which can open MS Office files and PDFs. Unfortunately, the PDF viewer kind of sucks, but the Word document viewer seems pretty decent. And, of course, I was able to install a NES emulator, which is awesome to have right on your phone.

Here's another "from the archives" post that's been sitting in my drafts folder since March 27, 2007.  At the time I'd been getting into playing Sudoku and was running Kubuntu on my home desktop and laptop.  All I wanted to do was play Sudoku on my computer.  And apparently it didn't go very well.  At the time I wrote:

Why is it that there don't seem to be any good desktop sudoku games for Linux? At least, I haven't found any that meet my requirements. The three in Ubuntu's repositories sure don't.

My requirements for a sudoku game are pretty simple. Beyond the basics of creating a new puzzle and telling me if I'm right when I complete it, I only want one thing: notes. I just want an easy way to add notes to unfilled squares so that I can quickly reference the possibilities.

You wouldn't think that would be such a tall order. However, of the three sudoku games in the Ubuntu repositories, none of them do this. Gnusudoku doesn't even claim to support notes. GNOME-sudoku claims to support them "simply" by clicking at the corner of a box. However, it doesn't work. Maybe you have to run it under GNOME?

And last but not least, we have ksudoku. This one does support notes, and they do work, but they suffer from the same problem as the rest of this app: the user interface makes absolutely no freaking sense at all. Even this might be forgivable if there were a manual or even some kind of guide on the website, but there isn't. The status bar does flash back and forth between "helpful" tips, one of which says to use "RMB" to make notes, but that's it. It took me a few minutes and some experimentation to figures out that "RMB" is supposed to be an abbreviation for "Right Mouse Button". But even with that, I still can't figure out how the hell notes are supposed to work. I can't seem to get more than one number into a note and things disappear if I right-click too many times.

I assume those problems have probably been fixed by now.  I mean, it's been almost 13 years, so either they're fixed or the project is dead.  But I seem to recall this being sort of emblematic of open-source games at the time: there weren't that many of them and even fewer were any good.

By now, I'm sure that the situation is much better, if only because enough time has passed for more games to mature.  But, unfortunately, I don't really care anymore.  These days I don't want to sit down in front of a computer just to play video games.  I just don't have that kind of time.  If I'm going to play a game, it's the kind that runs on my phone and which can be dropped at a moment's notice.

Coincidentally, one such game that I just downloaded the other day is Open Sudoku, which is actually pretty nice as Android Sudoku apps go.  It's open-source and you can find it on F-Droid as well as in the Play store.  Like most open-source games, it doesn't have a lot of frills or fancy graphics, but it's easy to play and does the job.

Author's note: Here's another little rant that's been sitting in my drafts folder for years. Twelve years, to be precise - I created this on March 28, 2007. That was toward the end of my "government IT guy" days.

I'd forgotten how much of a pain the internet filtering was. These days, I hardly think about it. The government job was the last time I worked anyplace that even tried to filter the web. And e-mail filtering hasn't been something I've worried about in a long time either. These days, the filtering is more likely to be too lax than anything else. And if something does get incorrectly filtered, you generally just go to your junk mail folder to find it. No need for the rigamarole of going back and forth with the IT people. It's nice to know that at least some things get better.

I'm really starting to hate spam filters. Specifically, our spam filters at work. And our web filters. In fact, pretty much all the filters we have here. Even the water filters suck. (Actually, I don't think there are any water filters, which, if you'd tasted the municipal water, you would agree is a problem.)

I asked a vendor to send me a quote last week. I didn't get it, so I called and asked him to send it again. I checked with one of our network people, and she tells me it apparently didn't get through our first level of filters. So she white-listed the sender's domain and I asked the guy to send it again. It still didn't get through.

As I've mentioned before, our web filters also block Wikipedia.

Note: This is a short entry that's been sitting in my drafts folder since March 2010, i.e. from half a career ago. My "new" job at the time was with an online advertising startup. It was my first and only early-stage startup experience. In retrospect, it was useful because it exposed me to a lot of new thing, in terms of not only technology, but people, processes, and ways of approaching software development (looking at things from the QA perspective was particularly eye-opening). It was not, however, enjoyable. I've also worked for later-stage startups and found that much more enjoyable. Sure, you don't get nearly as much equity when you come in later, but there's also less craziness. (And let's face it, most of the time the stock options never end up being worth anything anyway.)

Wow. I haven't posted anything in almost six months. I'm slacking. (Note: If only I'd known then that I wouldn't publish this for nine years...)

Actually, I've been kind of busy with work. I will have been at the "new" job for a year next month. The first six months I was doing the QA work, which was actually kind of interesting, as I'd never done that before. I did some functional test automation, got pretty familiar with Selenium and PHPUnit, got some exposure to an actual organized development process. Not bad, overall.

On the down side, the last six months have been a bit more of a cluster file system check, if you get my meaning. Lots of overtime, throwing out half our existing code-base, etc. On the up side, I've officially moved over to development and we're using Flash and FLEX for our new product, which are new to me.

The good part: FLEX is actually not a bad framework. It's got its quirks, but it's pretty powerful and, if nothing else, it beats the pants off of developing UIs in HTML and JavaScript. And while it's not my favorite language in the world, ActionScript 3 isn't bad either. It's strongly typed, object oriented, and generally fairly clean.

The bad part: Flash is not so nice. It wasn't quite what I was expecting. I guess I assumed that "Flash" was just design environment for ActionScript programming. Actually, it's more of an animation package that happens to have a programming language bolted onto it. The worst part is that our product requires that we do the Flash portion in ActionScript 2, which seriously sucks. I mean, I feel like I'm back in 1989. And the code editor in Flash CS4 is...extremely minimal. As in slightly less crappy than Windows Notepad. I am sersiously not enjoying the Flash part.

(Note: On the up side, none of this matters anymore because Flash is now officially dead.)

Note: This is an article I started in October of 2012 and never finished. Fortunately, my feelings on the issue haven't changed significantly. So I filled it out into a real entry. Enjoy!

As I alluded to in a (not so) recent entry on switching RSS readers, I'm anti-cloud.

Of course, that's a little ambiguous. The fact is, "cloud" doesn't really mean anything anymore. It's pretty much come to refer to "doing stuff on somebody else's server." So these days we refer to "having your e-mail in the cloud" rather than "using a third-party webmail service," like we did 15 years ago. But really it's exactly the same thing - despite all the bells and whistles, GMail is not fundamentally different than the Lycos webmail account I used in 1999. It still amounts to relying entirely on some third-party's services for your e-mail needs.

And if the truth were known, I'm not even really against "the cloud" per se. I have no real objection to, say, hosting a site on an Amazon EC2 or Windows Azure instance that I'm paying for. It's really just the "public cloud." You know, all those "cloud services" that companies offer for free - things like GMail and Google Calendar spring to mind.

And it's not even that I object to using these servies. It's just that I don't want to rely on them for anything I deem at all important. This is mostly because of the often-overlooked fact that users have no control over these services. The providers can literally cut you off at a moment's notice and there's not a thing you can do about it. With a paid service, you at least have some leverage - maybe not much, but they generally at least owe you some warning.

There are, of course, innumerable examples of this. The most recent one for me is Amazon Music. They used to[i] offer a hosting service where you could upload your personal MP3 files to the Amazon cloud and listen to them through their service. I kinda liked Amazon Music, so I was considering doing that. Then they terminated that service. So now I use Plex and/or Subsonic to listen to my music straight from my [i]own server, than you very much!

As a result, I have my own implementation of a lot of stuff. This includes running my own calendar server. This is a project that has had a few incarnations, but that I've always felt was important for me. Your calendar is a window into your every-day life, a record of every important event you have. Do you really want to trust it to some third party? Especially one that basically makes its money by creating a detailed profile of everything you do so that they can better serve ads to you? (I think we all know who I'm thinking of here....)

For several years I used a simple roll-your-own CalDAV server using SabreDAV. That worked fine, but it was quite simple and I needed a second application to provide a web-based calendar (basically a web-based CalDAV client). So I decided to switch to something a little more full-featured and easier to manage.

So these days, I just run my own OwnCloud instance. At it's core, OwnCloud is basically a WebDAV server with a nice UI on top of it. In addition to nice file sync-and-share support, it gives me web-based calendar and contact apps with support for CalDAV and CardDAV respectively. It also has the ability to install additional apps to provide more features, such as an image gallery, music players, and note-taking apps. Most of the more impressive apps are for the enterprise version only, or require third-party services or additional servers, but all I really wanted was calendar and contact support.

To get the full experience, I also use the OwnCloud apps on my laptop and phone to sync important personal files, as well as the DAVx5 app on my phone to synchronize the Android calendar and contacts database with my server. Overall, it works pretty well and doesn't really require much maintenance. And most important, I don't have to depend on Google or Amazon for a service that might get canned tomorrow.

Note: This is another "from the archives" essay that I started two years ago.  It was inspired by this talk by Kevlin Henney as well as the general low quality of the unit test suite in the product I was working on at the time.  If you're browsing YouTube, I generally recommend any of Kevlin Henney's talks.  He's a bit like Uncle Bob Martin - insightful and entertaining with a deep well of experience to back it up.

Does your code have GUTs?  It should.  If you're a professional programmer, you really should have GUTs.  And by "GUTs", I mean "Good Unit Tests".

Unless you've been living under a rock for the last decade or two (or you're just new), the industry has decided that unit tests are a thing all "Serious Engineers" need to write.  For years I've been hearing about how you need to have unit tests, and if you don't you're not a real programmer.  So I decided to get with the program and start writing unit tests.

And you know what?  I failed miserably.  It was a serious train wreck.

Why did I fail?  Because I listened to those blog posts and podcast episodes that just tell you to write unit tests, and then I went out and just tried to write unit tests.  My problem was that I didn't know what a good unit test looked like.  So I wrote tests, but they were brittle, hard to understand, hard to maintain, and not even especially useful.  The result was that after a while, I concluded that these "unit test" people didn't know what they were talking about and had just latched onto the latest fad.  And then every year or two I would try again and reach the same conclusion.

So the real question is not "do you have unit tests?", but "do you have good unit tests?"  Because, as I alluded to above, bad unit tests are not necessarily better than no unit tests.  A bad unit test suite will break all the time and not necessarily give you any useful information, which makes maintaining it more costly than just dropping it.  

What do GUTs look like?

Good unit tests have three main properties:

1. Readability
2. Maintainability
3. Durability

What does this mean?  Well, readability means just what it sounds like - that you can read and understand the test cases.  Without being an expert in the system under test, you should be able to look at a test case and be able to figure out what behavior it's testing.

Likewise, maintainability means the same thing as it does for production code - that you should be able to update and expand the test suite without undue effort or pain.

Durability simply means that your tests should not break willy-nilly.  Obviously there are lots of potential changes you can make to code that would break your tests, but the tests should not break unless they really need to.  So a durable test suite should not start failing because internal implementation details of the code under test were changed.

Guidelines for writing tests

So how to you write tests that have those three properties?  Well, I have a few guidelines and common practices that I use.  At work, I routinely find myself suggesting these things during code reviews.  Some people might disagree, but I've found these to be helpful and I'll try to justify my opinions.  Hopefully you'll find them useful.

Note: for the sake of simplicity and consistency, examples will be of a blogging system written in PHP, but these principles and ideas are by no means specific to such a system.  I just use that because I happen to have plenty of examples at hand so that I don't have to come up with fake ones.

Test names

Naming your tests is an important, but often-overlooked aspect of test writing.  Your test cases should have descriptive names.  And by "descriptive", I mean it should tell you three things:

1. What behavior is being tested.
2. What the conditions of the test are.
3. What the expected outcome is.

If you use the "generate test cases" feature of your IDE, or something like that, you might end up with test names like testSaveEntry or testPublishEntry.  These are bad names.  For one thing, they only tell you the name of the function they're testing.  For another, they guide you into a convention of a one-to-one mapping of test classes and methods to production code classes and methods.  This is limiting and unnecessary.  You should have as many test classes and methods as you need.  I sometimes have an entire test class just to test one production method.  I don't recommend that as a general rule, but there are cases where it makes sense.

When in doubt, I recommend choosing a test naming convention.  I often use "test<name of method under test>_When<conditions of test>_<expected result>".  So, for example, if I was writing a test to check if the publishEntry() method throws an exception when the database update fails, I might name it testPublishEntry_WhenDbUpdateFails_Throws.  Or if I wanted to test that a text sanitizing function HTML encodes any angle brackets that it finds in the input, I might call it testSanitize_WhenDataContainsHtml_ReturnsDataWithTagsEscaped.  Obviously you can use a different convention, or no convention at all, but the point is that the test name tells you everything you need to know about what is being tested.

One thing to note here: the above names are kind of long.  I know - that's OK.  Remember that these are test names.  They're not methods that somebody is going to be calling in other code.  Nobody is ever going to have to type that name again.  So don't get hung up on the length.

Also of note, you should think about what the expected behavior is.  Things like testSaveEntryWorks are not enlightening.  What does "work" mean?  The same goes for things like testSaveEntryMakesExpectedDatabaseCalls.  OK...what are the expected database calls?  If you can't come up with something specific, that probably means you need to think more about your test, maybe even break it into multiple tests.

A good guideline to keep in mind is that it should be possible to write a pretty-printer that can read the names of all your test methods and print out a low-level specification for your system.  Ideally, you should be able to figure out how everything works just by looking at the test names.

Granularity

Big tests are bad.  Why?  Because the bigger the test is, the more things can break it.  This is the same reason that big functions and methods are bad - the more they do, the more opportunity there is for something to go wrong.  So if you can, it's better to keep things small.  Just as each function should do one thing, and do it well, each test should test one thing, and test it well.

And when I say "one thing", I really mean one thing.  Ideally, each test should have one, and only one, reason to fail.  The goal is that when a test fails, it should be pretty obvious what went wrong.  So if the only thing a test asserts is that X = Y, and it fails, then you can be pretty sure X isn't equal to Y.

On the other hand, if you're asserting a bunch of different things, it's harder to pinpoint the problem.  Furthermore, since most test frameworks will fail the test at the first failed assertion, you can end up with one failure masking another, i.e. the first of several assertions fails, so you fix it, and then the next assertion in that test fails, etc.  

So if you need to check a bunch of things, then write a bunch of tests.  Don't try to cram them all into one test.

Don't assert too much, but always assert something

A corollary to rule that a test should only test one thing is that a test should test at least one thing.  In code reviews for junior developers, I sometimes see tests that don't actually contain any assertions.  They have a bunch of setup code that configures mocks to make a method run, but no explicit assertions or call expectations on mocks.  All it really asserts is "this method doesn't throw an exception."

Needless to say, that's not a very strong assertion.  Heck, I can write methods that don't throw an exception all day long.  If the test fails, all you have to do is delete all the code in the method you're testing and there you go - it won't throw anymore.  Granted, your application won't do anything, but at least the test suite will pass.

So always make sure you're making some explicit assertion.  Otherwise, there's no point in writing the test.

Readability

Ideally, your tests should be a low-level specification - they should be clear and detailed enough that you could hand the tests to a developer and they could reverse engineer every part of your system based on that.  You probably won't actually accomplish that, but that's the goal we're shooting for in terms of coverage and clarity.

So given that, it's obviously very important that tests be readable.  A test that's hard to read isn't doing its job of communicating the intent of the system.  But what, exactly, does "readable" mean?

In my view, it's a combination of things.  Obviously, the action of the test should be clear.  You should be able to look at the test and easily be able to see what it's doing without having to search around through other code.  But you also want the test to be simple enough that you can easily understand it and not miss the forest for the trees.

One simple strategy to address this is to adopt a test structure convention.  This is typically an "arrange-act-assert" layout, where you group all of the setup code, action of the code-under-test, and the assertions into their own sections, so it's clear where each one starts and ends.  Where people sometimes get into trouble with this is when using mocking frameworks that require you to declare expectations and assertions on mocks up-front before they are called (you usually have to do "arrange-assert-act" in that case, but it's the same idea).  What I often see is people declare some a mock, declare some some assertion on it, then declare another mock with it's assertions farther down the test method, etc.  This makes the test harder to read because you have to hunt through all the setup code to determine which mocks have assertions on them.  There isn't a single place you can look at to see the expected result.

Another good strategy is the judicious use of utility functions.  This is especially the case if you have lots of similar tests with a substantial amount of setup code, which is not an uncommon case.  So, for example, if you have a method that calls a remote web service, you might want to have a bunch of different test methods that check all the different error conditions and return values, the mock object setup for many of those is probably very similar.  In fact, it might be the same except for one or two values.  You could handle that by just doing a copy-and-paste of the setup code and changing what you need, but then you end up with a sea of almost identical code and it becomes harder to see what the differences between the tests are.  The solution is to encapsulate some of that setup code in a utility method that you can just call from each of the tests.  This allows you to have your mocks be configured in one place and keep all those extraneous details out of the test body.

The one thing to watch out for with utility methods is that they should be as simple as possible.  When I can, I try to make them take no parameters and just give them a name that describes how they configure my mocks.  Remember that all the setup doesn't have to be encapsulated in a single function  - you can call multiple utility functions that configure different things.  In cases where it makes sense for the utility methods to take parameters, I try to limit how many they take, so as to keep the scope of what the function does clear.  And if you find yourself putting conditionals or loops in your utility functions, you might want to think about your approach.  Remember, the more logic there is in your tests, the more likely it is that your tests will have bugs, which is the last thing you want.

Conclusion

These are just some of the techniques and considerations that I employ when writing unit tests.  They've been working for me so far, but your mileage may vary.

My main goal here is just to give some ideas to people who are new to this unit testing thing.  Remember, writing unit tests is a skill that you have to learn.  This is especially the case when doing test-driven development.  Despite what you may have been led to believe, it's not a simple and obvious thing that you can just do cold.  So be patient, try different things, and take your time.  It's the journey, not the destination, that's the point.

Note: I started this entry two years ago and it's been sitting in my drafts folder ever since.  However, while the links might not be news anymore, the underlying issue is the same.  So I cleaned it up for another From The Archives entry.

A while back, there was a story going around about how the guy who invented the password strength rules that you see all over the web now regrets it.  That made me think about how we approach these kinds of issues and the advice we give to non-technical users.

Security is one of those areas of computing where there are a lot of cargo cults.  Relatively few people, even among IT professionals, seem to have a good handle on how to secure their systems.  So they rely on guidelines like these from the "experts", often following them blindly without any real understanding of the rationale.

And you can't really blame them - security is hard.  Even knowing what you need to defend against can be a tall order.  And with some of the biggest companies in the world being compromised left and right (for example, the Equifax hack, which should scare the heck out of you if it doesn't already), it's clear that this is not a resource problem that you can just buy your way out of.  Not even big tech companies are immune, so what chance does the average user have?

Well, unfortunately, for things like the Equifax breach, the average user doesn't have much to say about it.  Once a third-party has your personal information, you really have no choice but to rely on them to secure it.  And if they don't do a good job, well...you're sorta just out of luck.  I mean, you can always sue them, but let's be realistic: for private individuals, the amount of time and money required to do that is prohibitive.  It's cheaper and less painful to just absorb the loss and get on with your life.

Passwords are a different story, though.  Those are one of the few pieces of security that are (mostly) under the control of the user.  So we as professionals can offer some guidance there.  And if the top passwords revealed from various database breaches are any indication, we should offer some.

These days, there's really only one piece of advice that matters: get a password manager and use it for everything.  I like KeePass, but 1Password, LastPass, and a number of other good programs are available.  These days we all have more website logins than we can realistically remember, so it's impossible to follow the old advice of using strong passwords AND not reusing them AND not writing them down.  By using a password manager, we compromise on the "not writing it down" part and write down our passwords securely so that we can keep them strong and unique without making our lives difficult.

Of course, there will always be a few passwords that you just need to remember.  For instance, the master password for your password manager.  For these, the standard advice is to use long passwords containing number, letters, and special characters.  Probably the easiest way to do this and still keep the password memorable is to use a passphrase.  So rather than one word, use a short phrase containing several words and insert some punctuation or other special characters.  For example, the password Bob has _17_ "Cats"! isn't that hard to remember, but it's 20 characters long and  contains letters, numbers, capital and lower-case letters, punctuation, and spaces.  Yeah, it's harder to type and remember than "12345", but it's way easier than something like "UD09BhbjH7" and it fulfills the complexity requirements.

For more important accounts, you can also do things like enabling two-factor authentication, which adds another layer of security.  Typically this involves sending a code to your phone via text message or an app like Google Authenticator and entering that when you log in.  Even this isn't fool-proof (see SIM swapping scams), but it's one more hoop that someone trying to access your account has to jump through.

So forget the annoying rules about changing passwords every month and things like that.  Pick a handful of good passwords for the stuff you really need to type out and just use a password manager for everything else.  There's no reason to remember a bajillion obscure bits of information if you don't need to.  After all, that's why we have computers in the first place.

Author's note: Here's another random "from the archives" post.  This is from May 6, 2007.  It's more of a "personal reminiscience" type post, more in keeping with the "personal journal" nature of a blog than my usual subject matter.  But hey, it's my website, so I can post whatever the heck I want.

I live in Corning. It's a small city in central New York, about 30 minutes from the Pennsylvania border. As locations for software developers and other technologists go, it's not quite in the middle of nowhere, but it's close.

However, there are still advantages to be had. For example, there are the air shows. It just so happens that the Elmira/Corning regional airport is also the location of the National Warplane Museum (note: that has changed - now it's the Wings of Eagles), and the airport just happens to be across the street from the local mall. So when we went shopping on Sunday, we got a nice view of the day's airshow while we were driving around. The part we saw was a bright red biplane doing loops, barrel rolls, flying in low over the highway exit. It was actually very cool. I wish I'd had a camera....

Commentary: I remember watching this plane as my wife and I walk though the parking lot of the Tops supermarket in the plaza across the street from the mall.  I don't remember exactly why we were there (it wasn't our normal market) or what we'd done before, but I do remember that it was a sunny afternoon and I kept looking up at the red biplane doing loops and barrel rolls overhead.  It was a symbol of freedom - rolling unfettered through the blue sky, no cares about what was going on below.  I was a little jealous - I'd been feeling trapped in a job that I increasingly disliked and I wanted to be like that plane.

It's funny how images like that will stick with you sometimes.  It's been seven years, but I still remember that plane, even though I don't remember any of the context.  In fact, the only other thing I remember doing that day was working in my garden and thinking about that plane.  Perhaps it's not a coincidence that my garden was one of my main sources of escapism....

Author's Note: This is another entry in my "from the archives" series.  I've decided it's time to clear some of those out of my queue.  This was an unfinished stub entry I last edited on April 20, 2007, when I was still working in county government.  I actually still have some recollection of the incident that inspired this entry, nearly seven years later.  There wasn't much to the original entry itself (as I said, it was kind of a stub), so this post will feature some commentary on my own comments.  Yeah, I could try to actually finish the post, but I only vaguely recall what my original point was and I'd rather make a different one now.

Question: what is the purpose of having a "help desk" when all they do is forward support requests to the programmers and analysts? Isn't that the opposite of help?

I admit that's not entirely fair. Our help desk does all sorts of things and often solves small problems on their own. So it's not like they're just a switchboard.

It just annoys me when they pass on a request without even trying to get any more details. For instance, when they get e-mail requests and just copy and paste the body of the e-mail into our help desk software. Then they print out the ticket on a sheet of blue paper and lay it on the desk of the programmer or analyst who last worked with the department or system in question.  There's no indication that they even asked the user any questions.  It seems like I'd be just as well off is the user just e-mailed me directly.  At least we wouldn't be killing trees that way.

Commentary: I don't even really remember what the partiuclar issue that prompted this was anymore.  I just remember coming back to my desk one day, seeing a blue print out of a help desk ticket, and realizing from what it contained that the "help" desk had done no analysis or trouble-shooting on it whatsoever.

This was one of those moments that was a combination of anger and shaking my head in resignation.  I had a lot of those in 2007.  I guess it was just part of the realization that government IT just wasn't for me.  In fact, it was that August that I took the job with eBaum's World  and moved into full-time development, which is really what I wanted to be doing all along.  So really, it all worked out in the end.

Author's note: This one from the archives was written back in March 2008. This is something I've seen a couple of more times since then. A new developer, usually fairly young, comes in and wants to rewrite everything using whatever technology or method he's most familiar with. It seems to be not so much a problem with learning the new codebase as it is developer hubris. "My chosen way is obviously the right way, so clearly that's how we should be doing things." The people who do this don't seem to stop and think about whether their chosen method or tool might not be the best choice in the current situation, or whether implementing it might just be more trouble than it's worth. Perhaps it's just a lack of experience - they haven't had enough rewrites go south to realize that this isn't always a good idea. Either way, enjoy the rant!

Here's a little tid-bit for you young, aspiring developers out there: When you start a new job, it's usually a good idea to wait until your second week before you suggest rewriting the entire existing codebase.

I say this because we recently hired a new developer. He's obviously very bright and knows his stuff, but his first suggestion, on his first day, was that we should rewrite our entire codebase from a custom PHP framework to Symfony. After all, it would probably only take a couple of months.

Now, I don't know much about Symfony, but I have nothing against against it. And, in all fairness, our current framework is a bit of a mess. So I'm not dead-set against the very notion of switching. However, the current framework has been tailored to our needs and it is perfectly adequate. I'm sure that Symfony is very nice, but it's not immediately obvious what compelling advantages it would offer. After all, we already have an application framework that does what we need and which all three of the established developers are very familiar with. Given that we already have a full plate as far as projects go (hence hiring a new developer), it's hard to imagine how you could make a business case for spending 2 or 3 months learning Symfony and converting the whole site to use it. And if the new guy things that we're going to convert in parallel with adding features to the existing codebase (which he actually mentioned), he's out of his mind.

The thing that really bothers me about this, though, is that the new guy suggested this on his very first day. As far as I know, he hasn't even written a line of code yet and he's already writing up the business case for switching to Symfony. Even though he's barely looked at our codebase and so couldn't possibly have any idea what kind of effort this would entail.

So, to all the developers out there, next time you start a job on an established development team, at least take the time to learn what technology they're using and why before you suggest replacing it. Do a project or two with the current stack. Talk to the other developers about the history of the codebase and the company culture. Pushing to replace major technologies before you've even written a single line of code just makes you look like a tool.

Author's note: In yet another installment of "From the Archives", here is a little entry that I wrote up back in September of 2007, but never bothered to publish. It's a little dated, but sadly it still has the ring of truth to it. Of course, there has been some improvement. A lot of hardware doesn't come with a CD anymore - they just give you a pamphlet with a link to the crappy software instead. But anyway, I thought this entry was based on an interesting quote, so here it is.

On an episode of Hanselminutes a month ago, Scott and Carl spoke with Jeff Atwood of Coding Horror fame about the "ultimate developer rig" he built for Scott. During the course of discussion, Jeff uttered what is, undoubtedly, the best quote about software developers that I've ever heard.

Nobody hates software more than software developers.

Not only does it sum up the feeling of true software geeks everywhere, but it also offers a sad commentary on the state of the computer hardware business.

This quote was uttered in the context of a discussion about "mouse software," i.e. those useless CDs full of crap-ware that get thrown in the package with even fairly standard mice. They typically include drivers, a control panel applet, some kind of tool that runs in the system tray, and probably a few other things.

As Jeff and Carl rightly pointed out, the reason they, myself, and many other programmers hate such software is that it is, quite simply, useless. Nobody needs to install the software that comes with a standard wheel mouse because their operating system (no matter what it is) already supports it. In fact, Carl went so far as to say (half-kidding, I hope) that if he caught any of his employees installing "mouse software," they'd be fired on the spot. Any half-competent programmer or IT person should know better.

Sadly, this phenomenon isn't just limited to mice. USB media devices seem to be the worst offenders. Cameras and MP3 players in particular always seem to come with some kind of software that's supposed to help you move files onto or off of the device. Of course, as it's always third-rate crap-ware, as the vendor makes their money on the hardware and just throws in the software as an after-thought. But the worst part is that there's no reason you should even need it, because any decent MP3 player or camera really ought to "just work" as a USB mass storage device (with the notable exception being the iPod, which is decent, if overrated).

So really, it's not that software developers hate software, it's that we hate crappy, unnecessary software. That's probably because we know what we need and can easily judge the value of a program. We can tell when all a program does is the digital equivalent of busy-work. Being handed a CD of code that doesn't do anything useful feels like being condescended to. That angers us.

Author's note: This is another entry that's been sitting in my drafts folder for years and which, after three glasses of wine, I've now decided to publish. This one is from April 2007, about four months before I left the government IT world for the world of commercial product development. It's funny how my posts are much less humorous when I'm enjoying my job than they are when I'm frustrated. But, then again, working with non-technical end users generates a lot more funny stories than working with top-notch developers. Oh, well.

The corporate IT world can be frustrating at times. The IT department is often viewed as a cost center, a necessary liability, rather than a helpful resource. As a result, it often seems to the IT people like the entire organization is against them. This leads to frustration, apathy, hostility, and using a sniper rifle to pick off people walking in from the parking lot. Thus I present a few tips to keep your IT department on your good side.

1. Don't report a problem by just saying you're "getting error messages" on your computer. We're the IT department, not the Psychic Friends Network. You're going to have to be a little more specific if you want any help.
2. If you do call about "getting error messages," then at least have the decency to tell us what the messages say. I know fixing computer problems may seem like magic at times, but we do need something to go on. If you dismissed the error dialog, didn't take a screenshot, didn't write down the message, don't remember what it said, don't know what application raised it, and don't remember what you were doing when it came up, there's not a hell of a lot we can do for you.
3. Please realize that "computer people" don't automatically know everything there is to know about computers. I may work in the IT department, but that doesn't mean I can fix your computer myself. It also doesn't mean I can tell you how to do something in your CAD or GIS software. It's not like I practice drawing maps in my free time. We specialize too, you know.
4. When requesting custom software, don't expect me to read your mind. I understand that you might not think of some things until after you start using the software. I have no problem with adding things later. In fact, I expect it. However, I really don't appreciate getting attitude because the program doesn't do X when nobody ever mentioned to me that it needed to do X.
5. When you ask how to do something, pay attention to the answer. I don't mind showing you how to add a border to that cell in Excel. I do mind showing you how to do it once a week for a month. And the same goes for clearing out your deleted items in Outlook.

Author's note: This is a piece I wrote back in 2006, but didn't publish because I was afraid people might find it offensive. I found it while looking through the backlog of unpublished/unfinished entries in my drafts folder and thought it was funny enough to share. It's a satire of "magic" frameworks like Ruby on Rails (which was the big thing at the time), which are sold as the solution to all your problems. Thinking about "Rails evangelist", my mind naturally jumped to the new testament evangelists, so this is sort of a merger of the two. No offense intended to any Christians in the audience - I just thought software and eternal salvation made for a funny juxtaposition. But maybe I just have a weird sense of humor. Enjoy!

That's it - the "programming problem" will soon be solved. Next week is the official first, last, and only ever release of the ultimate web development framework - Ruby on the Cross.

What is it

What is RC? It's the ultimate web development tool based on the Christian faith. Think of it as a cross between Ubuntu Christian Edition and Ruby on Rails, but infinitely more productive. It was inspired by the Book of Matthew, chapter 17, verse 20:

And Jesus said unto them, Because of your unbelief: for verily I say unto you, If ye have faith as a grain of mustard seed, ye shall say unto this mountain, Remove hence to yonder place; and it shall remove; and nothing shall be impossible unto you.

Using this principle, RC is able to reach untold levels of productivity.

What does it do?

As you probably already know, Ruby on Rails takes all the pain out of developing CRUD applications by taking your database schema and automatically generating models, controllers, and default views for it, giving you a working, if simple, web application just by running a few commands. RC takes this one step farther: it generates your domain logic for you.

That's right, you heard me: Ruby on the Cross is capable of doing your business analysis for you. In fact, not only can it generate your domain object, but it generates fully functional controllers; rich, AJAX-driven views (which are fully backward-compatible with Mosaic 1.0, by the way); and even writes your database schema. Best of all, it is 100% guaranteed to be future-proof and to be maximally performant. Thus there will never be any need to maintain the generated program!

How does it work?

At this point, you're probably thinking that this is too good to be true. How could anyone possibly promise those things?

Well, remember the quote from Mat.17:20? By drawing on the faith of the programmer, RC is able to miraculously harness the power of God to create genuinely perfect software. There are no algorithms to debug, no trade-offs or design compromises to make. Just the pure invokation of divine power, which not even Murphy's Law can resist.

Hard to believe? It shouldn't be. Just think about all the things people thank God for. If He can get someone a parking space or help the high school basketball team win their big game, why can't He write software?

How do you use it?

Using Ruby on the Cross is quite simple. You simply need to run a few shell commands. You start with:
ruby cross praise jesus
This command initializes Cross and prepares your system for an infilling of the Holy Spirit.

The next command is:
ruby cross supplicate <app_name>
This command forms a prayer, asking the Almighty to design your application. Note that no configuration, or even knowledge of the problem domain, is necessary for this step. Being omniscient, the Lord is able to determine exactly what your customer needs, and will need in the future, with perfect precision.

Lastly, you simply need to ask God to put the project files on your PC so that you can deploy them. To do this, you simply run:
ruby cross create ex nihilo <app_name>
This will cause the completed project source code to appear on your hard drive. Note that you do not need to specify a path on the filesystem - the Lord knows the most appropriate place and will put the code there. And don't worry about source control: God has already added the code to your RCS - not that you're ever going to need to change it.