LinLog

I had a workflow revelation on my work machine the other week: Corporate IT pushed a Windows 10 update that brought me up to a version that actually supports running Linux GUI apps under WSL!

I've WSL GUI apps on my home Win11 system for a while, but having that on my work machine is huge.  I use WSL extensively there.  See, we've been standardizing our dev environment setup using Lando, which is basically a convenience layer on top of docker-compose.  It's a handy tool, but the relevant thing to note here is that it works by directly mounting your project code directory inside the Docker container.  This is convenient for Mac and Linux users, but is kind of a problem for Windows users.

As you may (or may not) know, WSL2 is great and out-performs WSL1 in every way except one: cross-filesystem performance.  As long as you keep everything inside the WSL filesystem, you're golden and everything is fast.  But as soon as you try to cross from the Windows filesystem to the Linux one, or vice versa, performance just falls off a cliff.  Sure, it's not a big deal if you just want to edit a file or something like that, but anything that does any remotely significant amount of filesystem access (e.g. running an npm install) is just painful.  In my experience, it's not unheard of for the performance penalty to be on the order of 10x. 

Clearly that's not something you want to endure as part of your normal workflow.  The simple fix is to do all your work inside WSL, which for me means installing Lando in WSL and hosting my code inside WSL.  The only problem is managing the code.  If you want to do that in Windows, you need to do it over a network share, which works, but isn't exactly a great experience.  It also causes various permissions and file ownership issues for apps like Git that actually care about that.

That's where the WSL GUI apps come in.  Rather than dealing with the network share hassle, you can just install your favorite GUI tools inside WSL and run them just like you're on native Linux.  Problem solved!

Well, mostly solved.  Sadly, not everything runs on Linux.  In particular, there's no Linux port of SourceTree, which is currently my graphical Git client of choice.  But it's not that I particularly like SourceTree - it's just that I hate it less than all the other Git clients I've tried.  So I was forced to try some other options.

This part did not go well.  I tried a few different options, including GitFiend (which was nice, but would randomly crash under WSL) and Git-Cola, which was also decent, but which I had to drop because if left alone it would occasionally lock up and then somehow take down the entire system if I tried to close it.  I have no idea how it managed to do that (presumably some bug in WSL's GUI layer), but that's a different problem.  I also attempted to try Gittyup, but I couldn't, because it only offered Flatpak packages.  And, of course, the Flatpak daemon (or whatever it's called) won't install on WSL because it's missing some of the system-level stuff that it uses.  But that's a different post.

Eventually, I declared GUI bankruptcy and decided to actually learn how to use Fugitive, the Git plugin for Vim.  Turns out that Fugitive is actually pretty good and figuring it out was easier than finding a good graphical Git client.  But that's also a story for another post.

In any event, having WSL GUI apps is pretty nice.  Now I can do all my work in WSL, and still have both GVim and PHPStorm, if I need it, without having to pay the cross-OS performance price.  So I can have nice things and good performance.  Yay!

As I mentioned a while ago, I replaced my desktop/home server this past summer.  In the process, I switched from my old setup of Ubuntu running Trinity Desktop to plain-old Ubuntu MATE, so I've been getting used to some new software anyway.  As part of this process, I figured it was time to take another look for OneDrive clients for Linux.

See, I actually kind of like OneDrive.  I have an Office 365 subscription, which means I get 1TB of OneDrive storage included, so I might as well use it.  I also happen to like the web interface and photo-syncing aspects of it pretty well.

However, I'm slightly paranoid and generally distrustful of cloud service providers, so I like to have local copies and offline backups of my files.  This is a problem for me, because my primary Windows machine is a laptop, and I don't want to pay the premium to put a multi-terabyte drive in my laptop just so I can sync my entire OneDrive, and scheduled backups to a USB disk are awkward for a laptop that's not plugged in most of the time.  Now, I do have a multi-terabyte drive connected to my Linux desktop, but for a long time there were no good OneDrive sync clients for Linux.  In the past, I had worked around this by using one-off sync tools like Unison (which...mostly worked most of the time) or by setting up an ownCloud sync on top of the OneDrive sync (which worked but was kind of janky).  However, but those depended on syncing from my Windows laptop, which was OK when I had 20 or 30 gigabytes of data in OneDrive, but at this point I'm well over 100GB.  Most of that is archival data like family photos and just eats up too much space on a 500GB SSD.

Enter InSync.  InSync is a third-party file sync tool that runs on Windows, Mac, and Linux and supports OneDrive, Google Drive, and Dropbox.  It has all the bells and whistles you'd expect, including file manager integrations, exclusions, directory selection, and other cool stuff.  But what I care about is the basics - two-way syncing.  And it does that really well.  In fact, it totally solves my problem right out of the box.  No more janky hacks - I can just connect it to my OneDrive account and it syncs things to my Linux box.

The only down-side to InSync is that it's proprietary (which I don't mind) and the licensing is confusing.  The up side is that it's not actually that expensive - currently, the pricing page lists licenses at $30 USD per cloud account.  So if you only want to sync OneDrive, it's $30 and you're done.  However, there's also an optional support contract and there's some difference between "legacy" licenses (which I think is what I have) and their new subscription model.  Frankly, I don't fully understand the difference, but as long as it syncs my OneDrive and doesn't cost too much, I don't really care.  

So if you're a OneDrive user and a Linux user, InSync is definitely worth a try.  I don't know about the other platforms or services (I assume they're all similar), but OneDrive on Linux works great.

This is a note to my future self about an annoyance with Nextcloud.  If you're not aware of it, Nextcloud is basically a fork of ownCloud, which is a "self-hosted cloud" platform.  Basically, they both provide a bunch of cloud-based services, like file sync and share, calendar, contacts, and various other things.  I switched to Nextcloud last year because ownCloud was lagging way behind in its support for newer PHP versions.

Anyway, I noticed a rather annoying issue where Nextcloud was leaving hundreds of stale auth tokens in the database.  Apparently, I'm not the only person this has happened to.

While Nextcloud has a menu item to revoke and remove stale sessions on their settings page, it's on a per-item basis.  So if you have hundreds of stale sessions, the only way to remove them is to go through, one by one, and click the menu and select the "revoke" option.  Needless to say, this is terrible.

The less annoying solution is to just go straight into the database and delete them there.  You can just run something like:
DELETE FROM oc_authtoken WHERE last_activity < <whatever_timestamp>;
That might be ugly, but at least it doesn't take forever.

It's important to note that, in addition to being annoying, this is evidently also a performance problem.  From what I've read, it's the reason that authenticating to my Nextcloud instance had gotten absurdly slow.  The app responded fine once I was logged in, but the login process itself took forever.  It also seems to be the reason why my hosting provider's control panel has been showing I'm way over my allotted MySQL execution time.  After deleting all those stale sessions, not only is login nice and snappy again, but my MySQL usage dropped off a ledge.  Just look at this graph:

As you can see, January is a sea of red, and then it drops off to be comfortably under the limit after I deleted the old sessions.  The Nextcloud team really needs to fix this issue.

It looks like my old go-to editor, Komodo IDE, is being retired. I find this slightly sad, since I was a Komodo Edit and Komodo IDE user for many years.  Of course, it's been several years since I used Komodo, but it still feels like a loss to see something you were once very invested in being put out to pasture.  Although it is good to see that ActiveState chose to open-source Komodo IDE rather than just letting it die, so bravo!

Of course, when you read the rationale, you can't really disagree.  The number of free, high-quality editors and IDEs makes "generic code editor" not exactly the best business to be in.  Especially when you're tied to an archaic, end-of-life framework like XUL.  And given that they don't exactly have a large team working on Komodo (never more than 4 people, according to the article), rewriting sounds like it would be a pretty huge effort.

Looking at my blog history and tag list, it seems I started using Komodo Edit some time before November of 2008, used that for several years, and then upgraded to Komodo IDE.  I posted about looking for a new IDE in March 2017.  So I was a Komodo user for 8+ years.  Not  bad run.  It was a good IDE for a long time.  Sadly, I was one of those people who didn't like the UI revamp that came out in version 10.

However, I was somewhat lucky in my timing, because by the time I dropped Komodo, Visual Studio Code had come out.  So I was starting to use Vim seriously in the era where language servers were becoming a thing.  The importance of the Language Server Protocol (LSP) for making "Vim as IDE" viable cannot be understated.  This makes things way easier and better.  All you need is a Vim plugin that speaks LSP (I use coc.nvim), and you can get all the same code intelligence that VS Code does.  That means you can get all of the goodness of Vim without having to sacrifice your intellisense.  It's a good time to be coding.

It's the end of an era. (Cue overly dramatic music.)  I've been using ownCloud as my personal file/caldav/carddav server for years.  This week, I finally decided to switch to NextCloud.  This is my story.

The thing is, I actually remember when NextCloud split from ownCloud.  At the time, I was working on a (now-defunct) product that involved ownCloud.  Basically, my company's core business at the time was data backup, so we had a lot of servers with big disks and were looking for a way to monetize that extra space.  The idea at the time was to do that by integrating a "file sync and share" product into our offerings, and that product was a rebranded ownCloud Enterprise.  Of course, the "file sync and share" space was already pretty crowded, so that product never gained much traction, but it did help me get more into ownCloud and the company even paid to send me to their user conference in Berlin, where I got to meet their team (who, at the time, seemed not-very-impressed with the whole "NextCloud" thing) and see some sites.  So it was actually a great experience, even if the product didn't pan out.

Anyway, despite my affection for ownCloud, my motivation for this change was actually pretty simple and prosaic - I was upgrading my home server (that'll be another post), and I didn't want to downgrade shit.  See, I actually run two ownCloud instances - one on my local network for accessing various media files, and another in my web hosting, for caldav/carddav and files that I want to be highly available.  For my home instance, I was doing a fresh install of the latest Ubuntu MATE on  a brand-new box.  This shouldn't be an issue, except that MATE comes with PHP 8.1, but for some reason, ownCloud only supports PHP 7.4.

Yes, you heard that right - 7.4.  That's the newest version that's officially supported.  The last 7.x release.  The one that's no longer actively supported and has less than six months of security updates left.  That one.  That's what they still expect me to use.

For my previous home box, I believe I'd actually hacked up the source a bit to make it work (since I don't think I depended on anything that didn't work in 8.x), but week I was sick and I just didn't feel like it.  Depending on a version that's about to lose security fixes is crazy anyway.  So I figured I'd "upgrade" to NextCloud, since they actually recommend PHP 8.1.

For my home server, I just did a fresh install, which is fairly straight-forward.  The only annoying part was the Apache configuration, and that was only annoying because I was running NextCloud on a non-standard port and forgot to add a "Listen" directive. 🤦‍♂️ For this instance, there was no real need to do any migration, because the only data I had in there was the (very small) list of users - the rest was just files, which can be trivially re-indexed.

Upgrading the instance on my web hosting was another story.  Since that had my carddav and caldav data, I really did need to migrate that.  I was also already several versions behind on my updates - it was running ownCloud 10.3, whereas 10.8 was current.  However, this turned out to be a blessing in disguise.

You see, NextCloud includes support for migrating from an ownCloud instance.  The thing is, they only support specific migrations.  In my case, the relevant case was that you can migrate from exactly ownCloud 10.5 to NextCloud 20.  Sadly, it took me a couple of tries to realize that the version migration matrix are exact, so there was no path to directly migrate from ownCloud 10.3 to NextCloud.  So I had to use the auto-updater to update ownCloud 10.3 to 10.4, and then manually update ownCloud 10.4 to 10.5 (because the auto-updater wanted to go all the way to 10.8).  Then I could follow the migration process and manually update to NextCloud 20.  From there, I was able to use the NextCloud auto-updater four times to upgrade to the current version.

So the upgrade process was...tedious.  Not really "hard", but definitely tedious.  The directions are pretty clear and simple, it's just a lot of steps to get to a current version of NextCloud.  But at least none of the steps were particularly complicated or prone to error.  As data migrations go, it could be much worse.  And the best part is that it maintained URLs and credentials, so I didn't even have to reconfigure my caldav/carddav clients.

As far as NextCloud itself goes, it seems...pretty much like ownCloud, but nicer.  They've made the UI prettier (both for the web interface and the client app), added a nice dashboard landing page, and made some other cosmetic improvements.  They also seem to have a wider range of installable apps, which is nice.  I haven't had all that long to play with it yet, but so far it seems like a distinct upgrade.

A while back, I posted about a tool called Duet, which allows you to convert an iPad or Android tablet (or even phone) into an external laptop display.  It actually works quite well, and allows you to use either WiFi or USB connections for your tablet monitor.  It also support using the touch screen on the tablet to control your desktop, which is pretty cool.

However, I did eventually discover an issue with it.  It seems that, on my work laptop (but not my personal one), the "energy efficient" setting doesn't properly support all resolutions.  It's a really weird bug, as the other two performance settings ("high power" and "pixel perfect") both work fine, and everything works fine on my personal laptop, but "energy efficient" only works when the resolution is set to maximum on my work laptop.  On the up side, their support people have been very responsive and I can just use a different setting, so it's not a big deal.

Anyway, as part of trying to collect more info on this bug for Duet's testing team, I signed up for a trial of Duet Air to see if I could reproduce the issue through that (spoiler: I could).  Duet Air enables Duet's "remote desktop" feature, which allows you to use not only mobile devices, but other laptops as external displays.

It's actually a pretty slick feature.  You just create an account and sign into all of your devices with it.  Then you can go to the "remote desktop" tab in Duet and choose the device you want to connect to.  The paradigm is that you use the "display" device to select what you connect to.  So, for example, if I want to have four monitors for my work machine, I can open up Duet on my home laptop, select my work laptop, and the home laptop becomes a wireless display.

So far, it's working pretty well.  It's easy to use and set up, performant, and it's a tool I'm already using.  It's also fairly cheap at $25/year.  I think I'll probably continue using it after the trial.

A few weeks ago, I decided to set up a DIY home intercom system.  This was motivated by the fact that my son has been doing home-school and we set him up a workspace in the basement.  This isn't a problem per se, but my wife usually doesn't go down there with him if he's doing independent work, which means there's often yelling up and down the stairs.  This is, shall we say... somewhat distracting when I'm trying to work.

I did a little searching for intercom systems, thinking I might buy some hardware, but decided that looked like too much work.  We'd have to find a home for it, and then you might not hear it if you were on the other side of the house, unless I put them everywhere, which is an even bigger pain.  Besides, it seemed like there should be an app for that.  And since we pretty much have our phones close to hand most of the time, that would be more convenient than dedicated hardware anyway.

Turns out there is an app for that.  A number of them, actually.  The one I decided to go with was Zello, which is a fairly simple walkie-talkie app.  I went with this one for a few reasons:

1. The mobile app is free, at least for personal use.  (There's a PC version too, but that's only for paid corporate accounts.)
2. It's in the Amazon and Google Play app stores.
3. It's easy to set up.
4. It's really easy to use.

The setup process for Zello was pretty basic.  For my son, I decided to just put it on an old Kindle Fire that I had laying around.  It can just sit on the desk, plugged in and ready to use whenever we need to talk to him.  My wife and I just put the app on our phones.  From there, you just create an account (which only requires basic information) for each device using the app, and then send a contact request to the other accounts.  Once your request is accepted, that person will appear in your contact list.

Actually talking to other people is even simpler.  You just tap on the person's account from your contact list and then you get a screen with a great big "talk" button in the middle.  When you want to talk to the person, you just press and hold the button and start talking, just like an old-fashioned walkie-talkie.  When you're done, you release the button.  From what I can tell, the connection is not in real-time - it seems like the app records your message and then delivers it, so you are less subject to the vagaries of the network.  But barring networking issues, the delay seems to be pretty short - a few seconds in most cases.

The app also has a few other features, including very basic text messaging.  There's also a "channels" feature, which I haven't used yet.  That's their "group voice chat" feature.  Presumably the idea is to mimic a dedicated frequency for a CB radio.  The primary use-case for the commercial version of Zello seems to be for fleet dispatchers, so the interface seems geared toward a simple replacement for a traditional radio system.

Overall, the app works pretty well.  It was easy to set up and it has definitely saved some frustration in terms of yelling back and forth across the house.  Also, my son seems to like using it.  He even ends is messages with "over and out".  So I count this as a win.

The other week I was wondering: can I use my tablet as an external monitor?  I mean, it would be nice to have a portable monitor so I can do dual-display on my laptop wherever I go.  And I usually have my tablet with me.  And it's got a nice 10" display with decent definition.  So why not use it as an extra monitor where I can toss my Slack window or something like that?

Unfortunately, it doesn't seem like there's an out-of-the-box way to do this with my Windows laptop and Galaxy S6 Lite tablet.  But the good news is that there's software that will make it work pretty painlessly.

The solution I ended up going with was Duet.  It's a combination of a desktop app and mobile app working in tandem that works for Windows, Mac, Android, and iOS.  There is a service you can create an account for, but it's not necessary - you can use the apps locally without signing up for anything.  (In fact, I haven't even tried creating an account yet.)  By default Duet connects via a USB connection to your device, but you can also enable an option to connect wirelessly over WiFi.  The WiFi connection is fine, but at least on my network there was a lot of mouse lag when trying to use apps on the tablet.  It's not unusable, but it is unpleasant.  The USB connection is super-responsive, though, so I normally just use that.  As a nice extra, Duet even allows you to use the touchscreen of the tablet, so if your laptop has a touchscreen (which mine does), your secondary screen will work just the same.  Very slick!

The only down side of Duet I've seen so far is that it's not free and there doesn't seem to be a demo version.  There's no cost for the desktop component, but the Android app is about $10.  That's not exactly a fortune, but it's enough to be annoying if you try it and it doesn't work.  Thus my first attempt was with Spacedesk, which is completely free.  However, I didn't actually get a chance to try it out, as the desktop driver install experience was...not great.  For some reason, it took an uncomfortably long to for the installer to run.  Like, after half an hour it was still working.  So I clicked "cancel".  And then, after another hour, the rollback finally completed and I closed the installer.  I have no idea what it was doing, but that was enough to make me not trust it.

Duet, on the other hand, installed fast and worked the first time.  The settings are pretty minimal.  For the connection, you can change the framerate, performance setting, and resolution.  System-wide, you can toggle notifications and "Duet Air", which is what they call the WiFi connection.  To connect the tablet, you pretty much just plug it in.  Duet on the tablet will prompt you to connect the screen and Duet on the laptop will also detect the connection and add the display.  (Of course, if you're using a WiFi connection, it's not that simple.  But it's still pretty simple.)  After that, the tablet just behaves like a regular touchscreen display.  Unfortunately, it only supports one device at a time, so you can't plug in two tablets, or your tablet and your phone, but that's not a huge deal.

So far I'm pretty happy with Duet.  It's a nicely done little utility that "just works" and does a useful thing.  Definitely worth checking out if you have a device that would make a good extra monitor.

A couple of weeks ago I got the notification on my laptop that I could now upgrade to Windows 11.  And since I was feeling optimistic that day, I clicked "yes".

I'm not going to do a "review of Windows 11" post, though.  Not because I'm lazy or pressed for time (though I don't deny those charges), but really just because I don't have that much to say.  I mean, so far Windows 11 is fine.  And I don't mean that in the sarcastic, room-on-fire-this-is-fine meme sort of way.  My experience has been genuinely fine.  It's not a phenomenal, life-changing upgrade, but I haven't had any problems either.

For the most part, I haven't really noticed much of a change from Windows 10.  Yeah, now windows have more rounded corners and the UI in general got kind of a face lift, but those are mostly cosmetic changes.  They added some handy window management features that I use on occasion, but I haven't discovered any major features that strike me as must-have's.  

The one change I did immediately notice was the start menu.  I really don't like the new start menu.  I think it's much less useful than the one from Windows 10.  For one, the default position is in the middle of the screen, which seems like a pointless change.  However, there's a setting to easily change that.  But beyond that, it doesn't allow much customization and seems much more focused on being a glorified search box than a menu.  You can still pin items to the start menu, but the option to arrange them into groups is gone.  Also, the pinned area is now smaller and paginated, which is kind of annoying.

Fortunately, that can be changed too.  There are a few options out there for start menu replacement in Windows 11.  I went with Stardock's Start11, which give you quite a few options in terms of customizing the start menu experience, including versions of the Windows 10 menu and the "classic" Windows 7 style menu.  On top of this, it gives you a number of other settings to manipulate the look and behavior of the start menu and taskbar, such as controlling transparency and texture, swapping out the start button image, and controlling click behavior.  It's actually quite well done, and with a $6 price tag, it's kind of a no-brainer if you don't like the new menu.

A few weeks ago, I was looking into Typescript a bit.  I've heard lots of good things about it, but never had a chance to play with it.  However, I got tasked with some updates to my company's portal site.  (While not technically my team's responsibility, the portal team was swamped, so I agreed to make the required updates to support a  back-end feature my team added.)  And, of course, the portal team uses Typescript.

Naturally, most of the editing recommendations for Typescript are focused on Visual Studio Code.  But I like Vim, so I did a quick search and found this article, which led me to CoC (which I choose to pronounce "coke", like the soda), which stands for the slightly ungrammatical "Conquer of Completion".  It's a plugin for NeoVim and Vim that essentially does Intellisense (code completion, context popups, etc.) using language servers.

If you're not familiar, the Language Server Protocol (abbreviated LSP, though that always makes me think of the Liskov Substitution Principle) was developed by Microsoft for VS Code.  It's essentially a way to make Intellisense work without the editor having to implement support for each language.  It does this by defining a protocol that "clients" like an editor can use to communicate with a "language server".  The language server is a stand-alone program that can provide code intelligence for a particular language, but is not directly tied to any particular editor.  The server can then be called by any client that implements the protocol, which means that the editor itself doesn't actually have to know anything about the language to implement advanced editing features - which is huge.

Anyway, CoC is an LSP client for Vim.  And I have to say, it's awesome!  I've messed with a few code completion and LSP plugins in the past, but I never really got them to work right.  They were either difficult to configure, or required Vim to be built with particular non-standard options.  But CoC was dead-simple to set up.  The only catch is that you have to install the language servers separately, but it turns out that's super-simple as well.  (The ones I've used so far can all be installed through NPM.)

I'm still getting used to it, but having CoC is a game changer for Vim.  I'd given up on having this level of intelligence in my editor.  I mean, for something that supports as many languages as Vim, building it the old-fashioned way just isn't feasible.  But when you can use the same language servers as more modern editors to do the heavy lifting, suddenly it's no longer crazy.

The next step is to look into the available commands and customizations for CoC and see what I can come up with to optimize my experience.  So far it's a pretty cool tool and it definitely makes the development experience nicer.  I want to see what else I can do with it.

Author's note:  Happy new year!  I thought I'd start the year off with another old post that's been sitting in my drafts folder since March 24, 2013.  This time, though, I'm going to provide more inline commentary.  As usual, the interjections will be italicized and in parentheses.

You see, this post is on LnBlog, and specifically what is (or was) wrong with it.  If you don't know, LnBlog is the software that runs this website - I wrote it as a "teach yourself PHP" project starting back around 2005.  I've been improving it, on and off, ever since.  So in this post, I'm going to show you what I thought of it back in 2013 and then discuss what I think now and what has changed.  Hopefully it will be somewhat enlightening.  Enjoy!

The year is (relatively) new and it's time for some reflection. In this case, reflection on past code - namely LnBlog, the software that runs this site.

I've come a long way from LnBlog, which as my first "teach yourself PHP" project. I've now been doing full-time professional PHP development since 2007 and can reasonably claim to have some expertise in it. And looking back, while the LnBlog codebase is surprisingly not horrifying for someone who had a whopping two months of web development experience going into it, it's still a mess. So it's time to start slowly refactoring it. And who knows? Blogging my thought process might be useful or interesting to others.

(Back to now: I actually did blog some of this stuff, but not until 2017 or so.  And I still agree with that initial assessment.  The code had plenty of problems then and it still does.  If I were starting fresh today, I'd probably do almost everything differently.  But on the other hand, I've seen much worse in much newer code.  And in the last three years or so I've been making slow and steady improvements.)

The Issues

There are lot of things about LnBlog that need changing. A few of them are functional, but it's mostly maintenance issues. By that I mean that the code is not amenable to change. It's not well organized, it's too hard to understand, and it's too difficult to make updates. So let's go over a few of the obvious difficulties.

1. The plugin system

I have to face it - the plugin system is an unholy mess. The entire design is poorly thought out. It's built on the premise that a "plugin" will be a single PHP file, which makes things...painful. Any plugin with significant functionality or a decent amount markup starts to get messy very quickly. The "single file" limitation makes adding styles and JavaScript ugly as well.

On the up side, the event-driven aspect works reasonably well. The code for it is a bit nasty, but it works. The main problem is that there aren't really enough extension points. It needs a bit more granularity, I think. Or perhaps it just needs to be better organized.

(Back to now: I still agree with most of this, except perhaps the thing about extension points.  So far, the only place where that's been a real problem is when it comes to inserting markup mid-page.  But yeah, the whole "a plugin is one file" thing was ill-conceived.  The good news is that it's totally fixable - I just need to figure out some design conventions around splitting things out, which hasn't been a priority so far.)

2. The templating system

This one is also an unholy mess. The idea isn't bad - allow any file in a theme to be over-ridden. However, I tried to abstract the template files too much. The files are too big and contain too much logic. Also, the simple template library I'm using is more a hindrance than a help. I'd be better off just ditching it.

I've also been thinking of getting rid of the translation support. Let's face it - I'm the only person using this software. And I'm only fluent in one language. Granted, the translation markers don't cause any harm, but they don't really do anything for me either, and accounting for them in JS is a bit of a pain.

(Back to now: The only thing I still agree with here is that the existing templates are a mess.  But that has nothing to do with the template system - I just did a bad job of implementing the template logic.  I'm working on fixing that - for instance, I added some Jinja-like block functionality to the template library.  I had considered re-writing the templates in Twig or something, but it quickly became obvious that that would be a huge amount of work, that it would be difficult to do in a piece-wise fashion, and it's not clear that the payoff would be worth it.  Likewise with the translation markers - taking them out would be a bunch of work for almost zero payoff and the JS thing isn't really that big a deal.  Besides, if I ever changed my mind again it's WAY more work to put them back in.)

3. The UI sucks

Yeah, my client-side skills have come a long way since I built LnBlog. The UI is very Web 1.0. The JavaScript is poorly written, the style sheets are a mess, the markup is badly done, and it's generally "serviceable" at best.

As I realized the other day, the style sheets and markup are probably the worst part. Trying to update them is difficult at best, which is exactly the opposite of what you want in a theme system. In retrospect, my idea to replace files wholesale rather than overriding seems misguided. They're too fragmented. When it comes to the style sheets and JavaScript, this also hurts performance, because there are a lot of files and everything is loaded in the page head.

(Back to now: This is pretty much still accurate.  I've been slowly improving the UI, but it's still not looking particularly "modern".  That's not such a big deal, but the templates and CSS are still a pain-point.  Really, what I need to do is rework the theme system so that I can easily make lighter-weight themes, i.e. I should be able to just create one override CSS file and call it good.  I have the framework for that in place, but I have yet to actually go through the existing themes and make that work.)

4. Too much compatibility

When I first started writing LnBlog, I had a really crappy shared web hosting account. And by "really crappy", I mean it offered no database server and had safe-mode and the various other half-baked PHP "security measures" enabled by default. So I actually built LnBlog to be maximally compatible with such an environment.

These days, you can get decent hosting pretty cheap. So unless you can't afford to pay anything, there's no need to settle for such crappy hosting. And again, let's be honest here - I don't even know anyone other than me who's using this software. So supporting such crappy, hypothetical configurations is a waste of my time.

In addition, I really put an absolutely ridiculous number of configuration settings into LnBlog. The main config file is extensively documented and comes to over 700 lines. That's completely nuts and a pain to deal with. It contains a lot of settings that are pointless and that hardly anyone would ever want to override. And most of those could be moved into a GUI rather than having to edit a file. There's absolutely no reason for many of those settings.

(Back to now: This is also still true.  I've been looking at redoing the config system, but that's another one of those things that is a big change because it has tendrils all through the code.  I have been moving some stuff out of the main blogconfig.php file, and I've been avoiding adding to it, but there's still a lot there.  For the most part, it's not a huge issue, since most of the things you would want to configure are through the UI, but still....)

5. No real controller structure

I knew nothing of MVC or design patterns when I first wrote LnBlog. As a result, the "glue" code is in the form of old-style procedural pages. They're messy, poorly organized, and hard to maintain. A more modern approach would make things much easier to deal with.

(Back to now: The old "pages" are dead in all but name.  A handful of them still exist, but they're three-liners that just delegate to a controller class.  The bad news is that it's pretty much just two monolithic controller classes with all the old logic dumped into them.  So that sucks.  But they have dependency injection and some unit test coverage, so this is still an improvement.  And I've at least got a little routing groundwork laid so that I could start breaking off pieces of functionality into other classes in the future.)

The Problem

While I'd like to fix all this stuff in one shot, there are three big problems here:

1. That's a lot of stuff, both in terms of the number of tasks and the amount of code involved.
2. I no longer have the kind of free time I did when I first wrote this.
3. I'm actually using this software.

Of course, those are two sides of the same coin.  LnBlog isn't huge, but it isn't tiny either - the codebase is upwards of 20,000 lines.  That wouldn't be a big deal if I were working on it as my full-time job, but this is a side-project and I can devote maybe a couple hours a day to it sometimes.  So major surgery is pretty much out.  And the third factor means that I need to be careful about breaking changes - not only do I not want to break my own website, but I also want to avoid having to do a lot of migration work because writing migration scripts is not my idea of a fun way to spend my free time.

(Back to now: This is always a problem with open-source and side projects.  Nothing has changed here except, perhaps, my development process.  After that year I spent learning about the Personal Software Process, I started using some of those methods for my personal projects.  The main change was that, when making any kind of a big change or feature addition, I actual do a semi-formal process with a requirements and design phase and review phases.  It sounds kind of silly for a personal project, but it's actually extremely useful.  The main benefit is just in having my thoughts documented.  Since I might be going a week or more between coding sessions on any particular feature, it's insanely helpful to have documentation to refer back to.  That way I don't have to remember or waste time figuring things out again.  And by having design- and code-review phases as part of my development process, I have a built-in reminder to go back and check that I actually implemented all those things I documented.  Having the whole thing written out just makes it much easier when you have long gaps in between work sessions.)

General commentary from the present: So as you can see from the above comments, I've fixed or am fixing a lot of the things that bothered me about LnBlog eight years ago.  In the last two or three years I've put a lot of work into this project again.  Part of it is because I actually use it and want it to be better, but part of it is also "sharpening the saw".  I've been using LnBlog as an exercise in building my development skills.  It's not just coding new features, like the flurry of development in the first two years or so that I worked on LnBlog, it's cleaning up my past messes, adding quality assurance (in the form of tests and static analysis), updating the documentation and figuring out how to balance responsible project management with limited resources).  It's an exercise in managing legacy code.

To me, this is a useful and important thing to practice.  As a professional developer, you will have to deal with legacy code.  In my day job, I've had to deal with code that was written by our CEO 10+ years ago when he started the company.  Software is a weird combination of things that live a week and things that live forever, and there's seldom any good way to tell which group the code will be in when you're writing it.  So while it's important to know how to write code correctly the first time, it's also important to know how to deal with the reality of the code you have.  And no, "let's rewrite it" is not dealing with reality.  And when you have a code-base that's 15 years old, that you're actively using, and that you originally wrote, it's a great opportunity to experiment and build your skills in terms of modernizing legacy code.

And that's just what I'm doing.  Slowly but surely, LnBlog is getting better.  I've implemented a bunch of new features, and in the process I've worked on my design and analysis skills, both at a product level and at a technical level.  I've fixed a bunch of bugs, which makes my life easier.  I've implemented additional tests and static analysis, which also makes my life easier by finding bugs faster and giving me more confidence in my code.  I've improved the design of the system, which again makes my life easier because I can now do more with less effort.  Sure, there's still plenty do to, but I've made lots of progress, and things are only getting better.

Author's note: Here's another old article that I mostly finished, but never published - I'm not sure why.  This one is from way back on August 23, 2013.

I was working for deviantART.com at the time.  That was a very interesting experience.  For one, it was y first time working 100% remote, and with a 100% distributed team, no less.  We had people in eastern and western Europe, and from the east coast to the west coast of the US.  It was also a big, high-traffic site.  I mean, not Google or Facebook, but I believe it was in the top 100 sites on the web at the time, according to Alexa or Quantcast (depending on how you count "top 100").

It also had a lot of custom tech.  My experience up to that point had mostly been with fairly vanilla stuff, stitching together a bunch of off-the-shelf components.  But deviantART was old enough and big enough that a lot of the off-the-shelf tools we would use today weren't wide-spread yet, so they had to roll their own.  For instance, they had a system to do traits in PHP before that was actually a feature of PHP (it involved code generation, in case you were wondering).

Today's post from the archives is about my first dealings with one such custom tool.  It nicely illustrates one of the pitfalls of custom tooling - it's usually not well documented, so spotting and resolving issues with it isn't always straight-forward.  This was a case of finding that out the hard way.  Enjoy!

Lesson learned this week: object serialization in PHP uses more space than you think.

I had a fun problem recently.  And by "fun", I mean "WTF?!?"  

I got assigned a data migration task at work last week.  It wasn't a particularly big deal - we had two locations where user's names were being stored and my task was to consolidate them.  I'd already updated the UI code to read and save the values, so it was just a matter of running a data migration job.

Now, at dA we have this handy tool that we call Distributor.  We use it mainly for data migration and database cleanup tasks.  Basically, it just crawls all the rows of a database table in chunks and passes the rows through a PHP function.  We have many tables that contain tens or hundreds of millions of rows, so it's important that data migrations can be done gradually - trying to do it all at once would hammer the database too hard and cause problems.  Distributor allows us to set how big each chunk is and configure the frequency and concurrency level of chunk processing.

There's three parts to Distributor that come into play here: the distributor runner (i.e. the tool itself), the "recipe" which determines what table to crawl, and the "job" function which performs the actual migration/cleanup/whatever.  We seldom have to think about the runner, since it's pretty stable, so I was concentrating on the recipe and the job.

Well, things were going well.  I wrote my recipe, which scanned the old_name table and returned rows containing the userid and the name itself.  And then I wrote my migration job, which updated the new_name table.  (I'm fabricating those table names, in case you didn't already figure that out.)  Distributor includes a "counter" feature that allows us to trigger logging messages in jobs and totals up the number of times they're triggered.  We typically make liberal use of these, logging all possible code paths, as it makes debugging easier.It seemed pretty straight-forward and it passed through code review without any complaints.

So I ran my distributor job.  The old_name table had about 22 million rows in it, so at a moderate chunk size of 200 rows, I figured it would take a couple of days.  When I checked back a day or two later, the distributor runner was reporting that the job was only 4% complete.  But when I looked at my logging counters, they reported that the job had processed 28 million rows.  WTF?!?  

Needless to say, head-scratching, testing, and debugging followed.  The short version is that the runner was restarting the job at random.  Of course, that doesn't reset the counters, so I'd actually processed 28 million rows, but most of them were repeats.  

So why was the runner resetting itself?  Well, I traced that back to a database column that's used by the runner.  It turns out that the current status of a job, including the last chunk of data returned by the crawler recipe, is stored in the database as a serialized string.  The reason the crawler was restarting was because PHP's unserialize() function was erroring out when trying to deserialize that string.  And it seems that the reason it was failing was that the string was being truncated - it was overflowing the database column!

The source of the problem appeared to be the fact that my crawler recipe was returning both a userid and the actual name.  You see, we typically write crawlers to just return a list of numeric IDs.  We can look up the other stuff st run-time.  Well, that extra data was just enough to overflow the column on certain records.  That's what I get for trying to save a database lookup!

Author's note: I started the draft of this article way back on July 1, 2013. Sadly, it's still pretty relevant.

So I'm built myself a nice little web-based bookmarking app. I wanted something that would both give me some insight into how I use my bookmarks and also save me from worrying about syncing bookmarks between multiple browsers on multiple devices. And since I've regained my distrust of "the Cloud" with the demise of Google Reader, I decided to write my own. (Note from the future: Yes, I know seven years ago, but I still don't really trust cloud services.) If you're interested, check out the GitHub page. Maybe one day I'll make a real, official release of it. I call in Lnto, with "ln" being the UNIX "link" command and a tie-in to "LnBlog", the software that runs this site. (For LnBlog, the tie-in was to "ln" for the natural logarithm, i.e. "natural bLOG". Get it? I swear I thought it was funny at the time.)

One must-have feature for such an app is a "bookmark this page" feature. With native browser bookmarks, this is built in. With a web app...not so much. So the solution is to either write an extension for every browser I want to support (which is a lot of work), or just write a bookmarklet - a little piece of JavaScript that you can bookmark and run with a single click. Since this is a personal project that I'm doing in my limited free time, the latter seemed like the obvious choice.

There's just one problem here - mobile browsers. In addition to my laptop and my desktop, I have an Android phone and a Kindle Fire that I want to support. And while the actual bookmarklet code works just fine on all of those devices, actually bookmarking it isn't quite so easy. Because they're mobile browsers, you can't just drag the link to the toolbar as you would on the desktop.

Until recently, Firefox Mobile handled this well. (Author's note: We're back to the current time now, not 2013.) It would allow you to bookmark a bookmarklet like a normal bookmark. You just had to press the link and select a "bookmark this link" item from the menu. Then you could just bring up the bookmark screen when you were on a page and it would run the bookmarklet. However, with the updates for Firefox Mobile 81, that doesn't work anymore - the javascript: URL scheme doesn't seem to get executed when you invoke the bookmark. And other browsers don't seem to support bookmarking the bookmarklet in the first place. This link suggests that it's possible using bookmark syncing, but I'm not sure if that still works and I don't really want to turn on syncing anyway.

What I eventually did was just create a page that I can paste a URL into and it will do the same thing as the bookmarklet. It's not great, but it's serviceable. At some point, maybe I'll get around to creating an Android app. Then I'll have some native integration options to work with.

Well, I just set up Let's Encrypt on my home server for the first time.  When I was finished, my first thought was, "Damn, that was awesome!  Why didn't I set that up a long time ago?"

If you're not familiar with Let's Encrypt, it's a non-profit project of the Internet Security Research Group to provide website operators with free SSL certificates.  The idea is to make it easy for everyone to have SSL properly enabled for their website, as opposed to the old days when you had to either buy an SSL certificate or use a self-signed one that browsers would complain about.

I didn't really know much about Let's Encrypt until recently, other than then fact that they provide free SSL certs which are actually trusted by browsers.  And really, that was all I needed to know to be interested.  So I decided to try it out on my home server.  I was already using them on this website, that that was a slightly different situation: my web host integrated Let's Encrypt into their control panel, so all I had to do to set up a cert for one of my subdomains was click a button.  Super convenient, but not really any learning process there.

It turns out that setting up my home server to use the Let's Encrypt certs was pretty painless.  The recommended method is to use certbot, which is a tool developed by the EFF.  It basically automates the entire process of setting up the certificate.  Seriously - the entire process.  It's actually way easier to set up a Let's Encrypt cert with certbot than it is to make your own self-signed cert.  You just need to run a command, answer a couple of questions, and it will get the certs for each of your sites, install them, and keep them updated.  The only catch is that you need root shell access and your web server has to be accessible via port 80 (for verification purposes).

Compared to the old self-signed cert I was using, this is way easier.  You don't have to generate any keys, or create a CSR (Certifiate Signing Request), or edit your server config files.  Running certbot takes care of everything for you.  So if you haven't tried Let's Encrypt and you're running a site that could use some SSL, I definitely recommend it.

Author's note: Here's another article from the archives. This one was written way back on March 17th, 2007. At the time I was still working for the county government and using Linux on all my home computers. This rant came out of frustration at the interaction of not-fantastic bespoke software and annoying semi/anti-features in a commercial flash drive.

We don't have that in the Linux world. Most of our software is free or open-source and is installed from a single trusted distributor. We don't have to worry about whether a package is bundled with spyware, a crappy browser toolbar, or a useless systray icon.

This was brought home to me today when I had to deal with a user's flash drive at work. His department has a number of laptops that can't connect to our network, but this crappy application that the state foisted on us requires them to move data from back and forth between our file server and the laptops. The easy solution to this is just to use flash drives.

Well, this user's flash drive broke and the department didn't have any spares, so he went out and bought a 1GB Cruzer Micro. Nice enough drive, plenty of capacity for his purposes. The only problem was, it didn't work with The Crappy Software Foisted On Us By The State (TCSFOUBTS, pronounced 'ticks-fouts').

You see, TCSFOUBTS is one of those half-baked, government consultingware type of systems. It has a bazillion features and is highly customizable. On the down side, it's slow as all hell, has a terrible user interface, and is difficult to configure. For example, some customization options require manually editing MS Access database tables and quite a number involve hand-editing INI files or other text files.

Now, having planned ahead, the TCSFOUBTS people built in a feature to let users automatically copy their data onto a removable drive with just a couple of clicks. However, by "removable drive," what they really meant is "a path." For this feature to work, you have to configure TCSFOUBTS with the drive letter for your removable device.

By now, you probably see where I'm going with this. TCSFOUBTS needs the drive letter, which has to be configured by the system administrator. However, unlike the old flash drive, the Cruzer Micro has a bootable partition that displays as a CD-ROM drive. (Note from the present: this was before booting from USB was a common thing.) Of course, that's the first partition, and so get the drive letter where TCSFOUBTS wants to copy its files.

What was on that first partition? I don't remember. Some useless utility software that nobody asked for. One of those "value-adds" that they put on flash drives because they want to use the space for something. But it meant that the drive was stuck with an unusable partition that I didn't immediately have the ability to remove. So for the user to use this drive, they would have to hand-edit some INI file, and then edit it back when they got a different drive. And since the users for this system were all non-technical, that sounded less than appealing.

Eventually the user decided to just not bother and requisition an "official" drive. It was just a frustrating series of events. It goes to show how a few misplaced assumptions can completely mess up a user experience.

Author's note: Welcome to another episode of "From the Archives". This is the stub of an article that I wrote twelve years ago, on April 25, 2008. At the time, KDE 4.x was freshly into stable release. I was a use and fan of KDE at the time, and there had been a lot of hype about how awesome version 4 was going to be. My initial reaction, however, was...not so great.

This is actually slightly relevant because I have resurrected the GUI on my "home desktop", by which I mean my "home server". This is the box sitting under my desk in the basement that runs Ubuntu 18.04 and runs various web and media server software. It does have a GUI installed, but I hadn't really used it in years - in part because GNOME didn't really work well on it. This isn't super surprising, since it's an old box with just the integrated graphics chip. But it's got more than enough memory and processing power for the workload I want it to do, so there's not really any point in upgrading.

Anyway, due to the current CONVID-19 pandemic I'm now working from home and sitting at that desk all day every day, so I decided to fix up my desktop. Part of my cleanup process was to axe GNOME and install the Trinity Desktop Environment (TDE). This is a project I just recently discovered and immediately fell in love with. It's essentially a fork and continuation of KDE 3.5. It's since evolved into its own thing, apparently, but it's still noticeably KDE-like, so I'm very comfortable in the UI. Just like the original KDE 3.5, TDE is powerful, intuitive, generally pleasant to use, and works quite well on my not-so-new system. It doesn't have the fancy graphical bells and whistles, but I never really cared much about that anyway. I would definitely recommend it to any old-school KDE 3.x fans.

Anyway, here are my thoughts from the time. To be fair, I haven't used "real" KDE to any extent since this, so I'm sure all of my complaints have been addressed. But then again, I don't really care. I'm happy with TDE. Enjoy!

Kubuntu 8.04 was released yesterday (note: again, this was twelve years ago). That means it's upgrade time for me again.

This time around, Kubuntu comes in 2 varieties: the "rock-solid" KDE 3 version, and the KDE 4 remix. I had been intending to get on the leading edge and install the KDE 4 version. However, just to be on the safe side, I decided to give the live CD a try first. And after messing around with it for half an hour or so, I'm glad I did.

Bottom line: I think I'm going to wait for KDE 4.1. Or maybe 4.2.

I just don't care for 4.0.3. It definitely looks different...but not better. I just didn't see any new features that looked even remotely interesting, let alone compelling. The splash screens were kind of nice, and the plasma widget effects on the desktop were pretty neat, but that's about it.

There seemed to be a lot more down sides. Of course, I'm not sure how many of these are KDE 4 issues and how many are Kubuntu issues, but I found them annoying either way. Here's my list:

1. The window style for the default desktop theme is unbearably ugly. It's too dark and too monochromatic. I guess somebody must like it, but it really bothers me.
2. Where's the control panel? I don't see it and the context-sensitive configuration panels don't have all the options in them.
3. In fact, where the heck are the other options?
4. What the heck happened to Amarok and the other applications? The UI is completely different and it feels like half the features are missing.

I could go on, but why bother? There's just no reason for me to upgrade at this point.

A while back I came upon this article on using Vim as a PHP IDE.  It's got some very good pointers, although the author may have gone a little over the top on trying to mimic the behavior of PHPStorm.  I haven't tried all of those plugins, but quite a few of them are in my Vim config.

If nothing else, the article gives you a good sense for just how powerful Vim is when it comes to extending its behavior.  It's actually pretty impressive.

Earlier this year,  finally read through The Pragmatic Programmer.  It's been sitting on my bookshelf for...at least 15 years now.  I'd read parts of it before, but never went through the whole thing.  Anyway, it contains a section on choosing an editor, and one of the things they stress is extension.  You need to be able to customize your editor to extend its functionality to suit your workflow.  The idea is that the editor is one of your primary tools of code-craft and you need to make it truly yours.  You need to learn to use it well and completely, to make it an extension of your hand.

So far I'm finding Vim to be a pretty good choice in that regard.  Partly this is due to its raw power, but to a large extent it's also due to its age and the ecosystem around it.  Vim is a very old tool and it's very mature and stable.  This is a very good thing, because it means that there are lots of examples and documentation for how to use and customize it.  If you want do do something new with Vim, there's a good change that someone already wrote a plugin to do it.  And if not, there are plenty of plugins out there that you can use as examples. 

Being mature and stable also means that things are unlikely to change out from underneath you.  Sure, new features are still being added, but the basic functionality has been largely unchanged for a while.  This is what you want from a good tool.  If you're going to invest a lot of time and effort to get good at using a tool, you want that investment to retain its value.  You don't want to spend three months every couple of years re-learning because the tool vendor decided to re-architect their product or switch to the latest trendy programming language.

So while Vim may be old and boring, there's something to be said for being old and boring.  When was the last time you saw an old, boring person on The Jerry Springer Show?  Doesn't happen.  New and interesting may be exciting, but there's a reason why telling someone "may you live in interesting times" is supposed to be a curse.

Here's another "from the archives" post that's been sitting in my drafts folder since March 27, 2007.  At the time I'd been getting into playing Sudoku and was running Kubuntu on my home desktop and laptop.  All I wanted to do was play Sudoku on my computer.  And apparently it didn't go very well.  At the time I wrote:

Why is it that there don't seem to be any good desktop sudoku games for Linux? At least, I haven't found any that meet my requirements. The three in Ubuntu's repositories sure don't.

My requirements for a sudoku game are pretty simple. Beyond the basics of creating a new puzzle and telling me if I'm right when I complete it, I only want one thing: notes. I just want an easy way to add notes to unfilled squares so that I can quickly reference the possibilities.

You wouldn't think that would be such a tall order. However, of the three sudoku games in the Ubuntu repositories, none of them do this. Gnusudoku doesn't even claim to support notes. GNOME-sudoku claims to support them "simply" by clicking at the corner of a box. However, it doesn't work. Maybe you have to run it under GNOME?

And last but not least, we have ksudoku. This one does support notes, and they do work, but they suffer from the same problem as the rest of this app: the user interface makes absolutely no freaking sense at all. Even this might be forgivable if there were a manual or even some kind of guide on the website, but there isn't. The status bar does flash back and forth between "helpful" tips, one of which says to use "RMB" to make notes, but that's it. It took me a few minutes and some experimentation to figures out that "RMB" is supposed to be an abbreviation for "Right Mouse Button". But even with that, I still can't figure out how the hell notes are supposed to work. I can't seem to get more than one number into a note and things disappear if I right-click too many times.

I assume those problems have probably been fixed by now.  I mean, it's been almost 13 years, so either they're fixed or the project is dead.  But I seem to recall this being sort of emblematic of open-source games at the time: there weren't that many of them and even fewer were any good.

By now, I'm sure that the situation is much better, if only because enough time has passed for more games to mature.  But, unfortunately, I don't really care anymore.  These days I don't want to sit down in front of a computer just to play video games.  I just don't have that kind of time.  If I'm going to play a game, it's the kind that runs on my phone and which can be dropped at a moment's notice.

Coincidentally, one such game that I just downloaded the other day is Open Sudoku, which is actually pretty nice as Android Sudoku apps go.  It's open-source and you can find it on F-Droid as well as in the Play store.  Like most open-source games, it doesn't have a lot of frills or fancy graphics, but it's easy to play and does the job.

Pro-tip: the voice recognition feature on your phone is not as smart as you.

Here's the situation: I was reading a political philosophy article that was discussing some ideas of Karl Marx because I was a philosophy major and this is the sort of thing we do in our free time.  Anyway, for some reason I wanted to look up the term "bourgeoisie".  The article didn't use that term, but it came to mind and I wanted to see if my understanding of how Marx used it was correct.  So I started typing the word into Duck Duck Go on my phone and suddenly realized - I don't know how to spell it.  (In my own defense, the word originates in French, which is notorious for spelling things in a way that is not intuitive to native English speakers.  Especially the ones who don't speak French.)

So, being a card-carrying computer geek, what do I do?  In a fit of blind optimism, even though I know better, I push the "microphone" button on my virtual keyboard!  I might not know how to spell "bourgeoisie", but I do know how to pronounce it.  Why struggle trying to figure out the spelling when I can just say it and let Android do the spelling for me?

Yeah, that didn't go so well.  Apparently Google voice control doesn't even know that word.  I tried several times and the closest it got was the quasi-phonetic spelling "booshwazee".  Although, oddly enough, searching Duck Duck Go for "booshwazee" actually does show the results for "bourgeoisie".  So in that sense, I guess it did work.  Kind of.  But not really.

Moral of the story: don't count on your apps to be more literate than the average user, especially in a multi-lingual context.  In retrospect, this should have been obvious.  What was I thinking?

Almost three years ago I wrote a post about setting up a browser plugin for KeePass.  That plugin was chromeIPass and it worked pretty darn well.

Now fast-forward to a few months ago.  My wife's laptop broke down and I had to re-install Windows on it.  In the process, I tried to set up chromeIPass and discovered that it's dead!  Well, mostly dead, anyway.  It's open-source, and the source is still available, but it's no longer available in the Chrome app store.  So it's effectively dead.

So I started looking for alternatives.  The good news is that there's a fork of chromeIPass called KeePassHTTP-Connector. That still exists in the Chrome store.  However, it's also been discontinued!  Apparently it's deprecated in favor of KeePassXC-Browser which is a similar plugin for KeePassXC.  Apparently KeePassXC is a cross-platform re-implementation of KeePass.  I'm not sure why that's needed, since KeePass is written in C# and runs under Mono, and .NET core is now cross-platform anyway, but whatever.  The one nice thing about that browser plugin is that it uses a KeePassNatMsg plugin to communicate with KeePass.  Apparently that's more secure because it doesn't involve talking over HTTP.  But apparently it doesn't work correctly with "real" KeePass.  At least, it didn't for me - the plugin segfaulted when I tried to configure it.

Luckily, I did find a currently supported plugin that actually seems fairly good - Kee.  This is actually intended for a separate password manager, also called Kee, which I gather is some kind of paid service based on KeePass.  (Or something.  To be honest, I didn't really look into it - I only cared about the browser plugin.)  The Kee plugin is based on the old KeeFox plugin for Firefox, but this one also runs in Chrome.  It uses the KeePassRPC plugin for communication with KeePass.

If you used KeeFox in the past, this plugin is equally painless to use and configure.  Just install the KeePassRPC plugin, fire up KeePass, and install the browser plugin.  Kee will automatically attempt to connect to the RPC server and KeePass will prompt you to authorize the connection by bringing up a window with an authorization code.  Just enter that code into the window that Kee opens and click "connect".  Done!  Now when you visit a site that's in your KeePass database, Kee will put icons you can click in the login boxes and auto-populate the login form.  (The auto-population can be turned off - the convenience of that functionality is fantastic, but the security is iffy.)

So at least there's still a good, supported KeePass browser plugin out there.  I suppose this is one of the pitfalls of "roll your own" systems based on open-source software.  Since KeePass doesn't bundle a browser plugin, like many of the proprietary password managers do, we're forced to rely on the community, which can be both good and bad.  The bad comes when the "community" is basically one guy who eventually loses interest.  And while it's great that the source is there for anyone to pick up, it's important to recognize that adopting a new software project requires a substantial time commitment.  Free software is free as in "free puppy", not "free beer".

Here's a lovely UX anti-pattern I discovered today.  I encountered this message while trying to uninstall MediaMonkey, which is a Windows-based music/media manager.

The anti-pattern here is not so much with the pop-up asking if you'd like to "get MediaMonkey Gold for FREE" (note the all-caps on "free"), it's the fact that clicking the "yes" doesn't actually do that.  It just takes you to the generic MediaMonkey download page.  That page says nothing at all about getting the gold version for free.  It just has the comparison matrix for the free vs. gold versions and gives you links to download the free version or to buy the gold version.  Not a word about getting it for free.

This is rather disappointing.  Not because I particularly care about getting MediaMonkey Gold (I would have just bought it if I did), but because up until that moment I had a pretty good opinion of MediaMonkey.  It's not that I didn't like it - I was uninstalling it because I just don't use it.  It's actually a very nice app with lots of good features.  It's just that in the past year the only thing I've used it for is ripping a bunch audio CDs to MP3, and apparently that's a feature that's only available during the "trial period" of the free version.  I haven't used the app in about six months, so my trial period has expired and I didn't feel like paying for something that I use maybe twice a year, hence my decision to just uninstall it.

This just seems like a really shady tactic to discourage people from uninstalling the application.  Offering free stuff to people who threaten to stop using your product seems a bit iffy in the first place, but if you're going to do that you should at least actually send them to a place where they can get the free stuff.  Don't say "you want free stuff?" and then redirect them to the main sales page.  That's the kind of thing you'd expect from a dodgy banner ad, not an otherwise respectable, high-quality software package.

Last year I bought a new car.  And I mean a new new car - a 2019 model year right.  My last two "new cars" were low-mileage "pre-owned" (i.e. "used") cars.  They were fine, but they didn't have any bells or whistles.  In fact, my last one didn't even have power locks or windows.

The new car has all that stuff, though.  And one of those bells and whistles is an entertainment center with a touch screen and support for Android Auto.  This was actually something I really wanted, as opposed to having just the integrated options.  My reasoning was that with Android Auto, I can keep the "brains" of the system in my phone, which can be upgraded or replaced, whereas integrated systems are likely to become outdated and maybe even discontinued before I get rid of the car.

The Reality

The reality of Android Auto, however, is not as great as I'd hoped it would be.  Much of the reason for this is that it's primarily driven by the voice-control features in Google Assistant.  There's some support for typing and menu-driven UI, but it's intentionally limited for "safety reasons."  For example, you can't type into Google Maps while you're driving, nor can you scroll beyond a certain limit in the Plex app, because paying too much attention to the screen is dangerous.

You may have noticed I put "safety reasons" in "sarcasm quotes".  That's because the voice control can sometimes be so frustrating that I find myself more distracted by it than if I could just search for what I needed from a menu.  I end up angry and yelling at the console or just picking up my phone and using it directly rather than the car's console interface.

Let me give you an example.  While driving with my son, he asked to listen to some music.  He wanted to listen to "My Songs Know What You Did in the Dark (Light Em Up)" by Fallout Boy.  So I said to Google, "Plan light 'em up by Fallout Boy."  Google replied, "OK, asking Amazon Music to play 'My Songs Know What You Did in the Dark (Light Em Up)' by Fallout Boy."

Great!  The music started playing and I heard, "Do you have the time, to listen to me whine."  I looked at the screen and Amazon Music was playing Green Day.  Why?  I have no idea.  So I tried again and asked Google, "Play My Songs Know What You Did in the Dark by Fallout Boy."  Once again, Google replied "OK, asking Amazon Music to play 'My Songs Know What You Did in the Dark (Light Em Up)' by Fallout Boy."  And this time, it played the right song.  It claimed to be asking Amazon the same thing both times, so why did one work and the other didn't?  Who knows?

This wouldn't be a big deal if it were an isolated incident, but it's actually pretty common when using Android Auto.  Sometimes it gives you what you ask for, sometimes it doesn't.  Sometimes is claims it's giving you what you asked for, but gives you something else.  And sometimes it just doesn't give you any response at all.

And it's not just Google - I have an Amazon Echo and Alexa does the same sort of thing.  It seems to be a bit more reliable than Google, but it still has problems.

When it works...

The thing is, voice interfaces are really great...when they work as you'd expect.  When you know what you want and the system correctly understands you and maps your request to the right thing, they're like the computers on Star Trek.  And to be fair, they do work well often enough to make them useful.  It's just that when they don't work, I find them unreasonably infuriating.

I think the reason for this is that the discoverability of voice interfaces is limited.  On the one hand, yes, you can just ask your device anything.  But the world of "things you can ask" is so huge that it's overwhelming - it's hard to tell where to start.  And when something as simple as "play X song" doesn't work, it's not obvious where to go next.  Try also including the artist?  Maybe a variation on the title?  All you can do is keep trying random things and hope that one of them works.  Sometimes you stumble on the right thing, and sometimes you don't and just give up in disgust.

It's kind of like trying to use a command-line interface, but with no help and no manual.  OK, so the one command you know didn't work.  What next?  Well, you can try some variations on the arguments you passed it.  Maybe sprinkle in a common option that might be applicable.  But ultimately, it's just blind guessing.

When you're using a graphical interface, you still end up guessing, but at least it's a bounded search space - you're limited to what's on the screen.  Also, the options usually have at least some description of what they do.

If nothing else, these issues with voice interfaces are a good way to relate to non-technical users.  You've seen non-technical users get confused, frustrated, and angry when they encounter a task that they don't know how to accomplish or a problem that they can't seem to fix?  Well, that's how I feel when Android Auto tells me one thing and then does another.  Maybe they'll fix that some time soon....

It's time for the third, slightly shorter, installment of my ongoing series on refactoring my blogging software.  In the first part, I discussed reworking how post publication was done and in the second part I talked about reworking things to add Webmention support.  This time, we're going to talk about two mini-projects to improve the UI for editing posts.

This improvement is, I'm slightly sad to say, pretty boring.  It basically involves fixing a "bug" that's really an artifact of some very old design choices.  These choices led to the existing implementation behaving in unexpected ways when the workflow changed.

The Problem

Originally LnBlog was pretty basic and written almost entirely in HTML and PHP, i.e. there was no JavaScript to speak of.  You wrote posts either in raw HTML in a text area box, using "auto-markup", which just automatically linkified things, or using "LBCode", which is my own bastardized version of the BBCode markup that used to be popular on web forums.  I had implemented some plugins to support WYSIWYG post editors, but I didn't really use them and they didn't get much love.

Well, I eventually got tired of writing in LBCode and switched to composing all my posts using the TinyMCE plugin.  That is now the standard way to compose your posts in LnBlog.  The problem is that the existing workflow wasn't really designed for WYSIWYG composition.

In the old model, the idea was that you could compose your entire post on the entry editing page, hit "publish", and it would all be submitted to the server in one go.  There's also a "review" button which renders your post as it would appear when published and a "save draft" button to save your work for later.  These also assume that submitting the post is an all-or-nothing operation.  So if you got part way done with your post and decided you didn't like it, you could just leave the page and nothing would be saved to the server.

At this point it is also worth noting how LnBlog stores its data.  Everything is file-based and entries are self-contained.  That means that each entry has a directory and that directory contains all the post data, comments, and uploaded files that are belong to that entry.

What's the problem with this?  Well, to have meaningful WYSIWYG editing, you need to be able to do things like upload a file and then be able to see it in the post editor.  In the old workflow, you'd have to write your post, insert an image tag with the file name of your picture (which would not render), add your picture as an upload, save the entry (either by saving the draft or using the "preview", which would have trigger a save if you had uploads), and then go back to editing your post.  This was an unacceptably workflow clunky.

On top of this, there was a further problem.  Even after you previewed your post, it still wouldn't render correctly in the WYSIWYG editor.  That's because the relative URLs were inconsistent.  The uploaded files got stored in a special, segregated draft directory, but the post editor page itself was not relative to that directory, so TinyMCE didn't have the right path to render it.  And you can't use an absolute URL because the URL will change after the post is published.

So there were two semi-related tasks to fix this.  The first was to introduce a better upload mechanism.  The old one was just a regular <input type="file"> box, which worked but wasn't especially user-friendly.  The second one was to fix things such that TinyMCE could consistently render the correct URL for any files we uploaded.

The solution - Design

The actual solution to this problem was not so much in the code as it was in changing the design.  The first part was simple: fix the clunky old upload process by introducing a more modern JavaScript widget to do the uploads.  So after looking at some alternatives, I decided to implement Dropzone.js as the standard upload mechanism.

The second part involved changing the workflow for writing and publishing posts.  The result was a somewhat simpler and more consistent workflow that reduces the number of branches in the code.  In the old workflow, you had the following possible cases when submitting a post to the server:

1. New post being published (nothing saved yet).
2. New post being saved as a draft (nothing saved yet).
3. Existing draft post being published.
4. Existing draft post being saved.
5. New (not yet saved) post being previewed with attached files.
6. Existing draft post being previewed with attached files.

This is kind of a lot of cases.  Too many, in fact.  Publishing and saving were slightly different depending on whether or not the entry already existed, and then there were the preview cases.  These were necessary because extra processing was required when an entry was previewed with new attachments because, well, if you attached an image, you'd want to see it.  So this complexity was a minor problem in and of itself.

So the solution was to change the workflow such that all of these are no longer special cases.  I did this by simply issuing the decree that all draft entries shall always already exist.  In other words, just create a new draft when we first open the new post editor.  This does two things for us:

1. It allows us to solve the "relative URL" problem because now we can make the draft editing URL always relative to the draft storage directory.
2. It eliminates some of those special cases.  If the draft always exists, then "publish new post" and "publish existing draft" are effectively the same operation.  When combined with the modern upload widget, this also eliminates the need for the special "preview" cases.

The implementation - Results

I won't get into the actual implementation details of these tasks because, frankly, they're not very interesting.  There aren't any good lessons or generalizations to take from the code - it's mostly just adapting the ideosyncratic stuff that was already there.

The implementation was also small and went fairly smoothly.  The upload widget was actually the hard part - there were a bunch of minor issues in the process of integrating that.  There were some issues with the other part as well, but less serious.  Much of it was just integration issues that weren't necessarily expected and would have been hard to foresee.  You know, the kind of thing you expect from legacy code.  Here's some stats from Process Dashboard:

As you can see, my estimates here were not great.  The upload part involved more trial and error with Dropzone.js than I had expected and ended up with more bugs.  The draft workflow change went better, but I ended up spending more time on the design than I initially anticipated.  However, these tasks both had a lot of unknowns, so I didn't really expect the estimates to be that accurate.

Take Away

The interesting thing about this project was not so much what needed to be done but why it needed to be done. 

Editing posts is obvious a fundamental function of a blog, and it's one that I originally wrote way back in 2005.  It's worth remembering that the web was a very different place back then.  Internet Explorer was still the leading web browser; PHP 5 was still brand new; it wasn't yet considered "safe" to just use JavaScript for everything (because, hey, people might not have JavaScript enabled); internet speeds were still pretty slow; and browsing on mobile devices was just starting to become feasible.  In that world, a lot of the design decisions I made at the time seemed pretty reasonable.

But, of course, the web evolved.  The modern web makes it much easier for the file upload workflow to be asynchronous, which offers a much nicer user experience.  By ditching some of the biases and assumptions of the old post editor, I was more easily able to update the interface.

One of the interesting things to note here is that changing the post editing workflow was easier than the alternatives.  Keeping the old workflow was by no means impossible.  I kicked around several ideas that didn't involve changing it.  However, most of those had other limitations or complications and I eventually decided that they would ultimately be more work.  

This is something that comes up with some regularity when working with an older code-base.  It often happens that the assumptions baked into the architecture don't age well as the world around the application progresses.  Thus, when you need to finally "fix" that aspect of the app, you end up having to do a bit of cost-benefit analysis.  Is it better to re-vamp this part of the application?  Or should you shim in the new features in a kinda-hacky-but-it-works sort of way?

While as developers, our first instinct is usually to do the "real" fix and replace the old thing, the "correct" answer is seldom so straight-forward.  In this case, the "real" fix was relatively small and straight-forward.  But in other cases, the old assumptions are smeared through the entire application and trying to remove them becomes a nightmare.  It might take weeks or months to make a relatively simple change, and then weeks or months after that to deal with all the unforeseen fallout of that change.  Is that worth the effort?  It probably depends on what the "real" fix buys you.

I had a project at work once that was a great example of that.  On the surface, the request was a simple "I want to be able to update this field", where the field in question was data that was generally but not necessarily static. In most systems, this would be as simple as adding a UI to edit that field and having it update the datastore.  But in this case, that field was used internally as the unique identifier and was used that way across a number of different systems.  So this assumption was everywhere.  Everybody knew this was a terrible design, but it had been that way for a decade and was such a huge pain to fix that we had been putting it off for years.  When we finally bit the bullet and did it right, unraveling the baked-in assumptions about this piece of data took an entire team over a month.  At an extremely conservative estimate, that's well over $25,000 to fix "make this field updatable".  That's a pretty hefty price tag for something that seems so trivial.

The point is, old applications tend to have lots of weird, esoteric design decisions and implementation-specific issues that constrain them.  Sometimes removing these constraints is simple and straight-forward.  Sometimes it's not.  And without full context, it's often hard to tell when one it will be.  So whenever possible, try to have pity on the future maintenance programmer who will be working on your system and anticipate those kind of issues.  After all, that programmer might be you.

I've posted several entries in the past about the benefits of password managers.  Well, I just read a very interesting article that also detailed some of the risks of password managers.

The article is by a security professional who offers a very good take on some of the aspects of password management that I've rarely considered.  For instance, using a password manager can very much entail putting all your eggs in one basket.  For instance, what if you get malware that steals your master password?  What if you forget the password?  That might seem far-fetched, but you never know - you could hit your head, have a stroke, or any number of things.  So in addition to security, there are things like recovery strategy to consider.

While I've been guilty of making blanket statements that "everybody should use a password manager," I now see that that's a mistake.  I still believe password managers are a very good for many, if not most people, but it needs a more nuanced assessment.  Depending on your risk profile and tolerance, you might want to avoid putting all your eggs.  You might want to avoid password managers altogether, or use them only for low-value, or perhaps use multiple password vaults to partition things by importance.

The point is that security is not a one-size-fits-all thing.  There are lots of use-cases and it's important not to get stuck in thinking that yours is the only one or even the most common or important one.  Consider the situation and the trade-offs involved before making a decision or recommending a course of action to others.

I posted an entry a while back about passwords.  Well, turns out that Microsoft is no longer recommending rotating your passwords and is removing the password expiration policies from Windows.

So yup, I was right.

But really, what was the point of always changing your passwords anyway?  Well, I guess it does mitigate against the possibility that a re-used password has been compromised somewhere else.  But what's the cost?  Users can never remember their passwords.  And since coming up with a good, memorable password is hard, they're not going to put in the effort. Instead, they'll come up with some pattern that meets the minimum complexity requirements and just cycle through it. 

Is this better?  I'm no expert, but it's not clear to me that it is.If you want extra protection, there are better ways to get it.  For example, setting up two-factor authentication.  That's also a bit of a pain for users, but at least it provides more real protection.

And on a semi-related side-note, I'd like to point out KeepassAndroid now integrates with Firefox Mobile.  I've mentioned this app in some previous posts about KeePass.  I do a lot of my non-work web browsing on my phone these days, so this app is really a necessity for me.  This new integration is really nice because it means I can now get the same user experience on both my laptop and my phone.  I just tap on the login box and get promoted to auto-fill the form.  Perfect!

Note: This is a short entry that's been sitting in my drafts folder since March 2010, i.e. from half a career ago. My "new" job at the time was with an online advertising startup. It was my first and only early-stage startup experience. In retrospect, it was useful because it exposed me to a lot of new thing, in terms of not only technology, but people, processes, and ways of approaching software development (looking at things from the QA perspective was particularly eye-opening). It was not, however, enjoyable. I've also worked for later-stage startups and found that much more enjoyable. Sure, you don't get nearly as much equity when you come in later, but there's also less craziness. (And let's face it, most of the time the stock options never end up being worth anything anyway.)

Wow. I haven't posted anything in almost six months. I'm slacking. (Note: If only I'd known then that I wouldn't publish this for nine years...)

Actually, I've been kind of busy with work. I will have been at the "new" job for a year next month. The first six months I was doing the QA work, which was actually kind of interesting, as I'd never done that before. I did some functional test automation, got pretty familiar with Selenium and PHPUnit, got some exposure to an actual organized development process. Not bad, overall.

On the down side, the last six months have been a bit more of a cluster file system check, if you get my meaning. Lots of overtime, throwing out half our existing code-base, etc. On the up side, I've officially moved over to development and we're using Flash and FLEX for our new product, which are new to me.

The good part: FLEX is actually not a bad framework. It's got its quirks, but it's pretty powerful and, if nothing else, it beats the pants off of developing UIs in HTML and JavaScript. And while it's not my favorite language in the world, ActionScript 3 isn't bad either. It's strongly typed, object oriented, and generally fairly clean.

The bad part: Flash is not so nice. It wasn't quite what I was expecting. I guess I assumed that "Flash" was just design environment for ActionScript programming. Actually, it's more of an animation package that happens to have a programming language bolted onto it. The worst part is that our product requires that we do the Flash portion in ActionScript 2, which seriously sucks. I mean, I feel like I'm back in 1989. And the code editor in Flash CS4 is...extremely minimal. As in slightly less crappy than Windows Notepad. I am sersiously not enjoying the Flash part.

(Note: On the up side, none of this matters anymore because Flash is now officially dead.)

Note: This is an article I started in October of 2012 and never finished. Fortunately, my feelings on the issue haven't changed significantly. So I filled it out into a real entry. Enjoy!

As I alluded to in a (not so) recent entry on switching RSS readers, I'm anti-cloud.

Of course, that's a little ambiguous. The fact is, "cloud" doesn't really mean anything anymore. It's pretty much come to refer to "doing stuff on somebody else's server." So these days we refer to "having your e-mail in the cloud" rather than "using a third-party webmail service," like we did 15 years ago. But really it's exactly the same thing - despite all the bells and whistles, GMail is not fundamentally different than the Lycos webmail account I used in 1999. It still amounts to relying entirely on some third-party's services for your e-mail needs.

And if the truth were known, I'm not even really against "the cloud" per se. I have no real objection to, say, hosting a site on an Amazon EC2 or Windows Azure instance that I'm paying for. It's really just the "public cloud." You know, all those "cloud services" that companies offer for free - things like GMail and Google Calendar spring to mind.

And it's not even that I object to using these servies. It's just that I don't want to rely on them for anything I deem at all important. This is mostly because of the often-overlooked fact that users have no control over these services. The providers can literally cut you off at a moment's notice and there's not a thing you can do about it. With a paid service, you at least have some leverage - maybe not much, but they generally at least owe you some warning.

There are, of course, innumerable examples of this. The most recent one for me is Amazon Music. They used to[i] offer a hosting service where you could upload your personal MP3 files to the Amazon cloud and listen to them through their service. I kinda liked Amazon Music, so I was considering doing that. Then they terminated that service. So now I use Plex and/or Subsonic to listen to my music straight from my [i]own server, than you very much!

As a result, I have my own implementation of a lot of stuff. This includes running my own calendar server. This is a project that has had a few incarnations, but that I've always felt was important for me. Your calendar is a window into your every-day life, a record of every important event you have. Do you really want to trust it to some third party? Especially one that basically makes its money by creating a detailed profile of everything you do so that they can better serve ads to you? (I think we all know who I'm thinking of here....)

For several years I used a simple roll-your-own CalDAV server using SabreDAV. That worked fine, but it was quite simple and I needed a second application to provide a web-based calendar (basically a web-based CalDAV client). So I decided to switch to something a little more full-featured and easier to manage.

So these days, I just run my own OwnCloud instance. At it's core, OwnCloud is basically a WebDAV server with a nice UI on top of it. In addition to nice file sync-and-share support, it gives me web-based calendar and contact apps with support for CalDAV and CardDAV respectively. It also has the ability to install additional apps to provide more features, such as an image gallery, music players, and note-taking apps. Most of the more impressive apps are for the enterprise version only, or require third-party services or additional servers, but all I really wanted was calendar and contact support.

To get the full experience, I also use the OwnCloud apps on my laptop and phone to sync important personal files, as well as the DAVx5 app on my phone to synchronize the Android calendar and contacts database with my server. Overall, it works pretty well and doesn't really require much maintenance. And most important, I don't have to depend on Google or Amazon for a service that might get canned tomorrow.

About a year and a half ago, I wrote an entry about the first step in my refactoring of LnBlog.  Well, that's still a thing that I work on from time to time, so I thought I might as well write a post on the latest round of changes.  As you've probably figured out, progress on this particular project is, of necessity, slow and extremely irregular, but that's an interesting challenge in and of itself.

Feature Addition: Webmention

For this second step, I didn't so much refactor as add a feature.  This particular feature has been on my list for a while and I figured it was finally time to implement it.  That feature is webmention support.  This is the newer generation of blog notification, similar to Trackback (which I don't think anyone uses anymore) and Pingback.  So, basically, it's just a way of notifying another blog that you linked to them and vice versa.  LnBlog already supported the two older versions, so I thought it made sense to add the new one.

One of the nice things about Webmention is that it actually has a formal specification that's published as a W3C recommendation.  So unlike some of the older "standards" that were around when I first implemented LnBlog, this one is actually official, well structured, and well thought out.  So that makes things slightly easier.

Unlike the last post, I didn't follow any formal process or do any time tracking for this addition.  In retrospect I kind of wish I had, but this work was very in and out in terms of consistency and I didn't think about tracking until it was too late to matter.  Nevertheless, I'll try to break down some of my process and results.

Step 1: Analysis

The first step, naturally, was analyzing the work to be done, i.e. reading the spec.  The webmention protocol isn't particularly complicated, but like all specification documents it looks much more so when you put all the edge cases and optional portions together.  

I actually looked at the spec several times before deciding to actually implement it.  Since my time for this project is limited and only available sporadically, I was a little intimidated by the unexpected length of the spec.  When you have maybe an hour a day to work on a piece of code, it's difficult to get into the any kind of flow state, so large changes that require extended concentration are pretty much off the table.

So how do we address this?  How do you build something when you don't have enough time to get the whole thing in your head at once?

Step 2: Design

Answer: you document it.  You figure out a piece and write down what you figured out.  Then the next time you're able to work on it, you can read that and pick up where you left off.  Some people call this "design".

I ended up reading through the spec over several days and eventually putting together UML diagrams to help me understand the flow.  There were two flows, sending and receiving, so I made one diagram for each, which spelled out the various validations and error conditions that were described in the spec.

That was really all I needed as far as design for implementing the webmention protocol.  It's pretty straight-forward and I made the diagrams detailed enough that I could work directly from them.  The only real consideration left was where to fit the webmention implementation into the code.

My initial thought was to model a webmention as a new class, i.e. to have a Webmention class to complement the currently existing TrackBack and Pingback classes.  In fact, this seemed like the obvious implementation given the code I was working with.  However, when I started to look at it, it became clear that the only real difference between Pingbacks and Webmentions is the communication protocol.  It's the same data and roughly the same workflow and use-case.  It's just that Pingback goes over XML-RPC and Webmention uses plain-old HTTP form posting.  It didn't really make sense to have a different object class for what is essentially the same thing, so I ended up just re-using the existing Pingback class and just adding a "webmention" flag for reference.

Step 3: Implementation

One of the nice things about having a clear spec is that it makes it really easy to do test-driven development because the spec practically writes half your test cases for you.  Of course, there are always additional things to consider and test for, but it still makes things simpler.

The big challenge was really how to fit webmentions into the existing application structure.  As I mentioned above, I'd already reached the conclusion that creating a new domain object for the was a waste of time.  But what about the rest of it?  Where should the logic for sending them go?  Or receiving?  And how should sending webmentions play with sending pingbacks?

The first point of reference was the pingback implementation.  The old pingback implementation for sending pingbacks lived directly in the domain classes.  So a blog entry would scan itself for links, create a pingback object for each, and then ask the pingback if its URI supported pingbacks, and then the entry would sent the pingback request.  (Yes, this is confusing.  No, I don't remember why I wrote it that way.)  As for receiving pingbacks, that lived entirely in the XML-RPC endpoint.  Obviously none of this was a good example to imitate.

The most obvious solution here was to encapsulate this stuff in its own class, so I created a SocialWebClient class to do that.  Since pingback and webmention are so similar, it made sense to just have one class to handle both of them.  After all, the only real difference in sending them was the message protocol.  The SocialWebClient has a single method, sendReplies(), which takes an entry, scans its links and for each detects if the URI supports pingback or webmention and sends the appropriate one (or a webmention if it supports both).  Similarly, I created a SocialWebServer class for receiving webmentions with an addWebmention() method that is called by an endpoint to save incoming mentions.  I had originally hoped to roll the pingback implementation into that as well, but it was slightly inconvenient with the use of XML-RPC, so I ended up pushing that off until later.

Results

As I mentioned, I didn't track the amount of time I spent on this task.  However, I can retroactively calculate how much code was involved.  Here's the lines-of-code summary as reported by Process Dashboard:

For those who aren't familiar, the "base" value is the lines of code in the affected files before the changes, while the "total" it the total number of lines in affected files after the changes.  The magic number here is "Added & Modified", which is essentially the "new" code.  So all in all, I wrote about a thousand lines for a net increase 700 lines.

Most of this was in the new files, as reported by Process Dashboard below.  I'll spare you the 31 files that contained assorted lesser changes (many related to fixing unrelated issues) since none of them had more even 100 lines changed.  

It's helpful to note that of the 717 lines added here, slightly less than half (315 lines) is unit test code.  Since I was trying to do test-driven development, this is to be expected - the rule of thumb is "write at least as much test code as production code".  That leaves the meat of the implementation at around 400 lines.  And of those 400 lines, most of it is actually refactoring.

As I noted above, the Pingback and Webmention protocols are quite similar, differing mostly in the transport protocol.  The algorithms for sending and receiving them are practically identical.  So most of that work was in generalizing the existing implementation to work for both Pingback and Webmention.  This meant pulling things out into new classes and adjusting them to be easily testable.  Not exciting stuff, but more work than you might think.

So the main take-away from this project was: don't underestimate how hard it can be to work with legacy code.  Once I figured out that the implementation of Webmention would closely mirror what I already had for Pingback, this task should have been really short and simple.  But 700 lines isn't really that short or simple.  Bringing old code up to snuff can take a surprising amount of effort.  But if you've worked on a large, brown-field code-base, you probably already know that.

Yesterday I started part one of 723 in what will be my ongoing attempt to get my photo collection under control.  This is a task that I've been thinking about and putting off for literally years because it's so daunting.  But when one of our cats died yesterday (rest in peace, Loki) and I was looking for pictures of him, I realized that it's time to actually do it.

My problem is simply that I have an absolutely huge collection of pictures and videos.  Ever since we got our first digital camera, I've been very liberal about taking pictures on vacations and such.  I mean, they're free, so why not take a bunch so that at least one comes out nice?  And, of course, ever since we got smart phones, I've been very liberal about taking pictures all the freaking time.  I have the OneDrive app on my phone configured to automatically sync them to the cloud and then down to my laptop, so I have multiple copies and they get included in my normal laptop backups.

All that works great, but the issue is that I now have well over a decade worth of pictures and videos - over 12,000 files taking up about 60GB.  How the heck do you find anything in that many files?  My current organizational "system" (if you can call it that) is just to manually sort pictures from my camera roll into "albums", i.e. folders.  That's fine as far as it goes, but it's not very helpful if I want to find anything at a more granular level, e.g. pictures of a specific cat.

Step 1 of 723: Make a Plan

Ultimately, I want to be able to have some nice photo albums and still be able to search for individual photos based on date and content, e.g. pictures of my son from when he was two years old.  I  also want to be able to able to easily create and share photo albums, both with family and with myself through other devices, e.g. tablets, TVs, etc.

And, by the way, I really don't want to be locked into one application or service.  I've been collecting these photos for over 15 years and based on my experience with the software industry it's not unlikely that my collection will outlast any vendor I happen to pick.

So the plan is to start by adding tags to all my pictures.  Yes, this will take along time, but the idea is that once I have all the pictures tagged with the people and places that are in them, I can more easily search them, which will allow me to refine my tags as well as pick out meaningful groupings for albums.

To do the tagging and management, I decided on Digikam.  I picked this for several reasons:

1. It's open-source and cross-platform (it runs well on Windows, despite being a KDE project), so it avoids lock-in on that level.
2. It can write the tags I'm setting directly to the picture metadata.  So while Digikam has and uses its own database, I'm not tied to that.  The canonical source for metadata is the images themselves, which avoids a different level of product lock-in.
3. It's pretty fast and makes it easy to navigate images.  The hierarchical tag structure Digikam supports is actually pretty nice for navigation.
4. It's extremely powerful and gives you a wide range of editing capabilities and publishing/export options.

So far, I'm still working my way through adding all the takes.  Obviously that's going to take a while.  The process is tedious, but Digikam is working pretty well for adding the tags.  The interface takes a little getting used to, if only because there are so many options and tools, but once I got the hang of it, the tagging was pretty easy and went pretty fast.

The up side of the tedious tag adding is that I have an excuse to go back through many years worth of memories.  There are a lot of pictures in there that I haven't looked at in years and it's nice to be reminded.  This is also a good opportunity to clean up duplicate pictures and ones that are just junk, e.g. out of focus, too dark to see, or contain mostly my finger.

Last year I posted about adopting Vim as my default editor.  I've been leveling up my Vim knowledge since then, so I thought I'd write a summary of the current state of my ad hoc Vim IDE configuration.

First, let's start with some sources of information and inspiration.  I found Ben McCormick's Learning Vim in 2014 series to be very useful.  The Vim as Language post in particular offers an eye-opening perspective on the beauty of using Vim.  I've also been working my way through Steve Losh's Learn Vimscript the Hard Way, which is a great source for information on customizing your Vim configuration.  And if you want a little inspiration to give Vim a try, here is Roy Osherove's "Vim for Victory" talk from GOTO 2013.

So how has my Vim adoption gone?  Pretty well, actually.  When I look back at my original post on looking for a new editor, it's pretty clear that a sufficiently customized Vim meets all my criteria.  However, to be fair, it did take a while to get it sufficiently customized.  The customization I've done wasn't actually that hard, but it took some time to figure out what I needed, what I could do, and then Google how to do it.  But paradoxically, that's one of the strengths of Vim - it's been around long enough that pretty much everything that you might want to do either has a plugin available or has been documented someplace on the web, so you rarely need to write any original code.

My Plugins

These days there are actually quite a few plugin managers available for Vim.  The nice thing about this is that they all support the same plugin format, i.e. GitHub repositories laid out in the standard ~/.vim directory format.  I'm currently using Plug because it provides an easy mechanism for selective or deferred plugin loading (in the case where you have a plugin that's not always needed and slows down Vim startup).

Here are some of the goodies my Plug plugin list currently contains:

• scrooloose/nerdtree - A nice file explorer plugin that provides some enhancements to Vim's native file browser.  Pretty much a must-have for every Vim setup.
• ctrlpvim/ctrlp.vim - A fuzzy file finder that works on buffers and other stuff too.  Just press ctrl+p and start typing the name you want.
• jlanzarotta/bufexplorer - A handy plugin to list and switch between the current buffers.  Think of it as like the tab strip at the top of other editors, but easier to deal with from the keyboard.
• tpope/vim-dispatch - A nice plugin for running external programs asynchronously.  By default, external command execution blocks the rest of the UI until the command is done.  This is fine sometimes, but not others.  Dispatch integrates with other plugins and provides a way to run things in the background and get their output back into Vim.
• tpope/vim-surround - Provides a Vim movement that helps you manipulate surrounding entities.  Good for things like changing quotes, HTML tags, etc.
• Chiel92/vim-autoformat - Provides an interface to various code formatters.  I use it as a replacement for the JSON beautifying feature that I loved so much in Komodo.
• mileszs/ack.vim - A quick and easy integration of the ack! text search tool.  Like the built-in grep, but better.
• joonty/vim-sauce - Sauce is a handy little plugin for managing multiple configuration files.  It's also useful for adding the concept of a "project" to Vim.  I use it to create project-specific configurations that handle all the customization that would be done in the project file of a regular IDE.
• janko-m/vim-test - A unit test runner plugin that handles many different tools.
• vim-airline/vim-airline - An enhanced status line that's both pretty and displays some useful information.
• w0rp/ale - The Asynchronous Lint Engine, this offers syntax and style checking with inline error notifications, just like in a regular IDE.
• majutsushi/tagbar - A tool for displaying the tags in the current file, similar to the structure browsers found in IDEs.

Needless to say, I also made a number of other customizations to my Vim configuration.  My full work-in-progress Vim configuration is in this repo if you're interested.  I do not hold this up as a great example of how to configure Vim, but it's working for me so far and, as previously noted, it actually wasn't all that much work.

The IDE Functionality

So what functionality do I have with this setup?  Well, it turns out I actually get most of what I previously had with Komodo.  Of course, I need to integrate with a few external packages for this, the key ones being Exuberant CTags, which indexes identifiers in the code, and ack for text search.  I also need various external formatters and linters, though the specific programs depend on what language I'm coding in.  Nothing fancy, though - they're pretty much all command-line executables that you can just drop someplace in your path.

So here's what I get for my trouble:

• Insanely powerful key bindings.  I mean sersiously powerful - I don't think there's anything you can do in Vim that can't be bound to a keystroke.  And it's usually pretty easy.  Just the other week I defined a couple of ad hoc key bindings to help me add translation keys to a web template.  It's really a beautiful thing.
• Inline syntax and style checking.  Using ALE in conjunction with the appropriate external linters, I get the same kind of inline checking I can get in PHPStorm.
• Navigating identifiers.  Using Vim's ctag support, it's possible to navigate uses and definitions of a particular, for example, much like the code browsing abilities of PHPStorm.  Of course, it's not perfect because ctags lack knowledge of the context of the identifier, but it's not bad.  (And to be fair, I've seen the code navigation in PHPStorm and Komodo fall over on more than one occasion.)
• Searching.  Between CtrlP and Ack, I have some nice facilities for searching for or within files.  Again, very similar to what I had in Komodo or PHPStorm.
• Project management.  Between NERDTree and Sauce, I have some decent support for the concept of project.  They give me a nice file tree navigation panel and the ability to define project-specific configuration.

Conclusion

The short version is that this Vim experiment is going pretty well.  Granted, it is somewhat more work than figuring out a traditional IDE.  But on the other hand, it's not that bad and actually isn't as much work as I thought it would be.

In terms of functionality, I find that I haven't actually given up very much.  In fact, if you're talking about multi-language IDEs, I'm not even sure I've given up anything I care about.  It turns out that Vim is remarkably capable and the plugin ecosystem is very large and deep.

Would I recommend this to someone who's never touched Vim before?  Probably not.  But if you're familiar with Vim and interested in trying a new way of working, it might be worth a shot.  At worst, you'll improve your Vim skills and enjoy using a nice, fast editor that doesn't eat up half a gigabyte of RAM when it's just sitting there doing nothing.

Backups are important.  Everybody in IT knows this.  After all, whenever someone tells you they lost some important file, the first question is always, "Well do you have a backup?"

Nevertheless, many of us are lax in setting up our own backup solutions.  I know I was for many years.  And while that might not be forgivable for people who really should know better, it is understandable.  After all, implementing a proper backup solution is a lot of work and the benefits are not immediately obvious.  Think about it: it requires an ongoing investment of both time and money which, in the best-case scenario (i.e. nothing goes wrong), will never pay off.  Is it any wonder that even IT professionals aren't eager to put a lot of effort into something they actively hope never to need?

But about a year ago, I finally put a home backup solution in place.  That solution was CrashPlan.  But now CrashPlan is discontinuing their home plans, which means I've been forced to figure out something else.

The old setup

CrashPlan's "home" plan was actually pretty nice.  Perhaps too nice, since apparently it wasn't profitable enough for them to keep offering it.  The subscription was priced at $150 per year and that included up to ten computers in your household and unlimited cloud storage.  On top of that, their agent software supported not only backing up to the cloud, but also backing up to external hard drives and other computers running the CrashPlan agent.  And it ran on Windows, Linux, and Mac!

Currently, I have three computers in active use in my house: my laptop, my wife's laptop, and my desktop, which is really more of home server at this point.  I had both of the laptops backing up to the cloud, while the desktop backed up to both the cloud and an external hard drive.  I've got close to a terabyte of data on the desktop, so the external drive is important.  I do want a copy of that data off-site just in case of a catastrophe, but I'd rather not have to suck that much data down from the cloud if I can avoid it.

I'm happy with this setup and I wanted to keep something equivalent to it.  I feel like it provides me with sufficient protection and flexibility while not requiring me to buy extra hardware or pay for more cloud storage than I need.

The alternatives

Turns out this setup isn't quite as easy to replicate as I had hoped.  There are plenty of home backup services out there, but most of them don't offer the same range of options.  For instance, many are Mac and Windows only - no Linux.  Many offer limited storage - usually a terabyte or less, which I'm already pushing with my largest system.  Many are cloud backup only - no local drives.  And when you add up the cost of three systems, most of them are more expensive than CrashPlan Home was.

On their transition page, CrashPlan recommends that existing home customers either upgrade to their small business plan or switch over to Carbonite.  I briefly considered upgrading to the small business plan, but the price is $10/month/device.  So I'd go from $150/year to $360/year for basically the same service.  That doesn't sound like a great deal to me.

Carbonite, on the other hand, is one of the options I considered the first time when I settled on CrashPlan.  They're offering a 50% discount to CrashPlan customers, so the initial price would only be $90.  Presumably that's only for the first year, but even after that $180/year is only slightly more than CrashPlan.  However, from what I can see Carbonite doesn't support Linux on their home plan - they only do Linux servers on their office plan.  I also don't see an option to back up to an external drive.  Although it does support backing up external drives to the cloud...for another $40/year.

Plan B

After doing some research and thinking about it for a while, I eventually decided to skip the all-in-one services.  Yeah, they're nice and require almost zero work on my part, but I wanted some more flexibility and didn't want to pay an arm and a leg for it.  However, I didn't want to completely roll my own solution.  Writing simple backup scripts is easy, but writing good backup scripts with proper retention, special file handling, logging, notifications, etc. is a lot of work.

If you don't want a full-service backup solution, the alternative is to go a la carte and get a backup program and separate cloud storage provider.  There are a number of options available for both, but after some research and experiments, I decided to go with CloudBerry Backup for my backup program and Backblaze B2 as my storage provider.

The choice of Backblaze B2 was kind of a no-brainer.  Since this is long-term backup storage, performance is not a huge concern - it was mostly a question of capacity and price.  B2 has standard "pay for what you use" billing and the prices are extremely reasonable.  Currently, storage is $0.005 per gigabyte and there's no charge for uploads.  So for the volume of data I have, I'm looking at storage costs of $5 or $6 per month, which is pretty cheap.

The backup program was a different story.  I tried out several options before settling on CloudBerry.  Most of the options I tried were...not user friendly.  For me, the CloudBerry UI had the right balance of control and ease of use.  Some of the other solutions I tried were either too arcane or simplified things too much to do what I wanted.  CloudBerry uses a wizard-based configuration approach that makes it relatively painless to figure out each step of your backup or restore plan.  I find that this allows them to expose all the available options without overwhelming the user.

As far as capabilities, CloudBerry pretty much has what I wanted.  It supports Windows, Mac, and Linux using multiple storage solutions, including local file systems, network file systems, and various cloud providers.  Beyond that, the feature set depends on the version you use.  There's a free version, but I went with the paid desktop version because it supports compression and encryption.  The licenses are per-system and they're a one-time change, not a subscription.  The basic "home" licenses currently run $50 for Windows and $30 for Linux, which I think is pretty reasonable.

Results

So far, the combination of CloudBerry and B2 for my backup solution is working well.  I've been using it for about five months and have all three of my systems backing up to the cloud and my desktop also backing up to a USB hard drive.  The process was largely painless, but there were a few bumps along the way.

As I mentioned in a previous post, as part of this process I moved my desktop from Windows to Linux.  As it turns out, setting up the two laptops to have CloudBerry back up to B2 was completely painless.  Literally the only annoyance I had in that process was that it took quite a while for the confirmation e-mail that contained my license key to arrive.  So if you're a Windows user, I can recommend CloudBerry without reservation.

Linux wasn't quite as simple, though.  I had a number of problems with the initial setup.  The interface is very stripped-down compared to the Windows version and doesn't offer all of the same options.  I also had problems getting the backups to run correctly - they were repeatedly stalling and hanging.  Fortunately, the paid license comes with a year of support and I found the CloudBerry support people to be very helpful.  In addition, they seem to be actively working on the Linux version.  I initially installed version 2.1.0 and now they're up to 2.4.1.  All of the issues I had have been resolved by upgrading to the newer versions, so things are working well now.

I had initially been a little concerned about the per-gigabyte and per-transaction pricing, but so far it hasn't been an issue.  I found Backblaze's storage cost calculator to be pretty accurate and the per-transaction charges are not significant.  The cost has been basically what I initially estimated and I haven't had any surprises.

Overall, I'm very happy with this solution.  The price is reasonable and, more importantly, it provides me with lots of flexibility.  Hopefully I'll be able to keep this solution in place for years to come.

The first part of my LnBlog refactoring is kind of a large one: changing the publication logic.  I'll start by giving an overview of the old design, discussing the new design, and then looking at some of the actual project data.

History

LnBlog is a very old system.  I started it back in 2005 and did most of the work on it over the next two years.  It was very much an educational project - it was my first PHP project (in PHP 4, no less) and only my third web-based project.  So I really didn't know what I was doing.

As you might expect, the original design was very simple.  In the first version, "publishing an entry" really just meant creating a directory and writing a few files to the disk.  As the system grew more features, more and more steps were added to that process.  The result was a tangle of code where all of the logic lived in the "controller" (originally it was a server-page) with some of the actual persistence logic encapsulated in the domain objects.  So, roughly speaking, the BlogEntry object knew how to write it's metadata file to the appropriate location, but it didn't know how to properly handle uploaded files, notifications, or really anything else.  

Originally, LnBlog used a server-page model, with each URL being a separate file and code being shared by including common configuration files and function libraries all over the place.  Shortly prior to starting this project, I had consolidated the logic from all those pages into a single, massive WebPages class, with one public method for each endpoint - a monolithic controller class, for all intents and purposes.  This still isn't great, but it gave me a common entry point to funnel requests through, which means I can better control common setup code, handle routing more easily, and generally not have the pain of dealing with a dozen or more endpoints.

Anyway, this WebPages class served up and edited entries by directly manipulating domain objects such as BlogEntry, Blog, BlogComment, etc.  The domain objects knew how to save themselves to disk, and a few other things, but the majority of the logic was right in the WebPages class.  This worked fine for the main use-case of creating an entry from the "edit entry" page, but it made things very awkward for the less-used locations, such as the Metaweblog API endpoint or scheduled publication of drafts.

Furthermore, the publication logic in the WebPages class was very hairy.  All types of publication flowed through a single endpoint and used a common code path.  So there were flags and conditions all over the place, and it was very difficult to tell what needed to be updated when making changes.  Bugs were rampant and since there was no test automation to speak of, testing any changes was extremely laborious and error prone.

The New Design

There were two main goals for this refactoring:

1. Create a new Publisher class to encapsulate the publication logic.  The idea was to have a single entity that is responsible for managing the publication state of blog entries.  Given a BlogEntry object, it would know how to publish it as a regular entry or a static article, unpublish it, handle updating or deleting it, etc.  This would give us a single entity that could own all the steps involved in publishing.
2. Create unit tests around the publication process.  The logic around the entire process was more complicated than you'd think and the old code was poorly structured, so it broke with disturbing regularity.  Hence I wanted some automated checking to reduce the risk of bugs.

So the design is actually pretty straight-forward: create a "publisher" class, give it operations for each of the things we do with blog entries (create, update, delete, etc.), copy the existing logic for those cases into the corresponding methods, update the endpoints to use this new class, and call it a day.  

So it was mostly just a reorganization - there wasn't any significant new logic that needed to be written.  Simple, right?  What could go wrong?

Results

While I was happy with the result, this project turned out to be a much larger undertaking than I'd assumed.  I knew it was going to be a relatively large task, but I was off by a factor of more than two.  Below is a table summarizing the project statistics and comparing them to the original estimates (from PSP data I captured using Process Dashboard). 

When I originally did the conceptual design and estimate, I had planned for relatively modest changes to the Article, BlogEntry, and WebPages classes and the creators.php file.  I also planned for new Publisher and BlogUpdater classes, as well as associated tests and some tests for the WebPages class.  This came to 29 hours and 946 new or changed lines of code across nine source files.  Definitely a big job when you consider I'm working in increments of two hours or less per day, whenever I get around to it.

In actuality, the changes were much larger in scope.  I ended up changing 27 additional files I hadn't considered, and ended up creating two other new utility classes (although I did ditch the BlogUpdater class - I no longer even remember what it was supposed to do).  The resulting 3138 lines of code took me 64 hours spread over five months.

Effects of Testing

I did test-driven development when working on this project.  I've found TDD to be useful in the past and it was very helpful to me here.  It was also very effective in meeting my second goal of building a test suite around the publication logic.  PHPUnit reports statement coverage for the Publisher tests at 97.52% and close to 100% coverage for the tested methods in the WebPages class (I only wrote tests for the endpoint that handles creating and publishing entries).

More importantly, using TDD also helped me to untangle the logic of the publication process.  And it turns out there was actually a lot to it.  I ended up generating about 2000 lines of test code over the course of this project.  It turns out that the design and unit test phase occupied 65% of the total project time - about 41 hours.  Having a comprehensive test suite was immensely helpful when I was rebuilding the publication logic across weeks and months.  It allowed me to have an easy check on my changes without having to load all of the code I worked on three weeks ago back into my brain.

Sadly, the code was not such that I could easily write tests against the existing code.  In fact, many of the additional changes came from having to break dependencies in the existing code to make it possible to unit test.  Luckily, most of them were not hard to break, e.g. using an existing file system abstraction layer, but the work still adds up.  It would have been very nice to have an existing test suite to prevent regressions in the rewrite.  Unfortunately, even integration tests would have been awkward, and even if I could have written them, it would have been very hard to get the level f coverage I'd need to be confident in the refactor.

Conclusion

In terms of the results, this project worked out pretty well.  It didn't really go according to plan, but I got what I was looking for in the end - a better publication design and a good test suite.  However, it was a long, slow slog.  Maybe that was too big a slice of work to do all at once.  Perhaps a more iterative approach could have kept things moving at a reasonable pace.  I'll have to try that on the next big project.

Today, we're going to talk a little about design and refactoring.  As a case-study, we're going to use a little blogging application called LnBlog.  You probably haven't heard of it - it's not very popular.  However, you have used it, at least peripherally, because it's running this site.  And you also have at least a passing familiarity with the author of that application, because it's me. 

Motivation

Software is an "interesting" field.  The cool new technologies, frameworks, and languages get all the press and they're what everybody wants to work with.  But let's be honest: it's generally not what makes the money.  I mean, how could it be?  It just came out last week!

No, if you have the good fortune to work on a grown-up, profitable product, it's almost certainly going to be the "old and busted" tech.  It might not be COBOL or Fortran, but it's almost certainly "legacy code".  It might be C++ or Java or, in our case, PHP, but it's probably old, poorly organized, lacking unit tests, and generally hard to work with.

I work on such a product for my day job.  It's a 10-year-old PHP codebase, written in an old-fashioned procedural style.  There are no unit tests for the old code, and you couldn't really write them even if you wanted to.  Sure, there's a newer part with proper design, tests, etc., but the old code is the kind of stuff that "works for the most part", but everybody is afraid to touch it because it's so brittle and tightly coupled that God alone knows what will break when you make a change.

This also applies to LnBlog.  It was my very first PHP application.  I started it way back in early 2005, in the bad old days of PHP 4.  Over the next two or three years, I managed to turn it into something that was relatively functional and full-featured.  And for the last ten years or so, I've managed to keep it working.

Of course, it hasn't gotten a whole lot of love in that time.  I've been busy and, for the most part, it worked and was "good enough".  However, I occasionally need to fix bugs or want to add features, and doing that is a truly painful process.  So I would very much like to alleviate that pain.

The Issue

Let me be honest: I didn't really know what I was doing when I wrote LnBlog.  I was about four years out of school and had only been coding for about six or seven years total.  And I was working mostly in Visual Basic 6 at the time, which just barely counts.  It was also only my third web-based project, and the first two were written in classic ASP and VBScript, which also just barely counts.

As a result, it contains a lot of questionable design decisions and overly-complicated algorithms.  The code is largely procedural, kind of buggy, and makes poor use of abstraction.  So, in short, it's not great.

But, in fairness to myself, I've seen worse.  In fact, I've seen a lot worse.  It does have a class hierarchy for the domain objects (though it's a naive design), an abstraction layer for data access (though it's inconsistently used), and a templating system for separating markup from domain logic (though the templates are an ungodly mess).  And it's not like I had a role model or mentor to guide me through this - I was figuring out what worked on my own.  So while it's not great, I think it's actually surprisingly good given the circumstances under which it was built.

The Goal - Make the Code "Good"

So I want to make LnBlog better.  I've thought about rewriting it, but decided that I wouldn't be doing myself any favors by going that route.  I also hold no illusions of a grand re-architecture that will fix all the problems and be a shining beacon of design perfection.  Rather, I have a relatively modest list of new features and bug fixes, and I just want to make the code good enough that I can make changes easily when I need to and be reasonably confident that I'm not breaking things.  In other words, I want to do a true refactoring.

If you haven't read Martin Fowler's book, the word "refactoring" is not a synonym for "changing code" or "rewriting code".  Rather, it has a very specific meaning: improving the internal design of code without changing the external behavior.  In other words, all you do is make the code easier to work with - you don't change what it does in any way.  This is why people like Bob Martin tell you that "refactor X" should never be an item in your Scrum backlog.  It is purely a design and "code cleanliness" activity, not a "feature" you can deliver.

So my goal with LnBlog is to gradually reshape it into what it should have been in the first place.  This is partially to make changing it easier in the future.  But more importantly, it's a professional development goal, an exercise in code craftsmanship.  As I mentioned above, I've worked professionally on many systems that are even more messed up than LnBlog.  So this is a study in how to approach refactoring a system.  

And So It Begins...

My intention is to write a number of articles describing this process.  I've already completed the first step, which is rewriting some of the persistence and publication logic.  I'm using the PSP to track my planned and actual performance, so I'll have some actual data to use in my discussion of that process.  Hint: so far, the two are very different.

With any luck, this project will enable me to draw some useful or interesting conclusions about good and bad ways to approach reworking legacy systems.  Maybe it will also enlighten some other people along the way.  And if nothing else, I should at least get a better codebase out of it.

So, as I mentioned before, I'm looking for a new go-to text editor/IDE.  So far, I've taken cursory looks at a few of the options.  These include:

1. PHPStorm.  As I mentioned in the last post, I use PHPStorm at work.  And while it's really good in a lot of ways, I'm not in love with it.  On the up side, it's got great code intelligence and the VI-mode plugin is actually quite good.  On the down side, it's a single-language IDE (well, not quite, but it's still got a limited set of supported languages).  I guess I could just buy the entire JetBrains IDE suite, but I'm not crazy about switching back and forth.  Also, PHPStorm is kinda heavy - like, Eclipse heavy - both in terms of memory footprint and, more important, conceptual weight of the UI.  So while I don't dislike it, I don't really have any enthusiasm for it.
2. Visual Studio Code.  I initially liked the look of VS Code.  It's got the cool Visual Studio intellisense, which is nice, and it seems to have a lot of extensions available.  The Vim-emulation plugin seemed fairly good, but not great.  The most popular PHP plugin, however, didn't seem to work out of the box at all.  I'm not sure why, though it could be its wonky install process/implementation (apparently it's written in PHP).  At any rate, I didn't care enough to look into it, though it might be worth taking a closer look at VS Code at some point.
3. Atom.  I liked the look of Atom.  I read a little about the philosophy behind it and I really wanted to like it.  But then I fired it up and tried opening one of my project directories and it didn't work.  And by "didn't work", I mean that Atom actually crashed, consistently, on this particular directory.  So that's a no-go.  A quick Google revealed that it might be a problem with the Git library, which could possibly be fixed by changing the index version on the repo, but frankly I don't care.  If I can't trust my editor to just open a directory, then I can't trust it at all.  I mean, I don't even care if my editor has Git support at all, so I'm certainly not going to accept crashing if it sees a repo it doesn't like.  
4. Sublime Text.  I've heard good things about Sublime.  I used to work with several people who really liked it.  Then I fired it up and immediately said, "What the heck is this?"  The UI is pathologically minimal, except for a gajillion menu items and the stupid friggin' mini-map (which I didn't like in Komodo either).  Configuration is done by editing a JSON file, but what the heck is with the weird out-of-box-experience?  The UI is extremely minimal and customization is done by editing a JSON file, which is weird (to be fair, VS Code and Atom do that too, but it's more forgivable because they're free), and the plugin manager was immediately confusing.  Seemed like getting used to Sublime might be a steep learning curve.
5. Vim.  Yes, you read that right - Vim.  I was surprised too.  Let me explain.

After trying out Sublime, my initial reaction was, "Geez, if I want something that complicated, why don't I just use Vim?"  And then I stopped.  And I said to myself, "Actually...why don't I use Vim?"  Good Vim emulation is one of my must-haves, and no plugin is ever gonna beat the real thing.  It's free, open-source, hugely customizable, has lots of available plugins, and is extremely well established.

The thing is, I knew Vim could be customized into a pseudo-IDE, but I'd never really thought of myself as a hard-core Vim user so I'd never tried it.  But the truth is that I've been a Vim user for a very long time, and for the last few years I've been actively trying to pick up more Vim tricks for use in Vim emulation plugins.  So while I didn't know an awful lot about customizing Vim, I'm very comfortable actually editing code in it.

And it turns out that actually customizing Vim isn't really that bad.  Heck, there are even package managers for Vim now!  There are also IDE-like configuration/plugin bundles you can install, such as spf13, but I quickly determined that those were too big and overwhelming.  However, they are good as a source of ideas and settings to copy into your own custom ~/.vimrc file.  That's actually part of the beauty of Vim - despite the fact that Vim script is a little weird and the configuration is in no way intuitive, there's enough information already out there that it doesn't really matter.

So over the course of a week or so, I pulled out some of the more interesting settings from the spf13 config, found a selection of plugins that I liked, and got myself a nice, functional Vim setup.  I even set up some symlinks and file syncing so that I can have my setup synchronized between home and work.  

Is it perfect?  No, but nothing ever is.  But so far, it's working pretty well.  And what's more, I actually enjoy using it.  It might not have the power of a specialized IDE when it comes to the more advanced features, but it's got a heck of a lot of power in general.  And the amount and types of customization you can do are amazing.  With a little effort, you can really shape the editor to your workflow, which is a thing of beauty.

A couple of weeks ago I started looking into new code editors.  I've been using Komodo for eight or nine years now, counting both KomodoIDE and KomodoEdit, but I'm growing dissatisfied with it.  In fact, I'm becoming unsatisfied enough that I think the time has come to move on.  However, I'm not sure I see an obvious candidate for a replacement.  So in this post, I'll talk about some of the things I find lacking in Komodo and what I'm looking for in an editor.

Why the change?

Let me start by saying that I'm not trying to bad-mouth Komodo here.  I have used it for a long time and it has served me well.  The people at ActiveState are nice and they do good work.  But the direction it seems to be heading in doesn't mesh well with my needs and I'm not sure the cost/benefit analysis of sticking with Komodo makes sense anymore.  Maybe this can provide some insight to the Komodo development team that can help them improve the IDE.

My dissatisfaction started a few months ago, when my team transitioned to a different product.  Unlike the last product we worked on, this one doesn't belong to just us.  This one is core to the company's business, and has several other development teams working on it, not to mention all the other groups that come in contact with it.  As such, there's an existing process and structure that we needed to adhere to, and it quickly became apparent that adhering to that was much easier if you were running the standard development setup, which is PHPStorm on Ubuntu.  I was running KomodoIDE on Windows, so this was a problem.  I was able to use WSL to get around the Linux part, but KomodoIDE just didn't offer the same features that I needed from PHPStorm.

So let's use that as a starting point.  What does PHPStorm offer that Komodo can't compete with at present?  Well, for purposes of the product I'm working on, here's the list:

1. Code intelligence.  This is by far the biggest factor.  I'm not just talking about intellisense here.  I mean finding and navigating the actual places in the code where a class, function, or method is used or defined.  And not just text search, but actually knowing about the class and its inheritance hierarchy.  PHPStorm is actually pretty awesome at that.  Komodo's code intel., at least least for PHP and JavaScript, is buggy at best and totally broken at worst.  The quality of the experience also seems to vary hugely depending on the codebase you're working with.  It's nice that they have something for code intel., but you can't really rely on it.
2. Validations.  PHPStorm integrates with phpcs, which is nice because we need to actually adhere to PSR-2.  Komodo doesn't have that built in.  This might seem like a trivial thing because, after all, I can always just run phpcs from the command line.  However, having the check done in the editor is actually hugely useful, because we have a lot of legacy code that doesn't adhere to PSR-2, which makes selectively running the checks for only the code you changed awkward.  Seeing violations in your editor gives you a nice, simple way to catch mistakes you made before they get to code review.
3. Symfony support.  While it's not built in, PHPStorm has a pretty good quality plugin for Symfony support.  We use Symfony, so this is really nice.  Komodo doesn't have that.

Those things are important, but they are specific to the new product I'm working on.  If these were my only complaints, I would happily continue using Komodo for the foreseeable future and just use PHPStorm for that one project at work.  But they're not.  The list of annoyances and things I don't like has been slowly growing over the years.  This includes actual problems and bugs, missing functionality, and road-map issues (i.e. disagreement with the direction I see Komodo going).  Here's a summary:

1. Kinda buggy.  I hate to say it, but I've seen a decent amount of weird errors, crashes, and just plain strange behavior in Komodo.  Not a huge number - it certainly hasn't been buggy enough to get switch editors - but it's still a non-trivial issue.  I'll give just a few examples here to give you the flavor of my complaints.
   1. Maybe it's just my perception, but it's rare for the error log or console to not have something in it.
   2. Sometimes when I'm doing things in the "places" panel, some keypress will trigger a change to the next pane of that panel.  I'm not sure what it is and I can't seem to do it on purpose.
   3. I'm constantly empty getting "find" panes accumulating in my bottom panel.  Again, I'm not 100% sure what causes them and can't seem to reproduce intentionally.
   4. I've tried numerous times on numerous versions of Komodo to set up the keybindings for "focus X pane", but they just don't seem to work at all.
   5. There are various places in the settings UI where one setting determines another, but it's not clearly indicated.  So you can change a setting, click OK, and your change will just be ignored because it's invalid, but there's no indication of that.  A good example is the new unit test runner.  For options other than "custom", the testing framework determines which parser is used, but you can still change the parser in the UI.  It just doesn't do anything.
   6. The syntax checking preferences for JavaScript allow you to set a custom JSHint version to use, probably because the integrated one is kind of old.  I've tried this numerous times and have never been able to get it to work.  Oh, and speaking of JSHint, I'm still kinda miffed that Komodo 10 removed the graphical JSHint configuration UI.  Now there's just a text area to enter your settings, so you have to go to the JSHint site and look up the options rather than just being able to pick from a list.
2. Pretty over functional.  In the last few releases, some of the tools have been revamped in such a way that makes them prettier, but in my opinion doesn't actually make them more useful.  The two big examples that spring to mind are version control and unit testing.
   1. In older versions of Komodo, my only complaint about the VCS integration was that there wasn't an easy way to do a commit of all pending changes in the project - I fixed that myself with a simple macro.  In the new version, they fixed that problem.  But at the same time, the VCS widget no longer displays changes for added files, which is a big pain.  I typically use my VCS diff for pre-commit code review, so it's kind of inconvenient to have to switch to another view to do that.
   2. The new unit test runner in Komodo 10.2 looks very pretty.  However, they removed the key bindings I was using to run the current test suite.  And it didn't detect my old test suites, so I had to recreate them.  They also changed the name-humanizing algorithm, so that test names that used to be rendered nicely in the runner aren't anymore.
3. Features I don't care about.  It feels like there have been a few of these added lately.  Some of them seem like things that are really just gimmicks that look good in marketing material but don't provide any value.  Others do provide genuine value, but seem tacked on to pad the feature set, i.e. they're useful, but I don't see the point of having them right in the IDE.  Some examples include:
   1. Collaboration.  You can do real-time editor sharing with someone else using KomodoIDE.  Cool!  And maybe it would be useful if I was doing remote pair programming with someone else who uses Komodo.  But I'm not, so I've never actually used it.  I suppose this could substitute for some of those "social coding" web editors, but this doesn't feel like a general enough use-case to want it integrated into my IDE.
   2. Sharing to kopy.io.  Upload code snippets directly to a code-sharing website.  Nice!  But again, that's something that's not hard to do without IDE support and that I seldom or never do it anyway.  And even if I did, I'd just create  gist on GitHub, not use a new site.
   3. Slack sharing.  You can share code in a Slack channel right from inside Komodo.  Great!  But again, I don't do this often and it's not clear how this is easier than just copy-and-pasting the code into Slack.
   4. Minimap.  A 10,000-foot overview of how your code looks that replaces the traditional scroll bar.  I think Sublime Text had this first.  Sure, it looks really cool, but does anyone actually use these things?  I don't spend a lot of time looking for code segments based on the shape of the text, so it's never been clear to me what useful information this provides.
   5. HTTP inspector.  This is actually an older feature - an HTTP proxy that allows you to inspect traffic.  And it's a genuinely valuable thing for a developer to have.  But you still have to set it up like any other HTTP proxy, so it's not clear how having this baked into the IDE is better than just using a stand-alone proxy app.  And again, this isn't something I need on a regular basis so I've never used it for actual work.
4. Features that are strange or hard to use.  There are also features that I use, or would like to use, that either don't make sense or don't quite do what I need.  In other words, they could be really good and useful, but they fall short.  For instance:
   1. Keyboard navigation.  Komodo actually has a pretty nice configuration interface for setting up keyboard shortcuts.  Unfortunately, a lot of the actual bindings you might want don't exist (or don't work, as previously noted).  But my big gripe is that navigating around the UI using the keyboard is difficult, particularly in the side/bottom panels.  Trying to navigate between fields within a pane often either doesn't work, gets stuck, switches you to the main editor, or otherwise fails in strange ways.  And as I mentioned earlier, trying to use the keyboard to navigate between different panes seems to just generally not work.
   2. Regex toolkit.  This seems like a really useful tool.  But I'll be darned if I can figure out how it's supposed to work.  Every now and then I try it and I always spend more time trying to figure out how it works than it would take so just write a one-off test script to test the regex.
   3. Publishing.  Komodo has a publishing tool that lets you push your code up to a remote system for testing.  That's a very nice and useful thing.  Except that it's actually a "synchronizer," by which I mean it only does two-way synchronization of files with a remote server.  Why?  What if I don't care what's on the server and just want to clobber it every time?  That's not such an uncommon occurrence with test servers.  In fact, for me, wanting to pull in changes from the remote servers is distinctly an edge-case, not something I'd want to happen by default.

I could probably go on, but I think I've made my point.  It's not that Komodo is a bad IDE - far from it.  But there are a number of rough edges and niggling little issues that are increasingly starting to bother me.  Choice of editor can be a very personal and subjective thing, and for me it just feels like it's time for a change.

What do I want?

So that leaves the question: what do I want in an editor or IDE?  Well, I'm not completely sure.  I've been using Komodo for a long time, and I do like a lot of things about it.  So let's start with a list of some of those things.  That should at least work as a jumping off point.

1. Good VI emulation.  I've been using VI emulation in my editors ever since I worked for deviantART.  I got a MacBook Pro when I started there and I found the weird keyboard layout completely infuriating, particularly when I switched back to my regular PC keyboard so I decided to just switch to VI key bindings since they're always the same.  Since then, I've gotten progressively more friendly with VI mode, to the point where coding without it feels strange.  Komodo includes fairly decent VI emulation out of the box, and it's relatively easy to add customizations, so any editor I pick up will need comparable VI emulation support.
2. Code formatters.  One of the really handy features of Komodo is built-in code formatters.  My most common use-case for this is copying some JSON out of the bowser network monitor, pasting it into Komodo, and formatting it so I can analyze it.  It would be really nice for a new editor to support something like that.
3. Light-weight.  A concept of "projects" is nice in an IDE, but sometimes I just want to quickly open a file.  So I'd rather not use an IDE that takes ages to start up and insists that everything be part of a project (I'm looking at you, Eclipse).  Komodo is pretty good in that regard - it can just start up and open a file without it being too much of an ordeal.
4. Extensibility.  No editor or IDE is ever perfect out of the box, so it's important to have a decent extension ecosystem around your editor.  Komodo is pretty good in this regard, with a community site offering a nice assortment of extensions.
5. Scriptability.  In addition to extensions, one of the things Komodo gets really right is giving you the ability to easily write simple scripts to automate things.  It lets you write user scripts in JavaScript that can be fairly easily hooked into the UI.  This is huge.  There are a lot of "small wins" you can achieve with this kind of scripting that will really improve your workflow.  Supporting extensions is great, but it's hard to justify integrating a feature into your IDE if you need, e.g., an entire Java build environment to write an extension for something you could do in five lines of shell.
6. Multi-language.  This is a big one.  If you regularly code in more than one language, having to learn a different IDE for each one is a real drag.  In the best-case scenario, you have to configure the same things for each editor.  In the worst-case scenario, you have to learn two completely different tools with completely different features.  Most of my professional work is in PHP and JavaScript these days, but I also do some Python, SQL, BASH and PowerShell scripting, and I'm trying to learn Haskell.  So given the choice, I'd rather learn one editor inside and out than have a bunch of language-specific editors that I use as "notepad with some extra features".
7. Cross-platform.  Right now I use Windows both at home and at work, but that's not set in stone.  I actually use Windows at work by choice (yeah, yeah, I know) - there are a few people with MacBooks, but most of the other developers use Ubuntu.  In the past, I've gone back and forth between platforms, so running on Windows, Linux, and MacOS is a hard requirement for me.  I don't want to be tied to one platform or have to learn a different IDE for each if I decide to switch.
8. The right feel.  This one is highly subjective, but it's important to me that my IDE feel smooth to use.  I want it to work with me, rather than feeling like I have to adapt to it.  This has always been my problem with Eclipse - I feel like the UI doesn't really adapt smoothly to what I want.  Something about coding in it just feels a little "off" to me.

 Time to experiment

So it's time to start doing some experimentation.  It took me a long time to finally settle on Komodo, so I'll probably go back and forth a few times.  

I've got lots of choices, to be sure.  Back when I settled on Komodo Edit, I was looking primarily at free and open-source editors.  My use of Komodo IDE grew out of that.  These days, I'm not as price-sensitive, so commercial IDEs are definitely on the table, provided they have reasonable non-Enterprise pricing (i.e. I'm not gonna spend $2000 on one).

Last time I was IDE shopping I looked at a bunch of Linux-only editors, but those are out.  I used Eclipse for a while, but I'm not inclined to revisit that.  I also used jEdit for a while, and while it was fairly nice, it doesn't appear to be getting much active development these days, and I'm not sure it fits my current requirements anyway.  But now we have things like Sublime Text, Atom, Visual Studio Code and the entire suite of Jet Brains IDEs.  So I've got lots of things to look at.  If nothing else, the process will be good for a few blog posts.

I discovered something enlightening and annoying this morning: PowerShell uses different execution policy settings for the 32-bit and 64-bit versions.  This is something I did not know and did not expect.

I'd seen this problem before, but could never figure out what was happening.  I'd try to run a command through PowerShell using another program, in this case Vim, and it would fail with the standard error about execution of my profile.ps1 being prohibited by the execution policy setting.  Yet when I bring up Powershell on the command line, everything works and the execution policy looks correct:

PS pgeer> Get-ExecutionPolicy -List Scope ExecutionPolicy ----- --------------- MachinePolicy Undefined UserPolicy Undefined Process Undefined CurrentUser Undefined LocalMachine RemoteSigned

Well, it turns out we're talking about two different versions of Powershell.  When I run it from the terminal, I'm using the 64-bit version.  But my Vim build is 32-bit, which apparently means that it's running the 32-bit version of Powershell.  And that version uses its own execution policy setting.  You can confirm that by explicitly opening "Windows Powershell (x86)" from the start menu and running Get-ExecutionPolicy.  

As for the fix, it's simple: just run Set-ExecutionPolicy RemoteSigned as admin, but do it in both the x86 and x64 versions of Powershell.

As for why Microsoft chose to make the different builds use different execution policies, I couldn't say.  It doesn't make much sense to me.  Although, to be honest, I'm not sure why I even need both versions on the same system.  Maybe to support interop between commandlets and unmanaged code?  Who knows?

In my last post about KeePass, I mentioned that you can integrate your KeePass password database with your web browser.  In this post, I'll tell you more about how to do that and why it's an extremely handy thing.

Why bother?

So why would you want to bother with integrating your browser with KeePass?  I mean, most browsers have a feature to remember your passwords anyway, so why not just use that?  Or if you want to use KeePass, why not just use that auto-type feature I talked about in the last post?

It's true, you could just use the password manager that's built into your browser.  Pretty much all of them have one, these days.  Most of them will even secure your data with a master password.  They may even synchronize your passwords to the cloud, so you can access them on more than one device.  Granted, that's pretty handy.

However, browser password managers generally just do passwords - they don't allow you to enter extra information or attach files like KeePass does.  They also don't work for things outside the web browser, like for security software such as VPN clients.  So they don't provide you with a single, secure location for all your important account information.  But more importantly, they're generally tied to a single browser.  Sure, Google Chrome can store and synchronize all my passwords, but what if I decide I don't like Chrome anymore?  Maybe I just bought a Mac and decided I really like Safari.  Is there an easy way to get my passwords out of one browser and into another?  I don't know.  

By using KeePass with a plugin for your browser, you can get the best of both worlds.  KeePass itself gives you more power and features than browser password managers and allows keeps you from being tied to a single browser.  Using a browser integration plugin adds on the ability to have the browser automatically fill in your username and password when you visit a website.  It's not quite as convenient as the browser-integrated password managers, but it still pretty good.  And it's definitely a lot easier than trying to use auto-type or copy-and-paste to fill in password forms.

What are my options?

In general, there are a lot of plugins available for KeePass.  Just look at the list.  Or maybe don't - you probably don't care about 90% of those plugins.  The main thing you need to know about is which browsers have plugins available. 

Short answer: Chrome, Firefox, and Safari.

Long answer: Chrome, Firefox, and Safari have proper browser plugins available.  The Chrome plugin also works in Vivaldi and possibly other browsers that are based on Chrome.  There are also form-filling plugins that work with Internet Explorer.  To my knowledge, there is no plugin support available for Microsoft Edge.

For this entry, I'll just talk about setting up a plugin with Chrome.  We're going to use a Chrome extension called ChromeIPass.  It adds a KeePass button to the toolbar in Chrome and can automatically detect login forms on webpages you visit.  It works with a KeePass plugin called KeePassHttp.

First, you need to install the KeePassHttp plugin.  Start by going to the KeePassHttp website and clicking the "download" link, or just download it directly here.  Sadly, KeePass doesn't have a nice way to install plugins - you just have to copy the plugin file to the KeePass plugins folder on your system.  Inconvenient, but fortunately not something you need to do very often.  On most computers, this will be C:\Program Files (x86)\KeePass Password Safe 2\Plugins.  So just copy the KeePassHttp.plgx file that you downloaded and paste it into that location.  Since this is a system directory, you will probably be prompted to grant access.  Click "continue" to copy the file.  Note that if KeePass is running, you will need to close and restart it for it to detect the plugin.

Now that the KeePassHttp plugin is installed, KeePass will be able to communicate with Chrome.  You just need to install the ChromeIPass extension.  You can do that by going to the Chrome web store page here and clicking the "Add to Chrome" button.  

So...now what?

OK, now that ChromeIPass is installed, what do you do with it?  Well, not really much until it's time to log into a site.  So pick a site that's in your KeePass database and go there - I'll use sourceforge.net for this example because it's a pretty standard login form.

The first time you try to log into a site using ChromeIPass, you'll need to connect it to your KeePass database.  You should notice a KeePass icon is now in your toolbar.  Make sure KeePass is running and click that button.

You should see a "Connect" button.  Click that and KeePass will prompt you to add a new encryption key for the KeePassHttp plugin.  This is a security mechanism - the KeePassHttp plugin encrypts its communication with your KeePass database and this is just the initial step where it sets that up.  Don't worry about the details right now - just type in a unique name for the key, maybe based on your browser and computer, e.g. "Laptop - Chrome".  You only have to do this the first time you connect a browser to your database - after that, the encryption is automatic.

Now that ChromeIPass is connected to your KeePass database, you can click the ChromeIPass button in your toolbar and click the "Redetect Credetials Fields" to fill in your username and password.  Alternatively, you can just refresh the webpage and they should be auto-filled.  You won't see anything in the browser yet, but KeePass itself ill prompt you to allow access to the password for this site.  You can check the "Remember this decision" box to not be prompted to allow access the next time you visit this site.

(I should probably stop to acknowledge that this thing of having to grant a site access to your KeePass database before you can log in is kind of a drag.  I agree, it is somewhat annoying.  This is actually a security feature of KeePassHttp - that's the portion of this that runs inside KeePass itself and allows the ChromeIPass extension to talk to it.  It actually has a lot of security-related settings.  This is actually a good thing, though, because it essentially provides a way for other programs to read your KeePass database, and you want to make sure that malware or dodgy websites aren't able to do that.  However, if you want to disable some of these settings, like prompting to allow access, you can do that by going into KeePass and selecting the "Tools > KeePassHttp Options" menu item.  The KeePassHttp documentation has some more information on the available settings.)

The good news is that now you're done!  After you allow access to KeePass, ChromeIPass will automatically fill in your username and password.  If you selected the "remember" option when allowing access to the site, ChromeIPass will automatically fill in your login info the next time you visit the site, no action required.  You will only have to allow access the first time you visit a new site of if you elect not to have KeePass remember the approval.

If you're so inclined, ChromeIPass has a number of other features, as detailed in the documentation.  For instance, it can save or update entries automatically when you enter a password into a webpage; it has a built-in password generator that lets you create strong passwords right in the browser; it can customize the login fields for non-standard login forms; and it provides a handy right-click menu to fill in passwords and access other functionality.  

Hopefully this will help get you started.  Using a password manager is a must for keeping your accounts secure these days, and integrated browser support makes using one that much easier, which means you're more likely to keep using it.

You should be using a password manager.  If you're a technical person, this is probably not news to you - you're very likely already using one.  

This article is for the non-technical people.  People like my wife (hi, honey!) and my mom.  People who access a lot of websites and have a lot of passwords to remember.

Security 101

So why is using a password manager a good idea?

Well, you may have seen guidelines for cyber security that tell you things like:

1. Don't write down your passwords.
2. Don't reuse passwords on different sites.
3. Don't use short, easy to guess passwords.
4. Don't use passwords that are easy to figure out from public data (like a birthday that anyone can get from your Facebook profile).

Such guidance raises the question: if I have to use long passwords that aren't related to anything in my life, and I can't reuse them or write them down, how the hell am I supposed to remember them?!?

This is a totally reasonable question.  Yes, ideally we would all memorize a hundred different 32-character-long, randomly generated passwords.  But in real life, nobody can actually do that.  So a password manager is a good compromise.

What is a Password Manager

My mother has a little paper "password book" that she keeps in a drawer next to her computer.  When she has to create a new account for some website, she writes down all the login information in that book so that she can look it up later.

A password manager is the digital equivalent of that password book.  It's an application that lets you record your login information and them look it up later.  Most password managers have lots of other handy-dandy features as well, but that's the core of what they do.

So how is this different from, say, writing down all your passwords in a Word document on your desktop?  Well, a password manager encrypts all your data.  It requires a "master password" to decrypt your information, so if some nasty hacker steals that file, they won't be able to actually read it.  

Is this as secure as just memorizing all your passwords?  No.  But as we said, nobody can do that anyway, and this is one heck of a lot more secure than the alternatives, i.e. reused or weak passwords.  With a password manager, you can still have strong, unique passwords for all your sites, but you're relieved of the burden of having to remember them all.

About KeePass

There are a number of password managers out there, but the one I'm going to talk about is KeePass.  It's a free, open-source password management application that will run on Windows, Linux, and Mac, and has compatible apps available for iOS and Android.  KeePass works offline (i.e. it requires no internet connection and doesn't depend on any online services), but it's possible to sync your KeePass passwords between devices using file sync tools like DropBox or OneDrive.  So it provides you some flexibility, but you aren't beholden to a single company that can get hacked or go out of business.

KeePass creates files password files that end with ".kdbx".  You can open those files from within KeePass or double-click on them in Window Explorer.  When you try to open one, KeePass will prompt you for the master password to that file.  Every KDBX file has its own master password.  This allows you to do things like create a password file to share with the rest of your family, and have a different one for the accounts that are just yours.  (That's a topic for a different post.)

One of the handy extra functions of KeePass is that each entry in your password save can have a bunch of extra data associated with it.  For example, you can add custom fields and attach files to each entry, which are handy for things like account validation questions and activation files for software licenses.  Basically, you can keep all the important information in one place.  And since KeePass encrypts your entire password file, it will all be kept secure.

Using KeePass

So how do you use KeePass?  Let's walk through it.

Step 1 - Download

The first thing you need to do is to get yourself a copy of KeePass.  You can go to this page and click the download link for the "professional edition".  (There's not really anything "professional" about it - it's just a newer version with more features.)  When that's done, you can double-click the file to install it like any other program.

You can also install KeePass through Ninite.  If you're not familiar with Ninite, I strongly recommend you check it out.  It's a great tool that makes it brain-dead simple to install and update a collection of programs with just a few clicks.  You basically just select a bunch of applications you'd like to install from a list, click a button, and you get an installer program you can run to put everything you selected on your computer.  And if you run that program again later, it will actually update any of those programs that have a newer version.  It's very slick and a real time-saver.

Step 2 - Create a password safe

 Next, open up KeePass and click "File > New".  You will be prompted to choose where you want to save your new password database.  Choose a name and folder that work for you.  Remember - your password database is just a regular file, so you can always move or rename it later if you want.

After that, you should get a dialog that looks like this:

This shows several options for securing your password safe.  But don't worry about that - the one you really want is the first one, "master password".  So choose a password and type it in.  If you click the three dots on the right, KeePass will display the password as you type, so that you don't have to re-enter it.

There are two important things to note when choosing a master password.  First, since it's going to protect all your other passwords, you want to make it good.  KeePass provides a password strength meter to help you judge, but the main things to bear in mind are that you want a range of different characters and you want it to be long.  And no, ten letters does not qualify as "long" - it should be more of a passphrase than a password.  One common technique is to use a full sentence, complete with capitalization and punctuation (an maybe some numbers, if you can work them in).  That will generally give you a pretty strong password, but it will still be easy to remember.

The other important thing to remember is that the password to a KDBX file is your encription key for that file.  That means that the only way to decrypt the file is with that password.  If you forget your master password, your data is gone forever.  So write down your master password and keep it in a safe place until you're certain you've committed it to memory.  And if you want to change your master password later, make sure to make a backup copy of your KDBX file first.

After you've chosen a master password, you should see a screen that allows you to configure some of the settings for your password file.  However, you don't really need to worry about this - those are all optional.  You can safely click the "OK" button to just continue on.

Step 3 - Organize your passwords

Alright!  You now have a password database set up.  You should see a list of groups on the left and a list of password entries on the right, like in the image below.  These are the sample groups and entries that KeePass creates by default.  They're just to give you an idea of how to use your password database - you can safely delete them at any time.

You can click on each group at the left to see what entries it contains.  The groups are basically like folders in Windows.  There's a top-level folder, and it contains a bunch of sub-folders and each of those sub-folders can contain other folders.  So in the screenshot, you can see that "NewDatabase" is highlighted in the group list.  That's the top-level folder for my example database.  You can see on the right that it contains two entries.  You can move an entry into another folders by dragging it from the entry list on the right onto one of the folders on the left.

Step 4 - Create passwords

To add a password entry to your database, select "Edit > Add Entry" from the menu.  That will bring up the entry edit screen.  This is the same screen you'll see when you double-click on the title of an existing entry, except that it is mostly blank.

There are a lot of tabs and options on this screen, but you don't really need to worry about those.  The main things are right in front of you: the entry title, user name, and password.  You'll probably also want to fill in the URL field with the web address of the site this information is for.  This will come in handy if you want to use a KeePass plugin for your web browser (which we'll cover in another post).  When you're done entering your info, click the OK button to create the entry.  You should then select "File > Save" from the menu or push the "save" button on the toolbar to save the changes to your password database.

You'll probably notice that there's already a password filled in.  KeePass will generate a random password for new entries.  You are free to change this yourself or use the button to the right of the "repeat" box to generate other random passwords using different rules.  KeePass has a password generator that lets you specify the allowed characters and length for a random password, which is handy for those sites that insist on specific password length or complexity rules.

Step 5 - Getting your passwords out

Now let's back up and say you've just started up your computer, are logging in to some website, and want to get a password out of KeePass.  The first thing you need to do is open up your password database.  You can do this by double-clicking on it in Windows Explorer or by opening up KeePass then selecting your database from the "File > Open" menu.  When you open the database, you'll be greeted by a screen asking you to enter your master password - you know, the one you came up with in step 2.  (Hint: remember that you can click the button with the three dots to display the password as you type it.)  After you enter your master password, the database will be decrypted and you'll find yourself and the entry browsing screen from step 3.

There are several ways to get your passwords out of KeePass.  Here's the list in order of preference:

1. Use a browser plugin to automatically fill in login forms.  Since most of the passwords you end up creating are for websites, setting up your browser to fill in the forms from your KeePass database makes life much easier.  I'll talk about how to do that in the next post.  But don't worry - it's not all that hard.
2. Use auto-type.  This is a feature of KeePass where you to click a button in the KeePass window and it will automatically send the keystrokes for your username and password to the last window you used.  So, for example, you would navigate to the login page of a site in your web browser, click in the username box, and then switch over to the KeePass window and click the auto-type button on the toolbar (the one that looks kind of like some keyboard keys - hover your cursor over the buttons to see the descriptions).  By default, the auto-type feature will type your username, the "tab" key, your password, and then the "enter" key.  This will work for probably 90% or more of login pages, but it's not universal, so be aware of that.
3. Copy them to the clipboard.  If all else fails, you can always just copy your passwords to the clipboard so that you can paste them into another window.  KeePass makes this fairly easy.  In the main password list that you saw in step 3, when you double-click on the actual username or password for an entry in the list, it will copy that to the clipboard.  This saves you having to open up the entry edit screen and copy things there.  You can then switch to another window and paste the data into a login screen.
4. Just read it.  Last, but not least, you can always go low-tech and just read the passwords out of the window.  Just double-click the name of your entry, then click the "three dots" button to make the password visible.  Clearly this is not great, but sometimes it's necessary.  For example, you will need to do this when entering a password on a system that doesn't have KeePass installed, such as to login into your Amazon or Netflix account when setting up a Roku or other streaming media system.

Conclusion

With any luck, I've made this "password manager" thing sound good enough for you to try it out.  You really should look into it.  Password reuse has become increasingly dangerous, with hackers trying the usernames and passwords they harvested from one hack on other sites just to see if they work.  Password cracking tools have also advanced a lot in recent years, including information gleaned from previous attacks, so relying on things like "133t 5p34k" passwords is no longer good enough.  A decent password manager, if used consistently with randomly generated passwords, will provide you with a good trade-off between convenience and security.

Disclaimer: This is yet another "note to self" post.  If you're not me, feel free to ignore it.

After God alone knows how many years (at least six, since I have posts related to it from 2010), it's finally time up upgrade the version of Mercurial that I have installed on my shared web hosting account.  This is a shared hosting account with no shell access - nothing but web-based tools and FTP.  I also don't know what OS it's running - just that it's some form of Linux.  So I've been putting this off for obvious reasons.

Unfortunately for me, the defaults for repository creation in Mercurial 3.7 turn on general delta support by default.  That isn't supported by the old version I was running (1.7), so my choices are to either use the now non-standard, older, and less efficient format my repositories, or just bite the bullet and upgrade.  So I did the latter, since the version I had was pretty ancient and I was going to have to do it eventually anyway.

Fortunately, my hosting provider supports Python 2.7, which gets you most of Mercurial.  However, there are some C-based components to Mercurial.  Since I have no shell access to the hosting server, and there are probably no development tools installed even if I did, I had to try compiling on a VM.  I was able to do that by spinning up a Fedora 24 VM (on the assumption that they're running RHEL, or something close enough to it), and doing a local build.  The only caveat was that apparently my provider is running a 32-bit OS, because building on a 64-bit VM resulted in errors about the ELF format being incorrect.

Once the Fedora VM was up and running, I was able to do a build by running the following:
sudo dnf install python-devel
sudo dnf install redhat-rpm-config
cd /path/to/mercurial-3.8.x
make local

That's about it.  After I had a working build I was able to copy the Mercurial 3.8 folder to the server, right over top of the old version, and it just worked.  Upgrade accomplished!

Tonight I finally got around to fixing my trackpad.  Again.

So here's the thing: like most Windows-based laptops, my little Lenovo ultrabook has a trackpad with two "button" regions at the bottom.  They're not separate physical buttons, but they act as the left- and right-click buttons.  However, rather than force you to use the right-click pseudo-button, the Synaptics software the comes with the laptop lets you turn off the right-click button and use two-finger-click as the right-click, a la a MacBook Pro.  I find this preferable, in particular because my laptop centers the trackpad under the space bar, rather than in the actual physical center of the unit, which means that when you're right-handed, it's easy to accidentally hit the right-click button when you didn't mean to.

Prior to upgrading to Windows 10, I had this all set up and it was fine.  After the upgrade, not so much.  Sure, the same options were still there, but the problem was the that option to turn off the right-click button did just that - it turned it completely off!  So rather than clicking in that region doing a standard left-click, that part of the trackpad was just dead, which was even worse than the problem I was trying to fix.

Luckily, it turns out that the functionality is still there - the Synaptics people just need some better UX.  I found the solution in this forum thread on Lenovo's site.  It turns out you can just change the value of the registry key HKEY_CURRENT_USER\Software\Synaptics\SynTP\TouchPadPS2\ExButton4Action to 1, which is apparently the standard left-click action.  By default, it's 2, but when you turn off the "Enable Secondary Corner Click" (i.e. right-click to open context menu) feature in the Synaptics UI, that gets changed to 0, which is apparently the "do nothing" action.

Long story short: my mouse is much better now.

So as part of my ongoing professional improvement program, I've been working my way through PSP: A Self-Improvement Process for Software Engineers.  And by "working", I mean I'm doing the exercises from the SEI website and everything.  So far it's actually quite an interesting process - I'd definitely recommend it to any professional software developer, even if you're not interested in using the process, just as "food for thought".  You can pick up a relatively cheap used copy of the book on Amazon (it is technically a textbook, so new ones are a little pricey).  I'll have to write a post on it when I'm finished.

Anyway, the PSP uses line-of-code counting to estimate program size, defect densities, etc.  So I thought it would be nice to be able to run line count reports right from within Komodo.  Fortunately, it turned out to be pretty easy.  I just lifted some code from Nathan Rijksen's "Open Terminal Here" macro as a starting point and went from there.  The macro just takes the selected files or directories in you "places" pane and appends them to a custom command.  I've used cloc as my line-counting tool of choice for a number of years, but you can change the command to be whatever you want (even "wc -l" if you really want).  I also added in a little code to calculate and run from a common base directory, so that you wouldn't get a full, absolute path for every item in the report.  The command output is sent to the Komodo output pane.

The code for the macro is below.  Or, if you're feeling lazy, you can just download the tool here and drop it in your toolbox.

/**
 * Adds a "Count LOC" menu item to items in the Places widget.
 * This will run a line-counter on the selected paths and display the command output.
 * Based on the "Open Terminal Here" macro by Nathan Rijksen.
 *
 * Usage: Update the "command" variable to contain whatever command you want to run.  The paths to the files selected
 * in the places pane will be appended to this command.
 *
 * If the "use_common_directory" variable is true, then the macro will calculate the deepest common directory of all
 * the selected files and will run the command from that directory, passing relative paths.  For example, if you
 * select files /foo/bar/baz/buzz.js and /foo/bar/fizz/fuzz.css, the command will be run from /foo/bar and will be
 * passed the paths baz/buzz.js and fizz/fuzz.css.
 * If this variable is set to false, then the command will be passed the full, absolute paths to all files and no
 * working directory will be specified for it.
 *
 * @author Peter Geer
 * @contributor Nathan Rijksen
 * @contributor Mathieu Strauch
 * @version 0.1
 */
/*global ko, extensions:true */

// Register namespace
if ((typeof extensions) == 'undefined') {
    extensions = {};
}
extensions.CountLOC = {};

(function() {
    
    var command = "cloc --by-file --force-lang=PHP,phtml",
        use_common_directory = true,
        label = 'Count LOC',
        id = 'contextCountLOC',
        sibling,
        d,
        mi,
        callback,
        longest_common_path;
    
    longest_common_path = function(list) {
        var longest_item = list[0].substr(0, list[0].lastIndexOf('/')),
            i = 0;
        for (i = 0; i < list.length; i++) {
            while (longest_item !== '' && list[i].indexOf(longest_item) < 0) {
                longest_item = longest_item.substr(0, longest_item.lastIndexOf('/'));
            }
            if (longest_item === '') {
                break;
            }
        }
        return longest_item;
    };
    
    callback = function(e) {
        var i = 0,
            cmd = command,
            curr_dir = null,
            uris = ko.places.viewMgr.getSelectedURIs();
        
        if (uris.length === 0) {
            return;
        }
        
        // Clean up the URIs.
        for (i = 0; i < uris.length; i++) {
            uris[i] = uris[i].replace(/^[a-zA-Z]+:\/\//,'');
            if (uris[i].match(/^\/[a-zA-Z]:\//)) {
                uris[i] = uris[i].substr(1);
            }
        }
        
        // If set, turn the absolute paths into relative paths.
        if (use_common_directory) {
            curr_dir = longest_common_path(uris);
            if (curr_dir !== '') {
                for (i = 0; i < uris.length; i++) {
                    uris[i] = uris[i].substr(curr_dir.length + 1);
                }
            } else {
                curr_dir = null;
            }
        }

        // Prepare command for each platform
        for (i = 0; i < uris.length; i++) {
            cmd += ' ' + uris[i];
        }

        // Run command, show output in bottom pane
        ko.run.command(cmd, {cwd: curr_dir, runIn: 'command-output-window'});
    };
    
    // Get places pane document object
    d = document.getElementById('placesViewbox').contentDocument,

    // Remove existing menu entry if it exists
    mi = d.getElementById(id);
    if (mi) {
        mi.parentNode.removeChild(mi);
    }

    // Get the sibling element which we want to insert our menu item after
    sibling = d.getElementById('placesContextMenu_rename');
    
    // Create our menu item
    mi = document.createElement("menuitem");
    mi.setAttribute("id", id);
    mi.setAttribute("label", label);

    // Add event listener for when the menu item is used
    mi.addEventListener('command', callback);

    // Append menu item to popupmenu
    if (sibling && sibling.parentNode) {
        sibling.parentNode.insertBefore(mi, sibling);
    }

}.apply(extensions.CountLOC));

Just a quick note on a small customization to Tiny Tiny RSS.  If you're not aware of TT-RSS, it's the online RSS aggregator that I switched to after Google Reader closed down.  It's got all the features I care about, has a fairly nice web UI and there are mobile apps and mobile-web front-ends available.  And despite what it says in the official system requirements about needing a dedicated server, it works perfectly on my el cheapo shared hosting account - wasn't even hard to set up.

Anyway, I recently switched the configuration for the web viewer from the "default" theme to the "night" theme, which is a white-on-black color scheme.  I decided to try that because it matches better with a lot of the development tools I'm using (Komodo IDE, Visual Studio, command prompts - all light text on dark backgrounds).  The only problem is that, unlike the default theme, which shows headlines of posts you've read in a different color, the night theme doesn't visually differentiate read and unread posts.

Fortunately, this is easy to fix.  You can just open up the preferences and there's an option to customize your stylesheet.  To change the highlight color, copy something like the following into the custom CSS box.

body#ttrssMain .hl .hlTitle a {
    color: #878787;  /* A slightly darker gray to use for "read" posts. */
}

body#ttrssMain .hl.active .hlTitle a,
body#ttrssMain .hl.Unread .hlTitle a {
    color: #CCC;  /* Normal headline color for the "night" theme. */
}

It's upgrade time again.  As I mentioned in my last entry, my desktop/home server had been getting short on disk space - and by "short on disk space," I mean I started to see Explorer reporting the free space as less than 1GB on my C: drive.  So it was time for a new hard drive.

Since the only real problem was my 100GB C: partition, I decided to go with more of a "start over" approach and just got a modest 120GB SSD.  My Windows 7 installation was about 5 years old, so it had accumulated a fair amount of cruft.  For instance, the winsxs folder alone had swollen to almost 14GB.  So I reasoned that a clean installation would probably fix most of my problems.

Along with the new hard drive, it was also time for some new software.  I started that out with an OS upgrade from Windows 7 Pro to Windows 8.1.  Yes, I know - everybody hates Windows 8.  But I think it's a good OS, damn it!  Sure, I'll grant you that the initial release had some rough edges, but aside from the questionable decision to remove the start menu (which is trivially fixed by installing Classic Start), the 8.1 update fixes every complaint I had.

As a change of pace, this time I decided to try the downloadable purchase of Windows 8.1 from NewEgg rather than waiting for physical media to ship.  It turns out that this process is actually pretty simple.  You place your order and then get an e-mail with a license key and a link to a downloader program.  You run that, give it your key, and it gives you several options for downloading the installation files.  One of these is to just download a bootable ISO image that you can burn to disk.  So it's actually not as wierd as I initially feared.  Of course, the one catch is that the downloader runs under Windows, so this probably doesn't work so well if you're a Mac or Linux user.

The one thing of note here is that this time I decided to save myself a few bucks and drop down from the Professional edition to the standard one.  I made this decision after considering the non-transferability of the OEM version and looking at the feature comparison matrix.  It turns out the the Pro versino contained exactly one feature that I cared about: the Remote Desktop Server.  So I reasonsed that if I could find a suitable remote access solution to replace RDP, then there was no need to buy the Professional edition.  And after playing around with TeamViewer for a few days, I decided I'd found that. 

It turns out that TeamViewer actually works quite well.  For starters, it's free for non-commercial use and it's in Ninite, so there's basically zero overhead to getting it set up.  Registering for an account is also free and helpful, though not strictly necessary.  The performance seems to be pretty good so far and it has the ability to start LAN-only connections or go through their servers for over-the-Internet access.  After using TeamViewer for a couple of days, I was more than convinced that I could do without the Windows RPD server.

Next on the list was a service to run Virtual Box.  As you may know, Virtual Box is a free system virtualization package.  It works pretty well, but it doesn't come with the ability to run a VM on boot (at least in Windows) - you have to wait until you log in.  To fix that, I installed VBoxVmService.  This is a little Windows service that will run selected VMs on boot without anyone having to log in and also offers a systray app that allows you to manage those VMs.  Previously, I had been using the similarly named VirualBoxService, which does essentilally the same thing but isn't quite as nice to use.  Of course, there are some limitiations, but for the most part it works well enough for my setup.  All I really wanted to do was have a Linux VM run on boot to serve as a web server because setting the stuff up on Windows was just too much of a pain.

While I was at it, I also decided to give Plex a try.  I'd previously been a little turned off by the requirement to create an account, even though I only wanted to use it on my LAN, but it turns out that actually isn't necessary.  The account registration is really only needed for remote access.  And with Android and Roku apps, Plex provided more functionality and required less work than my home-grown solution of using PHP scripts to generate customized web server directory listings for Roksbox.  That was all well and good, but just using Plex is much easier and seems to work better.

So far, things seem to be going pretty well and I'm happy with my new setup.  Granted, there are no radical changes to my setup, but in my days of Linux on the desktop, painful upgrades were not always such an uncommon occurrence, so I guess I'm a little battle-scarred.

One last thing to note is that I'm actually kind of impressed with Windows 7.  I days gone by, a Windows install lasting for 5 years of heavy use would be unheard of.  And even with seriously limited hard drive space, the system was rock-solid and actually performed pretty well.  If I'd been inclined to migrate it to the new drive, I probably could have kept that installation going for much longer.  Switching back to Windows was definitely a good move.

This entry started out as a quick discussion of consolidating my photos onto Amazon Cloud Drive.  However, that didn't happen.  So now this entry is about why.

This started out as an attempt to clean up my hard drive.  I have a 1TB hard drive in my desktop, divided into two partitions: 100GB for my system "C: drive" and the rest of it allocated to assorted data.  The problem is that my C: drive was down to less than 5GB free, so it was time to do some clean-up.

Part of my problem was that my locally synced cloud storage was all in my home directory on the C: drive, including Cloud Drive.  So the plan was to move my Cloud Drive folder to the D: drive and, in the process, move a bunch of my older photos into Cloud Drive.  After all, I've complained about Cloud Drive, but now that Amazon offers unlimited photo storage to Prime members, why not take advantage of it?

Well, I'll tell you why - because it doesn't work, that's why.  For starters, the desktop Cloud Drive app doesn't provide any way to set the local sync directory.  Luckily, I did find that there's a registry key for that that you can set manually.  So I did that, moved my Cloud Drive folder, and restarted the Cloud Drive app.  And then I waited.  And waited.  And waited some more for Cloud Drive to upload the newly added photos.  However, when I came back the next morning, the systray icon said that Cloud Drive was up to date, but when I looked at the website, my new photos weren't there.

OK, so plan B: try downloading the latest version of the Cloud Drive desktop app.  My version was from March, so maybe there were somce improvements since then. 

And here's problem number two: the new app isn't the same as the one I have.  As far as I can tell Amazon no longer offers a desktop sync app.  Now they just have a "downloader/uploader" app.  It's not sync, though - the process is totally manual.  And I can't find any link to the version I have.  Presumably it's been replaced by this lame uploader app.  I notice that the Cloud Drive website now omits any mention of sync and talks about accessing your data on your computer through the website.

OK, so no upgrade.  Plan C: try to get the version I have working.  That didn't work out, though.  I didn't have the installer anymore, so reinstalling was out of the question.  I tried deregistering the app and resyncing my cloud data, but now that was just broken.  Cloud Drive synced part of my documents folder, then just stopped and reported that it was up-to-date.

At that point, I decided to just give up.  I'd been thinking about switching to OneDrive anyway.  It works well enough and fixes the problems I have with Cloud Drive.  It also gives me 30GB of free storage and has pretty reasonable rates for upgrades - $2/month for 100GB or 1TB for $7/month including an Office 365 subscription.  Plus I've already got it set up on my desktop, laptop, phone, and tablet, so it's just a matter of getting my data into it.

So that's what I did.  I changed my OneDrive location to my D: drive by unlinking and relinking the app (which is required for Windows 7 - Windows 8.1 is much easier) and then moved over the pictures from my Cloud Drive as well as the old ones I wanted to consolidate.  Hopefully that will work out.  It's going to tak OneDrive a while to sync all that data - over 10GB - but it seems to be going well so far.  And unlike Cloud Drive, as least OneDrive has a progress indicator so you can tell that something is actually happening.

As for Cloud Drive, I think I'm pretty much done with that.  I'll probably keep the app on my phone, as it provides a convenient second backup of my photos and also integrates with my Kindle, but I'm not even going to try to actively manage the content anymore.  It seems that Amazon is moving away from the desktop to an all-online kind of offering.  That's all well and good, but it's not really what I'm looking for at the moment.  Too bad.

Continuing on the Komodo macro theme for this week, here's another little macro template that might come in handy.  This one is just a quick out outline for how to reference your project's root directory when running a command.

As you may know, Komodo's commands support a variety of interpolation variables to do things like insert file paths and other input into your commands.  The problem is that there's no variable to get the base directory of your current project - by which I mean the "project base directory" that you can set in your project properties.  Oh, sure, there's the %p and %P variables that work on the current project, but they don't get the project base path.  They get the path to the project file and the directory in which the project file is contained.  That's fine when your project file lives in the root of your source tree, but if you want to put your project files in another location, it doesn't help.

Sadly, there is currently no way to get this path using the interpolation variables.  However, it's pretty easy to get with a macro.  The only problem with that is that the macro syntax is a little odd and the documentation is somewhat lacking.  The documentation for ko.run.runCommand() does little more that give the function signature, which is bad because there are 17 parameters to that function, and it's not entirely clear which are required and what the valid values are.  Luckily, when you create a command, the data is stored in JSON format with keys that more or less match the parameter names to runCommand(), so you can pretty much figure it out by creating the command as you'd like it and then opening the command file up in an editor tab to examine the JSON.

Anyway, here's macro template.  Needless to say, you can substitute in the project base directory at the appropriate place for your needs.  In my case, I needed it in the working directory.

var partSvc = Cc["@activestate.com/koPartService;1"].getService(Ci.koIPartService),
    baseDir = partSvc.currentProject.liveDirectory,
    dir = baseDir + '\\path\\to\\OpenLayers\\build',
    cmd = 'python build.py';
ko.run.runCommand(window, cmd, dir, undefined, false, false, true, "command-output-window");

Several months ago, I posted a Komodo IDE macro to run a source control commit on the current project.  That was nice, but there was an issue with it: it only sort of worked. 

Basically, in some cases the SCC type of the project directory was never set.  In particular, if you focused on another window and then double-clicked the macro to invoke it, without touching anything else in Komodo, it wouldn't work.  While this scenario sounds like an edge-case, it turns out to be infuriatingly common, especially when you use multiple monitors.  The obvious example is:

1. Make some changes to your web app in Komodo.
2. Switch focus to a browser window and test them out.
3. See that everything works correctly.
4. Double click the commit macro to commit the changes.
5. Wait a while and then curse when the commit window never comes up.

I'm no Komodo expert, so I'm not sure exactly what the problem was.  What I did eventually figure out, though, is that Komodo's SCC API doesn't seem to like dealing with directories.  It prefers to deal with files.  And it turns out that, if you're only dealing with a single file, the "commit" window code will search that file's directory for other SCC items to work with.

So here's an improved version of the same macro.  This time, it grabs the project root and looks for a regular file in it that's under source control.  It then proceedes in the same way as the old one, except that it's much more reliable.

(function() {
    "use strict";
    
    // Find a file in the project root and use it to get the SCC type.  If we
    // don't find any files, just try it on the directory itself.
    // TODO: Maybe do a recusive search in case the top-level has no files.
    function getSccType(url, path) {
        var os = Components.classes["@activestate.com/koOs;1"]
                           .getService(Components.interfaces.koIOs),
            ospath = Components.classes["@activestate.com/koOsPath;1"]
                               .getService(Components.interfaces.koIOsPath),
            fileSvc = Components.classes["@activestate.com/koFileService;1"]
                                .getService(Components.interfaces.koIFileService),
            files = os.listdir(path, {}),
            kofile = null;
        // First look for a file, because that always seems to work
        for (var i = 0; i < files.length; i++) {
            var furi = url + '/' + files[i],
                fpath = ospath.join(path, files[i]);
            if (ospath.isfile(fpath)) {
                kofile = fileSvc.getFileFromURI(furi);
                if (kofile.sccDirType) {
                    return kofile.sccDirType;
                }
            }
        }
        // If we didn't find a file, just try the directory.  However, this
        // sometimes fails for no discernable reason.
        kofile = fileSvc.getFileFromURI(url);
        return kofile.sccDirType;
    }
    
    var curr_project_url =  ko.projects.manager.getCurrentProject().importDirectoryURI,
        curr_project_path = ko.projects.manager.getCurrentProject().importDirectoryLocalPath,
        count = 0;
    
    // HACK: For some reason, the SCC type on directories doesn't populate.
    // immediately.  I don't know why.  However, it seems to work properly on
    // files, which is good enough.
    var runner = function() {
        var scc_type = getSccType(curr_project_url, curr_project_path),
            cid = "@activestate.com/koSCC?type=" + scc_type + ";1",
            fileSvc = Components.classes["@activestate.com/koFileService;1"]
                                .getService(Components.interfaces.koIFileService),
            kodir = fileSvc.getFileFromURI(curr_project_url),
            sccSvc = null;
            
        if (scc_type) {
            // Get the koISCC service object
            sccSvc = Components.classes[cid].getService(Components.interfaces.koISCC);
            
            if (!sccSvc || !sccSvc.isFunctional) {
                alert("Didn't get back a functional SCC service. :( ");
            } else {
                ko.scc.Commit(sccSvc, [curr_project_url]);
            }
        
        } else if (count < 50) { // Just in case this never actually works....
            count += 1;
            setTimeout(runner, 100);
        } else {
            alert('Project directory never got a valid SCC type.');
        }
    };
    
    runner();
}());

So here's an interesting tidbit that I discovered this afternoon: you can use RewriteBase in a .htaccess file without knowing the actual base URL.  This is extremely useful for writing a portable .htaccess file.

In case you don't know, the RewriteBase directive to Apache's mod_rewrite is used to specify the base path used in relative rewrite rules.  Normally, if you don't specify a full path, mod_rewrite will just rewrite the URL relative to the current directory, i.e. the one where the .htaccess file is.  Unfortunately, this isn't always the right thing to do.  For example, if the .htaccess file is under an aliased directory, then mod_rewrite will try to make the URL relative to the filesystem path rather than the path alias, which won't work.

Turns out that you can account for this in four (relatively) simple lines:

RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond $1#%{REQUEST_URI} ([^#]*)#(.*)\1$
RewriteRule ^(.*)$ %2index.php [QSA,L]

All you need to do is substitute in your rewrite target for "index.php" and it "just works".  No changes to server configuration required and no need to edit the RewriteBase for the specific server.

Here's a little issue I ran across the other day.  I was setting up a VirtualBox VM with an Ubuntu guest and I wanted to add a shared folder.  Simple, right?  Just install the VirtualBox guest additions, configure the shared folder to mount automatically, and it "just works".

The only problem is the permissions.  By default, VirtualBox mounts the shared folder with an owner of root.vboxsf and permissions of "rwxrwx---".  So if you add your user account to the vboxsf group, you get full access.  Everyone else...not so much.  Of course, this isn't inherently a problem.  However, I wanted the Apache process to have read access to this shared folder because I have a web app that needs to read data off of it.  And I didn't really want to give it write access (which it doesn't need), so just adding it to the vboxsf group wasn't a good option.  What I really needed to do was change the permissions with which the share was mounted.

As far as I can tell, there's no way to get VirtualBox to change the permissions.  At least, I didn't see anything in the guest OS and there's no setting in the management UI.  Fortunately, you can pretty easily bypass the auto-mounting.  Since it's a Linux guest, you can just turn off the auto-mounting in the VirtualBox management console and add a line to your /etc/fstab.

There is one issue, though: you have to make sure the vboxsf kernel module is loaded before the system auto-mounts the share.  If you don't the mounting will fail.  However, forcing the module to load is easily accomplished by adding a vboxsf line to your /etc/modules file.

As for the line in /etc/fstab, this seems to work pretty well for me:
SHARE_NAME   /media/somedir   vboxsf   rw,gid=999,dmode=775,fmode=664   0   0

You know what?  Amazon Cloud Drive is a little bit of a pain.  I think it's time to start moving away from it. I think I'm going to give DropBox another look.

To be clear, I don't plan to stop using Cloud Drive altogether. For one thing, the Kindle integration is actually pretty nice.  And for another, they give Amazon Prime members a pretty decent amount of space for free.  And I find that the desktop Cloud Drive app actually works pretty well.  It's just the mobile app and the website that suck.

The mobile app

For starters, let me voice my new complaint with the mobile app: the semantics of syncing your pictures are rather poorly defined.  Or, rather, they're not defined — at least, not to my knowledge.  By that I mean: what happens when you delete a bunch of files from your Cloud Drive, but not from your phone's main picture storage?  I'll tell you what: the app tries to re-add them to your Cloud Drive. 

This has happened to me two or three times in the last week or two.  A couple of weeks ago I went through my Cloud Drive and deleted some of the pictures that were duplicated, out of focus, etc.  Now, at semi-random intervals, the app wants to re-upload them.

In fairness, this could be due in part to the fact that Amazon's apps all seem to share authentication and it seems to be semi-broken on my phone.  I've had this problem several times recently.  The Kindle app on my Galaxy Nexus will stop synchronizing and the settings will report that it is "Registered to .".  No, that's not a typo, my name is just missing.  And when this happens, I can't authenticate with the Cloud Drive app or the Cloud Player app either — they just reject my credentials.  So far, the only fix I've found is to deregister my device in the Kindle app settings and then re-register it.  That fixes the Kindle app as well as authentication in the other apps. 

The website

I've blogged about the Cloud Drive website and it's useless "share" feature before.  Well, despite that, the other day I decided to give that "share" feature a try.  Hey, maybe it won't really be so bad, right?

Good Lord, it was even worse than I'd imagined.

So my task was to share one or two dozen pictures of my son with my relatives.  Seems simple enough, right?  The pictures are already in my Cloud Drive, so I just need to send out some links to them.  As I noted in the last entry, Cloud Drive doesn't actually support sharing folders or sharing multiple files at once, so the only way to do this is by sharing one file at a time and copying the URL into an e-mail. 

As bad as this is, it turns out it's made even worse by the fact that Cloud Drive is now reloading the file list after you complete the "share" operation.  And to add insult to injury, the reload is really slow.  Granted, I have about 1500 images in that directory, but that shouldn't matter because the reload really doesn't need to happen at all.  I mean, nothing has changed in the folder view.  All that happens is that the UI locks up for a few seconds and then I get a "file has been shared" message.

So this only confirms my opinion that the "share" feature in Cloud Drive is unusable.  I mean, if sharing were just a matter of popping up the "share" dialog that holds the URL, copying it, pasting to another window, and then closing the dialog, that would be one thing.  It would suck, but I could deal with it.  But the brain-dead UI locking up for five seconds after each share just makes it too painful to even try.  I got through maybe half a dozen pictures before giving up in disgust.

Solution - DropBox?

So, clearly, I need another platform for sharing my pictures.  So I looked around and found myself mighily confused.  My requirements were pretty simple.  I wanted something that:

1. Had a free account tier,
2. Had an easy way to share by just sending someone a link,
3. And provided a simple uploading option.

I briefly considered using the Google+ "Photos" feature, but gave up on that when I realized the whole visibility/sharing thing wasn't completely obvious.

For now, I'm trying out DropBox.  I've had a free DropBox account for a couple of years, but never really used it for anything, so that was one less thing to sign up for.  The desktop app is pretty nice and allows me just drag files between Explorer windows and shows an icon of the file's sync status.  And sharing things via the web UI is dead simple, which is exactly what I was looking for.  So we'll see how that goes.  Worst-case scenario, I won't have lost anything.

I'm currently on my annual "this time I'm really going to start doing test-driven development (or at least something close to it)" kick.  And it seems to be going pretty well, so hopefully this time it will actually stick. 

As I said, I do this every year and testing usually ends up falling by the wayside.  Granted, this is partly just due to a lack of discipline and commitment on my part.  Getting started with a new practice takes effort, and sometimes it's hard to justify that effort when "I've been doing just fine without it for years."  But there's also the matter of the type of projects I work on.  Most of the projects I've worked on have not had an establish test suite or a culture of testing.  Whether it's a work-related project or an old personal project for before I heard the gospel of TDD, the norm has been a sizable, years-old code-base that has few if any tests and isn't really designed to be testable in the first place. 

Getting into TDD with a project like that can be daunting.  Picture PHP code littered with static method calls, system calls, and direct class instantiations at multiple levels.  "You need to inject a mock object?  Yeah, good luck with that."  The classes may be well-defined, but there's not much compartmentalization of responsibilities within them, so inserting test doubles is not always straight-forward.  That leaves you in the unenviable position of either having to rewrite a bunch of existing code to make it unit-testable or set up some real test data and turn the unit test into an integration test.  The first option is obviously preferable, but can be much more risky, especially since you don't already have tests to validate the behavior of the code you need to change.  And while the second approach is certainly better than no tests at all, integration tests are slow to run, cumbersome to set up, and much more prone to breakage when things change.  Faced with a mess like that, it doesn't seem that unreasonable to say, "You know what?  I've got things I need to get done.  I'll get back to those tests later."  But, of course, you never actually do.

This time, I'm doing things a little differently.  For starters, I'm not writing new tests for old code.  I'm just doing the new code.  I'll get to the old code when/if I get around to refactoring it.  That means that I don't have to worry about untestable code, which makes the entire enterprise about a thousand times simpler.  I'm also not working with PHP code for the time being.  I'm trying to do TDD on two projects — one for work that's in JavaScript and a personal one in Python.  For Python, I'm using unittest and mock which, so far, I'm finding to be less of a pain in the neck than PHPUnit.

For the JavaScript project, I'm using Jasmine, which brings me to the title of this post.  Since I'm trying to do TDD (or rather BDD), I quickly tired of alt+TABbing to a browser window and then hitting ctrl+R to reload the page with the test runner.  Sure, it's not a big deal, but it's just one more thing that gets in the way.  I wanted to do what I could do in Python and PHP, which is just hit a hotkey and have Komodo run the test suite right in the same window. 

Turns out, that was actually pretty easy to set up.  I just banged out a quick macro that opens up the Jasmine test runner HTML file in a browser tab or refreshes it if it's already opened.  I bound that to a hotkey and I'm good to go.  Sure, it doesn't use the Komodo IDE testing framework, but that's not the end of the world — I just move it to a split pane and it works out pretty well.  I even added some CSS to the spec runner to make it match my Komodo color scheme.

Here's a screenshot of the side-by-side runner and some (unrelated but public) code:

And here's the macro itself:
(function() {
    var uri = 'file:///path/to/test_runner.html',
        view = ko.views.manager.getViewForURI(uri);
    if (view === null) {
        ko.open.URI(uri, 'browser');
    } else if (view.reload) {
        view.reload();
    } else {
        view.viewPreview();
    }
})();

Thanks to this answer from StackOverflow, the key mappings in VI and Vim make soooo much more sense now.  It's all explained by this keyboard layout:

The more I use Vim, or the Vim emulation in Komodo, the less sense it makes to use the "escape" key to switch out of insert mode.  I mean, you have to move your hand way away from the home row just to reach it, which is exactly the opposite of how things are supposed to work.  But if you're using this keyboard, then it seems like a completely obvious choice.

Incidentally, after far too many years of suffering with the escape key, I've switched to using the other choices for getting out of insert mode — Ctrl-C and Ctrl-[.  Both do the same thing as the escape key, but require significantly less hand movement and so are faster to type.  Especially Ctrl-C, as most of us are used to using that for copying text to the clipboard.

Incidentally, since I'm using the Vim emulation for Komodo IDE/Edit, I should probably mention that it does support Ctrl-[ out of the box, but not Ctrl-C — that is, naturally, reserved for the "copy" operation.  However, it's easy to change that.  Simply create a new macro with the following code and set the key binding for it to Ctrl-C (you may have to unbind the standard "copy" operation first):

var scimoz = ko.views.manager.currentView.scimoz;
if (scimoz && scimoz.selText.length == 0) {
    ko.commands.doCommand('cmd_vim_cancel');
} else {
    ko.commands.doCommand('cmd_copy');
}

This macro preserves the standard copy behavior by checking if you have any text selected when it executes.  If you do, it does a standard "copy to clipboard".  If you don't, it switches the editor from "insert" mode to "normal" mode.

Edit (2014-10-09): An improved version of this macro can be found in this post.

So I finally got around to writing a little macro for Komodo IDE that I've been missing for a while.  It has a very simple task - perform a "source control commit" operation on the currently project.  You'd think this would be built in, and it wort of is, but it doesn't work very well. 

In Komodo IDE, the SCC commit action is tied to files and directories.  So if you want to commit an entire directory, you need to select it in the "places" pane (i.e. the file system browser).  And if you want to commit the directory that's currently set as the root of the file view, you either have to go up a level and select it or right-click on an empty spot in the file browser.  So, in other words, it's grossly inconvenient.

Hence this little macro.  Just create a new macro in your toolbox and paste this code into it (note that this doesn't work in Komodo Edit).  Note that this is a little hacky, as some of the SCC initialization seems to be asynchronous and I don't know what (if any) events are fired on completion.  But hey, it works, so close enough.

(function() {
    var curr_project_url =  ko.projects.manager.getCurrentProject().importDirectoryURI,
        fileSvc = Components.classes["@activestate.com/koFileService;1"]
                            .getService(Components.interfaces.koIFileService),
        kodir = fileSvc.getFileFromURI(curr_project_url),
        count = 0;
    
    // HACK: For some reason, the SCC type takes some time to populate.
    // I don't know if there's any event for this, so instead just try it again
    // if it's empty.
    var runner = function() {
        var cid = '',
            sccSvc = undefined;
            
        if (kodir.sccDirType) {
            cid = "@activestate.com/koSCC?type=" + kodir.sccDirType + ";1";
            sccSvc = Components.classes[cid].getService(Components.interfaces.koISCC);
            
            // Get the koISCC service object
            if (!sccSvc || !sccSvc.isFunctional) {
                alert("Didn't get back a functional SCC service. :(");
            } else {
                ko.scc.Commit(sccSvc, [curr_project_url]);
            }
        } else if (count < 10) { // Just in case this never actually works....
            count += 1;
            setTimeout(runner, 100);
        }
    };
    
    runner();
})();

After shamelessly reusing passwords for far too long, I finally decided to get myself a decent password manager. After a few false starts, I ended up going with KeePass. In retrospect, I probably should have started with that, but my thought process didn't work out that way.

Originally, my thought was that I wanted to use a web-based password manager. I figured that would work best as I'd be able to access it from any device. But I didn't want to use a third-party service, as I wasn't sure how much I wanted to trust them. So I was looking for something self-hosted.

I started off with PPMA, a little Yii-based application. It had the virtue of being pretty easy to use and install. There were a few down sides, though. The main one was that it wasn't especially mobile-friendly, so there were parts of the app that actually didn't work on my phone, which defeats the whole "works on any device" plan. Also, it really only supported a single user, so it's not something I could easily set my wife up on as well. (To be fair, the multi-user support was sort of there, but it was only half-implemented. I was able to get it basically working on my own, but still.)

More importantly, I wasn't entirely confident in the overall security of PPMA. For starters, the only data it actually encrypted was the password. Granted, that's the most important piece, that's sort of a minimalist approach to account security. And even worse, I wasn't 100% convinced that that was secure - it's not clear to me that it doesn't store a password or key in session data that could be snooped on a shared server. Of course, I haven't done an extensive analysis, so I don't know if it has any problems, but the possibility was enough to make me wary and I didn't really want to do an extensive audit of the code (there was no documentation to speak of, and certainly nothing on the crypto scheme).

The next package I tried was Clipperz. This is actually a service, but their code is open-source, so you could conceivably self-host it. I had a bit more confidence in this one because they actually had some documentation with a decent discussion of how their security worked.

The only problem I had with Clipperz was that I couldn't actually get it to work. Their build script had some weird dependencies and was a pain to deal with (it looked like it was trying to check their source control repository for changes before running, for some reason). And once I got it installed, it just flat-out didn't work. I was able to create a new account, but after that every request just returned an error out. And to make things worse, it turns out their PHP backend is ancient and not recommended - it's still using the old-school MySQL database extension. The only other option was the AppEngine Python backend, which wasn't gonna work on my hosting provider. So that was a bust.

It was at that point that I started to think using a web-based solution might not be the best idea. Part of this is simply the nature of the web - you're working over a stateless protocol and probably using an RDBMS for persistence. So if you want to encrypt all the user's data and avoid storing their password, then you're already fighting with the medium. A desktop app doesn't have that problem, though - you can encrypt the entire data file and just hold the data in memory when you decrypt it.

It also occurred to me that accessing my passwords from any computer might not be as valuable as I'd originally thought. For one thing, I probably can't trust other people's computers. God alone knows what kind of malware or keyloggers might be installed on a random PC I would use to access my passwords. Besides, there's no need to trust a random system when I always have a trusted one with me - namely, my phone.

Great! So all I really need is a password manager than runs on Android.

Well...no, that won't do it. I don't really want to have to look up passwords on my phone and manually type them into a window on my desktop. So I need something that produces password databases that I can use on both Android and Windows.

Luckily, KeePass 2 fits the bill. It has a good feature set, seems to have a good reputation, and the documentation had enough info on how it works to inspire some confidence. The official application is only Windows-based, but there are a number of unofficial ports, including several to iOS and Android. It's even supported by the Ninite installer, so I can easily work it into my standard installation.

For me, the key feature that made KeePass viable was that it supports synchronization with a URL. There are extensions that add support for SSH and cloud services, if you're into that sort of thing, but synchronizing via standard FTP or WebDAV is built right in. KeePass also supports triggers that allow you to automatically synchronize your local database with the remote URL on certain events, e.g. opening or saving the database.

For the mobile side, I decided to go with Keepass2Android. There are several options out there, but I chose this one because it supports reading and writing the KeePass 2.x database format (which not all of them do) and can directly read and write files to FTP and WebDAV. It's also available as an APK download from the developer's site, as opposed to being available exclusively through the Google Play store, which means I can easily install it on my Kindle Fire.

Keepass2Android also has a handy little feature called "QuickUnlock", which allows you one chance to unlock your database by typing just the last few characters of your passphrase. If you get it wrong the database is locked and you need to enter the full passphrase. This addresses one of my main complaints about smart phones - the virtual keyboards work to actively discourage good passwords because they're so damned hard to type. I chose a long passphrase which takes several seconds to type on a full keyboard - on a virtual keyboard, it's absolutely excruciating. This way, I don't have to massively compromise security for usability.

So, in the end, my setup ended up being fairly straight-forward.

1. I install KeePass on all my computers.
2. I copy my KeePass database to the WebDAV server I have set up on my web hosting.
3. I set up all my computers with a trigger to sync with the remote URL.
4. I install Keepass2Android on my phone and tablet.
5. I configure them to open the database directly from the URL. Keepass2Android caches remote databases, so this is effectively the same as the desktop sync setup.
6. Profit! I now get my password database synchronized among all my computers and devices.

I've been using this setup for close to a month now, and it works pretty darn well. Good encryption, good usability, plenty of backups, and I didn't even have to involve a third-party service.

For the last few years, I've been using this tool called "go" - an eminently unsearchable name. Basically, it's kind of like bookmarks for the command line - you can set directories to be remembered and go back to them with a single short command.

Anyway, shortly after I started using it, I posted a patch that added support for Powershell (which I call go-posh). That worked well and everything was fine. Then, a bit later, I added a few more small patches to the application, such as an option to print the directory path rather than changing to it.

Well, that was all well and good. I'd been using my patched version every day for quite a while. I even added it to my Mercurial repository for safe keeping.

What I didn't realize is that I never updated my original blog post with my second set of changes. I discovered this the hard way a couple of weeks ago, on the computer at my new job. When I set up go-posh on that computer, I just used the ZIP archive from my blog post, rather than cloning it from my Mercurial repository like I normally would. It worked just fine for a few weeks until I tried to run something like gvim (go -p sb/somefile.txt) and was informed that the -p option didn't exist.

I hate it when I do things like that. It's such a stupid mistake and it's extra embarrassing because it's been wrong for nearly three years.

Anyway, I've updated the old blog entry. That's a little out-of-date now anyway, so I also linked it to the project tracker page, which is the canonical source of information anyway. I even rebuilt the Mercurial repo to reflect the actual changes I made to the stock install, rather than the context-free initial import I had before.

So anybody who's interested should grab the code from there. The additions over what I used to have in the blog post download include the aforementioned -p option, as well as resolving of shortcut prefixes when they aren't ambiguous (e.g. if the shortcut is "downloads", you can just use "dow"), home directory detection for Windows, and improved support for the -o options. You can see more info in the README.txt file.

Well, the new version of Opera is now in stable release - Opera 15! This is the first version based on Webkit instead of Presto, Opera's in-house rendering engine. After using it for a week or so on OSX, I have to say that I'm both pleased and disappointed.

I'm pleased because, let's face it, Webkit is pretty great. Especially if you're a web developer. It's fast, supports most of the emerging standards, and has really great development tools. And while I've been an Opera fan for a while, the old JavaScript and HTML rendering engines have been showing their age for some time. They're slow by comparison to Webkit-based browsers and have a lot of quirks. So I'm very happy that I can now go to Sta.sh in Opera 15 and actually use it - in Opera 12 it's almost unusably slow.

I'm also pleased about the appearance. The new Opera is definitely pretty. The old version wasn't bad, but you can tell that they've had designers putting some time in on the new version. It looks very smooth and polished.

I'm less pleased about everything else. You see, Opera 15 introduces some fairly major UI changes. And by "major UI changes", I mean they've basically scrapped the old UI and started over from scratch. Many of the familiar UI elements have been removed. Here's a quick run-down of the ones I noticed:

• The status bar is gone.
• Ditto for the side panel.
• The bookmark bar is also gone.
• In fact, bookmarks are gone altogether.
• Opera Mail has been moved to a separate application (not that I miss it).
• The RSS reader has disappeared (maybe it went with Opera Mail).
• Opera Unite is MIA.
• Tab stacking has disappeared.
• Page preview in tabs and on tab-hover are gone.
• Numeric shortcuts for Speed Dial items have been removed (i.e. they took away what made it "Speed Dial" in the first palce).
• The content blocker is gone (which is just as well - it was kind of half-baked anyway).

And that's just the things I noticed. But the worst part is that those UI features were the only reason I still used Opera. I feel like I'm no longer welcome in Opera land. They've literally removed everything from the browser that I cared about.

And what have they given me in return? Well, Webkit, which is no small thing. But if that's all I wanted, I'd just use Chrome or Safari. It's nice, but it doesn't distinguish Opera from the competition.

So what else is there? Well, Speed Dial has gotten a facelift. In fact, Speed Dial has sort of turned into the Opera 15 version of bookmarks. You can now create folders full of items in your Speed Dial and there's a tool to import your old bookmarks as Speed Dial items. I guess that's nice, but I'm not seeing where the "speed" comes in. It seems like they've just changed bookmarks to take up more screen real estate and be more cumbersome to browse through.

They've also introduced something called Stash, which, as far as I can tell, is also just another version of bookmarks. But instead of tile previews like Speed Dial, it uses really big previews and stacks them vertically. They're billing it as a "read later" feature, but as far as I can tell it's functionally equivalent to a dedicated bookmark folder. I guess that's nice, but I don't really get the point.

And, last and least, there's the new Discover feature. This is basically a whole listing of news items, right in your browser. Yeah, 'cause there aren't already 42,000 tools or services that do that. One that's directly integrated into the browser is just the killer feature to capture the key demographic of people who like to read random news stories and are too stupid to use one of the gajillion other established ways of getting them. Brilliant play, Opera!

Now, I'll grant you - visually, the new Speed Dial, Stash, and even Discover look fantastic. They're very pretty and make for really nice screenshots. However, I just don't see the point. I can imagine some people liking them, but they're just not new or innovative - they've just re-invented the wheel in a way that's more convoluted, but not visibly better.

Overall, I get the feeling that Opera 15 was designed by a graphic designer. Not a UI designer, but a graphic designer. In other words, it was built to be pretty rather than functional. I know I've sometimes had that experience when working with a graphic designer to create a UI - you get mock-ups that look beautiful, but were clearly created with little or no thought to their actual functionality. So you end up with workflows that don't make much sense, or new UI elements with undefined behavior, or some little control that's just "thrown in" but represents new behavior that is non-trivial to implement.

Honestly, at this point I think it's just time to switch to Chrome. I already use it for all my development work anyway, so I might as well use it for my personal stuff too. I had a good run with Opera, but I just don't think the new version is going to meet my needs. Maybe I'll take another look when version 16 comes out.

This past week I finally decided to migrate to a new bug tracking system for my personal projects. Granted, my projects aren't particularly big (in most cases, I'm probably the only user), so it's not like I'm flooded with bug reports. But in my last two jobs I lived and died by our ticket tracking system and found the use of a good bug tracker to be extremely helpful for my development process.

For the last few years, I've been using Mantis BT for my personal bug tracking needs. Mantis is actually a very capable bug tracker and worked fairly well for my needs. However, Mantis is just a bug tracker. It doesn't have a wiki or much in the way of project planning tools, release management, or anything else, really. It has some basic roadmapping features and a changelog generator, but that's about it. And since I've become used to working with the likes of Jira, Trac, and Phabricator, I've come to want a little more than that.

On a side-note, the other thing about Mantis is that it's a bit cumbersome to work with. The UI is a bit...antiquated, for one thing. In fact, I've heard people refer to it as a disaster. There's a reason that the Mantis site only shows screenshots of the mobile app, as seen from the issue reporting screen below. The workflow is also a little weird when it comes to things like the changelog and roadmap features. It centers on fixed-in and target versions for individual bugs, which presupposes a more organized type of release planning than I'm looking to do.

So if not Mantis, what to do for an issue tracker? Obviously, the simple answer would be to install one of the popular project management packages, like Trac or Redmine. However, there are a couple of problems with this:

1. I run this site on a shared hosting account. A really cheap shared hosting account.
2. I'm kinda lazy.

To expand on the "cheap" part, my hosting provider is really targeted more at simple PHP-based sites that are administered through their control panel. So not only do I not have root access to the server, I don't even have shell access - just the control panel and FTP. So anything that requires running interactive commands for setup is out. And while my host does "support" Ruby and Python, in the sense that the interpreters are installed and accessible, it only does so through CGI and has a rather limited set of libraries installed.

This all leads into the second point, i.e. laziness. I could just get a more full-featured hosting provider. However, I've been happy enough with my current one, they're very affordable ($6/month), and frankly, I can't be bothered to go to the effort of moving all my stuff to a new server. I also can't be bothered to figure out the non-trivial steps to set up a semi-supported package like Trac or Redmine on my current host. It just isn't worth the time and energy to me.

So I decided to do a little more research. After some Googling, I ended up settling on The Bug Genie. So far, it seems to be a pretty decent compromise between Mantis and something like Trac. In addition to ticket tracking, it has an integrated wiki module, release and milestone tracking features, and even source control integration.

The initial setup was not as intuitive as I might have hoped. For starters, there was no easy way to migrate my data from Mantis to The Bug Genie. I ended up just migrating the issues themselves, minus comments, by using Mantis's CSV export. A highly sub-optimal solution, to say the least, but it wasn't important enough to me to make a project out of the migration. Second, the permission and user system, while it seems pretty powerful, is a bit more complicated and granular than I need. Lastly, the source control integration was a bit of a pain to set up. The actual configuration wasn't that difficult once I figured out what I needed to do, but I had to go to the project team blog to find the documentation I needed. Honestly, the worst part was the particular format of commit message needed to trigger the integration - it's quite verbose and not at all intuitive.

It's only been about a month of fairly light use, but so far I'm pretty satisfied with The Bug Genie. The UI has its quriks, but is modern and easy to use. It's pretty configurable and well documented, allowing you to customize project pages using the wiki module. Things like the release tracking and VCS integration are nice touches and seem to work quite well. All in all, it pretty much does what I wanted and does it reasonably well. I'm pretty happy with it.

It seems that Microsoft has decided to be a bit nicer about testing old versions of Internet Explorer. I just found out about modern.IE, which is an official Microsoft site with various tools for testing your web pages in IE.

The really nice thing about this is that they now provide virtual machines for multiple platforms. That means that you can now get your IE VM as a VMware image, or a VirtualBox image, or one of a few other options.

When I was using Microsoft's IE testing VMs a couple of years ago, they were only offered in one format - Virtual PC. Sure, if you were running Windows and using Virtual PC, that was great. For everyone else, it was anywhere from a pain in the butt to completely useless. This is a much better arrangement and a welcome change. Nice work, Microsoft!

With the imminent demise of Google Reader, and FeedDemon consequently being end-of-lifed, my next task was clear: find a new RSS aggregator. This was not something I was looking forward to. However, as things turned out, I actually got to solve the problem in more or less the way I'd wanted to for years.

The thing is, I never really liked Google Reader. The only reason I started using it was because I liked FeedDemon and FeedDemon used Google Reader as it's back-end sync platform. (And if you've ever tried to use a desktop aggregator across multiple systems, you know that not being able to sync your read items across devices is just painful.) But I seldom used the Reader web app - I didn't think it was especially well done and I always regarded the "social" features as nothing but a waste of screen space. Granted, the "accessible anywhere" aspect of it was nice on occasion, but my overall impression was that the only reason it was so popular was because it was produced by Google.

The other issue with Reader is that I don't trust Google or hosted web services in general. Paranoia? Perhaps. But they remind me of the saying that "If you're not paying for the product, then you are the product." And while I know a lot of people aren't bothered by that, I think that Google knows far too much about me without handing them more information on a silver platter. Furthermore, you can't rely upon such services to always be available. Sure, Google is huge and has ridiculous amounts of money, but even the richest company has finite resources. And if a product isn't generating enough revenue, then the producer will eventually kill it, as evidenced by the case of reader.

What I'd really wanted to do for some time was to host my own RSS syncing service. Of course, there's not standard API for RSS syncing, so my favorite desktop clients wouldn't work. But with FeedDemon going away as well, and having no desktop replacement lined up, I no longer had to worry about that. So I decided to take a chance on a self-hosted web app and gave Tiny Tiny RSS a try. I was very pleasantly surprised.

The installation for TT-RSS is pretty simple. I use a shared hosting account, and though the documentation says that isn't supported, it actually works just fine. The install process for my host consisted of:

1. Copy the files to my host.
2. Create the database using my host's tool.
3. Import the database schema using using PHPMyAdmin.
4. Edit the config.php file to set the database connection information and a couple of other settings.
5. Use my host's tool to create a cron job to run the feed update script.
6. Log in to the administrator account and change the default password.
7. Create a new account for myself.
8. Import the OPML file that I exported from FeedDemon.

That's it. Note that half of those steps were in the TT-RSS UI. So the installation was pretty much dead-simple.

In the past, I wasn't a fan of web-based RSS readers. However, I have to say that Tiny Tiny RSS actually has a very nice UI. It's a traditional three-pane layout, much as you would find in a desktop app. It's all AJAX driven and works very much like a desktop client. It even has a rich set of keyboard shortcuts and contextual right-click menus.

As an added bonus, there's also a pretty nice mobile site. While the latest release (1.7.5) actually removed the built-in mobile clientweb site, there's a very nice third-party JavaScript client available. It uses the same API as the mobile clients, so the installation pretty much consists of enabling the API, copying the files to your host, and editing two settings in the config file to tell it the path to itself and to TT-RSS.

But who cares about the mobile site anyway? There are native Android clients! The official client is available as trial-ware in the Google Play store. And while it's good, I use a fork of it which is available for free through F-Droid. In addition to being free (as in both beer and speech), it has a few extra features which are nice. And while I may be a bit cheap, my main motivation to use the fork was not the price, but rather the fact that the official client isn't in the Amazon app store and I don't want to root my Kindle Fire HD. This was a big deal for me as I've found that lately my RSS-reading habits have changed - rather than using a desktop client, I've been spending most of my RSS reading time using the Google Reader app on my Kindle. The TT-RSS app isn't quite as good as the reader app, but it's still veyr good and more than adequate for my needs.

Overall, I'd definitely recommend Tiny Tiny RSS to anyone in need of an RSS reader. The main selling point for me was the self-hosted nature of it, but it's a very strong contender in any evaluation simply on its own merits. In my opinion, it's better than Google Reader and is competetive with NewsBlur, which I also looked at.

I have a new side project. It wasn't one I was really planning on, but it's the fun kind - the kind that scratches an itch.

You see, I've become a big fan of the Roku set-top box. I have three of them in my house. And in addition to using them for Netflix and Amazon Instant Video, I also use them to stream video and audio from my desktop PC (which has become an ersatz home server). For this I use an application called Roksbox. It allows you to stream videos from any web server. So since I already had IIS running on my PC, all I needed to do was set up a new website, symlink my video directory into the document root, and I was ready to go.

The problem with Roksbox is that it's a bit basic and the configuration is a little clunky. For example, if you want to add thumbnail images and metadata to the video listings, you have two options:
1) Put all the information for every video in a single XML file.
2) Turn on directory listings for your server and use one XML file and thumbnail file per video.
The problem with the first option is that it's ludicrously slow. By which I mean the interface can freeze for upwards of 30 seconds while Roksbox parses the XML file. And the problem with the second method is that you end up littering your filesystem with extra files.

This was particularly the case when I looked at adding thumbnails for a TV series. All I wanted was for each episode to have the season's DVD cover art as a thumbnail image rather than the default image. Unfortunately for me, in order to do that Roksbox wants you to have one image file for each episode. So I either needed a bunch of redundant copies, or a bunch of symlinks, and neither of those options appealed to me.

So I decided to try something - faking the directory listing. That is, I wrote a couple of simple PHP scripts. The first read the contents of a directory and simulated the output of the IIS directory browsing page, but inserting thumbnail entries for each video. So, for example, if there was a video "Episode 1.mp4", the script would add an "Episode 1.jpg" to the listing. The second script worked in conjunction with a rewrite rule for non-existent files. So since the "Episode 1.jpg" didn't exist, the request would be sent to the second script, which would parse the request URL and then redirect to an actual, existing thumbnail image. The result was that I had my images for every episode in the season and I didn't have to create any pointless links or files.

This worked so well that, naturally, I took it to the next level. I started branching out and added more support for customizing listings, such as hiding and re-ordering directories. Then I started getting really crazy and wrote code to generate listings for entirely fictitious directories in order to create video playlists. I even started writing admin pages to provide a GUI for configuring this stuff, built an SQLite database to save the information, and created an ad hoc application framework to help manage it.

I think the reason this little project really caught my interest is that it's something that I actually use on a daily basis. I've been trying for a while now to find a side-project that I can use for picking up new languages or technologies, but nothing ever seems to stick. I'll play with something for a while, but them lose interest pretty quickly when some other obligation comes up. I think that's because those projects are just excuses - I'm doing them because I need a project in order to learn something new, but I have no particular interest in that particular project. But even though I'm not using anything new for this project, it grabs my attention because it's something that I want for myself. That's something I haven't had in a while, and I really like the feeling.

If anyone is curious, the current code is in my public mercurial repository here. I'm planning to put up a proper release of it when it's a bit more together. For now, I'm just having fun making it work.

This is just another "note to self" post, so that I have something to refer back to later when I forget this again. Nothing to see here - move along.

Without further ado, here's a link for how to configure ISC BIND on Windows. This should have roughly the important details, with the exception of file locations. Per the company guide, I have my BIND files in C:\Windows\System32\dns. Note that this is restricted to admin accounts, so it won't show up in explorer with my regular account. I lost 15 minutes figuring that out the other day. Also, here's a link on the managed-keys-zone error I was getting in the event log. That turned out to be the fix for my problem where BIND was starting, but DNS queries weren't working.

If you're not me and you're still reading this, the motivation for this post was configuring a new development VM for work on my home Windows box. Normally, we're a Mac shop, and the company provides MacBooks for anyone who requests a computer. However, mine broke down and is in the shop, so I had to set up my Windows box with the development environment.

There are just two problems here:
1) Our codebase, despite being PHP running on a Linux VM, is actively Windows-hostile.
2) We require wildcard DNS for subdomains.

The use of BIND is to address the second problem. For the occasions when someone tries to run the VM on Windows, BIND is used to send the DNS queries for our development domain to the VM. Of course, we could just set the Windows DNS server to the VM itself, but that has the down side that your DNS breaks when you shut off the VM.

And if you're curious about the first problem, all you need to know is that our codebase is full of directories named "v.", which is just fine on UNIX-based systems, but is an invalid directory name on Windows. There's a good reason for that - trust me - but it's too long to explain without the relevant context. (Of course, there's no reason that it has to be "v." - the guys who came up with that could have just used a different naming convention. But they didn't, and now we're stuck with it.)

I finally had a "new tech" adventure last week - I dabbled in Mozilla extensions for the first time.

I've been using ActiveState's Komodo Edit as my primary editor for PHP, Python, and JavaScript for the last few years. One of the extensions I always used was the TabSwitcher. I've found that I depend on that extension even more heavily now that I'm with deviantART simply because of the size of our code-base. While the Fast Open extension is great and has similar functionality, it simply doesn't fill the same role for me. Our code-base has thousands of file, many of which have similar names, which results in Fast Open slowing down and giving me too many results to be useful. Because TabSwitcher operates on a much smaller data set, it's much better for the common case where I have 20 or 30 tabs open and am actively switching back and forth between 8 or 10 of them.

The only problem is that TabSwitcher isn't compatible with Komodo 7. That's actually part of the reason I hadn't bothered to upgrade Komodo on my work computer yet. So I decided to remedy that by porting it to Komodo 7. While I was at it, I added a couple of small enhancements, such as updating the bookmark list when a bookmark is added.

You can download the extension here:
tabswitcher-1.0.0-ko.xpi
If you want to look at the source, it's in my Mercurial repository here:
http://hg.skepticats.com/tabswitcher/

This was my first exposure to XUL or Mozilla add-ons, so hopefully everything will work the way it should. I haven't seen any problems so far, at least.

Author's note: In yet another installment of "From the Archives", here is a little entry that I wrote up back in September of 2007, but never bothered to publish. It's a little dated, but sadly it still has the ring of truth to it. Of course, there has been some improvement. A lot of hardware doesn't come with a CD anymore - they just give you a pamphlet with a link to the crappy software instead. But anyway, I thought this entry was based on an interesting quote, so here it is.

On an episode of Hanselminutes a month ago, Scott and Carl spoke with Jeff Atwood of Coding Horror fame about the "ultimate developer rig" he built for Scott. During the course of discussion, Jeff uttered what is, undoubtedly, the best quote about software developers that I've ever heard.

Nobody hates software more than software developers.

Not only does it sum up the feeling of true software geeks everywhere, but it also offers a sad commentary on the state of the computer hardware business.

This quote was uttered in the context of a discussion about "mouse software," i.e. those useless CDs full of crap-ware that get thrown in the package with even fairly standard mice. They typically include drivers, a control panel applet, some kind of tool that runs in the system tray, and probably a few other things.

As Jeff and Carl rightly pointed out, the reason they, myself, and many other programmers hate such software is that it is, quite simply, useless. Nobody needs to install the software that comes with a standard wheel mouse because their operating system (no matter what it is) already supports it. In fact, Carl went so far as to say (half-kidding, I hope) that if he caught any of his employees installing "mouse software," they'd be fired on the spot. Any half-competent programmer or IT person should know better.

Sadly, this phenomenon isn't just limited to mice. USB media devices seem to be the worst offenders. Cameras and MP3 players in particular always seem to come with some kind of software that's supposed to help you move files onto or off of the device. Of course, as it's always third-rate crap-ware, as the vendor makes their money on the hardware and just throws in the software as an after-thought. But the worst part is that there's no reason you should even need it, because any decent MP3 player or camera really ought to "just work" as a USB mass storage device (with the notable exception being the iPod, which is decent, if overrated).

So really, it's not that software developers hate software, it's that we hate crappy, unnecessary software. That's probably because we know what we need and can easily judge the value of a program. We can tell when all a program does is the digital equivalent of busy-work. Being handed a CD of code that doesn't do anything useful feels like being condescended to. That angers us.

OK, that was a little confusing.

So after my initial attempt at a Mercurial hook to connect my public repository to my Mantis instance, I decided to rearrange things a little. Initially, I was using a "commit" hook on my local machine. However, for the public bug tracker with the public repository, it made more sense to put the hook on the server. So I switched it to an "incoming" hook in my hgweb.config.

Actually converting the script to work with my Linux based web host was pretty easy. Just one command, really:

#!/bin/bash

basedir='/path/to/base/hg/install/dir'
mantis='/path/to/mantis/wwwroot/scripts/checkin.php'
hg="$basedir/hg"

$hg log -vr $HG_NODE --style $basedir/mercurial/templates/multiline | php $mantis

Nice and simple, right?

The hard/annoying part was in the style template file. Seem I wanted my commit message to have multiple lines and indent the file list, like this:

Changeset 85:5f7504e02f1c by Peter Geer, Mon Jan 17 23:12:46 2011 -0500
Updated plugin for PHP5 and fixed blog root when taken from defined constant (fix bug 0000021).
lib/blog.php
plugins/menubar_breadcrumbs.php

Problem was, it just didn't work. I was using exactly the same template text as in the old hook script, but I was getting everything on the same line. By switching from "\n" to "\r\n" as my newline escape code, I was able to get newlines to work, but the indentation still didn't.

Guess what fixed it - switching from double to single quotes.

changeset = 'Changeset {branches}{rev}:{node|short} by {author|person}, {date|date}\n{desc}\n{files} '
file = ' {file}\n'

Seriously. That's it. No mention of that in the manual - they're supposed to be the same. In fact, the example of this in the manual uses double-quotes.

Oddly enough, using double-quotes worked just fine on the command line. I'm wondering if it has something to do with the fact that I'm using hgweb or something. Either way, that really sucked.

Yesterday I upgraded FeedDemon to version 4. In the process, I did something new - I paid for a license.

For the last year or two that I've been using it, FeedDemon has been ad supported. The full feature set was available for free, but there was a little ad in the lower left-hand corner. If you wanted to get rid of that, you could buy a license key, but frankly, I never really cared that much.

Now, however, the business model has changed. There's now a free "Lite" version with some missing features and a paid "Pro" version with all the features and no ads. One of the annoying parts of this was that the features in the "pro" version are not all new - the for-pay features include some things that were previously free.

This bothered me a little for a couple of minutes. Then I read the FeedDemon author's article about why he changed the business model. Apparently, FeedDemon is no longer run by NewsGator. It's still branded for them, but they let the author, Nick Bradbury, go and returned control of the program to him. Apparently he's now doing the independent developer thing and making his living selling FeedDemon.

Well, that pretty much sealed the deal. I've tried a number of RSS readers - Linux, Windows, and web - and FeedDemon is the only one I've ever really liked. This is a program I use every day, I want some of the "pro" features, and buying it would help out a fellow developer. And to top it all off, he's only asking a lousy $20 for it. That's a no-brainer. So I bought a license and I'd encourage any other FeedDemon users to do the same.

Incidentally, some of the comments on Nick's article were really depressing. With users like some of those, I don't envy him trying to make his living off FeedDemon. Supporting a one-man project is tough enough when you release it as open-source and you don't owe anything to anyone. It must be hard trying to do right by people who paid you money when they feel your work is lacking - even (especially?) if they're clearly being unreasonable. Working 9 to 5 (well, officially 8 to 5 in my case, but you get the idea) might not be sexy, but at least it's stable and they pay someone else to deal with unhappy customers.

One thing I did notice in the new version that I found a little disappointing was the change in some of the newspaper themes. FeedDemon basically displays feeds as an HTML document and has a number of themes that control the look of the feed list and allows you to set themes on a per-feed basis. Changing the theme is now a "pro" feature, and I really wanted this, as I have several feeds that I have set to use the Outland theme, which is a nice, clean theme that displays full entries rather than just summaries. However, this theme is gone in FeedDemon 4.

The up side is that, while the theme format may have changed in FD4, it hasn't changed much. The "styles" are still just that - CSS and XSLT. In fact, I still had the Outland.fdxsl2 theme file sitting in my "C:\Program Files (x86)\FeedDemon\Data\Styles" directory, so I just tried renaming it to Outland.fdxsl4, restarting FD, and guess what? It worked! The only real problem was that the border on the feed was a little off, so I commented out the border on the HTML element (line 24), and it was just fine.

Last time I mentioned that I'd set up Mantis with a Subversion hook. Well, I've been rethinking the Subversion part.

I was listened to a back episode of Hanselminutes last week in which Scott was interviewing SourceGear's interviewing Eric Sink about DVCS and it was a very interesting conversation. Source control is one of those things you don't usually think about as a developer. You learn how to use a source control system that's "good enough", and after that, just sort of take it for granted. You certainly don't spend a lot of time jumping back and forth between systems and trying out the possibilities.

I really liked the way Eric explained distributed version control. As he pointed out, most of the arguments for DVCS that you hear seem pretty pointless, such as "There's no central server," or "You can commit on an airplane." Well...so what? Why wouldn't I want a central server - how else do you determine the authoritative repository? And honestly, who does important work on an airplane?

Instead, Eric framed the discussion in terms of flexibility. The flip-side of the above stupid arguments is actually pretty compelling. With DVCS, there's no central server, but that just means that your central server is determined by convention, not by the software, and it's much easier to set up "slave" repositories for remote sites. Likewise, while being able to commit on a plane is pointless, there's definite value in not needing to have a connection to the server to do core version control operations. Or, to put it another way, I may not code on a plane, but I do often code in coffee shops with slow WiFi connections.

So, in light of that, I decided to give Mercurial a try. According to Eric, it's fairly user-friendly, especially for Subversion users, and it's fairly Windows-friendly (it even has a TortoiseHg client, for whatever that's worth). So, since I was thinking of resurrecting LnBlog, I decided to use that as my test-case.

That leads me back into the Subversion hooks tie-in. Since I went to all the trouble of setting up an issue tracker and hooking it to SVN, I figured I'd do the same with Mercurial. Fortunately, it wasn't too bad once I figured out what I needed to do. I was able to fairly easily adapt the Powershell script I wrote for Subversion to work with Mercurial. In fact, the Mercurial version was actually shorter.

Actually adding a commit hook in Mercurial is pretty simple. You just add a line to your .hgrc file:
[hooks]
commit = powershell \path\to\hg-commit-hook.ps1
It seems that Mercurial preserves the environment for hooks, so you don't seem to have to worry about absolute paths and such like you do in Subversion.

The changes to the script itself were fairly small. This wiki page had a few good examples that got me started. The two big things were the passing of data and generating the message to pass to Mantis. Mercurial actually passes data into hook scripts through environment variables rather than command-line parameters, which is nice in that you actually get meaningful variable names coming in. As for the message generation, Mercurial's log command allows you to specify a template for its output, including substitution variables and some simple formatting functions. The result is a nice, short script with only a couple of calls to hg:

[System.Reflection.Assembly]::LoadWithPartialName("System.Web") | Out-Null

$issue = hg log -vr $env:HG_NODE

# If we don't have an issue number, just exit.
if ($issue -match "\b(?:bug|issue)\s*[#]{0,1}(\d+)\b") {
   
   $style_file = @'
changeset = "Changeset {branches}{rev}:{node|short} by {author|person}, {date|date}\n{desc}\n{files}"
file = " {file}\n"
'@
   Write-Output $style_file | Out-File -Encoding ASCII -Width 1000 hgstyle

   $data = hg log -vr $env:HG_NODE --style hgstyle
   rm hgstyle

   # Keep the cast to string from clobbering line breaks.
   $data = [String]::Join("`n", $data)
   $postData = "secret=somePassword&message="
   $postData += [System.Web.HttpUtility]::UrlEncode($data)

   C:\Cygwin\bin\wget.exe -O req.out --post-data=$postData http://yourhost.com/path/to/mantis/scripts/do_checkin.php
   rm req.out
}

That's it - a measly two calls to Mercurial, including one to see if we even need to run the hook. By specifying the right template, we can get the entire message in one command. That template gives me something like this:
Changeset 70:cbdb625298ca by Peter Geer, Mon Dec 13 16:45:53 2010 -0500
Got rid of unneeded requires now that we have an autoloader (issue #0000005).
   lib/article.php
   lib/creators.php
Perhaps not quite as pretty as the Subversion version, but close enough for now. Perhaps if I feel like getting really fancy, I'll look at the Python API for Mercurial and try my hand at writing some in-process hook code.

Edit: Updated the script to put each of the listed file on a new, indented line.

I came across an interesting article on Task Driven Development the other day. I came to it via the recent Linus on branching story on Slashdot.

One of the first things mentioned in that article is that you need an issue tracking system and it needs to be integrated with your source control system. After a moment's consideration, I said to myself, "That's a good idea." I've been thinking lately about getting back into doing some open-source stuff as well as some educational projects for myself, and something like that might be helpful. In the past, I usually didn't bother to track bugs and features in my personal projects, unless you count post-it notes and unorganized text files, because, well, they were personal projects and it just didn't seem important. However, now I'm used to living and dying by Jira and am fully aware of how crazy it is to think I'm just going to remember all the bugs I find and potential features I think up.

So, to that end, I installed a copy of Mantis on my site. I settled on Mantis, after a good 5 minutes of careful research, based largely on the facts that the article recommended it and that the requirements were low enough that I could run it on my cheap-ass shared hosting account. Fortunately, it seems nice enough so far - not radically different from Bugzilla or Jira, but OK.

Anyway, I came across a slight snag in setting up the Mantis integration with my Subversion repository. You see, Mantis supplies a script that can be called from Subversion's post-commit hook that will update any issues referenced in the commit message. However, the script is written on the assumption that Mantis and SVN live on the same box. In fact, that is not the case - SVN lives on my local Windows 7 box, while Mantis lives on my hosting provider's box, to which I do not have shell access (yes, I'm that cheap).

With a little Googling, I was able to turn up a few useful resources to help me get this set up. This article gave a nice overview of SVn hooks and Mantis integration, while this one provided a nice perspective for SVN on Windows. However, I had to cobble these together to make it work on my system.

My main problem with the documentation I had was my lack of SSH access to the box on which I'm running Mantis - all the articles I found assumed you had it. So the first order of business was a simple gateway PHP script to wrap the command-line Mantis script that would normally be run via SSH. Therefore I quickly banged out this do_checkin.php script and dropped it in a web-accessible directory.

<?php
# Very cheap security measure, simply so that just *anybody* can't post issues.
if ($_POST['secret'] !== 'somePasswordHere') {
   exit;
}

$cmd = 'php checkin.php <<< ' . escapeshellarg($_POST['message']);
system($cmd);


Next was the Subversion hook. I'm using SlikSVN, and it turns out that this will only execute hook scripts that are .exe or .bat files. I know this because I tried to write a Powershell hook script and it didn't work. But no worries - it's easy enough to write a wrapper batch post-commit.bat file such as this:

@ECHO OFF

SET REPOS=%1
SET REV=%2

CD %REPOS%\hooks
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe %REPOS%\hooks\post-commit.ps1 "%REPOS%" "%REV%"


For the Powershell script, it was easy enough to just port the BASH script posted in the previously linkd article. However, instead of calling the Mantis checkin.php script by SSH, I actually just used wget to send a POST to the do_checkin.php posted above.

[System.Reflection.Assembly]::LoadWithPartialName("System.Web") | Out-Null

$REPOS = $args[0]
$REV = $args[1]

"$REPOS $REV"

$svnlook = 'C:\Program Files\SlikSvn\bin\svnlook.exe'

$auth = & $svnlook author -r $REV $REPOS
$dt = & $svnlook date -r $REV $REPOS
$changed = & $svnlook changed -r $REV $REPOS
$changed = $changed -join "`n"
$log = & $svnlook log -r $REV $REPOS
$log = $log -join "`n"
$data = "Changeset [${REV}] by $auth, $dt
$log
$changed"

$postData = "secret=somePasswordHere&message="
$postData += [System.Web.HttpUtility]::UrlEncode($data)

C:\Cygwin\bin\wget.exe -O req.out --post-data=$postData http://yoursite.com/path/to/mantis/scripts/do_checkin.php
rm req.out


This seems to work fine in my testing. We'll see if that changes once I start adding in bugs and features for my projects and then writing the code to complete them. Hopefully that process will be somewhat easier for being more organized.

In honor of Raymond Chen, it's link clearance time! This one is devoted to Windows utilities I've discovered over the last few weeks.

First up is the Elevation PowerToys. This is just a bunch of scripts related to privilege elevation. In particular, I like the "elevate" command included with this. It's kind of like sudo for UAC - you use it to elevate to admin with a UAC prompt rather than having to use runas.

Next is Shell Link Extension. As the name suggests, this is just a shell extension for creating symlinks and hardlinks. It's a bit rough around the edges, but it's nice not to have to drop down to a cmd.exe prompt.

On a more eye-candy note, I recently discovered PuttyTray, an improved version of PuTTY. It includes systray integration and supports configurable window transparency. The eye-candy support is fairly limited compared to the options available in the UNIX world (how many configuration options did aterm have just related to transparency?), but overall it's not too bad.

Similarly, I also just discovered PowerShell Glass. Basically, it's just a utility to enable Aero glass in PowerShell terminals, rendering the window transparent. While it's a nice idea, it doesn't seem to work that well. Sure, the window turns transparent, but I can't read what's in it half the time. The text and background coloring isn't quite right and isn't configurable to any great degree, so depending on what's behind your terminal window, it can be very hard to read.

Edit: Apparently I spoke too soon on the PSGlass colors. I Googled it yesterday and noticed a comment that the transparency seems to work best with the dark blue Aero color scheme. So I tried changing my color scheme and, lo and behold, it worked. Turns out I had my desktop set to the "frost" scheme, which looks like it's probably the worst possible setting for PSGlass. The "twilight" (dark blue) setting does seem to be the most readable, but it's a little too blue for my taste. I ended up settling on "slate", which still gives you readable text when the window is active, but is a little less "in your face" color-wise.

Yesterday I tried creating a playlist in MediaMonkey and synchronizing it with my Sansa e280 running RockBox. I never really bothered with trying to sync playlists to my MP3 player before, because syncing, well, anything that's not an iPod using third-party software is a little iffy. Needless to say, my fears were well founded.

While MediaMonkey's syncing feature is nice, I ran into several problems. First, and most obvious, MediaMonkey didn't like the layout of my media directory. You see, I have a main "Music" folder which contains a bunch of artist and album sub-folders, but also a bunch of miscellaneous MP3s at the top-level. MediaMonkey didn't see those MP3s for some reason. After the sync, they just weren't there - I had to copy them over manually.

My other problem was a combination of the playlists themselves, how RockBox on the Sansa works, and where I'm putting my music on the Sansa. A couple of weeks ago, I "upgraded" my Sansa by buying a 16GB microSDHC card for it to complement the 8GB of internal flash storage. This resulted in a change of organization - I now keep all my actual music on the microSD card and put podcasts and other spoken material on the internal storage. Turns out that this messes with MediaMonkey's assumptions about playlists.

There are several issues here. The first is that when MediaMonkey syncs to the microSD card, it sees it as a separate device and creates the playlists accordingly. The result is that the playlists use absolute paths within the microSD card. However, the real absolute path for files on the microSD card starts with the device path, i.e. <microSD1>. So, basically, the paths in the playlist are wrong for RockBox.

The second issue is that the playlists get synced onto the microSD card. This isn't really a problem, just now what I want. I want the files synced to the main RockBox playlist folder on the Sansa's internal storage.

The third issue was with those assorted MP3s in my root "Music" directory. Even though I set the sync path to <Path:2> in MediaMonkey's sync configuration, which should mirror the directory structure on my hard drive, when I ran the sync, the assorted files on my playlist got copied into a "Music" folder on the microSD card. I'm not sure why. But as long as that's the case, I decided to "solve" that problem by just moving the rest of the assorted files there too. They weren't being synced anyway, so why not.

I decided to solve the playlist path and location problems with a simple Powershell script. You can download a copy here. It simply reads your playlists, adjusts the paths, and moves them to the main device storage. The code is below:

$source = "MUSICSD"
$device = "Sansa e280"
$listDir = "Playlists"

function findDriveByLabel {
   $drives = [System.Io.DriveInfo]::GetDrives()
   foreach ($drv in $drives) {
      if ($args[0] -eq $drv.VolumeLabel) {
         return $drv.RootDirectory
      }
   }
}

$sd = findDriveByLabel $source
$dev = findDriveByLabel $device
$sdDir = [System.Io.Path]::Combine($sd, $listDir)
$devDir = [System.Io.Path]::Combine($dev, $listDir)

$lists = ls $sdDir

foreach ($lst in $lists) {
   $outFile = [System.Io.Path]::Combine($devDir, $lst.Name)
   $outFile
   Get-Content $lst.FullName | ForEach-Object {
      "/<microSD1>" + $_.Replace("\", "/")
   } | Out-File -Encoding utf8 -Width 1000 $outFile
   rm $lst.FullName
}

$remLists = ls $sdDir
if ($remLists -eq $Null) {
   rmdir $sdDir
}

So far, this part is working fine. I'm still not 100% happy with the sync experience, though. The missing files is the part that really bugs me. I'll have to do a little experimenting at some point and see if I can make it work. Only problem is that it takes a while to write 16GB of data to the device....

I've been catching up on some of my back episodes of .NET Rocks the past couple of days. I'm currently about 3 months behind on my podcasts, so I've got plenty to listen to.

Anyway, I was listening to show 537 with James Kovacs at the gym this morning, and he mentioned something interesting toward the end. When asked what he's been into lately, he said he'd been playing with Vim. Imagine that - a hard-core Microsoft guy playing with Vim. That just gives me a good feeling.

I've always been frustrated by how people in the Windows world seem to think "text editor" equals "Windows Notepad". Even experienced people. You always hear them refer to opening something in Notepad, or typing things into Notepad, etc. This from people who've been in the business for 10, 15, or even 20 or more years! I mean, they must be aware that there are better editors out there. Is "notepad" just used as a generic term, a shorter version of "text editor"? I wish somebody would explain that to me.

But getting back to the topic, it was nice to see James mention Vim. Despite Carl's and Richard's comments about VI being an antique, Vim really is remarkably powerful. Granted, it takes some effort to learn, but once you've got it down, you can do some pretty complicated stuff in just a couple of keystrokes. It's always nice to see that MS A-listers can recognize the power of tools like Vim, and not just us transplanted UNIX geeks.

Yeah.... So remember how I was messing with freeSSHd last weekend? I'm thinking that was a waste of time.

While the native shell thing in freeSSHd was cool, I had two big issues with it. First, the terminal was buggy. I got lots of weird screen artifacts when I connected to it - the screen not clearing properly, weird random characters hanging around after a lot of output, etc. Not the rock-solid terminal experience I'm used to.

The second thing was something that occurred to me after trying out a couple of SSH sessions. If freeSSHd is using a native shell, with native file paths, how is that going to work with SCP and SFTP? How does it account for drive letters and path delimiters? Turns out the answer (at least for SFTP) was restricting access to a certain pre-defined directory ($HOME by default). For SCP...well, I still haven't figured out how that's supposed to work. But either way, this is decidedly not the behavior I was looking for.

So instead, I decided to go the old-fashioned way and just install Cygwin and use their OpenSSH server. And you know what? It was completely painless. I just used this handy LifeHacker entry to get the auto-config command and the necessary settings, and I was done. I started the service, connected to the SSH server, and all my stuff was right where it was supposed to be. I have a solid, familiar terminal, access to my whole system via SCP and SFTP, and NT authentication worked out of the box. Heck, the hardest part was figuring out the funky non-standard UI of the installer's package selection dialog. I should have just used Cygwin from the beginning and saved myself some effort.

Well, I did my Windows 7 install the other day. One day later, I'm doing pretty well. Ran into some problems, but so far the results are not bad.

Unsurprisingly, the actual install of Windows 7 was pretty uneventful. Pretty much the same as a typical Ubuntu installation - selecting partition, entering user info, clicking "next" a few times, etc. Nothing to report there.

The initial installation of my core programs was pretty easy too, thanks to Ninite. They have a nifty little service that allows you to download a customized installer that will do a silent install of any of a selected list of free (as in beer) programs. So I was able to go to a web page, check off Opera, Thunderbird, Media Monkey, the GIMP, Open Office, etc., download a single installer, and just wait while it downloaded and installed each program. Not quite apt-get, but pretty nice.

My first hang-up occurred when installing the Ext2IFS. Turns out that the installer won't run in Windows 7. You need to set it to run in Windows Server 2008 compatibility mode. And even after that, it was a little dodgy. It didn't correctly map my media drive to a letter on boot. It worked when I manually assigned a drive letter in the configuration dialog, but didn't happen automatically. It was also doing weird things when I tried to copy some backed-up data from my external EXT3-formatted USB drive back to my new NTFS partition. Apparently something between Ext2IFS and Win7 doesn't like it when you try to copy a few dozen GB of data in 20K files from EXT3 to NTFS over USB. (Actually, now that I write that, it seems less surprising.) The copy would start analyzing/counting the files, and then just die - no error, no nothing. I finally had to just boot from the Ubuntu live CD and copy the data from Linux. Still not sure why that was necessary.

I also had some interesting issues trying to install an SSH server. I initially tried FreeSSHD, which seemed to be the best reviewed free server. The installation was easy and the configuration tool was nice. The only problem was, I couldn't get it to work. And I mean, at all. Oh, sure, the telnet server worked, but not the SSH server. When set to listen on all interfaces, it kept complaining that the interface and/or port was already in use when I tried to start the SSH server. When bound to a specific IP, it gave me a generic access error (literally - the error message said it was a generic error).

After messing around fruitlessly with that for an hour or so, I gave up and switched to the MobaSSH server. This one is based on Cygwin. It's a commercial product with a limited home version and didn't have quite as nice an admin interface, but seems to work sell enough so far. The one caveat was that I did need to manually open port 22 in the Windows firewall for this to work.

The biggest problem so far was with setting up Subversion. Oh, installing SlikSVN was dead simple. The problem was setting up svnserve to run as a service. There were some good instructions in the TortiseSVN docs, but the only worked on the local host. I could do an svn ls <URL> on the local machine, but when I tried it from my laptop, the connection was denied. So I tried messing with the firewall settings, but to no effect. I even turned off the Windows firewall altogether, but it still didn't work - the connection was still actively denied.

I started looking for alternative explanations when I ran netstat -anp tcp and realized that nothing was listening on port 3690. After a little alternative Googling, I stumbled on to this page which gave me my solution. Apparently, the default mode for svnserve on Windows, starting with Vista, is to listen for IPv6 connections. If you want IPv4, you have to explicitly start svnserve with the option --listen-host 0.0.0.0. Adding that to the command for the svnserve service did the trick.

Since I haven't posted anything in a really long time, with the exception of the Press Your Luck post of a couple weeks ago, I thought I'd do a little list to get back into the swing of things. So this is just a list of thing technical/computery that have made me happy, to some extent, over the course of this year.

1. Windows 7. I've preferred Linux for a long time, but I've got to say, Win7 is awfully nice. I fact, it's so good I'm actually thinking of dumping Linux on my main desktop at my next upgrade. And for reference, I haven't had Windows on my home desktop since 2002 - and I was dual-booting then.
2. Subversion. Not that Subversion is that great in and of itself (although it is a perfectly good revision control system), but this year my team finally migrated from CVS to Subversion, and it sucks so much less! Yes, you read that right - my company was actually using CVS for new projects until the beginning of 2010. It's nice to finally step forward into 2005!
3. The new versions of Opera. I've been an Opera user for a very long time, and version 10.50+ has gotten really nice. The updated UI is pretty nifty, Opera Link is a very handy feature I've always wanted, and I actually find Opera Unite really cool. I use it to access my home PC fairly regularly. I don't care how fast Chrome is, Opera's still better in my book.
4. Not coding in JavaScript and HTML. Our new product at work is implemented in Adobe's FLEX. Yeah, FLEX is based on Flash, which sucks, but it's actually a pretty nice framework. Way more pleasant to work with than JavaScript and HTML. Even if you include jQuery, which takes most of the suck out of JavaScript, FLEX is still nicer. Now if only Flash didn't suck so much.... Too bad there's no viable alternative. (Silverlight doesn't have the market penetration and nobody with half a brain thinks HTML 5 is comparable. And yes, that implies Steve Jobs doesn't have half a brain.)
5. Not using Web TimeSheet. Last year, our company was using Web TimeSheet for our time tracking. If you've never used it, suffice it to say it sucks beyond all rational belief. Seriously - I can't describe how bad it was. It took damn near 30 seconds to load the time entry form. And that wasn't even waiting for an HTTP response - it was actively loading a bajillion things during that time. Now, we're doing our time/task tracking in Jira instead. Sadly, Jira still sucks, but not nearly as bad as Web TimeSheet.
6. I have a window. For the first time ever, I have a desk by the window. The president of our company left for another job a couple weeks ago, and since we're not replacing him, the CEO gave our four-man dev team permission to move into his office. It's just big enough for all four of us (only slightly smaller than big-ass the cube we all shared), and two of the four walls are all window. And I got the corner where the two windows meet. This is a far cry from when I was in public service and had the "cube" that used to be a printer nook and was exactly large enough to hold my desk - and nothing else.
7. Whipping out SQL For Smarties. A couple months ago I actually had an excuse at work to whip out my copy of Joe Celko's SQL For Smarties and apply one of the chapters. I needed to store a nested folder structure in the database, so I turned to the trees and hierarchies section, did a little reading, and settled on the path enumeration method. One of my colleagues had suggested adjacency lists, which Celko calls "the most commonly used and worst possible" model for trees. I love that book - it always makes me happy when I have an excuse to use something from it.

Seven seems like a good number. Time to call it a night. I get up early enough to go to the gym and still make it to the morning scrum. Perhaps I can continue the list later.

After six months of silence, here's a random "I'm writing this so I don't have to keep Googling it" post.

The "no whammy" game show was called "Press Your Luck". According to Wikipedia, it was hosted by Peter Tomarken and featured announcer Rod Roddy, in case anybody cares that much.

That just jumped into my head again as I was merging my private dev branch in Subversion back to trunk. For some reason - probably a version mismatch between clients and the server, which is only on 1.4 - we've been getting a lot of tree conflicts on merge. So this morning I found myself watching the merge progress saying, "No conflict, no conflict, no conflict, STOP!"

Well, I feel stupid. I just finally got syntax highlighting for PowerShell to work in Vim. I did a little Googling and found my solution in the PDF document linked form this article. Why you need an 8 page PDF to explain this is a completely different issue.

Turns out I was just missing some lines in my filetype.vim file. I already had the syntax, indent, and file type plugins, I just didn't think to add the code to activate them (for some reason I thought that stuff was loaded dynamically). However, page 6 of that PDF gave the answer:

" Powershell
au BufNewFile,BufRead *.ps1,*.psc1 setf ps1

I just added those lines to my filetype.vim, below the corresponding entries for Povray files, and voilà! Syntax highlighting now works.

Well, it's official: the people who develop PHP are morons. Or, rather, the people responsible for adding namespaces to PHP 5.3 are.

Why do I say this? Because I just read an annoucnement on Slashdot that they've decided on the operator to use for separating namespace in PHP 5.3: the backslash (\).

Seriously? The friggin' backslash? What kind of choice is that? Last I knew they'd pretty much decided to go with the double colon (::), like C++, which at least makes sense. But the backslash?

What's worse, just look at the RFC listing the operators they were considering. In addition to the backslash, they had the double star (**), double caret (^^), double percent (%%), a shell prompt (:>), a smiley face (:)), and a triple colon (:::). For God's sake, it looks like they picked this list out of a hat. They might as well have just used the string NAMESPACESEPARATOR. It's no less absurd than any of those.

Now, let's be realistic for a minute. In terms of syntax, PHP is a highly derivative language. It's an amalgamation of Perl, C++, and Java, with a dash of a few other things thrown in.

Given that heritage, there's really only a handful of choices for namespace separators that even make sense. The first, and most natural, is the double colon (::). This is what C++ uses and it's already used for static methods and class members in PHP. So the semantics of this can naturally be extended to the generic "scope resolution operator." Keeps things clean and simple.

The second choice is the dot (.), which is what's used in Java, C#, Python, and many others. This is a bit unnatural in PHP, as dot is the string concatenation operator, but it at least offers consistency with other related languages.

Third is...actually, that's it. There are only 2 valid choices of namespace separator. And the PHP namespace team didn't pick either one. Nice work guys.

The Slashdot article also linked to an interesting consequence of the choice of backslash: it has the potential to mess up referencing classes in strings. So if your class starts with, say, the letter "t" or "n", you're going to have to be very careful about using namespaces in conjunction with functions that accept a class name as a string. Just what we needed. As if PHP isn't messed up enough, now the behaviour of a function is going to depend on the names of your classes and the type of quotes you use.

I guess I'm going to have to bone up on my C#, because PHP seems to be going even farther off the deep end that before. It was always a thrown-together language, but this is just silly. The backslash is just a stupid choice for this operator and there's just no excuse for it.

Here's a quick and useful tactic for dealing with file attributes on FAT32 drives. I got the idea from this post on the Ubuntu blog

My new MP3 player (which I'll be blogging about when I have more time) uses a FAT32 filesystem. I needed to change the attributes on some of file attributes so that it would show the media folders but hide the system folders. Why I needed to do that is another story. Anyway, the point is that there was no obvious way to do this from Linux and since charging the MP3 player seems to reset these attributes, I didn't want to have to rely on a Windows machine being handy.

After way more Googling than I thought necessary, I discovered that you can do this with good old mtools. The really old-school people in the audience will probably remember them from the days when floppy disks were still in common use. Well, it turns out that they can be used with USB mass storage devices too.

The first step, after installing mtools of course, is to set up a drive letter for your USB device in your ~/.mtoolsrc file. This can be done by adding something like the following:
drive s: file="/dev/sdb1"
mtools_skip_check=1
The first line associates the S: drive letter with the device file for my player. The mtools_skip_check line suppresses errors which, I believe, arise from the fact that this is a USB device, not an actual floppy disk. Either that, or there's something about the FAT that mtools doesn't like, but can still work with.

Once that's set up, I was able to simply use mattrib to change the file attributes and [cdoe]mdir[/code] to show the attribute-sensitive directory listing. The actual commands look something like this:
mdir S:
mattrib +h S:/TMP
mattrib -h S:/MUSIC
Note the use of the S: drive letter to prefix paths on the root of the device. The +h and -h flags turn the hidden attribute on and off respectively. Also note that you can have the device mounted while doing this - mtools doesn't need exclusive access as far as I know.

Eventually, I'll be scripting this so that I can (hopefully) run it automatically after charging my player. Ideally, that script would include some HAL or udev magic to detect the dynamically assigned device node and add that to the mtoolsrc file. When I get around to writing that, I'll post the result.

As I mentioned the other day, Kubuntu now ships with Strigi as the default desktop search engine. So, I decided to give it a try. I started the daemon a few days ago and left it to build an index.

My reaction, when I came back last nigh, was - and I quote - "What the %$@#?!?" (And yes, I actually said, "What the percent dollar at pound?!?")

You see, I had noticed that I now had only 500MB of free space left on my home partition. This was surprising because. I knew the partition was filling up, but I hadn't moved any substantial amounts of data around lately and I should have had at least 5GB left.

A quick du -sch `ls -A` revealed the culprit. My ~/.strigi directory had ballooned up to a whopping 7.5GB.

Let my repeat that. The Strigi index was up to 7.5 gigabytes. How? Why? What the hell was it doing? My beagle index was only about 950MB. Why the heck does Strigi need so much more space?

This will definitely merit a little research. I'm hoping this was some freak malfunction. But if not, then I guess I won't be using Strigi. I mean, 7.5 gigs? Come on!

Tomorrow marks my 2-month anniversary at my new job doing LAMP. And for most of that two months, I've been going back and forth on what editor or IDE to use.

My requirements for a PHP IDE are, I think, not unreasonable. In addition to syntax highlighting (which should be a given for any code editor), I need to following:

1. Support for editing remote files over SSH. This is non-negotiable.
2. A PHP parser, preferably with intellisense and code completion.
3. A file tree browser that supports SSH.
4. Syntax highlighting, and perferably parsers, for (X)HTML and JavaScript.
5. Search and replace that supports regular expressions.
6. Support for an ad hoc, per-file workflow. In other words, I don't want something that is extremely project-centric.
7. It should be free - preferably as-in-speech, but I'll take as-in-beer if it's really good.

So far, my preferred IDE has been Quanta Plus. It has all of the features I need and also integrates nicely with KDE. It also has a few other nice features, including context-sensitive help (once you install the documentation in the right place). However, the build of Quanta 3.5.6 that came with Kubuntu Feisty is kind of unstable. It crashes on me every few days, and for one project, I actually had to switch to something else because I was making heavy use of regex search and replace, which was consistently crashing Quanta. Also, while Quanta has a PHP parser with some intellisense, it's pretty weak and not in any way comparable to, say, Visual Studio.

My second heavier-weight choice is ActiveState's free KomodoEdit. This is a very nice, XUL-based editor. It's stongest feature is undoubtedly the PHP parser. It's really outstanding. For instance, it can scan pre-determined paths for PHP files and do intellisense for them. It even understands PHPDoc syntax and can add the documentation to the intellisense.

The down side is that, while Komodo does speak SFTP, the file browser tree only does local files. There is a Remote Drive Tree extension that adds this feature, but while it's better than nothing, it still isn't that good. I also don't much care for the look of Komodo or for the keyboard shortcuts. Those things are much easier to customize in Quanta.

After Quanta, my other old stand-by is jEdit. After installing the PHPParser, XML, and FTP plugins, this meets my needs. On the down side, the PHP parser doesn't do any intellisense (although it does detect syntax errors). The interface also feels a littly clunky at times, although it's much better than the average Java application and not really any worse than Quanta in that regard.

I took a brief look at a couple of Eclipse setups, but wasn't initially impressed by them. It might be worth looking at them again some time, but the whole process of getting and installing the appropriate plugins just seemed like a lot of trouble. Same goes for Vim. I'm sure I could get it to do most, if not all, of what I want, but it seems like an awful lot of trouble. And then, of course, there's the Zend IDE, which I don't really want to pay for. And besides, my one of my co-workers told me that, while it's a decent IDE, the real selling point is the integrated debugging and profiling, which won't work on our setup.

And so my intermitent search goes on. I'm hoping that the upgrade to Kubuntu Gutsy will fix the stability problems in Quanta, which is my biggest problem with it. I'm also hoping for some nice new features when KDE 4 comes along. But I guess I'll keep looking in the meantime.

My current project at work is to integrate a web application that runs one aspect of our site with the rest of the site. Part of this involves converting the database code from a custom DB abstraction library to use the standard PHP Data Objects (PDO) library that comes with PHP 5.1. I had never used it before, but it turns out PDO isn't a bad library. It has an object-oriented API, supports multiple databases, "prepared" (i.e. DBMS parametrized) queries, transactions, and so forth. Pretty much the basic features you need in a database abstraction layer. The only problem is that the error handling API is somewhere between "weird" and "completely frickin' insane."

You see, PDO has "configurable" error handling. That is, when a method reaches an error condition, it will either return an error code or throw an exception. Which one it does is determined by an attribute setting on the PDO connection object. (Technically, it may actually return the error code and throw an exception, but the return value is irrelevant once an exception is thrown, so it doesn't really make any difference.)

Let me repeat that, in case you missed it: PDO will either throw an exception or return an error code, depending on a setting of the PDO connection object. So it's possible to change one single constant and break error handling for an entire script.

Upon learning this, my initial reaction was: "That's the stupidest thing I've ever heard in my life!" I mean, honestly, who ever heard of configurable error handling? As if programmers don't have enough to worry about with database code, now we have to pick an error handling mode as well?!? And the worst part is that, when you read the error handling documentation, the default mode not only doesn't throw an exception, it doesn't even print a warning. It just leaves you to see the bad return value and manually inspect the error code on the object. You have to manually set it to turn on warning messages. Why?!? Isn't that why we have the error_reporting variable in php.ini? What the heck were they thinking?!?

What really kills me is that exceptions are optional, but object-oriented programming is mandatory. Unlike, say, the MySQL extension, PDO does not have a procedural version of the API, just the object-oriented one. And yet exceptions, which I think I can safely say are the object-oriented way of doing error handling, aren't even used by default. Shouldn't that be the only way of dealing with errors? Why is it better to make them optional? What's the rationale behind that?

So far, the only rationale I can think of is that the PHP community has what seems to be a disproportionate number of morons in it. That may be a little harsh, but it seems to me like there are a lot of third-rate PHP programmers and that the people who run the PHP project tend to go a little too far to cater to them. I can only assume that somebody thought that exceptions might be too hard for some PHP users, so they figured it would be better to use error codes by default. Kind of like how they figured securing your server was too hard, so they added the fake band-aid of safe_mode. Or how they figured a simple function call or array look-up was too hard, so the invented register_globals. Or how they figured that cleaning form input was too hard, so the invented "magic quotes."

PHP isn't a bad language, and I do enjoy working with it, but it has gone through a long list of really bone-headed design decisions. In some ways, it seems like it's succeeded in spite of itself. I guess this is really just evolution in action. Take a simple template engine, let it mutate into a full-blown programming language, and this is what you get: a platypus. Lots of endearing features and lots of brain-dead features, but it's still alive and still evolving.

It's time for another tale of IT pain. You remember CRAPS, the pathological police system we're required to use? Well, it struck again.

Today I had to travel to a nearby police agency to assist them with their CRAPS installation. They were having problems with the data transfer between the field units and the office. This is normally accomplished via a removable USB drive. In order to ease this procedure, CRAPS includes a feature to automate the copying of data files to and from the USB drive. Basically, the user clicks a button and the data files get compressed and moved in the appropriate direction.

The problem with this feature is that it's not very friendly from a configuration point of view. You see, CRAPS doesn't actually know anything about USB drives. It just knows about paths, and they're configured statically. So you actually have to tell the software, "use drive F: for the data transfer."

What's worse, CRAPS isn't even very smart about handling paths. As you probably know, when Windows detects a USB mass storage device, it assigns it the next available drive letter, so you can't depend on the same devince getting the same letter every time. However, CRAPS requires that a CRAPS administrator configure the drive letter ahead of time and it cannot be changed by a regular user. So the user ends up with, for example, a drive with multiple partitions, he can't use it until an administrator can reconfigure his system. Which sucks.

The first problem today was that CRAPS can't even join paths properly. We were having problems with the data transfer feature mysteriously failing on a couple of workstations. The USB drive path was correctly set to E: in CRAPS. However, just on a lark, because I know how cranky CRAPS can be, I tried chaning it to E:\. And you know what? It worked. *THWACK* (That's the sound of me smacking myself in the head.)

My second problem was partly Windows, partly the fact that I didn't set up this other agency's network. You see, on one workstation, the USB drive was being mapped to F:, but the primary network share was also being mapped to F:. The result? The network share clobbers the USB device and you can't access the USB drive until the network share is disconnected.

This is a fairly well known problem. As I understand it, the cause is that drive mapping is done on a per-user basis, and while network shares are mapped by the user, USB drives are mapped by a system account. There are a number of possible fixes, of course, but they all kind of suck - especially if you don't have any significant ownership over the system you're working on.

Nothing is ever as easy as it should be. Which is why "IT land" sucks.

The ESRI class is almost over. We're down to the last 2 of 16 lessons.

Today has been pretty boring for me. The instructor has been doing a great job, it's just that we're getting deeper into the details of managing a geodatabase. Since this is the first time I've ever actually worked with a geodatabase, the finer points are more or less lost on me. I can understand the concepts, but I have no frame of reference for applying them. In terms of practical knowledge, I don't even know enough to ask an intelligent question.

By way of contrast, the people from host agancy, the City of Hartford, really know their stuff. They've been to several ESRI classes, gone to the ArcGIS user conference, and work with an ArcGIS database regularly. It's clear that they have a good handle on this stuff and are really getting something out of this.

All in all, I file this experience under "pointless waste of time and money." I had a better time than I anticipated, and I now know something about ArcGIS geodatabases, but I really had no business attending this class in the first place. They should have sent someone with at least a basic knowledge of ArcGIS instead - or, at the very least, someone who isn't looking to jump ship at the first opportunity (and my supervisor does know I'm looking for work - I'm not trying to hide it).

So, to sum up, I did get something out of this experience and it was very nice to get out of the office for several days. But the benefit to my employer won't even come close to justifying the $1400 registration fee plus travel expenses. I really don't know what they were thinking.

More live-blogging today. Unfortunately, it turns out that's the only kind I'll be doing until the class is over. I'm staying at the Crowne Plaza hotel, and while they offer WiFi service, they charge $10/day for it! That's actually worse than the $4/2-hour block that they charge for the AT&T WiFi at the Barnes & Noble back home. At that price, I could get 5 hours of service, and there's no way I'm going to be online that long here. Fortunately, the PCs in the training room for the class have web access, so I can at least check my e-mail.

Yesterday afternoon and this morning we got more into the details of managing ArcGIS geodatabases. Things like connection methods, authentication, data loading, management tools, "gotchas", and so forth. Basically, the stuff I will probably never need to know.

I'm actually a little ambivalent about this class so far. On the one hand, it's absolutely mind-numbing at times. It's not that the class is bad, it's just that it's getting into details that have absolutely no relevance for me.

On the other hand, it's actually very interesting in an academic sense. After all, we're talking about a system that scales up to clustered, multi-terabyte databases. The ArcGIS server runs under Windows, UNIX, and Linux and supports pretty much all the major DBMSs - Oracle, SQL Server, DB2, Informix, and there was even some talk of people using Postgress and Sybase. So we're really getting a closer look at the architecture of a very complex, high-end system. Plus our instructor has been around long enough that he can talk about how things have evolved over the years and the direction the architecture has taken. We're not just getting the tedious technical details, but some insight into the layers of the system, the APIs involved, and how everything interacts on various levels.

So as a case study of a major major information system, this class is actually quite interesting. However, it's really a class on managing geodatabases, not a case study on ArcGIS. So while the concrete details are putting me to sleep, the high-level stuff was definitely worth hearing. As a programmer, you tend to look at and read about things on more of a code-level. It's good to see how the "big boys" handle complicated design issues.

Well, here I am at Constitution Plaza in Hartford, Connecticut. I'm actually here on a business trip. My employer sent me to a training course with ESRI entitled Data Management in the Multiuser Geodatabase.

There are a few things to note about the very fact that I'm here. In the 6 years I've been with my current organization (which is far too long), this is the first time I've been on an out-of-state trip. In fact, it's the first time I've been on a trip that lasted longer than a day. It's also the first off-site paid training class I've been sent to. Kind of seems like a waste, given that if things things go well, I'll be able to get the hell out of here before I have a chance to put any of this to use.

As for the class itself, I have no real complaints so far. It's actually hosted by the City of Hartford IT Department, which has 2 people attending. The instructor, Jim, has been in the GIS business forever and has been working for ESRI for 14 years. He's really nice and seems pretty knowledgeable, so I'm actually enjoying the class so far. In particular, learning about the system architecture was kind of cool. Of course, I'll probably never put any of the stuff he's teaching us to use, but at least this gets me out of the office for a week.

One cool thing I've discovered is that Python is apparently one of the favored languages for automating ESRI's system. In fact, the lap machines we're using for the class actually have Python 2.4 interpreters installed on them. I don't know if we'll do any scripting in the class at all, but I just found that to be really cool.

I know there are those in the Linux community who will regard this as equivalent to spitting on the cross while blaspheming the holy spirit, but sometimes I feel sorry for the people at Microsoft. They have a tough job and they are frequently blamed for things that aren't their fault.

What made me think of this was Jeff Atwood's follow-up to his Windows spyware post.

I understand the pressure to be backwards compatible. There's no end of Vista blowback based on minor driver compatibility issues. The "if it doesn't work, it's automatically Microsoft's fault, even if the software or hardware vendor is clearly to blame" mentality is sadly all too common. But given the massive ongoing Windows security epidemic, was defaulting regular users to Administrator accounts-- exactly like Windows XP, Windows 2000, and Windows NT before it-- really the right decision to make?

In many ways, Microsoft is in a perpetual no-win situation. On the security front, for example, they are besieged by the evil forces of malware. However, every time they try to improve the situation, end users scream blood murder. For example, was the weird "quasi-administrator with prompting" a good idea? Probably not, but it's better for the average user than silently allowing the installation of spyware, and yet everyone seems to hate it. But what's the alternative? To make accounts regular users by default? How would the average Windows user would feel about that? I don't know, but I have read a many comments by recent converts to Linux who seem to think that entering a password just to install some software is completely stupid and unreasonable, so I can't imagine it would be universally hailed as a great improvement.

And, of course, there's always the breakage that accompanies any OS upgrade. For example, remember Windows XP Service Pack 2? I seem to recall widespread complaints about things breaking when that was rolled out. And we're seeing the same thing now with Vista. And who do people blame? Is it the ISV who are still coding with Windows 98 in mind? No - they blamed Microsoft.

What really kills me about this situation is when I read Raymond Chen's accounts of the efforts of the Windows App Compatibility team. For example, consider this sample chapter from his book. From the cases he mentions there, it is completely obvious that Microsoft takes backward-compatibility seriously. Very seriously. In fact, you might even say they take it too seriously.

Think of it this way. On Linux you're lucky if you can get source compatibility for an application that's more than 5 years old. Microsoft has binary compatibility with a large range of programs that are 10 or 15 years old. They're working with third-party binaries, diagnosing obscure bugs, and implementing fixes to keep the applications working, even though it's by sheer luck that they ever worked in the first place. As a programmer, it's hard to overstate how impressive this is. And yet all anyone ever focuses on is the problems they didn't fix.

Then there's the political angle. There are lots of people out there who seem to think that Microsoft can do no good. Everything they do is viewed with suspicion. Anyone who works for Microsoft has to contend with accusations that he is either in on the conspiracy or is bowing down to "the man" every time he says something they MS-haters don't like. That's got to be at least a little demoralizing. And while a certain degree of animosity is certainly warranted (as it is with practically any large business), it's not like Microsoft has been running child sweatshops or dumping toxic waste in the local drinking water. It just seems way out of proportion.

So no matter what the people in Redmond do, it seems like there's always somebody pissed off at them. And it's a shame, because they really do do some good work. The .NET world has lots of neat technologies and is a very cool place for developers. Even though OpenOffice may be good enough for many, MS Office is by far the best office suite available. And, last but not least, Windows, despite it's flaws (and they are legion) is a very impressive feat of software engineering. Not to mention that Microsoft employs a number of very bright people.

So, to the Linux community, I say, let's give Microsoft a chance. I'm not asking you to like MS, or to start using any of their products. Let's just be honest and realistic. Most people in the community aren't shy about placing the blame on them, but give credit where credit is due. We rightly object when people blame free software for not being a panacea, what with hardware incompatibilities and the lack of certain software. We should at least hold MS to the same standard and not judge them for failing to satisfy everyone.

We all have those few programs we can't do without. For the non-technical user, the list might include Internet Explorer and Outlook Express. For the hard-core geek, it might be Vim/Emacs, GCC, and GDB. As for me, lately I've found that VMware is way up on that list.

This is particularly the case when it comes to testing and evaluating software. If it's anything even remotely "big," such as Microsoft Office, or if it's something I'm doing for work or casual research and am not planning to keep installed, I'll just break out a VM, install the software, and then blow the VM away when I'm done. In fact, I keep a directory full of compressed VM images of various pre-configured test setups for just this purpose. When I need to try something out, I decompress one of the images, do my thing, and then delete it when I'm all done. It's kind of like the old-fashioned "take a disk image" approach, only way, way faster. Honestly, VMware makes things so easy, it baffles me that people still bother with physical test machines for testing applications. It's so...1990's.

But VMware is great for regular work too. The performance is quite good, so if you have even a middle of the road system, you can run medium to heavy-wieght applications in a VM without too much pain. This is especially useful if you happen to be stuck running Windows, because some things are just so much easier to do in Linux, such as getting the tools you need. Of course, virtualization can't beat running natively, but flipping back and forth between your regular desktop and a VM is a lot less cumbersome than dual-booting or having two computers.

Of course, my whole-hearted adoption of virtualization is not without its price. This week I found myself looking up prices on new RAM sticks for my desktop and laptop. The main benefit I envisioned? I could comfortably run more than one instance of VMware! It's the perfect answer to, "What could you possibly do with 4GB of memory?"

If you've never used any virtualization software, you really need to check it out. It's a godsend. I use VMware Player because it's free and available on both Windows and Linux. QEMU is also a fairly nice cross-platform solution. And for the Windows-only crowd, there's always Virtual PC. They might take a little getting used to, but it's well worth the effort.

Eugenia at OSNews ran an editorial today regarding password storage in the popular Pidgin IM client (formerly known as Gaim). Her complaint was that Pidgin stores the user's passwords as plain text, which is obviously insecure. However, the developers strenuously object to her proposed solution: obfuscation.

Basically the situation is that Pidgen does not have an encryption scheme for IM account passwords. Some people have submitted patches, e.g. to use gnome-keyring, but the developers don't want to tie themselve to any particular keyring implementation. So, in the absence of any real security, some users have suggested adding fake security, such as ROT13 encoding the passwords. Their argument is essentially that such obfuscation is better than nothing. The Pidgin devs, on the other hand, argue that this does nothing but provide a false sense of security, which is actually worse than being insecure.

I think the Pidgin developers have a good point here. For one thing, the use-case for obfuscation seems to be a little shakey. The general idea is that obfuscation would keep your passwords safe from nontechnical users who would have local access to your machine, such as a parent or co-worker. Presumably they would be able to find the accounts file, but not figure out the obfuscated password. The problem with this is that tools and methods to "recover" IM passwords are all over the net. All you have to do is Google it. Obfuscation would only stop people who are too stupid to use a search engine and those who are too unmotivated to use one. And for those types of attackers, proper file permissions and locking your terminal when you get up from the desk would be sufficient anyway.

To me, this sounds like a "shut up" feature, i.e. the kind of feature a developer implements not because it is actually good or useful, but because the customer thinks it is good and useful. "Fine, I'll do it if it'll shut you up!" This doesn't add any real security and it's not clear that it would stop even non-technical users who actually want your password, but it would make users feel better. That's the kind of thing you do in the commercial world for a whiney customer. I don't think it has any place in the open-source world.

It seems the three big news sites in my aggregator - Slashdot, OSNews, and Digg - have all picked up this Wired story about Firefox. Apparently people are starting to complain that the Fox has gotten slow and bloated. I guess they haven't been paying attention.

As an Opera user, this is something of a sore spot for me. First, the contention that Firefox is getting slow is complete and utter BS. "Getting" is irrelevant. Firefox has always been slow. Granted, speed is relative, and while FF may be fast compared to plain-old Mozilla (now known as SeaMonkey), Opera has always been way faster than both of them. If you don't believe me, try running Opera and Firefox side by side on a Linux box with a 500HMz processor and less than 256MB of RAM. The difference is painfully obvious.

And speaking of RAM, Opera has always had a lower base memory footprint than Firefox. As an example, here's a quick, highly unscientific screenshot comparison of memory usage in Opera 9.20 and Firefox 2.0.0.3 running on Windows XP. Note that Firefox has 9 extensions enabled and 6 tabs loaded. Opera, on the other hand, has 24 tabs loaded.

The really interesting thing to note here is that Opera's memory usage is quite variable. The 22MB in the screenshot is when Opera is sitting minimized in the task bar. Once I maximize it and start browsing, the numbers go up. In fact, the memory usage got up to 120MB at one point, but as soon as I minimize the browser window, it drops back down to 20MB or so, presumably transferring the data to disk cache. When I bring the browser window back up and start switching between open tabs, the RAM usage slowly creeps back up. So Opera is actually quite smart about memory. Not so with Firefox - its memory usage remained static when I tried the same thing.

Of course, Firefox would probably have a significantly lower memory footprint with no extensions enabled. But then, what would be the point of using it? After all, extensions are one of the big selling points. It's also where most of the cool features are implemented.

I always thought that was one of the biggest problems with Firefox: it almost forces you to install a bunch of extensions. Out of the box, Firefox is a good browser, but it's nothing special. I suspect the development team is finally starting to realize that having lots of good features out of the box is important. The extensions are great, but finding and downloading them is a pain and many "regular" users simply can't be bothered (not to mention the compatibility issues). By integrating them into the core, you spread the benefit to the masses rather than just those with a technical bent. Plus, you can (in theory) get better performance with the native C/C++ in the core than with a JavaScript extension.

So I still use Opera everyday and use Firefox for web development. I find Opera faster and easier to use in many respects. But Firefox has the Web Developer, Firebug, and HTML Validator extensions, which are really compelling. Now if only Opera would implement an extension mechanism and allow you to set an external RSS reader, I'd be all set....

Yesterday, I was telling you about the ridiculous install procedure for CRAPS. Today I'd like to continue the discussion.

As you remember from last time, the recommended procedure for getting the MS Access databases CRAPS uses on your network is to just install all the software directly onto the C: drive of your file server. If you're in IT in any capacity, and are at least marginally competent, I shouldn't have to explain why this is really stupid.

The customization and configuration process is similarly disjointed. Some of this you do in a crowded graphical configuration dialog. Some of it you do by hand-editing INI files. I'd say it's about a 60/40 split. And, as an added bonus, for a few settings you actually have to hand-edit rows in one of the database tables. Unless you don't have Access 2000. Then you just live with the defaults.

From the care and attention they put into the installation and configuration process, it should go without saying that the CRAPS developers take security very seriously. Thus they decided to use "binary files (also known as flat files)" to store certain data on field units. After all, "Binary files use a format that is specific to the application that creates them," which means that other programs are "unable to interpret binary files that they did not create. This makes [CRAPS] data secure, as only [CRAPS] can interpret and use its binary files." And if you don't believe me, that comes directly from the CRAPS administration manual. Seriously. The only thing I changed was the name of the system. The manual also claims speed and smaller file size as benefits, despite the fact that field units are single-user laptops with Pentium IV processors and 80GB hard drives.

It's always a bad sign when the developers feel the need to justify their choice of data format in the user manual. So it probably comes as no surprise that when you actually look at one of these "binary files," it contains mostly serialized text data. It's definitely not encrypted and most of it isn't even really binary. With a little intelligence, it's not even too hard to figure out the format just by looking at it. It sure is a good thing that opening sequential files in Vim is such a well-guarded secret, or else the security benefits might not seem so compelling.

And even worse, some of the "binary files" referred to in the manual are just standard formats. For example, guess what the "binary" import and export files are? They're ZIP archives! With a different file extension!

The really sad part is that, for all its fauilts, CRAPS actually works fairly well. Yes, it's slow (despite the speed benefits of binary files), the configuration is tedious, the user interface is arcane, and the whole system has a feel of being held together with duct tape and bailing wire, but it does work. It truly is worse than failure.

Today I start a new series: Tales of IT. In this series, I will be discussing some of the more humorous things I come across in my work in the IT world. Some are funny ha-ha, some are funny-weird, and some are just funny-sad.

Let me start by telling you a little about what I do. I'm a "Systems Analyst" for a local governement agency. In my organization, that means that I do software development (that's my favorite part), analysis, some server administration, some help desk stuff - whatever comes up. So at this point, I can basically do some of everything.

Now let me tell you about the organization I work for. Being the government, it should go without saying that it's extremely disfunctional. One of the more annoying disfunctions of the IT department is what I call the "you touch it, you own it" theory of tech support. By that, I mean that, for any software system (especially big, complicated, or expensive ones), there is an analyst who "owns" that system. Ownership is determined on a first-come, first-served basis, so if you're the first person to touch a new system, you will own that system until you quit, retire, or die. The natural result of this is that nobody will ever volunteer for anything.

Now, you may be thinking that having a "local expert" on every system we use is a good idea. And you'd be right. It is a good idea to have somebody with a degree of in-depth knowledge around. That isn't the issue.

The problem is simply that the whole situation is completely demoralizing. When you own a system, you become the goto guy for all problems related to that system. That includes everything from the hardish analysis work of determining the proper system configuration all the way down to the grunt work of clicking next through client installations on a couple dozen workstations. If somebody calls the help desk with a problem related to your system, it gets passed straight back to you - the help desk doesn't even try to resolve it or even get more information.

This brings me to a system I own, the Crazy and Ridiculously Attrocious Police System, or CRAPS for short. CRAPS is the system used by the police force to write tickets, take accident reports, and so forth. I got stuck with it because it runs on the terminals in the police cars, and since I already owned the 911 dispatching software on those terminals, and my boss apparently didn't think I had enough pain in my life, I was the obvious choice.

I've got plenty of stories about CRAPS, but today I'll just tell you about the system itself. For starters, CRAPS has a "client-server" design. By that, they mean the system stores data in Access 2000 databases that sit on a file "server" and that the "clients" get pointed to the mapped drive or UNC path to those databases. Sort of.

You see, the CRAPS manuals and installation program seem a little confused about just what a server is. The installer has options to perform a "server install" and a "client and server install." What this choice really comes down to is "install just the .MDB files on this machine or install the .MDB files and the program files on this machine."

The important thing to note here is that both options are for this machine. That means that if you want to do what we did and put just the databases on your file server, the installed expects you to log in on the file server to run the installation. You can't just run from another machine and point the installer to the network location. I know, I tried. It fouls up things in the next stages of installation. It turns out that there's a reason the manual "strongly recommends" that you keep the default installation path of C:\CRAPS - things tend to get messed up if you don't.

Part of the reason things get messed up is that a basic CRAPS install is a 2-step process. First, you install the base system. Second, you install an add-on package with all the databases you need to actually get things done. The databases from both packages need to be on the server, but there is no "server install" option for the add-on databases. In fact, you can't even change the installation path. They get installed to the same path as your base system. Unless you changed the default path. In that case, some of the files get installed to the right place and some of them disapear into the ether, never to be heard from again. In fact, I've found that it's more reliable to just install everything on a workstation and then copy manually copy the data directory to the file server. It really shouldn't be, but it is.

At this point, CRAPS is installed on the file server, but is not yet functional. We'll explore how to make it work tomorrow.

As you may have noticed, I've taken to posting screenshots a bit more. When doing this, I find it helpful to add highlighting to draw attention to the relevant elements. I usually like to do this by drawing circles around things.

Since I have a really hard time drawing circles with the mouse, I figured out how to do it with selections in the GIMP. I record it here as much for my own benefit as for others, because it seems like I have to rediscover this every time I try to do it.

First, use the selection tool to select the area you want to circle. Then, from the "Dialogs" menu, open the "Paths" dialog. Click the "Selection to path" button (the red dot with black lines above and below) to turn you selection into a path. Then click the "Stroke path" button just to the right of that. You can set the stoke thickness in this dialog and change the color by changing the color used by the pencil tool. Click the "Stroke" button and the outline of your selection will be turned into a line or the given thickness and color.

Linux has spoiled me. I've gotten so used to things "just working" that I sometimes forget just how miserable the Windows world can be.

That probably sounds weird to some people, but it's true. Granted, lots of hardware vendors and ISVs don't even pretend to support Linux, game companies largely ignore it, and many online services don't seem to be aware of its existence. But there is an up side to that - no malware or crapware.

This was driven home to me he other day when I had to "fix" a user's flash drive. The flash drive in question was a SanDisk Cruzer Micro which the user had purchased to replace his "official" one which broke. The short version of the background is that this user needed a flash drive to work with a system I'll call TCSFOUBTS (pronounced 'ticks-fouts'), which stands for The Crappy Software Foisted On Us By The State (not its real name). Since the "official" keeper of the flash drives had no extras and it would take days or weeks to get a replacement (welcome to civil service), he just bought one on his own.

As with all such cases, this didn't work out well. TCSFOUBTS, being a crappy Widnows application, requires an administrator to set drive letter at which the flash drive will be mounted. However, SanDisk saw fit to include something called U3 on the drive, which adds an annoying systray icon that allows you to install and run software from the flash drive.. This apparently adds a second partition to the drive, which Windows sees as a CD-ROM. Naturally, this is the first partition, so instead of the writable portion showing up as D:, it shows up as E:. Needless to say, TCSFOUBTS was configured to use D:.

The up side is that SanDisk was considerate enough to provide a removal utility for this crapware. It would be better if they didn't include it in the first place, but as crapware purveyors go, this makes them a good citizen. Most of the companies that push this stuff go out of their way to make it difficult to remove. Funny how even annoying things can look good by comparison.

I've discovered why Firefox hates me. Apparently it's not just me, but broken code in Firefox.

I found the solution in this Mozillazine thread. It contained no mention of the error I got in the console, or of external feed readers simply not launching, but the patch to FeedConverter.js did fix my problem.

All I had to do was change the indicated section of /usr/lib/firefox/components/FeedConverter.js, start Firefox, and add the strings browser.feeds.handlers.application.args and browser.feeds.handlers.application.uriPrefix in the about:config dialog.

So now external feed readers work. In fact, they work better than they're supposed to, because this patch adds support for setting arguments to the external command, so no wrapper script is required. For Akregator, I just set "--addfeed" as the browser.feeds.handlers.application.args and /usr/bin/akregator as the command. Bingo!

A few weeks ago, I installed Beagle, a desktop search tool written for Mono, and it's KDE front-end Kerry. This combination has just blown me away.

I guess desktop search is like the microwave oven, cell phones, and the internet - it's hard to realize how truly useful it is until you've gotten used to it. I know I didn't. But so far Kerry-Beagle has saved me lots of time searching through those hundred-odd PDF files I've been using for research. Now that I'm used to it, I don't think I'd want to do without it.

The user experience is probably the best thing about Kerry-Beagle. Once you have the software and all its dependencies installed (and there are a lot of them - Mono, like the .NET framework for Windows, isn't especially small), it's dead simple to use. There's basically no configuration required, as it will just go right ahead and index your entire home directory. You type in your search terms and wait for the results to come up. They're displayed is a nice, paginated list of files, complete with icons and matching portions of text and sorted by relevance. Easy to read, easy to browse, and just generally great. If you don't believe me, just look at the screenshot:

My only other desktop search experience is with the (apparently now defunct) Kat desktop search application for KDE, which never really worked very well for me. I guess now I'll have to try out some of the Windows offerings. Anyone have any suggestions on which are the best packages?

Nothing ever goes right when I try to use Firefox. I don't know why. I never do anything that exotic. But for some reason, it always falls short.

This time, it's the new RSS features in Firefox 2. You click on a feed link and you get a nice, readable display of the content with a "Subscribe Now" button at the top that lets you subscribe to it with several online services or allows you to pick a desktop application. However, in my case, it doesn't work.

And when I say "doesn't work" I don't mean it doesn't work as well as I'd like, or that it pops up an error message. I mean it does absolutely nothing. I open a feed, point Firefox to a script to add it to Akregator, click the "Subscribe Now" button and...nothing. Absolutely nothing happens.

I know it's not the script, because it works perfectly on the command line. I know it's not something in my profile because I already tried doing this from a blank profile. The only conclusion is that the Ubuntu Edgy Firefox package is broken.

There was one piece of feedback I turned up. After clicking the subscribe button, two instances of an extremely obscure error message show up in the error console. The exact text is:
Error: [Exception... "ServiceManager::GetService returned failure code:" nsresult: "0x80570016 (NS_ERROR_XPC_GS_RETURNED_FAILURE)" location: "JS frame :: file:///usr/lib/firefox/components/FeedConverter.js :: FRS_addToClientReader :: line 338" data: no]
Source File: file:///usr/lib/firefox/components/FeedConverter.js
Line: 338
The referenced line of code is this:
var ss = Cc["@mozilla.org/browser/shell-service;1"]
.getService(Ci.nsIShellService_MOZILLA_1_8_BRANCH);
I have absolutely no idea why this is failing or how to fix it. That is particularly frustrating because it's the only thing I have to go on. Does anybody else have a clue?

We went to visit my parents this past weekend and, as always, I ended up doing a little tech support. This time, it was actually for my brother's new laptop. He needed to get his data off his ancient Windows 98 box and his best idea of how to do it was to transfer all 12GB of it to the new system over IM. I didn't bother to explain all the myriad ways in which this was completely crazy.

Moving data from a Windows 98 PC to a Windows XP laptop proved to be slightly more difficult than I initially anticipated. Plan A was to use a handy-dandy USB thumb drive. However, Windows 98 doesn't natively support USB mass storage devices, so I needed drivers. However, for the one device I did have drivers for, I couldn't get Windows to pick them up. They installed with no errors, but when I plugged in the drive, it wasn't recognized. So plan A was a complete failure.

Plan B was to set up a good, old-fashioned network share. That didn't work very well either, though. After setting both PCs to use the same workgroup, the 98 machine couldn't connect to the XP machine, and the XP machine couldn't even see the 98 machine. So, fool that I am, I tried the XP home network setup wizard. After that, the 98 box couldn't see the XP system anymore. Of course, it might have something to do with the fact that I didn't run that network setup disk XP prompted me to create on the 98 box. Not that I could have if I'd wanted to. The setup wizard apparently can't put the disk on a CD-R, the laptop has no floppy drive, and the 98 box can't do USB drives, so how was I supposed to make the disk?

So plan B was a bust. Apparently I'm just not smart enough to figure out the magic of how Windows networking functions. I must admit that I don't know much about it, but apparently it's another one of those opaque Windows things: it either "just works" the first time or it doesn't and you're completely screwed. I guess I'll have to do some reading to figure out exactly what went wrong there.

So that lead me to plan C: when all else fails, use Linux. I remembered that my mother's system runs Xandros, has an SSH server, and has a mostly empty 80GB hard drive. So I did a quick web search and found WinSCP, a free graphical SFTP and SCP client for Windows. I installed that on both Windows boxes and problem solved! Just upload all the data from Win98 to Xandros, and then download it from Xandros to XP. Plan C was a success!

Too bad it took so long. I wasted way too much time on trying to set up a network share, and I never did figure out what the problem was. So much for Windows being "easy to use."

Speaking of VMware, while researching some VMX file settings, I came across a very handy page on manually creating VM images. It even includes VMX file templates and pre-made empty VMDK disk image files. Very useful stuff.

Project: Burn a CD in VMware using a Windows guest to a Linux host.
Status: Near total success.

Research on the web led me to believe that it was possible for VMware to let a guest system burn CDs. Perviously, I had attempted this with a Linux guest on a Windows host and met with complete failure. This time, things were different.

On a Linux host, there was really nothing to it. The only configuration required was to set the CD-RW drive's deviceType in the virtual machine's VMX file to "cdrom-raw".

There was one catch, however. It seems that the CD burning wizard built into Windows XP doesn't like working in a VM. When I tried to burn a disc, the wizard was unable to detect any CD in the drive and refused to let me proceed.

The good news is that the other two applications I tested worked without a hitch. I tried burning a data disc with an OEM copy of Nero Express and an audio disc with iTunes. Both worked perfectly. And really, iTunes was the only one I even cared about, so I'd say things worked out pretty well.

Does anybody know how to change the network adapter that VMware Player bridges to?

I haven't used VMware in a while, but when I fired it up the other day, the network bridging just plain didn't work. This struck me as weird, because it used to work.

After some futile web searches, I eventually grepped around /proc and discovered /proc/vmnet/bridge0, which indicated that the network bridge was going to eth0. This explained the lack of connectivity, as eth0 is my unplugged ethernet card. I wanted it to go to ra0, my WiFi card. However, rerunning vmware-config-network.pl didn't seem to do anything. In fact, I couldn't even get it to ask me what device to use.

I could be way off track here, but I suspect this has something to do with the fact that I'm using the pre-built VMware kernel modules from the Ubuntu repositories. I almost think the device to bridge to is configured at compile time and built directly into the modules. If that's the case, that kind of sucks.

At least the NAT networking is fine, so at least I've got something. On the down side, it's a little inconvenient. Since VMware Player doesn't have shared directory support, I need to go through the network to access my file shares. Accessing the localhost from a NATed VM doesn't work so well. I can still mount my local shares on a remote host and go through a share on that host to get the data, but that's, er, just this side of crazy.

I took yesterday off work and, naturally, I had a wierd helpdesk request on my desk when I got back. One of our outlying locations needed help with a UNIX-based server that had gone down. This was wierd because I didn't think we had any UNIX-based systems.

Up until now, I thought the only UNIX-based system we had at work was part of our financial system. At least, I think we have one there. I'm not allowed to touch those servers. And I don't want to touch them, because in our organization, as soon as you touch a system, whether it's hardware or software, you automatically become responsible for maintaining it for the rest of your life. All I know is that I've seen KSH scripts sitting on the printer a couple of times, and the financial system is the only place they could be coming from.

But it turns out that this other department has a security system that uses a UNIX box. But not a good UNIX. It's a really old version of SCO UnixWare. The only other UNIX that old that I've ever used was the Digital UNIX server I dialed into in college to check my e-mail.

And when I say old, I mean positively primitive. No bash, no less (!), no grep -r, really old X server running Mwm. Basically, a classic example of the bad old days of UNIX.

Of course, in fairness to SCO (assuming they deserve any fairness), the computer was at least 7 years old and had never been updated. The UnixWare version was probably 8 to 10 years old. Linux wasn't that great when I got into it 6 years ago. I can only imagine what RedHat and SuSE were like 7 or 8 years ago. Although I can't imagine they were any worse to use than this.

In the end, it doesn't really matter how bad the OS was. They're replacing the security system with a Windows-based one used in other departments, so I only had to get the UnixWare box to limp along long enough for the new system to come in. And I didn't really even have to mess with the software, since the problem was with the hardware.

Why is my USB so messed upon Ubuntu? First my cell phone data cable doesn't work properly, and now my MPIO FL100 MP3 player doesn't work right. What's the deal?

Let me rewind and give some background. A few years ago, my parents gave me a 128MB MPIO FL100 MP3 player for my birthday. It's a nice enough player, with a display and decent controls, but it's not Linux-friendly. In other words, it doesn't work as a block device, so you can't just mount it as a USB hard drive. It requires special software.

Fortunately, such software exists for Linux. The MPIO project over at SourceForge hosts it. It has both a command-line application (similar to an FTP client, for some reason) and a simple KDE-based front end. Of course, this project is now almost completely dead save for a post to the mailing list every month or two, so any hope of bug fixes or new features seems unfounded, but the software still works.

Or, it did until recently, when I noticed that this player no longer works properly under Dapper. I'm not sure exactly when the problem started - if it was the upgrade to Dapper or some upgrade since then. Either way, things are definitely no longer normal.

I'm now having trouble connecting to the device. I experimented this evening, and it seems I need to be root in order to connect to it. When connecting as a normal user, which used to work perfectly, I now get a message that the device cannot be found. Connecting as root, however, seems fine. The first time, at least. If I disconnect the software and try again, I get a message like this:
mpio: src/io.c(766): mpio_io_read: libusb returned error: (ffffff92) "No error"
mpio: src/io.c(816): mpio_io_version_read: Failed to read Sector.(nread=0xffffff92)
mpio: src/mpio.c(409): mpio_init: Unknown version string found!
Please report this to: mpio-devel@lists.sourceforge.net
mpio: src/mpio.c(410): mpio_init: data=0x8052820 len=64
mpio: 0000: 00 00 00 00 00 00 00 00 00 00 00 00 20 20 20 20 ............
mpio: 0010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
mpio: 0020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
mpio: 0030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
mpio: src/mpio.c(171): mpio_init_internal: WARNING: no internal memory found
It seems to work again if I physically disconnect the device.

This is growing to be a pain. At this point, I guess the main question is, how much effort do I want to put into fixing support for a now obselete device? I'm guessing the answer is going to be "not that much."

As a programmer, I appreciate how great it is to be able to read and modify the source code for the software I use. Really I do. On more than one occasion, I've even availed myself of the source to fix bugs or implement features in software that I use. As a good member of the FOSS, I naturally sent these patches to the program maintainers. It's a great system.

However, I think the fact that source is available causes its importance to be overplayed by people in all parts of the community. I think things would be a lot better for everyone if we de-emphasized the idea of users building their software from source.

Now, I'm not saying that source code shouldn't be easily available. By all means, keep that link to the source tarball in a prominent location on the download page. Anyone who wants the code should be able to find it with no trouble.

What I am saying is that we should try to keep end users away from compiling from source unless they really need to do so.

Some people will say that all users should know how to compile things from source. They'll say it's easy and it's a good learning experience. They'll also say it's convenient, in that it will work on any variety of UNIX. They're also wrong on all counts.

First, I've seen quite a number of users on web forums who, though they apparently build programs with some regularity, haven't learned a damned thing from it. You know what they've learn? "Type ./configure, make, and make install as root." That's not learning, that's mimicry. In fact, I've seen forum postings where users couldn't figure out why ./configure was returning a file not found error on a package which I knew for a fact didn't use autotools. That's no better than the Windows monkey who sits there reading instructions on where to click.

Building things from source can be dead simple. But if you don't have everything you need, it can be a huge pain. Many users are simply ill equipped to deal with the problems that might come up, from missing libraries, to missing header files to an inappropriate build environment. The simple truth is that no system is guaranteed to have everything needed to build every program. So when the error messages start piling up, what do the users think? "Why can't I just run a setup.exe?"

And did I mention that managing programs installed from source is a pain? Not only is there not necessarily any easy way to uninstall such programs, but the simple fact that they won't be registered with your package manager can plunge you into dependency hell. The easy solution is, of course, to compile from source and build your own RPM or DEB or whatever. But doing that right isn't trivial and doing it half-assed is still a pain.

And what benefit is there to compiling something from source? Well, if there's no binary available, then the answer is obvious. Likewise if you need to apply a patch or enable a non-standard compile-time option of some kind. But where there is already an acceptable binary available, what's the benefit?

Really, there isn't any benefit. If you can use a binary package, then building from source is a waste of time. Despite what some people like to claim, compiling from source will not make the program run faster. Oh, I know you can set compiler flags to optimize for your hardware, but trust me: you won't see the difference. There are some types of applications for which the right processor-specific optimizations can make a significant difference, but for most desktop applications, it just doesn't. Any speed gains you do get are typically too small to notice.

My recommendation is that users should build from source only as a last resort. If there's a package for your distribution, use that. If there's no package for your distribution, but there is for another (e.g. you're using Ubuntu, but there are only Fedora RPMs), try converting it with alien or some similar tool. If that fails, then you can think about building from source. Going straight to the source is time consuming and just makes things more complicated than they need to be.

Periodically, you see stories making the rounds about how online applications are going to completely supplant traditional desktop applications. In fact, these ideas have recently been extended to encompass the entire desktop, with the rise of web-based "operating systems."

It sounds great, doesn't it? All your data and all your applications would be available from a central server. You could use any computer, anywhere in the world, to access your personal, customized desktop, and it would always be exactly the same.

However, over the last month or so, and this week in particular, I've experienced the perfect proof that such ideas are, well, over-rated. That proof is internet outage.

Yes, Time Warner's Road Runner cable internet service has been very unreliable the last month or so. It's normally pretty good, but I've been experiencing frequent outages, usually for several hours at a time.

With wide broad-band availability, many of us have started to take high-speed, always-on connections for granted. Putting your entire desktop online is great when you know you will always be able to access it. But if everything is online, then when your connection goes down, your computer is completely useless.

The everything online philosophy also seriously limits the usefulness of laptops. I know that may sound shocking to some people, but the truth is that you can't get free WiFi access everywhere. In fact, there are many places where you can't even get paid WiFi access. For instance, my wife sometimes takes the laptop to work on slow days, where they have no web connection (though I'm not sure why) and no wireless access points nearby. On those days, it's nice that OpenOffice and Klickity (her new addiction) are desktop, rather than web, applications.

Not that I have anything against web applications. They're great! It's just that, like everything else in the world of computing, they've been over-hyped. Not everything needs to - or should - be a web application. Not every web application has to use AJAX. Not every program will benefit from using the latest trendy technology. And, finally, one that seems to have finally sunk in: not every application has to incorporate XML in some way, whether it makes sense or not.

Quickly following up on yesterday's tip to newbies, the advent of HAL and volume managers means that you no longer have to do things like mess around with udev scripts to accomplish automounting and such.

See, this is the beauty of HAL. At the system level, all it does is keep an up-to-date list of hardware. That's all. The HAL daemon just tracks when devices are added, removed, or changed and doesn't actually do anything.

All the actual actions are carried out by client applications, such as the desktop volume manager. The beauty of using a volume manager is that it's just a regular program run by a regular user. That means that you can customize things on a per-user basis without even needing root access. You can even run more than one of them if you really want.

So if you're trying to set up auto-mounting or some form of auto-execution, just use your freakin' volume manager. Don't mess around with udev or hotplug scripts. Don't use old solutions like the autorun program or supermount. Just use the HAL solution. And if distribution you use for your desktop doesn't support HAL, then I suggest you change distributions. Handling removable drives no longer sucks in the Linux world and there's no point in suffering when you don't have to.

Listen up Linux newbies! I'm going to let you in on a little secret: the "auto" and "noauto" options in /etc/fstab don't mean what you think they do.

Why do I bring this up? Because if you frequent forums like LinuxQuestions.org, you've probably seen threads like this one, asking about getting USB thumb drives to auto-mount or not auto-mount. Invariably, somebody suggests adding an "auto" or "noauto" to /etc/fstab.

To an inexperienced user, this might seem like a natural suggestion. "Auto" means auto-mount, so that should do the trick, right? And if you specify "noauto" then the device should not auto-mount. It makes sense and it's easy to do. Perfect!

Unfortunately, the system doesn't work that way. If you read the man page for the mount command, you'll see that the auto and noauto options actually control whether or not the filesystem is mounted when you run the "mount -a" command. This command mounts all filesystems not specified as "noauto" and is typically run on boot to mount your local hard drives and network filesystems.

So, in other words, the "auto" option in /etc/fstab really just means "mount this device on boot." For USB thumb drives, this is pretty useless, as you don't normally have them plugged in when you first boot up. What you want is for the drive to be automatically mounted when you plug it into the system. However, /etc/fstab has absolutely nothing to do with this.

On modern Linux systems, such as Ubuntu, Fedora, or anything released since 2005 (except for Slackware), automounting of removable drives is accomplished through the magical combination of D-BUS, HAL, and a volume manager (which is often integrated into the desktop environment).

The good thing about this setup is that it's highly flexible. D-BUS is a generic message bus and HAL is a generic hardware abstraction layer, so they can handle most any kind of hardware hotplugging, not just USB drives or CDs. It can also work for printers, digital cameras, MP3 players, or what have you. The down side is that it's a fairly complex software stack with lots of dependencies, so if it's not working out of the box, you're pretty much screwed. Sure, you might be able to get them working eventually, but it will probably take some significant knowledge of the system.

So if you want auto-mounting and it's not working, the news is bad. But if you're just looking to turn off the auto-mounting or popping up a file manager or something, the news is good. In fact, if you're using GNOME or KDE, the news is great, because they both have graphical configuration for the volume manager. In KDE, for example, you can set custom actions by the type of media, so you can choose to auto-mount USB drives, but not DVDs, or choose to automatically play audio CDs and video DVDs.

So the lesson for today is: don't worry about the auto and noauto options in /etc/fstab. They don't do anything interesting. And if you're trying to get your USB or other removable media to mount when you insert them, they don't do anything at all.

And, lastly, if you see somebody spreading this myth of "auto" and "noauto" around, call them on it. Correct them and point them to this page. And, if you're a real jack-ass, laugh and mock them for their lack of skillz.

Will someone please tell me when MSDN started to suck? I remember back when I first started with Visual Basic, MSDN was really great. It was a wonderful reference source with lots of good material. The site was relatively quick and easy to use, the documentation was useful, and the examples tended to be at least moderately informative.

What the hell happened? Today I was looking up some information on using the XPathNodeIterator class in the .NET framework and Google directed me to the MSDN page for it. It was horrible!

The first thing I noticed was the truly massive page size. I literally sat there for seven seconds watching Opera's page load progress bar move smoothly from zero to 100%. And that's on the T1 connection at work!

The second problem is the class declaration, which says that it's a public, abstract class that implements the ICloneable and IEnumerable interfaces. There's nothing wrong with including that information per se. I personally don't think that including the code for the declaration is particularly helpful, as they could just as easily say that in pseudo-code or English, but whatever. What I do object to is that they included this declaration in five different programming languages! Why?!?! Of what conceivable value is it to waste half a screen worth of text to display a freakin' declaration in VB, C#, C++, J#, and JScript? Is the average Windows programmer really so completely clueless that he can't decipher this information without a declaration in his particular language? It's ridiculous!

The third problem is the code samples. Or should I say "sample." There are three code blocks, each of which has exactly the same code, except translated into different languages - VB, C#, and C++. Again, why? Is this really necessary? And if it is, why do they have to display all three on the same page? Why not break out at least two of the samples into separate pages? It's just a pain to have to sort through lots of irrelevant information.

My last complaint is the content of the example itself. Maybe this is just a product of my not yet being too familiar with .NET or with object-oriented enterprise-level frameworks in general, but the code sample just struck me as kind of bizarre. The goal of the algorithm was to iterate through a set of nodes in an XML file. To do this, they created an XPathDocument object and got an XPathNavigator object from that. Fine. Then they selected a node with the navigator object to get an XPathNodeIterator object. OK, I get that. Then they saved the current node of the iterator, which returns an XPathNavigator. Umm.... And after that, they selected the child nodes from the navigator to get another XPathNodeIterator, which they then used to actually iterate through the child nodes.

Is that normal? Do people actually write code like that? I mean, I can follow what they're doing, but it seems like an awfully circuitous route. Why not just go straight to from the initial navigator to the final iterator? You can just chain the method calls rather than creating a new variable for each object that gets created, so why not do that? I suppose the charitable interpretation is that the example is intentionally verbose and general for instructive purposes. But to me, all those extra object variables are just confusing. It makes for another, seemingly redundant, level of indirection. Maybe I'm atypical, but the direct approach makes a lot more sense to me.

Well, after a bit of experimenting, I implemented my first quick-and-dirty site-specific fix in Opera. It wasn't even that hard.

The motivation came when I received a site update e-mail from theotaku.com, which I apparently registered with at some point. I had completely forgotten about it. I reaquainted myslef with it and rediscovered the fairly decent selection of images they have.

The only problem was that their homepage layout was completely garbled in Opera. It consists of a bunch of div tags that divide the content area up into news entries. However, the divs end up being crushed down to almost nothing and the text spilling out and running together. It turns out the problem was a height: inherit line in the stylesheet for the class applied to those divs. I'm not sure what the purpose of that line was, but removing it fixed the problem.

Getting the site to render correctly for me turned out to be quite simple, once I figured out how to do it. I ended up simply downloading a copy of the (single) stylesheet for the page, removing the problem line, and setting it as "my style sheet" in the site preferences dialog. That allowed me to simply change the view mode from author mode to user mode and ta-da! The page now renders correctly.

After weeks of mind-numbing IT type stuff, I'm finally getting back into programming a little. I've been playing with the .NET XML libraries the past couple of days. In particular, the System.XML.XPath library, which I found quite handy for accessing XML configuration files. So, after reading up a bit on XPath, XSLT, and XML in general, I was naturally overcome with a fit of optimism and decided to look at converting LnBlog to store data in XML files.

Currently, LnBlog stores it's data in "text files." What that really means is that it dumps each piece of entry meta into a "name: value" line at the beginning of the file and then dumps all the body data after that. It's not a pretty format in terms of interoperability or standardization. However, when you look at it in a text editor, it is very easy to see what's going on. It's also easy to parse in code, as each piece of metadata is one line with a particular name, and everything else is the body.

This scheme works well enough, but it's obviously a bit ad hoc. A standard format like XML would be much better. And since PHP is geared mostly toward developing web applications, and XML is splattered all over the web like an over-sized fly on a windshield, I figured reading and writing XML files would be a cinch.

Little did I know.

You see, for LnBlog, because it's targeted at lower-end shared hosting environments, and because I didn't want to limit myself to a possible userbase of seven people, I use PHP 4. It seems that XML support has improved in PHP 5, but that's still not as widely deployed as one might hope. So I'm stuck with the XML support in PHP4, which is kind of crappy.

If you look at the PHP 4 documentation, there are several XML extensions available. However, the only one that's not optional or experimetal, and hence the only one you can count on existing in the majority of installations, is the XML_Parser extension. What is this? It's a wrapper around expat, that's what. And that's my only option.

Don't get me wrong - it's not that expat is bad. It's just that it's not what I need. Expat is an event-driven parser, which means that you set up callback functions that get called when the parser encounters tags, attributes, etc. while scanning the data stream. The problem is, I need something more DOM-oriented. In particular, I just need something that will read the XML and parse it into an array or something based on the DOM.

The closest thing to that in the XML_Parser extension is the xml_parse_into_struct() function, which parses the file into one or two arrays, depending on the number of arguments you give. These don't actually correspond to the DOM, but rather to the sequence in which tags, data, etc. were encountered. So, in other words, if I want to get the file data into my objects, I have to write a parser to parse the output of the XML parser.

And did I mention writing XML files? What would be really nice is a few classes to handle creating nodes with the correct character encoding (handling character encoding in PHP is non-trivial), escape entities, and generally make sure the document is well-formed. But, of course, those classes don't exist. Or, rather, they exist in the PEAR repository, but I can't count on my users having shell access to install new modules. Hell, I don't have shell access to my web host, so I couldn't install PEAR modules if I wanted to. My only option is to write all the code myself. Granted, it's not a huge problem, so long as nobody ever uses a character set other than UTF-8, but it's still annoying.

Maybe tomorrow I can rant about the truly brain-dead reference passing semantics in PHP 4. I had a lovely time with that when I was trying to optimize the plugin system.

Yay! I can finally do it! I can finally use Opera and Akregator together! Well, at least to a certain extent.

Yesterday I discovered this blog entry by zeroK on this very subject. The basic concept is so simple it's brilliant: define a custom protocol. Opera allows you to modify the handler programs for protocols like mail, telnet, etc. Well, the solution is to simply define a feed:// protocol and set the handler to your RSS aggregator.

Unfortunately, there's really no such thing as a feed:// protocol, so you need some JavaScript. For feeds linked in the page header, the solution was to use the modified bookmarklet that extracts the links and pops up a list with feed:// substituted for http://.

As for the handler application, I banged out a little shell script using DCOP calls and KDialog to add a feed to a selected group. I didn't use the Akregator command line options because they don't seem to work when you're embedding Akregator in Kontact.

The only problem with this is that it doesn't work with Opera's built-in RSS icon. Changing the protocol on the linked RSS feeds with a user JavaScript just seems to make them stop working altogether.

Hopefully Opera will eventually add a setting to configure an external feed reader. While I love Opera as a web browser, I never really cared for the mail client. And since the RSS reader is based on the mail client, I don't like that either. In fact, not only is the feed reader based on a mail client I don't like, but it seems to work more like a mail client than an RSS aggregator. I tried it out again the other day and I really hate it. I'd much rather have something with the three-panel layout like Akregator or SharpReader, so I don't think I'm going to be switching any time soon.

But, at any rate, at least I'm making progress in this department.

Imagine my surprise. As per my usual Saturday routine, I dropped Sarah off at work, stopped in to Barnes & Noble, grabbed some magazines, and sat down in the café to have some breakfast. Little did I know, as I browsed a copy of Linux Format, that I'd actually find a column on myself.

Well, OK, it wasn't actually about me, but rather about my work. You see, the June 2006 issue of Linux Format included a column on LnBlog 0.6.4, my weblog system. It wasn't a feature article or anything, but it was on the first page of the "Hot Picks" section, which I thought was pretty cool.

This is kind of nice timing, since I've been trying to start building up a bit of a community around LnBlog. It started out as a personal learning project, but it's grown into a very capable blogging system. It's got the basics, like comments and trackbacks, multiple users and blogs, anti-spam plugins, and, in the newest release, blogging API support and optional rich-text editor plugins. It has a theme system and a plugin system with API documentation. In other words, all the technical makings for community development.

So, if you're a PHP programmer, a web designer, or just an average user looking for a blogging system for your web site, give LnBlog a look. It's light-weight, featureful, configurable, and has plenty to keep users and developers interested. I'll be making a few more posts about the technical aspects of LnBlog in the near future, so you'll get a more in-depth look later. If you're looking to get involved in an open-source project, please give it a look.

Let's continue my "bad argument" theme from yesterday with a news item from Slashdot. This one is regarding a ZDNet report that Microsoft representative Alan Yates claimed the MS Office XML format is preferable to the Open Document Format (ODF) for performance reasons. For support he cites this ZDNet blog by George Ou, which compares MS Office to OpenOffice.

Now, it may or may not be true that the Office XML format can be implemented more efficiently, in terms of space and manipulation speed, than ODF. I really don't know enough about the issue to say one way or the other. However, I do know that the ZDNet blog entry Mr. Yates cited doesn't say much of anything about it either. In other words, the argument is bad because the cited evidence doesn't really support the conclusion.

So what does the blog entry say about Office XML and ODF? Well, it gives an example of two 16 sheet spreadsheet files with the same data, one in Office XML format and one in ODF format. The author found that OpenOffice Calc used around three times as much memory and 100 times the CPU time as Excel.

Does this mean that Office XML is faster than ODF? No. It just means that Excel 2003 is faster than OpenOffice Calc 2.0. Given that OpenOffice has a history of being slow and bloated (version 2 is better, but still not great), this should come as a surprise to no one. I'm sure that if you compared the two programs using the same file format, Excel would still be faster.

So as far as the file formats go, this entry is comparing apples to oranges. And why shouldn't it? After all, Mr. Ou's original point was to compare the speed of OpenOffice to MS Office, which is exactly what he did. He's just had his words taken out of context by a Microsoft spokesman to support an argument he never tried to make.

You know what pisses me off? Bad arguments. As a philosophy major, I read lots fo bad arguments. As a "philosophy hobbyist," I still do. And nothing makes me madder than those arguments that are so hideously wrong you don't even know where to begin explaining the problem.

I ran into one of those at work today.  Because of HIPAA and other such laws, we're looking into encryption products (laptop drives, database, e-mail, etc.), and I was stupid enough to volunteer for the assignment. Of course, it proably doesn't matter, because the whole effort is doomed from the start. There is absolutely zero buy-in from our IT staff on the idea of deploying encryption products. The only person who is even remotely pleased with the idea is the boss, and I'm the only one of the staff who isn't dead-set against it. The network administrator is particularly against the idea, and without his support, it just isn't going to happen.

Anyway, the particular comment that pissed me off was concerning anti-virus filtering in our mail system. Basically, one of our network people was concerned that we might get encrypted viruses, and because they're encrypted, the anti-virus filters wouldn't be able to catch them. As support, this person cited a virus report from earlier this week about a "new" virus that works by enclosing the executable in a password-protected ZIP archive with the password in the message body. This "encryption" stops the virus filter from catching it, but if we just blocked all encrypted files, we wouldn't have to worry about it.

Now, to me, this entire argument sounds like complete bull. Of course, I'm not an expert on e-mail, network security, or encryption, so I could be wrong. If that's the case, somebody please correct me.  But the more I think about it, the more I feel like this is one of those arguments that's just barely true enough that you can make it with a straight face, and yet still have it be completely misleading.

First, I certainly never suggested that we allow any old encrypted file through the mail filters. Just because a file is encrypted doesn't automatically make it trustworthy. An attacker could certainly find a freeware symetric encryption utility from FooBaz Questionable Software, LLC, use it to encrypt a virus, and send it to everyone in the world along with the decryption key and instructions on how to get the naked pictures of Pamela Anderson out of the encrypted file. That would just be a variation on the password-protected ZIP file trick.

My choice would be to use a standard public-key system, like PGP or GnuPG. If you stick to allowing just encrypted messages in that format, then the "virus problem" goes away.  After all, the whole point of public-key cryptosystems is that the recipient of the encrypted file already has the decryption key before the file is even encrypted. Hell, the recipient is the one who generates both of the keys. To send public-key encrypted mass-mail, you'd have to encrypt the malicious attachment separately for each recipient. And since many recipients won't have a key pair, or the attacker won't the recipient's public key, the target audience is dramatically cut at the outset.

Plus you can have accountability in public-key cryptosystems.  After all, that's what digital signatures are for - so you can know who sent a message. If you're really paranoid, you could only accept encrypted attachments from messages signed by someone you trust.

Of course, nothing about public-key cryptography can prevent a someone with your public key from intentionally sending you a virus. And that's where the "just true enough" part comes in. Yes, an encrypted virus sent by a malicious attacker trusted by user won't be detected by the mail filters. Is this a problem? Well, if you have to do business with people you can't trust, then I guess so. But if you don't publish your public keys and don't do business with 13 year old script-kiddies, I don't see it as a big concern.  Besides, this is negated by the other anti-encryption argument this person has been pushing: that the data we're dealing with isn't really important enough to bother with.

So let me get this straight: we can't do e-mail encryption because we're swapping unimportant data with untrustworthy people. My question is: then why are we even bothering? We ought to just lock the doors and start browsing the want ads!

Sigh.... I'm done blowing off steam now. Time to start working on my CV.

Today's Daily Grind had a link to Tim Haines's thoughts on why he doesn't use Live.com as his home page. I've only looked briefly at Live.com, but I have to say I tend to agree with Tim. However, my real question is, do people actually still use home pages?

I haven't really used home pages in quite a while. My homepage on my home PC is still set to Yahoo! mail, despite the fact that I bought a domain two years ago and now only check it once every couple of weeks. At work, I never even bothered to set a home page. You know why? Because I'm an Opera user, and I configured by browser to always restore my last session. So, in other words, I never actually see my home page. I just open up my browser and immediately pick up where I left off last time.

Apparently, from Tim's informal poll, there are a fair percentage of people who have their browser start on a search page, like Google. I don't know why, though. Opera and Firefox both support configurable inline searches in the address bar, after all. When I want to search Google, I just open a new tab and type "g search term" to run my search. I also set up "define word" and "wiki search term" to search Dictionary.com and Wikipedia in the same way. It's so convenient I honestly don't know how anyone can go back to actually visiting the main search page after discovering this.  

The other day, I really noticed something for the first time. I realized the complete and total suckiness of synchronous mounting.

As you may or may not be aware, I use autofs to access my removable media, including my USB flash drives. I also have my autofs rules set up to mount volumes synchronously, so that when the writing is done, I can just yank the drive out, secure in the knowledge that there won't be any pending writes.

Well, earlier this week, I downloaded a couple of MP3s and decided to take them to work with me. They only totaled about 20MB, so I decided to put them on my JumpDrive. Putting this much data on my JumpDrive at once is actually pretty rare for me. Since the drive is only 32MB, I usually only put small files (< 1MB) on it and transport things like MP3s on CDs.

Well, when I started copying the MP3s, I noticed that the process was going slow. Unreasonably slow. I'm talking about transfer rates of 32KB per second, with an estimated completion time of 20 minutes. Now, I don't know much about USB flash drives, but I knew this couldn't possibly be right.

Well, a little experimenting revealed that the synchronous writes were the problem. When I mount a volume asynchronously, there is no problem at all and the write finishes in less than a minute. What's the deal with that? It's one thing to have a performance penalty associated with synchronous writing, but I'm just floored by the sheer magnitude of that penalty. I mean, 32KB per second?!? I can get faster transfer rates over a network! Why are the writes so slow? There must be some rational explanation for this.

I'm a little pissed off at BitPim right now. I'm also pissed off at whatever idiot decided it would be a good idea to emulate MacOS.

I just finished using BitPim to get some pictures off my cell phone. Now, for whatever reason, BitPim saves everything is downloads in its own data folder. Apparently it wants to be a one-stop PIM suite that integrates with your cell phone. Whatever.... I think that's a complete waste of time, but I don't know of any other program that runs on Linux and lets me access my phone, so I use it BitPim anyway.

Anyway, today I was stupid enough to try to use the "save" feature for pictures to move them out of the BitPim folder and put them with the rest of my images. I have a folder of other images from my camera, so I figured I'd put them there. However, before I clicked the "OK" button in the save dialog, I realized that I don't completely trust BitPim and decided I'd better check if that folder already had files with the same names as the ones I was saving. It did, so I tried to click the "cancel" button. The only problem was, some moron decided it would be a good idea to put the "OK" button on the right, where the cancel button normally is. By the time I realized this, I'd already clicked "OK" and BitPim happily overwrote my existing files without so much as a warning.

Now, I'm not really too upset about the lost images. They weren't really that important to me. In fact, I don't even remember what they were of. I'm just irritated because the loss was so easily avoidable. For one thing, it's not really that hard to check if a file path already exists. Just a confirmation box would have been enough. In fact, a confirmation box was the least they could do, since I was "saving" multiple files, which means the file picker only showed directories, so I couldn't actually see if there were any files already in the target path.

The second thing, i.e. the position of the OK button, really gets on my nerves. I mean, I knew I wanted to cancel the operation, so I did what I always do: I immediately went for the right-most button. It was completely automatic; I didn't even realize what I was doing until it was already too late. After all, I work at a computer every day, using Windows at work and KDE at home, and the right-most button is alwasy - always - the "cancel" or "no" button. Why would they switch it to the other side? Are they stupid or something?

Most likely, somebody thought it would be a good idea to emulate the MacOS style, kind of like the GNOME people do. And, as Tog pointed out, putting the "OK" button on the right is the more natural design decision. However, Tog also admits that it just doesn't matter anymore. Once Windows hit 90% desktop market share, such small design choices, even if they were correct before, became wrong. In a perfect world, where everyone came to your software with no bad habits, then maybe putting "OK" on the right would be the right choice. But we don't live in a perfect world, and if your software is cross platform, then the odds are that most of the people who use it will be used to Windows. And for any user who is used to Windows, putting "OK" on the right is unequicially wrong.

I guess I'm just sick of developers, especially in the free software world, doing things differently just for the sake of being different. Or perhaps I should say for the sake of not being like Microsoft. There's plenty of room to do new and innovative things without worrying about petty little details like button position. It's not doing your users any favors. This is especially true for people like me, who use Windows at work and Linux (or whatever your favorite OS is) at home. And since I can't choose not to use Windows, guess which piece of software I'll be looking to dump....

You know, years ago I thought the disconnect in user experience didn't matter. I figured that the people who complained about moving back and forth between Windows and Linux were just lamers looking for something to complain about. After a while, though, it really does start to grate on you. It's not an immediately fatal problem, but more like an itch that you can't quite reach. It's not a big deal at first, but it doesn't go away slowly builds up to the point where it drives you insane. That's part of the reason that I use KDE these days - because it can easily be configured to work like Windows.

I got a great new toy yesterday. It's a $13 cable. But not just any cable - it's a data cable for my cell phone. Combine it with BitPIM and you have the solution to a problem that's been bothering me for a while now.

You see, my wife and I both have Verizon cellular phones. If you don't already know, the deal is that you sign a two-year service contract with Verizon and they give you an obscenely huge discount on any phone that you buy at the same time. I'm talking like 50% or more off the normal retail price. And sometimes they even have brands that are buy one, get one free on top of the discount. So unless you're filthy rich, you can't afford not to get a new phone every time you renew your contract. This is kind of a win-win situation, since you get new hardware at a good price and Verizon gets to keep down the amount of old hardware on their network.

The only thing that bugs me about this is that every time you get a new phone, you have to sit down for an hour or so and enter all your contacts into the new phone. Sure, if you buy the expensive phone they'll give you a data kit that can sync your calendar and contacts with Outlook, but as a Linux user that doesn't do me much good.

Another annoyance surfaced a year ago when I got our current phones. This time I got camera phones - a pair of Samsung SCH-A670 camera phones, to be precise. I actually only got them because they weren't much more expensive than the non-camera model (like $20), but it turns out that having the camera is kind of handy sometimes. The only problem is getting the pictures off the camera and onto the computer. Sure, you can sign up for Verizon's picture messaging service which has some kind of web interface, but that adds something like $10 to your monthly payment. You can also send the pictures by e-mail, but then Verizon charges you $0.25 per message. That seems pretty silly when all I want to do is move the pictures the three feet from my camera to my hard drive. I ought to be able to do that for free. And it turns out I can.

A couple of weeks ago, I saw a comment on digg.com that made reference to BitPIM. It's basically a PIM application that's designed to sync with cell phones. It can sync calendar and contacts, download pictures, ring tones, text messages, and various other things. There's even a feature that lets you browse the phone's file system. And the best part? It's written in Python with wxWindows, so it works on Windows, Linux, and MacOS.

(Note: There is no Ubuntu package for BitPIM, so I had to download the RPM and convert it with alien. The only problem I had was an error about being unable to find libtiff.so.3. The correct way to fix this is probably to install that version of libtiff. However, I used the "cheap symlink" fix, which was to create a symlink named libtiff.so.3 that pointed to the version that's actually on my system, which was libtiff.so.4. It worked for me, but your mileage may vary.)

So now I can get all the data on my phone onto my PC. BitPIM even has an option to import and export vCards and other formats, so it's possible to sync my phone with Kontact, even if it is in a round-about way. Now all I have to do is find out if there's any way to make KDE talk directly to my phone. The ability to sync with KAddressBook and KOrganizer would be great. An ioslave to access the phone's filesystem would be great too. I think I feel a new project coming on....

I really need to upgrade my system so that I can try different web browsers. I've been using the Linux version of Opera for several years now because it was the only decent browser that ran well on my aging AMD K6, but it's really been starting to piss me off lately.

The thing is, I really like Opera, and have for a while. I liked it so much that I actually paid for a license to version 6.0 and then paid again to upgrade to 7.0. It's very fast and has a pretty good user interface.

The problem is that it doesn't feel like Opera is making progress. There are lots of little things that are missing or haven't worked right for a long time. Allow me to give a few examples.

• It took them until version 7.60beta to get printing right. Then, in the 8.0beta releases, I started having trouble again.
• There is no configuration interface to the address bar search functionality.
• There is no user interface for configuring the menu bar, although it can be done using the INI files.
• There is no single font changing, like the "larger" or "smaller" menu items in IE or Firefox. There's a setting for the minimum font size, individual settings for each text element, and a "zoom" feature, which scales the entire page. I find this somewhat annoying.
• There is no ad blocking functionality and Opera won't add one because, apparently, they feel it's hipocritical to have an ad blocker when they have banner ads in the freeware version. Of course, as a paying customer, I frankly don't give a damn how they feel about it. I want an ad blocker!

There have, of course, been many "revolutionary" new additions to Opera since I started using it. I even use some of the small ones. However, the big things are the "web suite" features, like the e-mail client (which I don't like), the RSS aggregator, and the chat application. These are all well and good, but I don't want a web suite - I want a web browser. I already have an e-mail client and an RSS reader - KMail and Akregator, and they're part of a PIM suite that integrates with my entire desktop. The only reason I don't use the web browser that goes with them is that it's slower than molasses on a cold day in January.

The problem I just had, though, was the last straw. I couldn't log into my Newegg account in Opera. After putting in my user ID and password, I got to the "processing login" page and was never redirected. I don't know why. Maybe it's related to the cache problem I had a few minutes before, where Opera showed me an old version of my shopping cart. (I've had cache mishaps in Opera for years, although they don't happen too often.)

I think it's just time for a change. Opera is nice, and there are a lot of features in it that I like, but it feels like it just isn't getting any better. Or, rather, it's not getting better in a way that matters to me. And with Firefox and it's massive number of useful extensions (I really miss the web developer toolbar when I'm using Opera), Opera just doesn't feel like as obvious a decision as it used to.

I always feel a little sad when I start moving away from a piece of software that I've loved for a long time, but it's always proved to be for the best. I suppose it will probably be the same way with Opera. Of course, I have to upgrade my system before it's feasible to switch, so maybe Opera 8.0 final (or even 8.1) will be good enough to keep me a loyal user. But I'm not going to hold my breath. On the up side, if I do drop Opera, I'll be moving to Firefox or Konqueror, both of which are old friends, so it won't be as jarring a change as usual.

Emulation is great. No need to dual-boot or go out and find a replacement for hardware that broke down years ago. Just fire up an emulator and you've got a whole different platform running in a convenient little window on your desktop.

Although video game emulators like ZSNES and MAME are nice, my current favorites are DOSBox and QEMU. DOSBox is a DOS emulator, kind of like DOSEMU, but more focused on games. I currently use it for exactly one thing - playing Bandit Kings of Ancient China, one of Koei's classic strategy games. This was literally the first PC game I ever bought. I got it and a copy of Tetris from a game store back around 1991 or 1992. I've since lost the copy of Tetris (which was the only program I ever bought that came with a 5.25" floppy disk), but I jealously guarded the Bandit Kings disks for years. They finally went bad a few years ago, but fortunately I had a backup of the game files, so I can still play my original copy. In fact, I even still have the instruction manual and poster that came with it. For some reason, I just never get tired of that game.

QEMU is cool for an entirely different reason. It's an open-source "process emulator" that supports full system emulation. In other words, you can use it kind of like VMWare, meaning you can install an old copy of Windows onto a file on your hard drive. This is a very handy thing to be able to do. For one thing, it allows you to use all that old Windows-only software you probably have lying around. It's also very nice for testing purposes if you're a web developer, since you will almost definitely want to test your pages in Internet Explorer. And after all, Wine is a big pain in the neck to install and configure, while installing a virtual copy of Windows is pretty simple. Also, you can have multiple QEMU disk images with different versions of Windows and Internet Explorer. I don't think you can do that with Wine.

The only problem with these programs is that they require more CPU power than I really have. My trusty 500MHz AMD K6 has served me well, but it can't really handle heavy emulation. However, QEMU does have an "accelerator" kernel module which is supposed to speed things up considerably. Unfortunately, the accelerator module is not free software and is only available in binary form. Plus it only works with the current CVS, not the any of the stable releases. Oh well. I'm compiling the CVS source now, so we'll see how it works. Hopefully I won't need it once I finally get around to upgrading my system.

You know what I need to write an article about? Automount. It's a great feature, it's very useful in these days of removable USB drives, and yet nobody seems to know about it.

If you don't know about automount, it's simply a feature whereby you can configure devices to be mounted on demand. So if, for example, you have an NFS share that you only access every once in a while, you can set up automount to handle it and then every time you access the mount point, if the share isn't mount, the system will mount it for you. And after the share is unused for a certain amount of time, it gets unmounted automatically. This magic happens through the combination of the kernel level autofs filesystem and the userland automount daemon.

This is really great for USB thumb drives. You can set it up so that you just stick the drive in a USB port, open it up in a file manager, do your stuff, close the file manager, and after two seconds the drive is cleanly unmounted and you just yank it out. Of course, there can be some complications. For one thing, this doesn't work if you're using a 2.4 series kernel and have multiple USB storage devices. To make this really work, you need udev, which is only supported on 2.6 kernels. That's because USB storage device names are not static - the first device you plug in gets the first available device name. Udev, however, lets the kernel assign static names to particular devices based on things like device serial numbers. Of course, it might be possible to fake this out in a 2.4 kernel by using hotplug and sniffing the output of dmesg, but that's not the easiest or most pleasant way to do things.

I wonder why it is that automount isn't more widely used by Linux distributors? Is it just because they like to handle things on the desktop level rather than the system level? I've heard that many distributions do things like automatically creating icons on the desktop when you plug in a USB stick, but, as a Slackware user, I've never actually seen it. I guess the thinking is that by using D-BUS or hotplug or whatever they use to do that, they can achieve greater flexibility than simply mounting. Maybe they see things like the Windows XP device dialog box as a goal. Although I'm not actually too fond of that feature of XP, I guess it might be useful to people new to computers.