LinLog

F# on Mono

pageer@skepticats.com (Peter Geer)

I learned something cool yesterday - F# runs on Mono! That means I can mess with it without having to use VMware!

I got interested in F# from hearing about it on Hanselmintes and .NET Rocks. For those who haven't heard of it, F# is a functional programming language, similar to OCaml, built on .NET. It's actually not an "official Microsoft product," but rather a project out of Microsoft Research, which is pretty cool.

Incidentally, working Microsoft Research is on my list dream jobs. I mean, how many organizations can boast of having had two Turing Award winners on staff? How could you not want to work someplace like that?

Anyway, the thing that excites me about F# is the combination of functional programming and its status as a first-class .NET language. I've been meaning to get up to speed on functional programming for a few years now, but I've just never gotten around to it. Learning LISP or ML always seemed on par with refreshing my Prolog and Ada skills - an interesting exercise, but not profitable in terms of marketability. I mean, how many Standard ML listing have you ever seen on Monster?

However, it looks like functional programming may start pushing more into the main stream. The current trend in hardware is that CPU speeds are flattening out and performance gains are being made by adding more processors or cores. However, most code today is not written to use more than one core/proc at a time. So we're going to have to start parallelizing our code to fully take advantage of the hardware. That's where functional programming comes in. Pure functions, by definition, have no side effects. So if you're writing pure functional programs, parallelism becomes much easier, as you have no worries about thread safety and whatnot.

So with F# I can now learn functional programming while using .NET. This means that I can leverage some of my existing knowledge while learning the new language, which always makes things go faster and smoother. It also means that this learning experience has some vague marketability, i.e. I can at least count it as .NET experience. In other words, it's not one of those "off in left-field" learning ventures like if I took up Intercal or APL. I'm not going to feel (as much) like I could be making better use of my time.

Anyway, it turns out that installing F# on Ubuntu 8.04 wasn't quite as painless as I had hoped. On the up side, the F# site does supply a ZIP archive with generic Mono-compatibile binaries and full source (under the MS shared-source license). However, it seems the binaries don't quite work right with Mono 1.2.6. That's fixable, though, thanks to Laurent Le Brun's article on using F$ 1.9.4.17 on Mono. Basically, the important thing is to remember to pass mono the --runtime=v2.0.50727 option when running the F# compiler or F# binaries.

I haven't been blogging much lately, but hopefully I'll be posting back in the coming months with tid-bits on F#. It's been a while since I tried to learn a new language, especially a non-procedural one, so I'm looking forward to it.

MIME type trivia

pageer@skepticats.com (Peter Geer)

Random bit of trivia: apparently the correct MIME type for JavaScript files is application/x-javascript.

We actually discovered this by accident at work this morning. Edwin was trying to figure out why Apache wasn't compressing a new JS file he'd added to the site and discovered that the file was being served as application/x-javascript instead of the expected text/javascript. This raised the question: "What the hell is wrong with Apache?" A little Googling then revealed the answer: "Oh. there's nothing wrong with Apache. It's just us."

So far, all the general MIME type references I've checked list JavaScript as application/x-javascript. Who knew? I've been using text/javascript for years. So has Edwin. Heck, it's even in the examples in the HTML 4.01 spec. I just figured that was the "official" type. Apparently I was wrong. Guess I'll have to have Lumberg send me another copy of that memo.

Account for cache

pageer@skepticats.com (Peter Geer)

Here's another interesting debugging tid-bit for you. Assuming, that is, that anyone but me cares about such things.

This past Friday, we rolled out a new feature of the site. Without giving away any identifying details, it involved a change to the way data related to user activity is tracked in one of my modules. In the past, we tracked this "user activity data" (UAD) via a batch calculation. Every night we ran a cron job that analyzed various database tables and calculated some results that we then stored in the database. It was a fairly intenxive calculation that touched a lot of tables, so it didn't make sense to do it in real time.

For this new site feature, I had to switch the module over to an event-driven approach. That is, rather than the batch calculation, we're now keeping a "running tally" of the UAD based on user actions. This means that the users can now see this information change in real time, which is good, but it also means it's now hard/impossible to determine exactly how a particular user arrived at their current UAD.

So here's the catch: for reasons I won't get into, I need to track the per-day UAD change. I did this by simply adding a "daily change" column in the database that I update along with the UAD score. Then, every night, we run a cron job to zero-out that column in the database. The SQL query it runs is simply:
update uad_table set daily_change = 0;
For non-coders, that just sets the daily_change to 0 for every row, i.e. every user, in the database. Nice and simple. For a query like that, there isn't much testing you can do - it either works or it doesn't.

With that in mind, I was quite surprised this morning to learn that this simple query was apparently not working for all users. While the daily change column was indeed cleared for some users, it was not cleared for some others. But, for SQL above, that is absolutely impossible. The only way that could happen is if MySQL was badly, badly broken.

But, as with all things in software, the devil is in the details. I failed to account for two things in this equation: my database update code and caching.

First, my update code. The relevant portion of my PHP update code looked something like this:
$this->uad += $adtl_uad; $this->daily_change += $adtl_uad; $this->save();
In this, the save() method writes the object properties to the database. This is pretty standard stuff. The problem is: what happens if the database row changes between the time the object fetches it and the time it writes the updates? The answer: the change will be clobbered and the object will write bad data. So if the user is doing something that changes his UAD and if things are timed just right, we could end up in a situation where this code reads a "daily change" value of 500, our cron job updates the database to make it zero, and then the update code writes back a "daily change" value of 500 plus something, thus clobbering our update. However, this alone could not be the problem because the time between reading the data and writing it is so small - way less than a second. The timing would have to be just right, and the odds of that happening often enough to account for the number of un-updated users are just too small.

No, the real problem was when we combined this update code with cachine. You see, we use memcached to cache the results of database queries in order to take some of the load off of our database servers and increase performance. I was using memcached for the UAD retreival, so whenever I retreived a user's UAD from the database, I would first check if we had a memcached copy of it. If so, I'd use that and bypass the database altogether. If not, then I'd query the database and then save the results in memcached. When I wrote the updated UAD back to the database, I would clear the memcached data for that record so that it would be refreshed on the next access.

If you're sharp, you may have picked up on the problem now. Where does the cron job clear memcached? It doesn't. So what happens if I run the cron job and then someone accesses a user UAD record for which there is still memcached data? Well, the PHP script is going to get the cached data that still has a non-zero "daily change" value, update that value, and write it back to the database, effectively canceling out the column zero-out for that record. Doh!

My solution was simply to change the PHP method that updates the UAD. Rather than calculating the UAD in code and writing the result back to the database, I now just let the database do the calculation. The parameterized SQL looks something like this:
update uad_table set uad = uad + :adtl_uad, daily_change = daily_change + :adtl_uad where uid = :uid

The moral of the story is two-fold. First, scale changes everything. On a smaller site that didn't need memcached, the race condition in the update code probably never would have been an issue.

Second, keep your database access consistent. The reason this problem popped up is because I was handling data through cache-aware domain objects in the web application, but going straight to the database in the cache-ignorant cron job. Ideally, I would have used the same update method in both the cron job and the web application. The only problem with that is that running one bulk update query in the cron job is a lot faster than iterating over 400,000 rows of user records to run a PHP method that has the same effect. Which is yet another example of how little in software is ever as simple as it seems.

Remember the magic properties

pageer@skepticats.com (Peter Geer)

You know what's a good way to feel really beaten-down at the end of the day? Spend 2 hours debugging a 1-line problem.

That's just what happened to me the other day. And, while I place some of the blame on PHP in order to make myself feel better, it was actually due to my own forgetfulness and tunnel-vision. If I had just taken a minute to step back and consider the possibilities, I probably would have had it figured out much faster.

Since last week, I have been working on the data access code for our site. Basically, we're going from using ad hoc SQL queries scattered throughout the codebase to a model-based design, where we have an abstract data model class which handles database access and then classes that inherit from that. You know, sort of an ActiveRecord pattern, a la Ruby on Rails or CakePHP.

Anyway, things were going along quite nicely. That kind of code conversion is always a little tedious, but I hadn't hit any major road-blocks and all my unit tests were passing.

Well, Tuesday afternoon, I hit a snag. I'm not sure at what point it happened, but I ran my unit tests, and suddenly I was seeing red. Lots of red. Everything was failing!

Well, I immediately when into homing mode. I started taking out tests and trying to isolate a single failure. It turns out that the first failure was in the test case for my class constructor. A little more debugging revealed the problem - none of the class members were being populated from the database. And yet, there was nothing in the error logs and the connection to the database wasn't failing.

Now, the way our (custom) abstract model class works is that it reads the table layout from the database, stores that in the object, uses it to determine the primary key of the table, and then retrieves a row based on the primary key value you pass. What was happening was that the constructor was running, getting the table data from the database (which is how I know the connection was functioning), and running the code to retrieve the row, but the values of the class members were never set.

Well, needless to say, I was baffled. Before too long, I thought I had narrowed it down to the method that actually retrieves the database row. We're using PDO (PHP Data Objects) for database access in the model class and I was using the feature of PDO whereby it will return a row from the database as or into an object. You can pass it an object and it will set each column as a field in the object. Alternatively, you can pass PDO a class name it it will return a new object of that class with the appropriate fields set from the database. You can also have it return an stdClass, which is PHP's generic object. I thought I was onto something when I discovered that fetching into the current object or fetching into a new object of the same class didn't work, but fetchin a generic stdClass did. So, of course, I eventually decided to work around the problem by re-coding the method to use the stdClass and just populate my object from that. Imagine my surprise when, even after that, and verifying that I was getting the row from the database, my unit tests still failed.

Well, to cut a long and boring story short, the problem was the magic __set() method. This is one of the "magic" functions PHP lets you define on a class. This particular one takes a key and a value and is executed every time you try to assign a value to a class property that doesn't exist. Well, we had defined that in the child class, not the model class, to do an update on the database in certain conditions and when I was re-writing it, I forgot to account for the default case - when you we don't want to update the DB, just assign the value to the key. It was one line, $this->$key = $value, which I had put inside an "if" block instead of out. So every time my code tried to do something like $this->foo = 'bar', the statement ended up being a no-op. Same thing when PDO tried to populate the object. But populating the field with the table layout worked fine, since that field was set in the class definition rather than being created at run-time.

So what's the moral of this story? Well, I guess there are several. First, tunnel-vision in debugging is a bad thing. I got hung up on the PDO object fetching and wasted lots of time trying to figure out why it wasn't working when the problem was actually someplace completely different.

The second lesson is that you need to stick to a process - in this case, I should have been doing real test-driven development. The reason this took so long is that I let far too much time elapse between the change to the __set() function and running my unit tests. If I had been doing real TDD and had run my tests after every change, I would have instantly been able to pinpoint the problem.

The last lesson is simply that software development is about managing complexity, and managing complexity is hard. Hacking away, designing as you go, relying on raw brain-power is all well and good, but it's not sustainable in the long run. When you run out of gas, or you're having a bad day, you need something to fall back on. This episode reminded me of that the hard way.

This seems to be a recurring theme for me. The longer I develop software, the more obvious it becomes that I need to imrpove my process - even whey I'm working by myself. It's kind of funny, really. The more I learn about software development, the less I feel like I know what I'm doing. There's just so much I have left to learn, it seems a little overwhelming. I know that doesn't make me inadequate - in fact, they say wisdom is knowing what you don't know. But it can still be a bit...disquieting at times.

When null is not null

pageer@skepticats.com (Peter Geer)

What follows is the saga of a bug that took about 2 hours to pinpoint and 5 minutes to fix. It reminds me of one reason I some find dynamically typed languages annoying: implicit casting.

Here's the scenario: The manager tells me he noticed some weird output on one web page. Specifically, this one particular field, which should be empty, is showing the word "null." I own this field, so it's my job to fix it.

Naturally, I start by looking at the code. The site is written in PHP, so I figured that the field is null in the database and PHP is just printing out the word instead of the empty string, as I'd expect. But that's not it. There's actually already a check for null in the data retrieval code. So, just to check, I look at the relevant row in the database. And guess what - the field in question has the value "null." And just to be clear, I don't mean the field is null, I mean it contains "null" - as a string.

Now the plot thickens. This row is user-submitted content. This particular field has a default value of NULL (that's actual, database null) and users need special permissions in the application to add or edit it. The user in question does not have these. Another check of the database reveals that there are a number of similar records, submitted by users without the proper permissions, that have the string "null" in this field. However, it's not all of them, which means it's not related to the default field value. So something is weird.

Off the top of my head, I knew that there are only two places in the code where a user can set that field. One was the main submission form, the other was an AJAX-based update page. A quick check of the database revealed that the updated timestamps on the affected records were empty, so naturally I concentrated on the submission form. Of course, this turned out to be the wrong decision, as the submission form was correct. It turns out that the updated timestamps for user-submitted records never actually got updated.

So I checked out my other possibility - the AJAX update page. It turns out that the server-side code was correct. The null was actually coming from the JavaScript!

When I added the problem field to this page, I made one bad mistake. You see, since only privileged users can edit the field in question, I decided that the server-side code would simply not put it on the page for non-privileged users. That's fine. However, I failed to account for the dynamically-created edit boxes. You see, because this page is actually a listing page with an edit feature, we were actually dumping the database field values into DIVs and then replacing these DIVs with input boxes when the user clicked the "edit" link. We then took the values of those boxes and sent an HTTP request to the server.

Well, if you're paying attention, you may see what happened. I wasn't careful with my JavaScript. When I added the ability to edit this one field, I didn't check if it actually existed in the page. I just used our JavaScript framework to grab the value and then dump it into the POST data I was sending to the server. Well, if the DIV for that field didn't exist, then the value I got back was null. And when I put that in the POST data, it was type-cast to the string "null." So, when the server-side page got the data, it saw that string, not the empty object value, and dutifully stored it in the database.

The moral of the story: when working in a weakly-typed language, you still have to be careful about your variable types. In fact, you have to be more careful. Because the compiler/interpreter won't help you find problems and you can't always count on implicit conversions being done the way you mean. Especially in JavaScript.

PHP IDE mini-review

pageer@skepticats.com (Peter Geer)

Tomorrow marks my 2-month anniversary at my new job doing LAMP. And for most of that two months, I've been going back and forth on what editor or IDE to use.

My requirements for a PHP IDE are, I think, not unreasonable. In addition to syntax highlighting (which should be a given for any code editor), I need to following:

Support for editing remote files over SSH. This is non-negotiable.
A PHP parser, preferably with intellisense and code completion.
A file tree browser that supports SSH.
Syntax highlighting, and perferably parsers, for (X)HTML and JavaScript.
Search and replace that supports regular expressions.
Support for an ad hoc, per-file workflow. In other words, I don't want something that is extremely project-centric.
It should be free - preferably as-in-speech, but I'll take as-in-beer if it's really good.

So far, my preferred IDE has been Quanta Plus. It has all of the features I need and also integrates nicely with KDE. It also has a few other nice features, including context-sensitive help (once you install the documentation in the right place). However, the build of Quanta 3.5.6 that came with Kubuntu Feisty is kind of unstable. It crashes on me every few days, and for one project, I actually had to switch to something else because I was making heavy use of regex search and replace, which was consistently crashing Quanta. Also, while Quanta has a PHP parser with some intellisense, it's pretty weak and not in any way comparable to, say, Visual Studio.

My second heavier-weight choice is ActiveState's free KomodoEdit. This is a very nice, XUL-based editor. It's stongest feature is undoubtedly the PHP parser. It's really outstanding. For instance, it can scan pre-determined paths for PHP files and do intellisense for them. It even understands PHPDoc syntax and can add the documentation to the intellisense.

The down side is that, while Komodo does speak SFTP, the file browser tree only does local files. There is a Remote Drive Tree extension that adds this feature, but while it's better than nothing, it still isn't that good. I also don't much care for the look of Komodo or for the keyboard shortcuts. Those things are much easier to customize in Quanta.

After Quanta, my other old stand-by is jEdit. After installing the PHPParser, XML, and FTP plugins, this meets my needs. On the down side, the PHP parser doesn't do any intellisense (although it does detect syntax errors). The interface also feels a littly clunky at times, although it's much better than the average Java application and not really any worse than Quanta in that regard.

I took a brief look at a couple of Eclipse setups, but wasn't initially impressed by them. It might be worth looking at them again some time, but the whole process of getting and installing the appropriate plugins just seemed like a lot of trouble. Same goes for Vim. I'm sure I could get it to do most, if not all, of what I want, but it seems like an awful lot of trouble. And then, of course, there's the Zend IDE, which I don't really want to pay for. And besides, my one of my co-workers told me that, while it's a decent IDE, the real selling point is the integrated debugging and profiling, which won't work on our setup.

And so my intermitent search goes on. I'm hoping that the upgrade to Kubuntu Gutsy will fix the stability problems in Quanta, which is my biggest problem with it. I'm also hoping for some nice new features when KDE 4 comes along. But I guess I'll keep looking in the meantime.

Time matters too

pageer@skepticats.com (Peter Geer)

Debugging tip: When debugging code that is in any way time-sensitive, make sure that the time on the computer you're working on is correct.

This particular nugget-o-wisdom would have come in handy for me last week.

You see, I had recently been working on the advertising module for our site, integrating the old stand-alone ad system with our new CMS. I got a but report, so I made some changes to the code. But after I started testing my changes, I realized that orders for new ad campaigns were being accepted, but kept randomly disappearing. It made no sense. I would place a test order and it would go through. When I opened up a MySQL console, I could see the record in the database. But when I went into the administration portion of our CMS, it wasn't there. And when I checked the database directly after that, the record had just disappeared. Gone without a trace.

How could this happen?

The answer is simple: timing. When the administrative interface is loaded, it does some cleanup to the data base. In particular, it cleans out any "stale" orders that have been placed - ads that customers entered, but never paid for. So if an ad is left unpaid for more than a day, it will get deleted.

But the ads I was entering were no more than a couple of minutes old. So why were they getting deleted? The logic in the database-cleaning code was correct. The "stale record" time in the configuration table was correct. So the records shouldn't have been deleted.

The problem was, I forgot about two things:
1) None of the clocks on our test servers (which are VMWare Server virtual machines) are right.
2) The public portion of our site and the management portion run on different servers.

The short version is that I ordering ads on one server and trying to view them in the admin interface running on another server. However, the system clocks of those two servers weren't even remotely close - the content server where I created the orders was a day or so slow and the admin server was several days fast. So as soon as the admin server saw the newly created ads, it immediately thought they were "stale" and deleted them.

That particular "bug" actually came back and bit me again today. I guess I'm going to have to talk to the guys and see if we can set up the test servers to run a periodic ntpdate so that I don't have to worry about that sort of thing again.

Versions matter

pageer@skepticats.com (Peter Geer)

I ran into an interesting PHP bug at work the other day. Actually, it was a bug in PDO in a particular version of PHP.

Here's the situation. I needed to write a script to do some fairly extensive data massaging for a database integration. Since we have PHP installed on pretty much all of the production and development servers, I wrote it in PHP. It worked perfectly on the test server, so I figured it was safe to use it in production. And I was right, to a certain extent: it was safe in the sense that it didn't break anything - it's just that the script didn't work.

I won't get too much into the details of the script, as it's kind of long and not particularly interesting, but here's some sample code with the key points: while ($rec = $data_stmt->fetch()) { $db->beginTransaction(); $get_id_statement->execute(array($rec['something'])); $id = $get_id_statement->fetchColumn(); $update_statement->execute(array($somevar, $id)); $db->commit(); } The upshot here is that I was looping through the result set from a query and performing other select and update queries based on the current record. Simple enough, right?

The error message I received when I tried to run the script on our production database server was this:

PDO::prepare(): SQLSTATE[HY000]: General error: 2014 Cannot execute queries while other unbuffered queries are active. Consider using PDOStatement::fetchAll(). Alternatively, if your code is only ever going to run against mysql, you may enable query buffering by setting the PDO::MYSQL_ATTR_USE_BUFFERED_QUERY attribute.

The wierd thing about this is that, as I said, the script worked perfectly on the development server. And the development servers are supposed to be configured the same as the production servers. Of course, that obviously wasn't the case, so we ended up spending the next hour or so messing around with various code configurations with regard to transactions, prepared statements, and all that good stuff.

To make a long story short, it turned out that the explanation was simply that the development database server was running PHP 5.2.x and the production database server was running 5.1.x. The version on the production server still has an experimental version of PDO which, as it turns out, didn't work quite right. If I had just adjusted the IP address in the PDO connection string and run the script from our production staging server, which has PHP 5.2, it would have worked perfectly the first time. Unfortunately, I didn't discover that until I had already wasted an hour bug-hunting and messing with the script until I got it to work on PHP 5.1 (by moving all the fetch and fetchColumn method calls to fetchAll). Live and learn.

Insane library design

pageer@skepticats.com (Peter Geer)

My current project at work is to integrate a web application that runs one aspect of our site with the rest of the site. Part of this involves converting the database code from a custom DB abstraction library to use the standard PHP Data Objects (PDO) library that comes with PHP 5.1. I had never used it before, but it turns out PDO isn't a bad library. It has an object-oriented API, supports multiple databases, "prepared" (i.e. DBMS parametrized) queries, transactions, and so forth. Pretty much the basic features you need in a database abstraction layer. The only problem is that the error handling API is somewhere between "weird" and "completely frickin' insane."

You see, PDO has "configurable" error handling. That is, when a method reaches an error condition, it will either return an error code or throw an exception. Which one it does is determined by an attribute setting on the PDO connection object. (Technically, it may actually return the error code and throw an exception, but the return value is irrelevant once an exception is thrown, so it doesn't really make any difference.)

Let me repeat that, in case you missed it: PDO will either throw an exception or return an error code, depending on a setting of the PDO connection object. So it's possible to change one single constant and break error handling for an entire script.

Upon learning this, my initial reaction was: "That's the stupidest thing I've ever heard in my life!" I mean, honestly, who ever heard of configurable error handling? As if programmers don't have enough to worry about with database code, now we have to pick an error handling mode as well?!? And the worst part is that, when you read the error handling documentation, the default mode not only doesn't throw an exception, it doesn't even print a warning. It just leaves you to see the bad return value and manually inspect the error code on the object. You have to manually set it to turn on warning messages. Why?!? Isn't that why we have the error_reporting variable in php.ini? What the heck were they thinking?!?

What really kills me is that exceptions are optional, but object-oriented programming is mandatory. Unlike, say, the MySQL extension, PDO does not have a procedural version of the API, just the object-oriented one. And yet exceptions, which I think I can safely say are the object-oriented way of doing error handling, aren't even used by default. Shouldn't that be the only way of dealing with errors? Why is it better to make them optional? What's the rationale behind that?

So far, the only rationale I can think of is that the PHP community has what seems to be a disproportionate number of morons in it. That may be a little harsh, but it seems to me like there are a lot of third-rate PHP programmers and that the people who run the PHP project tend to go a little too far to cater to them. I can only assume that somebody thought that exceptions might be too hard for some PHP users, so they figured it would be better to use error codes by default. Kind of like how they figured securing your server was too hard, so they added the fake band-aid of safe_mode. Or how they figured a simple function call or array look-up was too hard, so the invented register_globals. Or how they figured that cleaning form input was too hard, so the invented "magic quotes."

PHP isn't a bad language, and I do enjoy working with it, but it has gone through a long list of really bone-headed design decisions. In some ways, it seems like it's succeeded in spite of itself. I guess this is really just evolution in action. Take a simple template engine, let it mutate into a full-blown programming language, and this is what you get: a platypus. Lots of endearing features and lots of brain-dead features, but it's still alive and still evolving.

The new job and new facts

pageer@skepticats.com (Peter Geer)

I started my new job yesterday. After a whopping 4 days of work, I'm actually feeling pretty good about it. I've been leaving the office feeling energized and enthused about my work - or at least not run-down (though the 1.5 hour commute fixes that). This is quite a change from my last job.

On Tuesday I started work on my first project. Of course, I really only got in a couple of hours of real work. Most of the day I spent with another programmer going over the database schema, application architecture, and all that good stuff. You know, the things you need to know to be able to sensibly make an addition to a substantial piece of software. I managed to get a bunch of work done Wednesday and today, though. It's really not that big of a project, it's just that I'm not used to the codebase and it's a somewhat large and complicated.

So far, this job is a lot more fast-paced than my last one. In fact, I've already learned a few things.

In PHP library files, you don't need to close the <?php ?> tag. You can just have the opening <?php and not have to worry about stray space at the end of the file. For some reason, I just never knew that.
Writing PHP will error reporting turned off really sucks. Well, technically it's not completely off. The errors are logged - to a 200MB file on another server. Which still sucks.
MySQL throws really weird errors when you try to declare a foreign key and give it the wrong size integer for the key column.
I had never used ti before, but the jQuery JavaScript library is very cool. For example, it allows you to select DOM nodes with XPath expressions. How cool is that? Though on the down side, the syntax is a little...weird.

I'm sure the learning is just beginning. Methinks this job will make for much better technical blog-fodder than the last one.