LinLog

The anti-agile

pageer@skepticats.com (Peter Geer)

I think I've put my finger on the reason that, despite being a rookie in it's practice, I'm so enamoured of Test-Driven Development. It's because TDD, despite being an "agile" programming practice, is the very inverse of agile methods. It's good, old-fashioned design. The only difference is that it's dressed up to look like programming.

At least that's what I get from reading Dave Astel's thoughts on TDD. He's an advocate of Behavior-Driven Development. This is an evolution of Test-Driven Development where the emphasis is shifted from testing units of code to describing the desired behaviour of the system. The idea is that rather than concentrate on writing a test for each class method, you figure out what the system should do and design your test cases around that rather than around the program structure. (Apologies if I mangled that explanation - I only learned about BDD today.)

This is somewhat similar to the other expansion of the initialism "TDD" that I discovered early in my research - Test-Driven Design. Note that the emphasis here is on using the unit tests as a design tool, i.e. specifying the behavior of the system. In other words, the unit tests are not written to do testing per se, but rather as a vehicle for expressing the low-level system specification.

Having studied formal development methods, it seems to me that, at a fundamental level, practitioners of TDD and formal methods are really doing pretty much the same thing. They're both trying to describe how a program is supposed to function. The TDD people do it by creating a test suite that asserts what the program should and shouldn't do. The formal methods people do it by writing Z specifications and SPARK verification conditions. Both are creating artifacts that describe how the system is supposed to work. The formal methods people do it with verifiable mathematical models, while the TDD people do it with executable test. They're just two different ways of trying to get to the same place.

In a way, this seems to be the opposite of what agile methods promote. Most of the agile propoganda I've read advocates doing away with specifications and documentaiton to the maximum extent possible. In fact, I've even read some statements that the amount of non-code output in a project should approach zero. And yet TDD can be viewed primarily as a design activity of the type that, using a different notation, would generate reams of paper. The only thing that saves it is the technicallity that the design language is also a programming language. It just seems a little ironic to me.

I suppose that's really the genius of TDD. It's the type of specification activity programmers are always told they should do, but repackaged into a code-based format that's palatable to them. It's sort of a compromise between traditional "heavy-weight" design methods and the more common "make it up as we go" design. I don't want to say it's the best of both worlds, but it's certainly better than the undisciplined, half-assed approach to design that's so common in our industry.

More on TDD

pageer@skepticats.com (Peter Geer)

I think I may have to break down and buy a book on test-driven development. Maybe Kent Beck's Test Driven Development By Example. I usually don't like to pay for programming books because they tend to be expensive, have low re-read value, and you can often get the information online. However, this is more a methodology topic than a technology topic, so that's not so true. Plus they don't have anything on TDD in the local Barnes & Noble, so I can't read it without shelling out. Rats!

I really just need to do some more reading and get a little more practice. There are still some aspects that I'm not quite getting. Then again, it's been less than a month since I started looking at TDD and I haven't had as much coding time as I would have liked.

The problem seems to be two-fold. The first is that I'm not yet certain how to write really good unit tests. Second, and the reason I was looking at Beck's book, is because I haven't seen any really good examples yet.

Most of the articles I've read suffer from the same pedagogical flaws as tutorials on object-oriented programming: they're too simple to be useful. An article doesn't provide enough space to work up to a good example. You either end up jumping right in on the deep end, which makes it useless for beginners, or you do something small, manageable, and not especially useful in real life.

Another problem I'm having is working with mock object frameworks like NMock. At this point I'm still working on where the line is between adding mock objects to my unit tests and refactoring things so that it's not an issue. I suspect this is because I have yet to internalize the line between unit and functional tests. Hence the need for a more extensive treatment.

It's only a matter of time and practice. Just my little experience with TDD so far make me feel that I've made significant progress as a developer. Which actually feels both good and bad. It's good to learn new techniques and improve. But on the other hand, I feel like kind of a schmuck for taking so long to get to the party. But at least I'm not the last one....

On motivation and brain rust

pageer@skepticats.com (Peter Geer)

Things at work have been slow lately. Really slow. The fact is we're severely over-staffed and there just isn't enough work to go around right now. This has left me with an amount of down-time that I find...uncomfortable.

You'd think this might be a good position to be in. After all, it leaves me plenty of time to read up on development practices, the latest technologies, and keep up on my tech blogs. I have spare time to experiment with Ruby on Rails and play with Mono. I can browse through Code Complete or Refactoring at my leisure. What more could a programmer want?

As I've been discovering, this isn't quite as nice in practice as it seemed at first. For starters, there are still the miscellaneous menial tasks that need to be done - software installations, security and configuration changes, and just general support calls. These are excellent for knocking you out of the zone. Furthermore, the constant threat of them makes it a bit harder to concentrate in the first place.

Second, while reading and experimenting to build your skill-set is great, you need the right environment to make it truly profitable. You need some degree of freedom and some goal to work towards. Or at least I do. I find I learn the most when I have something I'm trying to accomplish. I also need periodic breaks to process and assimilate what I'm studying. It just doesn't seem to work as well when my "goal" is to stave off boredom and my breaks are scheduled.

Last, I've found that it's becoming a challenge just to stay sharp at times like this. You see, I need some sense of purpose to stay sharp. I feel like I'm chained in a room for 8 hours a day being forced to do nothing but pour water back and forth from one bucket into another. It feels like my brain is starting to rust.

This is exactly the opposite of how to motivate programmers. We need crave interesting progblems to solve. Or, at the very least, some problems to solve. Playing the point-and-click Windows monkey won't do it and I can only stand to read so many hours a day.

The problem is, the more I try to spend my time improving my knowledge and skills, the more unbearable I find my condition. I feel like I should be "out there" putting what I learn to use, but instead I have to sit at my desk for...no particular purpose. And what's worse, it's sapping my mental and emotional energy. After being stuck in the office all day, trying to keep from going stir-crazy, I feel like I've got nothing left when I get home. It's turning into a vicious cycle.

Have other people been in a situation like this? How do you deal with it? I mean, short of quitting (which I'd do if not for that damn mortgage). Are their any coping strategies to hold me over until I can get out?

Tasting the Rails kool-aid

pageer@skepticats.com (Peter Geer)

This week I started dabbling in Ruby on Rails. After reading Scott Hanselman's and Mike Gunderloy's coverage of RailsConf, I almost felt like I had to. So far, I'm impressed.

For anyone who hasn't been paying attention, Ruby on Rails is the current "big thing" in developing web applications. Ruby is a dynamic, object-oriented programming language from Japan. So far, it seems to read like the bastard child of Python and Perl. But in a good way, if that's possible. Rails is an MVC web application framework for Ruby. Through the magic of Rails, you can literally create a working, if extremely basic, database application in less than 10 minutes. No joke.

I haven't had much time to play with RoR yet, but it seems really, really nice. It basically takes all the pain out of building data-driven web applications. Rather than spend your time messing around with database access - buidling the plumbing, designing insert/edit screens, and so forth - rails lets you automatically generate all that stuff. For example, in the tutorials, they just have you create a new Rails application, create a new MySQL database and table, and run a couple of commands. Then you can fire up a web browser and add records to the database. All without writing a single line of code!

The thing that's really nice about RoR is that it encourages good system design. The default Rails application has a proper model/view/controller architecture, includes testing infrastructure, and other good stuff. In other words, it gives you a good starting point. Contrast this with Visual Basic. Bask in the VB 6 days you used to see tutorials on data-bound controls that showed you how to build database applications without writing a single line of code. All you had to do was drag and drop a bunch of controls in the VB designer and voilà - instant application. The only problem was that VB 6 seemed to actively encourage writing really crappy code. For instance, it generated brain-dead default names for all those drag-and-drop controls - why name a control DataSource1 and force the user to change it when you can just prompt him] for th econtrol name? It also made no attempt separate the interface, data access layer, and business logic. In fact, doing that made the magical drag-and-drop features less helpful. And unit testing? Was that even possible in VB 6?

My only complaint about RoR so far is that there doesn't seem to be a lot in the way of documentation yet. Sure, there's the API reference, but not much in an instructional vein. And so far, I'm unimpressed with the two tutorials I've looked at. But I suppose I can let that slide. After all, RoR is quite new - building up a good documentation base takes time. Of course, by the time we get some good tutorials, I won't need them, so what do I case?

Learning TDD

pageer@skepticats.com (Peter Geer)

It's time to start getting my skills up to date. And not just learning the programming language de jour, either. Learning new languages is nice, but as they say, you can write COBOL in any language. No, it's time to get up to speed on methods and practices.

My current project is to get myself up to speed on test-driven development and unit testing in general. This was prompted in large part by listening to an old episode of Hanselminutes on dynamic vs. compiled languages. Scott made the point that TDD and dynamic languages are (or should be) linked. Since you can't rely on the compiler to catch errors early, you need to replace that with continuous unit testing.

That made sense to me. I'm a fan of strict languages and static analysis. In fact, I'm one of the last 9 guys on Earth who thinks Ada is a really great programming language. (Note: That is a made-up number. The actual number is 14.) Since you don't have that safety net with dynamic languages, you need to find something else that can pick up the slack. Since unit testing is good for lots of other things as well (like detecting regressions), it seemed like something that was worth seriously taking up.

I've started off my journey by playing with the Simple Test unit testing framework for PHP. I came across it by chance and decided to go with it because the site had some helpful introductory articles on unit testing. I have a C# project I'll be starting at work this week, so I'll also looking at NUnit soon as well.

At the moment, I'm still trying to wrap my brain around the way TDD works. Conceptually, it's not that complicated. It's just that it's a more bottom-up approach than I'm used to. When you write tests first, it seems like you almost have to start by writing the basic building blocks and work your way up to the more complicated methods. However, I tend to work in reverse, starting with the "meatier" methods first and writing the supporting methods as I figure out what I need. But maybe that's not the best way to work. I don't know. I've still got a lot of reading to do.

A debugging case study.

pageer@skepticats.com (Peter Geer)

The thing about software development is that there's an inherent conflict in it. In order to get the functionality we've come to expect these days, a program has to be fairly complicated. On the other hand, in order to keep an intellectual handle on the system, developers need to make it as simple as possible. Otherwise, you're liable to lose your grip on the system and end up staring a a screen full of code that is obviously correct, yet is nonetheless not working as expected.

I had one of those moments today. I was trying to add some caching to LnBlog's plugin system. I started off by adding some caching code to the recent entries plugin (look to the left - it's the 5th panel down in the sidebar). But then I thought better of it and decided I'd save myself some work by sharing the code through the base Plugin class. That way, all the plugins could use it without having to copy it over and over again. Perfect! However, when I tested it out, I noticed that section header for the calendar plugin had mysteriously disappeared. And yet, I hadn't even touched the calendar plugin. What was going on?

And so, my search began. The process went something like this.

Step 1: Check the new code

The first step is always to check the new code. If something changed and then something broke, then whatever changed is the most obvious culprit. In this case, the new code included a new sidebar output event handler for the recent entries plugin and switching back to the old handler eliminated the bug. However, I didn't see anything looked obviously wrong with the new code. To make matters even stranger, I discovered that I could eliminate the bug by using an event handler with the same signature as the old version, but which merely called the new one in its body. Say what?

Step 2: Check the calling code

The next place to look is one step back in the code. In this case, I looked at the event registry. When the sidebar output event is raised, the event registry handles calling the event handlers. This was also suspect, because I had recenty added data parameters to the event handlers. However, there didn't appear to be anything wrong here either.

Step 3: Check the affected code

Normally, this would come sooner, but my next step was to look at the calendar plugin. Since I hadn't actually changed anyting in this plugin, my assumption was that the problem must be an interaction with some of the new code. Since that wasn't panning out, it was time to try something else.

Step 4: Pulling it all together

As I was looking at the calendar code, it all became clear. The sidebar output event handler takes an optional parameter which can be set to turn off the display of the section heading and container DIV element (for AJAX calls). This parameter is called $nodiv and the default value is false. The eevent register now takes a data parameter. Somehow, a data parameter is getting passed on to the calendar plugin. So the problem was that data was inadvertently getting passed on to the calendar.

I went back to the event register code and looked at the data parameter code. The same parameter gets passed to every event handler. Back over to the new recent entries plugin. It takes an optional parameter, passed by reference, with a defalut value of false. However, when the handler got the value[bfalse[/b], it reassigned that parameter to an object for the current blog.

And there it was. I was assigning a value an optional reference parameter. (Which I only had to do because PHP doesn't do function overloading.) That change was then propogated to other event handlers. Since the calendar event handler depended on getting false as its parameter value, it didn't function as expected.

On the up side, it was a nice, easy, one-line fix. Isn't that always the way? Tracking down the bug always seems to take ten times as long as fixing it.

Sabotaging productivity

pageer@skepticats.com (Peter Geer)

This week, .NET Rocks had a great interview with Jeff Atwood. Jeff is a really insightful guy and listening to him was as much fun as reading his blog. In fact, this interview inspired me to start listening to other episodes of .NET Rocks. Well, that and the fact that Carl Franklin co-hosts Hanselminutes, which I also enjoy.

One the topics the interview touch on was Jeff's Programmer's Bill of Rights. It enumerates six things a programmer should expect if he is to be productive.

I found this both depressing and comforting. It's depressing because, as Jeff pointed out in the interview, these things are neither unreasonable nor hard to fix. You can basically just throw money at them without putting in any real effort. These conditions should not be widespread enough that anyone needed to bring them up.

As for comfort...well, it's just nice to know you're not alone. I'm currently one of those poor schleps Jeff talked about who's still working on a single 17" CRT monitor, a three year old PC, and sitting in a cubicle right next to one of the network printers. I'm not even within sight of a window and my cube is literally just barely big enough to fit my desk. I write my code in SharpDevelop because my boss won't spring for a visual studio upgrade. Two years ago it was, "Well, we'll wait for the 2005 version to come out instead of buying 2003." This year it was, "We'll wait for the 2007 version to come out instead of buying 2007." And last but not least, despite the fact that we write mostly reporting-heavy information systems, I use the version of Crystal Reports that came bundled with VS.NET because, as crappy as it is, it's the best thing available to me.

I have to agree with Jeff, Richard, and Carl. The message you get from a setup like this is clear: you are not important. We don't value you as a person, we don't value your work, and we're certainly not going to waste money on making your life easier. The net effect is that morale, especially among the more clueful people in my office, is in the gutter. There's misery galore and productivity is next to nothing. But fortunately we work for the government, so nobody notices. And no, that was not a joke.

Sometimes it seems like our environment is tailored specifically to sabotage productivity. It's kind of like the keyboard they put on the laptops that the police use. I'm the happless IT guy who has to do configuration and maintenance on these laptops, and I can tell you that the only explanation for those keyboards is that I did something really, really terrible in a past life. They're ruggedized keyboards made of semi-hard plastic. The problem is that they're so rugged that it's completely impossible to type on them. You have to use the two-finger method because the keys are too hard to press with your little fingers. Trying to type with any speed at all is completely futile. And yet the cops are somehow expected to type up tickets and accident reports on these things. It's a wonder they even give out tickets anymore. Actually, maybe that was the idea....

I suppose this is what I get for taking an IT job when I really wanted to be in software development. In retrospect, maybe I should have stayed a full-time student that extra semester or two, finished my damned thesis and looked for a job with a real software company. But I thought I needed some experience and this was the best offer I got, so I took it. Unfortunately, I was too inexperienced to know that crappy experience isn't necessarily better than no experience.

Though on the up side, when I took this job is when I moved in with my (now) wife. It also provided the money that paid for that engagement ring. So in some ways this was the right decision. It's just the professional advancement wasn't one of them.

Now I just need to finish my damned Master's thesis and get the hell out of here.

Why have bug tracking when you can use post-its ?

pageer@skepticats.com (Peter Geer)

I came across a weird article today. It's entitled Why Bugs should not be Tracked and the basic premise is that it's a bad idea to use bug tracking software.

What struck me as really strange about this article is the final conclusion. After telling us how bug tracking is bad and unnecessary, the author offers an alternative: write your priorities down on a piece of paper. In other words, do the extremely low-tech version of what bug tracking software is for. Say what?

The real problem that this article addresses is that bug tracking systems are easily abused. It's easy to enter bugs and just let them sit there until the end of time. However, if you're using something like a piece of paper or a spreadsheet, you have to dispose of the "stale" issues in one way or another, otherwise the whole thing becomes completely unmanagable.

There's an easy response to that, though: "Don't abuse your bug tracking system." Don't let bugs pile up. Cagetorize sanely. Do some house-cleaning when the number of outstanding bugs gets too big for comfort. It's not rocket science, just use the tool productively. Agile development is all well and good, but sometimes it seems like they're throwing the baby out with the bath water.

Sybase + ADO.NET = pain

pageer@skepticats.com (Peter Geer)

I've been wrestling with Sybase again lately. Both last Thursday and today, I had to write a couple of small programs to update and reformat values in some Sybase tables at work. The programs themselves were very simple and took maybe half an hour to write. However, due to the pain inherent in connecting to Sybase, this quick update turned into a several-hour ordeal.

The database in question is running on , which is about three major versions out of date.

While I wrote the original client program in VB6 with ADO 2.6, I have now switched to C#. This is partly because VB6 sucks by comparison to C# and partly because I'm trying to make my skillset more marketable. So when I wrote those little update programs, I did it in C# with .NET 2.0. However, that didn't go so well.

Method 1 for connecting to database in question, which runs Sybase SQL Anywhere 7, was to use the System.Data.OleDb classes and just use the provider that worked with ADO 2.6. But, of course, it didn't work with ADO.NET. I even tried following the Sybase how-to on ADO.NET, but the ExecuteDataReader always resulted in a "no such interface supported" COM exception. The only difference was that I was using the ASAProv.70 provider instead of ASAProv.80, which I don't have. Presumably the older provider doesn't do ADO.NET.

The second attempt was to use the .NET 2.0 System.Data.Odbc classes. My reasoning was that, since ODBC is an industry standard supported by virtually everyone, using straight ODBC should work. And it did...sort of. I was able to connect to the database and read records without incident. The problem was with updating using OdbcCommand objects.

The code I used was pretty unremarkable, which is why I was a little surprised when it failed. It looked something like this:
OdbcConnection dbconn = new OdbcConnection("DSN=SomeDSN); dbconn.Open(); /* Lots of unrelated code.... /* string sql = "update MyTable set Bar = ?, Baz = ? where ID = ?;"; OdbcCommand cmd = new OdbcCommand(sql, dbconn); cmd.Parameters.Add("Foo", OdbcType.Char, 15).Value = someVar; cmd.Parameters.Add("Baz", OdbcType.Char, 15).Value = someOtherVar; cmd.Parameters.Add("Foo", OdbcType.Int).Value = someIDVar; int result = cmd.ExecuteNonQuery();
There shouldn't really be much to go wrong there. However, executing the command resulted in the following exception:
Exception System.Data.Odbc.OdbcException was thrown in debuggee: ERROR [HY000] [Sybase][ODBC Driver][Adaptive Server Anywhere]General error: Host variables may not be used within a batch
Googling this message came up with this page from Sybase website, which helpfully tells me:

You have attempted to execute a batch which contains host variable references. Host variables are not supported within a batch.

That's it. Just a restatement of the error message. I find this to be fairly typical of Sybase documentation.

At this point, I was faced with two questions:
1) What the heck is a "host variable?"
2) Why am I getting a message that they aren't allowed?

The first question was answerable with a little Googling. As explained by this page from the IBM iSeries manual, a host variable is just what the name suggests: an external variable that gets used in an SQL statement. You can use them for input and output parameters when writing more complicated SQL programs.

For the second question, Google failed me, so I can only speculate as to the answer. My guess is that ADO.NET is translating the parameters into host variables rather than inserting the values directly into the SQL. Presumably my version of the ODBC provider doesn't support host variables, hence the error. It seems to make sense, but as I said, that's only a guess.

My final solution? After spending way too much time on fruitless searching and experimentation, I finally decided to just go the quick and dirty route of string concatenation. Instead of parameters, I just used a String.Replace to escape quotes and spliced the new field values directly into the SQL. It's ugly, and I wouldn't want to use it in a "real" program, but you know what? It works.

No zone => Windows script-foo

pageer@skepticats.com (Peter Geer)

It's been a very "IT" kind of afternoon.

I started out the morning well, got some coding done, and was in "the zone" up until about 10:50. The then fire alarm went off. I don't know why, because there wasn't any major fire. Maybe it was a drill/test, maybe some kid pulled the fire alarm. Who knows.

Anyway, I had to go stand outside in the unseasonably cold weather (it's in the 30°F range this week) for about 10 minutes. That sapped a great deal of my motivation. Then I had to stand in line for another 10 minutes while the guards checked IDs put people through the metal detector. That pretty much knocked me out of the zone for the rest of the day.

So this afternoon was devoted to Windows XP trouble-shooting, because it's reliatively brain-dead. I worked with a laptop user to finally track down a long-standing error message that had been showing up in one application. And by "track down" I mean I finally got to actually see the error message and quickly diagnosed it as a file permission problem. I would have done that a long time ago, except that these laptops have no connectivity to our network and are almost never available at a time and location where I can look at them. Plus nobody ever bothered to tell me the exact error message, which makes it a little harder to figure out what's going on. While I try to keep my skills sharp and up-to-date, I have to admit that I'm still woefully lacking in the telepathic debugging department.

Now that I've got that problem tracked down, I'm presented with another. This problem affects 20 odd laptops. How do I fix it on all of them without having to go around and physically touch each of them? Remote access is out, so the first obvious solution is to get the users to do it. However, that won't work because I can't give them the admin password. The second obvious solution is to pawn it off on the help desk, but they'd never go for that. After all, we operate on the "whoever touches it first is stuck with it forever" theory of assigning support tasks. Hardly an optimal algorithm, but that's the way it is and nobody with any power is willing to rock the boat.

That leaves me with one option: script that sum bitch! I might not be able to trust the users with complex instructions, but I can certainly trust them to dump some files on a flash drive and double-click an icon.

The only problem is administrator access. The Windows runas command prompts for passwords. I need something that can use a stored password from a batch file, preferably one kept in an encrypted file. Luckily, a little Googling turned up just such a tool: lsrunase. That looks like it should do the trick. Another quick Google for a Windows command reference to find the CACLS command and I should be good to go. All I need to do now is write the batch file and test it out on one of the laptops. But that'll have to wait for tomorrow.