LinLog

So I'm getting up to speed on go-lang for a project I'm working on.  Since my company provides us with access to PluralSight, I've been going through the courses and projects in their Go pathway.

So far most of the courses have been pretty decent.  However, the projects are a different story.  The first one I did was fine - not particularly enlightening, but OK.  The second one, the Go CLI playbook, to be perfectly frank, is garbage.  At least, the first module is garbage.  I can't really speak to the rest because I can't get past the first module.

Now, in my own defense, it's not like the first module is hard.  In fact, it's completely trivial.  It basically comes down to "run go env > module1.txt ; go env -json > module1.json and then submit the result.  It even comes with a test file you can run locally.  So far so good.

The problem is, it doesn't work on Windows.  Like, doesn't work at all.  For starters, if you're running Powershell, output redirected to a file defaults to UTF-16 encoding, which is standard for Windows, but not for Go apparently.  Second, the go env command generates environment variables in an OS-specific format.  On Windows, that means it generates a file that can be consumed by cmd.exe.  Which is fine, except the project test file blithely assumes that the environment file is going to be in UNIX shell format, so the "set" at the start of each line makes it choke.

Well, that sucks.  Guess I'll just use WSL.  When I run the commands there and run the tests, they pass.  Great!  Let's upload the result and get on with our lives.  Except that doesn't work, because the tests fail when run in PluralSight's validator.  I don't know why.  My best guess is that something in the upload process is mangling the encoding.  I do know that it apparently doesn't run the test file you upload, so you can't hack around whatever the bug is.

The worst part is that you can't progress to the rest of the project until you "pass" that section.  So I'm pretty much just out of luck.  Maybe the rest of the course has some great material.  Or maybe it's equally useless.  Who knows?  Maybe it's just this project - so far the others I've tried have been OK.  Although I did have one other module where all the tests passed locally, but failed on upload.  But in that case, I at least had a legitimate bug in my code.  Why the tests passed locally is a different problem, though....

The other week, somebody posted this article by Jared White in one of the chats at work.  It decries the "shocking immaturity" of the ecosystem around JavaScript and Node.JS.

I mean...yeah.  But it's not like this is news.  The Node ecosystem has been messed up for years.  Remember the left-pad debacle?  That was five years ago.  It's pretty clear that the ecosystem was messed up then.  So I guess this article just tells us that not much has changed.

To be fair, a lot of the stuff Jared complains about isn't really specific to the JavaScript ecosystem.  I've also been in the industry for 20 years and I can say from experience that bugs and hype are endemic to most of the industry and have been for quite some time.  For example, in the early days of Rails, I remember seeing a million variations on the "build your own blog in 10 minutes with Ruby on Rails" tutorials.  And yes, that's fine, you can make a simple demo app in 10 minutes.  Whoop-de-doo.  In reality, what that usually means is that on a two-month project you've saved yourself...maybe a day or two at most.  There are lots of tools and framework in lots of language ecosystems that are grossly over-hyped - it's almost standard practice in the industry.

As for bugs, I can't speak to Jared's experience.  In any software ecosystem, bugs are obviously common and not mentioning them is almost de rigueur.  I mean, if you're developing a framework or library, of course you're not going to advertise the bugs and limitations of your tool.  You want people to use it, not be scared away.  But I'm willing to accept Jared's assertion that the JavaScript world is uniquely bad.  I know my experience of client-side JS libraries is...not fantastic in terms of reliability or documentation.  So while I'm not sure he's right, I wouldn't be surprised if he was.

I do think his point about the learning curve is interesting and valid, though I don't know that it relates specifically to bugs.  I haven't gotten deep into many of the fancy new JavaScript frameworks, but they do seem to be staggeringly complex.  I started working with JavaScript way back in 2005, when all you needed to do what save your code in a text file, open that file up in a browser, and see what it did.  It was extremely simple and the bar to entry was ridiculously low.  Then, a few years ago, I decided to try out React, since that's the big new thing.   Just to do "Hello, World!", I had to get my head around their weird template syntax, install a transpiler, and run some kind of server process (I don't even remember - maybe that's changed by now).  And when I saw that work, I quit because I had actual work to do.  It's hard for me to remember what it was like to be a beginner, but I can imagine that this kind of an on-ramp would be pretty daunting, even with the dumbed-down tutorials.  Heck, it seemed like kind of a lot to me, and I'm an experienced professional!

Honestly, I kind of wonder how much of the problems Jared is seeing stem from the "youth" of the JavaScript ecosystem.  I'm not talking about the language, of course.  I'm thinking more of the historical and cultural part of the ecosystem.  Consider:

1. While JavaScript has been around for a 25 years, it was widely considered a "toy" language for the first 10 years or so.  Remember - JavaScript came out in 1995, but jQuery didn't come along until 2005.  And these days, building your site on jQuery is the equivalent of building your house out of mud and sticks.
2. In the roughly 15 years of JavaScript's non-toy lifespan, there's been a lot of churn in the web space.  And during much of that time, it was considered important for many businesses to support legacy web browsers.  I remember many times having to stop myself from using "new" features of JavaScript because, well, we still have to support old versions of Internet Explorer.  Yeah, nobody cares about that anymore (thank God), but it wasn't all that long ago that they did.  Heck, I remember getting yelled at in 2015 because I forgot to test something in IE9, which was released in 2010!
3. From 1 and 2 above, it's clear that, in terms of the evolution of the ecosystem, the 25 year history of JavaScript is really a lot less than 25 years.  In fact, it's probably only within the last five years or so that we've managed to shake off most of the legacy cruft and get adoption of modern stuff beyond the handful of early-adopters.
4. On the cultural front, it's been my experience that a lot of young people these days get into coding through web development.  This is not a bad thing - everybody has to start somewhere and the web is a relatively accessible and popular medium.  But it's also my experience that a lot of the people who create open-source tools and libraries are younger, because they're less likely to have families and other obligations and hence more likely to have the free time.  Again, this is not bad, but it means that the people writing the tools are disproportionately likely to be less experienced.
5. So while there are plenty of things in the JavaScript world that are old enough that they should be mature, we can see from 3 and 4 that this might not necessarily be the case.  When a tool is developed by relatively inexperienced coders and hasn't been widely used outside a relatively small circle for very long, it shouldn't come as a surprise when they have some issues.

Of course, I'm just spit-balling here - I could be completely wrong.  But the point is that developing a stable ecosystem takes time, and the JavaScript ecosystem hasn't actually had as much real time to develop as the calendar suggests.  I mean, there's still a hot new framework coming out every other week, so it doesn't seem like the ecosystem has even finished stabilizing yet.  Maybe in a few years things will settle down more and quality will improve.

Or maybe not.  We'll see.  In the mean time, we just have to make do with what we have.

So the other week I saw a cool toy I've been wanting on Amazon, and it was a good price.  No, not a tech gadget this time - a new set of dumbbells.  And since I didn't want anything for my birthday, my wife gave the approval to order them.

Specifically, I got a pair of PowerBlock Elite EXP dumbbells.  These things may look like toasters, but they're actually adjustable dumbbells that can go from 5 to 50 pounds in 2.5 to 5 pound increments.  The big selling point is that they adjust quick  You can do 10 pound adjustments just by moving a pin, which is super fast.  The 2.5 and 5 pound increments are little more involved, as those are adjusted by removing weight cylinders from the handles, and you can't do all increments, but it's still just pulling the pin, flipping a lever, and dropping a cylinder in or out.

Compare this to my old set of dumbbells, which adjusts up to 52.5 pounds.  It adjusts in 2.5 pound increments (and does all of them), but does it by swapping out plates and securing them using a screw clamp.  It works fine, but it's slow.  So, for instance, my current program has a bunch of supersets in it, and for most of them I'm using different weights for each of the two exercises.  So the "go immediately into the second exercise" was actually more like "spend about a minute changing the weights and then go straight into the next exercise".  Clearly this is not ideal in terms of executing the program, and it also stretches out the workout.

I'm pretty happy with them.  They took a little getting used to, but after using them for a week or two they work pretty well.  My main complaint is that the instructions were a little iffy.  It took me a little playing around to figure out exactly how to put in the pin, which direction things went, and so forth.  But it's not complicated, so it just took a few minutes of messing around.

On the down side, Amazon is now sending me suggestions for the thing I just bought.  They did the same thing when I bought my new laptop on Prime day.

I'm sure at some point Amazon will fix this, but it doesn't exactly say much for their recommendation AI.  I mean, this recommendation is a different model and a lighter weight, so it's not exactly the same thing, but come on.  Can one of the biggest tech companies in the world, with huge amounts of computing power and legions of brilliant people devoted to building their recommendation systems, really not figure out that I don't need to buy a slightly different version of the thing I just bought from them?  

I don't know - maybe there's a good reason for that and I just don't know enough about Amazon and/or machine learning.  It's just a really annoying thing that seems like it should be an obvious thing to fix.  Heck, I would imagine that the machine learning systems might even pick up on that on its own.  I mean, do people who buy a thing really buy it again a week later that often?  I'd think those numbers are all fed into the recommendation system and that they'd be low enough that the algorithm would drop recent purchases (especially larger ones) off the recommendations.

Then again, maybe the algorithm knows something I don't.  Maybe people re-buy the same thing all the time.  It could be they even do that with items that cost hundreds of dollars.  I doubt it.  But what do I know?

This morning I finally took a little time to read a few articles about NFTs.  Specifically, I was interested in what you actually get when you buy an NFT.

As I'd written in a previous post, this was kind of confusing to me.  I mean, I got the technical aspect - that an NFT is a blockchain transaction that's tied to some file that lives outside the chain.  So you have an immutable record that the transaction happened, which is good.  But when you read about people paying hundreds of thousands, or even millions of dollars for an NFT, what did they actually get for that money?

The short answer is: it depends.

It looks like, legally speaking, it depends on the terms of the sale agreement.  It's possible that a high-priced NFT was purchased under an agreement that transfers full ownership and copyright rights to the purchaser.  However, it's also possible that all they got was that transaction on the blockchain.

It seems like the default is that buying an NFT is like buying a collectible, but in digital form.  Think of, say, a limited-edition collectible like a toy, coin, or something similar that comes with some sort of certificate of authenticity.  When you buy it, you get the physical item, but you also get documentation with some sort of serial number and proof that that specific item was sold to you.  Having that kind of provenance can, in and of itself, be very valuable to collectors.

However, while it might be understood that the supply of a limited-edition item will be fixed, you generally aren't guaranteed that.  There's not necessarily anything to stop the manufacturer from making a crap-ton more of them and flooding the market, thus diluting the value of your purchase.

That's kind of like how it works with NFTs.  Unless you have a contract that says otherwise, it seems like you can't generally assume that you get anything other than the digital equivalent of a proof-of-purchase.  You get the exclusive rights to that certificate, but the only rights you get over the work associated with it are basically the rights of private ownership, i.e. to have and use it for your own, personal purposes.

To my way of thinking, it's kind of like the difference between buying an e-book from O'Reilly and downloading the same e-book from a torrent site.  Either way, you have a copy of the book that you can read.  Either way, you can give your friend a copy of it to read.  But if you buy it, you have a receipt to prove that you bought it.

Is there some value in that?  Yes, I think there is.  I can especially see that in the case of a "digital collectible", where a significant portion of the value could be in just having them in your collection.  (I do think paying large amounts for that is silly, but I also think paying large amounts for physical collectibles with no intrinsic value is silly.)  But for cases where there's no physical asset, and no ownership rights over the original digital asset, and nothing to stop the creator from minting more tokens for the same asset...?  I have a hard time imagining a "certificate of authenticity for Jack Dorsey's first tweet" being worth $2.5 million.

So I'm moderating my position a little.  I'm not going to say that all NFT is just dumb.  Depending on the transaction details, it can be a useful thing to buy.  But I think it really depends on the details.  I mean, if all you're buying is a "certificate of authenticity" for something that somebody else owns and anyone can get for free, that seems kinda silly.  And paying thousands of dollars for something like that seems really silly.  So while NFT might not be totally worthless, I still think a lot of the high-priced transactions we read about are nothing but speculation.  (Or money laundering, or tax avoidance, or whatever other financial tricks people pull with collectibles.)

I only recently learned about the existence of NFTs as a "thing".  If you are also unfamiliar, NFT stands for "Non-Fungible Token".  CNN has an explanation here, but t's essentially a unique digital token, stored on a blockchain, that represents some digital asset.  Apparently NFTs have recently become the latest hot trend, with some of them selling for hundreds of thousands of dollars.  

However, this Slashdot post points out that there are some potential issues with NFTs.  Basically, NFTs don't actually store the asset you're purchasing.  So if you buy an NFT for an image, the image doesn't actually live on the blockchain.  You've only bought a reference to the image, which is actually hosted someplace else.  So if whatever server or service is hosting that image goes away, you will no longer be able to access it, regardless of owning the NFT.

So I guess what I'm confused about is: what's the point?  I mean, what are you really getting when you buy an NFT?  In a theoretical sense, I can see how an NFT is better than "conventional" digital assets in that it's not tied to a particular service.  Your ownership of that item is recorded in you crypto wallet which is independent of any particular service an can be used with multiple NFT marketplaces.  And that's a good thing. 

But when you look at it functionally, there's not really much difference.  The actual asset still exists completely independent of the blockchain, so it's not like a physical asset - there might only be one token, but you can still infinitely duplicate the asset.  And as far as I can tell, buying an NFT doesn't actually mean you're purchasing the copyright for the asset.  So you're just buying a copy and there's nothing to stop anyone from making other copies.  And because the asset isn't stored on the blockchain, if you want to ensure you always have access then you need to download a copy of it.  So...how is this different from buying a service-specific digital asset?

It seems like the point is less the actual asset and more that NFT is the same thing as Bitcoin - that it's just a different way to speculate on blockchain stuff.  Especially when you're talking about something like spending $2.5 million for jack Dorsey's first tweet, it's hard to see any other rational explanation.  But even for less absurd cases, it's not clear to me what the practical benefit is.  The main reason that blockchain "works" for cryptocurrency is because the thing on the blockchain is the thing you're transferring.  As soon as you introduce a disconnect between the thing being traded and the thing on the blockchain, it seems like you lose a lot of the benefit of the blockchain.  

The other day I decided to go back to something I did when I was still working in the office: putting tech conference talks on in the background while I work.  Sometimes it's nice to have a little background noise if I'm not trying to concentrate too deeply on something (what I'm currently working on involved a lot of waiting for CI pipelines to run).  Of course, sometimes I my concentration level ramps up and I totally tune out the entire talk, but sometimes I don't and it ends up being interesting.

This particular afternoon I was listening to some talks from the GOTOpia Europe 2020 virtual conference.  This one had an especially good lineup, including talks from Dave Thomas, Kevlin Henney, and Allen Holub, who are some of my favorite presenters.  Cat Swetel, who I'd never seen speak before, also had a very interesting presentation on metrics that I would heartily recommend.

It might not be the same vibe as the live conferences, but it's at least nice to be reminded that the entire world hasn't been canceled due to the pandemic.  And there are always some interesting discussions at GOTO, so it's worth a watch.

I saw an interesting article on Slashdot recently about the vanishing of online scientific journals.  The short version is that some people looked at online open-access academic journals and found that, over the last decade or so, a whole bunch of them have essentially disappeared.  Presumably the organizations running them either went out of business or just decided to discontinue them.  And nobody backed them up.

In case it's not already obvious, this is a bad thing.  Academic journals are supposed to be where we publish new advances in human knowledge and understanding.  Of course, not every journal article is a leap forward for human kind.  In fact, the majority of them are either tedious crap that nobody cares about, of questionable research quality, or otherwise not really that great.  And since we're talking about open-access journals, rather than top-tier ones like Nature, lower-quality work is probably over-represented in those journals.  So in reality, this is probably not a tragedy for the accumulated wisdom of mankind.  But still, there might have been some good stuff in there that was lost, so it's not good.

To me, this underscores just how transient our digital world is.  We talk about how nothing is ever really deleted from the internet, but that's not even remotely true.  Sure, things that go viral and are copied everywhere will live for a very long time, but an awful lot of content is really just published in one place.  If you're lucky, it might get backed up by the Internet Archive or Google's cache, but for the most part, if that publisher goes away, the content is just gone.

For some content, this is a real tragedy.  Fundamentally, content on the Internet isn't that different from offline content.  Whether it's published on a blog or in a printed magazine, a good article is still a good article.  A touching personal story is no more touching for being recorded on vinyl as opposed to existing as an MP3 file.  I know there's a lot of garbage on the web, but there's also a lot of stuff that has genuine value and meaning to people, and a lot of it is not the super-popular things that get copied everywhere.  It seems a shame for it to just vanish without a trace after a few short years.

I sometimes wonder what anthropologists 5000 years from now will find of our civilization.  We already know that good quality paper can last for centuries.  How long will our digital records last?  And if the media lasts 5000 years, what about the data it contains?  Will anthropologists actually be able to access it?  Or are they going to have to reverse-engineer our current filesystems, document, and media formats?  Maybe in 5000 years figuring out the MPEG-4 fomat from a binary blob on an optical disk will be child's play to the average social science major, who knows?  Or maybe the only thing they'll end up with the archival-quality print media from our libraries.  But then again, given what the social media landscape looks like, maybe that's just as well....

Author's Note: This entry from my archives was written on March 18, 2007 and has been sitting in my drafts folder ever since.  Not sure why I didn't publish it at the time.  I think I was going to add more, but never got around to it.  At any rate, this was back then RSS feeds were a trendy, new-ish thing and this article was supposed to be a less technical discussion of what they are and why they're good.

These days, of course, RSS is passé, and when people refer to a "feed", it's usually coming from Facebook, Twitter, Instagram, or whatever vendor-locked service the kids are using this week.  I find this sad.  The idea of the open web was so promising, but not that much really came of it.  Instead of being spoon-fed our information and entertainment by big media companies via broadcast and print media, we're now spoon-fed our information and entertainment via the internet by big tech companies.  And this time, the content is not selected by gate-keeping editors, but by AI algorithms that are tuned to feed us whatever will keep us clicking, with little to no regard for whether it's true, useful, or even remotely good for us.

For the record, I still use RSS feeds all the time.  I use the Tiny Tiny RSS aggregator, which is quite nice, to read the various blogs and news that I'm interested in following.  I have accounts with a few of the big social media platforms, but I rarely ever read them and never post anything.  I find them to be a huge time-sink and not especially conducive to good mental health, and so better off avoided.  Of course, your mileage may vary, but just keep in mind that you don't need to look at these sites - if anything truly important happens, someone will tell you about it.  I mean, unless you're a shut-in with no friends or family.  In that case, maybe social media is a good thing for you.  

At any rate, these were my thoughts in 2007.  Perhaps they'll be interesting or enlightening.  Or perhaps entertaining in their naivete.  Enjoy!

If you frequent tech sites or weblogs, you've probably seen the RSS icon or the XML feed icon .  You may also have seen other icons or text links referring to XML feeds, RSS, or even podcasts.  In fact, if you're using a web browser other than Internet Explorer, you may have seen one of these icons pop up in the address bar or status bar.  In this article, I will try to explain, in layman's terms, what RSS is and why it is so useful and important to the future of the internet.

What is RSS?

RSS stands for Really Simple Syndication.  As this name suggests, it is a syndication format.  

By "syndication," we mean essentially the same thing as when we talk about syndicated television shows.  A syndicated TV show is one that is shown on multiple independent channels at the same time, as opposed to being exclusive to a single network.  So for example, a syndicated show in the United States might be broadcast by NBC, Fox, the Sci-Fi channel, and USA all at the same time.

RSS works the same way.  An RSS file, usually referred to as a feed, contains a list of recent updates to a site.  The site operators publish this file on the web site and allow other people to subscribe to it, which is really just a fancy way of saying they automatically download it on a regular basis.  These people can then "republish" the information, either by incorporating it into their own sites or simply reading it into a desktop application.  The idea is that if the site's operators update the RSS feed every time they update the site, anyone who subscribes to it will automatically get the next "episode" the next time he downloads the file.

But what would I do with it?

If you are not already familiar with RSS, you may be wondering why anyone would bother with this.  After all, if you just want to read the updates of a site, isn't it just as easy to read the home page?  

At this point, you may be thinking that this doesn't sound much different from just visiting the web site in question.  After all, why would you want to bother with this RSS thing when you can just go to the site's home page like you've been doing for years?  

You wouldn't be wrong to think that.  If you're talking about just one site, with one set of updates to track, then RSS doesn't make any sense.  It would just be a different way of doing the same thing.

The beauty of RSS is that, unlike a web page, you can easily write a program to break up the information and organize it in a useful way.  For example, you can have a script on your web site that takes RSS news feeds from CNN, the BBC, and others and puts them all together in a news ticker on your home page.  You can also have programs such as RSS aggregators, which pull together news items from multiple sites and display them together so that you can browse them quickly and easily.  

I will discuss some other uses of RSS, including the trendiest of them all, Podcasting, later in this article.  (Note from the future: I never actually did that.)  But before that, we need to cover why RSS is useful and separate the fact from the hype.

A brief technical digression

What makes RSS so useful and so widely applicable is that it is a standard format.  It is an application of XML, the eXtensible Markup Language, which is an industry standard markup language for use with structured information.  I won't bore you with a description of XML, but the upshot of this is that RSS files all contain a certain set of standard information which is always marked with the same standard tags.  This means that a program can easily go through the file and pick out particular pieces of information, like the title of a particular news item, without having to pay any attention to what the title actually says or how it is formatted for display.  And because RSS is based on XML, there is already a wide array of programming tools that can be used to create and manipulate the files.

This is in stark contrast to web pages.  Although HTML, the markup language used to build web pages, has a standard set of tags, there is no standard for how a page is structured.  So while there are fixed ways of defining lists, tables, and paragraphs in HTML, there is no agreed upon way to say, "This is a news item, this is its title, and this is the link to its page in the archives."  (Note from the future: With the advent of HTML 5 this is no longer technically true.  However, semantic markup is not universally or consistently applied, so it's still close enough.)  So while a human being can easily look at a page full of items and determine where one ends and the next begins, there is no simple and general way for a computer to do that.  Because everyone is free to pick how they want to denote those things, a program would have to analyze the content of the page and figure out what the author of each page was thinking.  Needless to say, this kind of mind-reading is not something computers are particularly good at.

Getting past the hype

You know what the computing industry is best at producing?  It's not software, hardware, or anything else you can buy in the local office supplies store.  It's hype.  And to really appreciate how good RSS is, you have to get past the hype generated breathless pundits who seem to think it will cure cancer, feed the starving, and bring peace to the world.  (Note from the future: Nobody gives two hoots about RSS or XML anymore.  Hype has a very limited life span.)

From a technical standpoint, there is absolutely nothing revolutionary about RSS.  It's just a standard way of formatting a text file.  You could even create an RSS file in Windows Notepad if you really wanted to.  

And when you think about it, using RSS feeds is just a fancy name for putting a text file on your web site and then letting people download it and mess around with the information it contains.  How is that revolutionary?  We could do that 20 years ago.  

However, it is important to remember that "revolutionary" and "innovative" are not the same as "good."  RSS is good because it provides a standard way of transmitting certain kinds of information in a way that puts all the power in the hands of the consumer, not the publisher.  It's not the technology itself that's revolutionary, but rather the way people have chosen to apply it.

The real reason RSS and the whole "Web 2.0" thing are important is not because of the technology, but because of the paradigm.  That paradigm is: the user is king.  In an era where Hollywood and the music industry are trying to tell you when and where you're allowed to use the music and movies you paid for, open standards put that power in your hands.  (Note from the future: People have forgotten about that aspect, so now "Web 2.0" means "the site uses AJAX."  Also, substitute "Hollywood" for "big tech companies.")

Lauren Weinstein had an interesting post a earlier this year discussing software developers' attitudes toward the elderly.  His main point is that developers tend not to think at all about the issues that older people have when working with computers. These include things like reluctance to or difficulty with learning new programs or ways of working; old hardware which they can't afford to upgrade; isolation and lack of access to help; and physical limitations, such as poor eyesight or reduced manual dexterity.

Of course, this is obviously not true of all developers (like Lauren, for example), but if we apply it to the general zeitgeist of the community, at least as you see it online, then there does seem to be something to this.  As a group, developers are very focused on "the coming thing", as Brisco County Jr. would say.  We all want to be ahead of the curve, working with the cool new technology that's going to take over the world.  We want to be on greenfield projects that are setting the standard for how to do things.  That's why otherwise intelligent programmers do or suggest crazy things like rewriting their conventional LAMP-based site in Go and ReactJS.  Of course, it's long been established that rewriting from scratch is almost always stupid and wasteful, but the fact is that while PHP might pay the bills, it isn't cool.

Of course, it isn't just because they want to be cool that developers like newer technologies.  There are plenty of other reasons.  Intellectual curiosity, for one.  Many of us got into this line of work because we enjoy learning new things, and there are always interesting new technologies coming out to learn.  Learning old things can be interesting as well, but there are a few problems with that:

1. Older technologies are less marketable.  Learning new tech takes a lot of time and effort, and if the tech is already on the way out, the odds of seeing a return on that investment of time, whether financial or just in terms of re-using that knowledge, are significantly lower.
2. Older tech involves more grunt work.  In other words, older programming technologies tend to work at a lower level.  Not always, but the trend is to increasing levels of abstraction.  That means that it will likely take more effort and/or code to do the same thing that you might get more or less for free with newer tech.
3. The problems are less fun.  This particularly applies to things like "supporting Internet Explorer", which Lauren mentions.  When you have to support both the old stuff and the new stuff, you generally have lots of problems with platform-specific quirks, things that are supposed to be compatible but really aren't, and just generally trying to work around limitations of the older tech.  These are the kind of problems that can be difficult, but not in a good way.  They're less like "build a better mousetrap" and more like "find a needle in this haystack".

So, in general, developers aren't usually super enthusiastic about working with or supporting old tech.  It's not really as bad as some people make it sound, but it's not really where most of us want to be.

Another factor is the way websites are developed.  The ideal is that you'd have somebody who is trained and experienced in designing user experiences and who is capable of considering all the use-cases and evaluating the site based on them.  That person could communicate that information to the designers and developers, who could incorporate it into their work and produce sites that are easy to use, compatible with assistive technologies, degrade gracefully when using less capable hardware or software, etc.  The reality is that this rarely happens.  In my experience:

1. Many teams (at least the ones I have experience with) have no UX designer.  If you're lucky, you'll have a graphic designer who has some knowledge or awareness of UX concerns.  More likely, it will be left up to the developers, who are typically not experts.  And if you're very unlucky, you'll have to work with a graphic designer who is fixated on pixel-perfect fidelity to the design and is completely indifferent to the user experience.
2. Most developers are on the young side.  (There are plenty of older developers out there, but the field has been growing for years and the new people coming in are almost all young.)  They're also able-bodied, so they really have any conception of the physical challenges that older people can have.  And it's hard to design for a limitation that you didn't think of and don't really understand.
3. While it's easy in principle, progressive enhancement and graceful degradation can be very tricky to actually pull off.  The main reason is that it's extremely easy to accidentally introduce a change that doesn't play well with some browser, doesn't display properly at some resolution, or what have you.
4. And let's not forget testing.  Even if you can build a site with progressive enhancement, proper accessibility, and attention to the needs of less technical users with little support available, you still need to test it.  And the more considerations, use-cases, and supported configurations you have, the more your testing space expands.  That makes it much harder and more time-consuming to make sure that all these things are actually present and working as intended for all users.

So what am I trying to say here?  I do agree with Lauren that supporting elderly users, disabled users, etc. is an important thing.  It's a thing that, as an industry, we should do.  But it's hard.  And expensive (at least compared to the way most shops work now).  That's not an excuse for not doing it - more like an explanation.

Every shop needs to find a balance between supporting a diversity of users and doing what they need to do within a reasonable budget of time and money.  While it's important to think about security and support new standards, I think that in recent years the industry has probably been a little too quick to abandon old, but still widely used technologies.  If nothing else, we should at least think more about our target user base and whether we're actually serving them or ourselves by introducing $COOL_JS_FRAMEWORK or dropping support for Internet Explorer.  I'm sure that in many (but not all) cases, dropping the old stuff is the right choice, but that shouldn't be the default assumption.

On a lighter note this weekend, here's a "funny because it's true" parody blog entry from earlier this year.  It pretty well sums up the sometimes-rampant insanity in the software industry.  You know, where literally every startup thinks they're Facebook or Google and is going to go from zero to sixty-three quadrillion hits a day, so, well, you have to be ready for that kind of scale on day one.

Of course, approximately 90% of those startups will be out of business in less than five years, but hey, you never know, right?  After all, the entire business model is "get big so you can IPO or get bought out".  Sure, you could try to build a profitable, medium-sized business, but that's basically the same thing as failure.  Besides, if you can attract venture capital, it's not your money you're blowing through anyway, so who cares?

The good news is that the industry seems to be moving away from that mindset a little bit in the last few years.  Or, at least, things seem to be moving more toward business-to-business sales, as opposed mass-market products that have no clear business plan beyond "get huge".  So maybe that's a good sign and this idea of copying every crazy/innovative thing that the big boys are doing will be a thing of the past soon.

Or maybe not.  But I can dream.

Author's note: Here's another article from the archives. This one was written way back on March 17th, 2007. At the time I was still working for the county government and using Linux on all my home computers. This rant came out of frustration at the interaction of not-fantastic bespoke software and annoying semi/anti-features in a commercial flash drive.

We don't have that in the Linux world. Most of our software is free or open-source and is installed from a single trusted distributor. We don't have to worry about whether a package is bundled with spyware, a crappy browser toolbar, or a useless systray icon.

This was brought home to me today when I had to deal with a user's flash drive at work. His department has a number of laptops that can't connect to our network, but this crappy application that the state foisted on us requires them to move data from back and forth between our file server and the laptops. The easy solution to this is just to use flash drives.

Well, this user's flash drive broke and the department didn't have any spares, so he went out and bought a 1GB Cruzer Micro. Nice enough drive, plenty of capacity for his purposes. The only problem was, it didn't work with The Crappy Software Foisted On Us By The State (TCSFOUBTS, pronounced 'ticks-fouts').

You see, TCSFOUBTS is one of those half-baked, government consultingware type of systems. It has a bazillion features and is highly customizable. On the down side, it's slow as all hell, has a terrible user interface, and is difficult to configure. For example, some customization options require manually editing MS Access database tables and quite a number involve hand-editing INI files or other text files.

Now, having planned ahead, the TCSFOUBTS people built in a feature to let users automatically copy their data onto a removable drive with just a couple of clicks. However, by "removable drive," what they really meant is "a path." For this feature to work, you have to configure TCSFOUBTS with the drive letter for your removable device.

By now, you probably see where I'm going with this. TCSFOUBTS needs the drive letter, which has to be configured by the system administrator. However, unlike the old flash drive, the Cruzer Micro has a bootable partition that displays as a CD-ROM drive. (Note from the present: this was before booting from USB was a common thing.) Of course, that's the first partition, and so get the drive letter where TCSFOUBTS wants to copy its files.

What was on that first partition? I don't remember. Some useless utility software that nobody asked for. One of those "value-adds" that they put on flash drives because they want to use the space for something. But it meant that the drive was stuck with an unusable partition that I didn't immediately have the ability to remove. So for the user to use this drive, they would have to hand-edit some INI file, and then edit it back when they got a different drive. And since the users for this system were all non-technical, that sounded less than appealing.

Eventually the user decided to just not bother and requisition an "official" drive. It was just a frustrating series of events. It goes to show how a few misplaced assumptions can completely mess up a user experience.

Author's note: This is based on some notes and links I started collecting in November of 2015.  The bulk of the commentary is actually relatively new - from last year, August 15th, 2019 - so it's not too out of line with my current thinking.

As might be clear from my review of Bertrand Meyer's Agile!: The Good, the Hype, and the Ugly, I've been rethinking the whole agile development craze that has swept the industry.

There are a number of good presentations online questioning the "agile" movement.  For a more provocative point of view, I recommend Erik Miejer's One Hacker Way talk.  Dave Thomas, one of the "pragmatic programmers", also has a good talk on the topic. There's also a good one by Fred George on the hidden assumptions of agile.

My current thought (note: we're back to 2019 now) is that "agile" has become pretty much a meaningless buzz word.  Pretty much everybody is "doing agile" now - or at least claiming they do.  It's come to mean "anything that's not waterfall".  And we all know that "waterfall" doesn't work, which is why everyone is doing "agile".  (Side note: Winston Royce, in his paper that initially described the waterfall process, actually says that it doesn't really work.  But, of course, that didn't stop people from trying.)

Not that agility is a bad concept.  It isn't - being flexible is a good thing.  Responding to change is almost a requirement in most shops.  The values in the Agile Manifesto are all good things to emphasize.  But none of that amounts to a process.  It's just a loose collection of principles and ideas that are useful, but aren't a road-map for how to build software.

That's why, in practice, most shops that "do agile" are using some variation on Scrum.  And while I have no problem with Scrum per se, it's hardly the be-all and end-all of development processes.  In fact, the main problem with Scrum is probably that it's not a software development process - it's more of a project management framework.  It doesn't have a whole lot to say about the details of how to code, how to test, how to manage defects and other quality issues, how to manage releases, etc.  It's up to each shop to figure that stuff out for themselves.

Of course, that's not bad.  Every business is different and you should expect that you'll have to adapt any process to a certain extent.  Scrum is useful in that it gives you a framework for tracking what needs to be done and creating a feedback loop to improve your process.  But you still have to actually use that feedback loop to improve your process, i.e. you have to do the hard work of self-improvement.  Simply going through the motions of what the "agile consultant" says you should do it's going to cut it.  As with everything else in life, there are no shortcuts.

Author's note: Here's another little rant that's been sitting in my drafts folder for years. Twelve years, to be precise - I created this on March 28, 2007. That was toward the end of my "government IT guy" days.

I'd forgotten how much of a pain the internet filtering was. These days, I hardly think about it. The government job was the last time I worked anyplace that even tried to filter the web. And e-mail filtering hasn't been something I've worried about in a long time either. These days, the filtering is more likely to be too lax than anything else. And if something does get incorrectly filtered, you generally just go to your junk mail folder to find it. No need for the rigamarole of going back and forth with the IT people. It's nice to know that at least some things get better.

I'm really starting to hate spam filters. Specifically, our spam filters at work. And our web filters. In fact, pretty much all the filters we have here. Even the water filters suck. (Actually, I don't think there are any water filters, which, if you'd tasted the municipal water, you would agree is a problem.)

I asked a vendor to send me a quote last week. I didn't get it, so I called and asked him to send it again. I checked with one of our network people, and she tells me it apparently didn't get through our first level of filters. So she white-listed the sender's domain and I asked the guy to send it again. It still didn't get through.

As I've mentioned before, our web filters also block Wikipedia.

Strong Opinions Loosely Held Might be the Worst Idea in Tech.  I think the name of that article pretty much says it all.

This is essentially something I've thought since I heard about he concept of "strong opinions loosely held".  I can see how it could work in certain specific cases, or be mindfully applied as a technique for refining particular ideas.  However, that only works when everyone in the conversation agrees that that's the game they're playing, and that's not usually how I've seen the principle presented anyway.  Rather, it's usually described in terms more like "be confident in your opinions, but change your mind if you're wrong."  And that's fine, as far as it goes.  But it's far from clear that this works out well as a general principle.

To me, "strong opinions loosely held" always seemed kind of like an excuse to be an jerk.  Fight tooth and nail for your way until someone and if someone proves you wrong, oh well, you weren't that attached to the idea anyway.  It seems to fly in the face of the Hume's dictum that "a wise man proportions his belief to the evidence."  Why fight for an idea you don't care that much about?  If you're not sure you're right, why not just weigh the pros and cons of your idea with other options?

I suppose the thing that bothers me the most about it is that "strong opinions loosely held" just glorifies the Toxic Certainty Syndrome, as the article's author calls it, which already permeates the entire tech industry.  Too often, discussions turn into a game of "who's the smartest person in the room?"  Because, naturally, in a discussion between logical, intelligent STEM nerds, the best idea will obviously be the one that comes out on top (or so the self-serving narrative goes).  But in reality, nerds are just like any group, and getting people to do things your way is orthogonal to actually having good ideas.  So these conversations just as often degrade into "who's the biggest loud-mouth jerk i the room?"

I'm not sure how I feel about the article's specific "solution" to preface your assertions with a confidence level, but I do empathize with the idea.  My own approach is usually to just follow the "don't be a jerk" rule.  In other words, don't push hard for something you don't really believe in or aren't sure about, don't act more sure about a position than you are, and be honest about how much evidence or conviction you actually have.  It's like I learned in my Philosophy 101 class in college - to get to the truth, you should practice the principles of honesty and charity in argument.  Our industry already has enough toxic behavior as it is.  Don't make it worse by contributing to Toxic Certainty Syndrome.

This XKCD comic pretty much sums up my reaction every time someone mentions electronic voting.  I usually explain it with an analogy involving Scotch tape and bubble gum, but same idea.

As a side-note, there's an interesting response to this here.  The criticism, essentially, is that the comic is comparing apples to oranges.  For aircraft and elevators we're mostly concerned about accidental failures, whereas for voting machines the issue is protecting against intentional attacks.  So planes and elevators are only considered "safe" because we're not counting "being blown up" as a valid scenario that they need to defend against.

That's a fair criticism, but that's not really the point. When I hear regular, non-techie people talking about "computerize voting", they're not interested in the electronic replacements for old mechanical voting machines.  They're interested in voting online, like you'd vote in poll on Facebook.  That's a very different problem than securing a computerized voting machine, and much harder to solve.

The other day I listened to a very interesting talk by Allen Holub on the #NoEstimates movement, which I've posted about before.  It's definitely worth a look if you have some time.  Mr. Holub makes the slightly controversial claim that all estimates are evil and we should all just stop doing them immediately.  His rationale for this is essentially that estimates are made up and harmful.  And, of course, he's not wrong.

I've always been a little skeptical of the #NoEstimates movement, but the thing it clearly get right is that estimates are frequently misused and misapplied.  Just about every developer has a story about estimates being interpreted as promises, or having to work late just to "meet your sprint commitments", or skimping on quality because the sprint is about to end and you need to close out your tasks.  And we won't even get into managers who try to use estimates as a performance evaluation tool - the dysfunction of this should be obvious to anyone with any experience.

Is it the estimates?

So estimates can be used for bad things.  That's not controversial.  But that's also not a problem with estimation per se.  I guess you could think of it as the software equivalent of "guns don't kill people, people kill people."  It's not that estimates are inherently bad, it's just that they're seldom used for good.  That doesn't mean that you should abolish estimates, but it does mean that you should use them with caution.

The thing about estimates is that, while they're often just guesses, they don't have to be.  In my study of the Personal Software Process (PSP), I learned about doing estimates based on historical data.  I've even seen it work - in my last job, I reached the point where my time estimates were within about 15% of actual task completion time, which I consider to be pretty good given that the estimates were in minutes.  Granted, it doesn't always work - sometimes you have tasks that are truly unprecedented or have too many unknowns - but it can be done.

The main problem with data-based estimation is that most organizations and developers are not equipped to do it.  For one thing, it requires data, and a lot of shops don't actually collect the kind of data that would be useful.  Most developers certainly don't collect the kind of data that the PSP estimation method uses.  It also takes more time than many people seem to want to devote to estimates.  It's not that much time, but still a lot longer than saying, "well, I guess that's about eight story points".

What's the point?

The real question here is: why do you want an estimate?  What are you trying to accomplish?

When I do PSP estimates, I'm using them to gauge my performance.  I'm looking at my historical data and using that to project out time, code volume, and defect counts.  I can compare these projections to my actual results, see if my performance is consistent, and if not, analyze the reasons and look for ways to improve in the future.

Yes, these are "estimates", but not in the same sense as most teams do them.  My manager never sees these "estimates"; I make no commitments based on them; nobody depends on their being accurate; in fact, nobody but me even knows what they are.  They're a part of my task planning, so they're short-term estimates - they go out two or three weeks at most, usually more like a few days.  I start with some requirements analysis, sketch out a conceptual design, and then do the estimate based on that.  So the process of coming up with the estimate actually helps me flesh out and validate my plan, which is valuable in and of itself.

The important thing to note in the above description is that my estimates are not targets.  If I'm working on a task and it takes 500% longer than my estimate, I'm the only one who knows and I don't care.  Sure, I'll try to figure out why that task took so much longer than I thought, but the fact that "I missed my estimates" has no relevance and no consequences.

But that's usually not how it works when your boss or your project manager (or even your scrum master) asks you for an estimate.  They want you to tell them when the task will be done.  And they'll be very disappointed if it's not done when you said it would be.  And that's where the dysfunction begins....  Because even in the best of circumstances, we don't really know exactly how long s development task will take.  And we seldom work in the best of circumstances.

So...no estimates?

The interesting thing about doing PSP estimates is that, until relatively recently, I had trouble not thinking of the estimates as targets.  Process Dashboard, my PSP support tool, has a progress bar that counts down the time remaining for your estimate and then turns red when you go over the estimate.  And I would actually get upset when that bar turned red!  Why?!?  There was no reason for that - there was literally nothing at stake!  I was just psychologically stuck in the wrong mindset.  And that's the same mindset that most people get stuck in when they talk about estimates.

So I'm starting to come around to this "no estimates" thing.  I still believe that estimates can be useful and productive, but they usually aren't.  And if your company is particularly enlightened and can do estimates in a way that's productive and avoids the pathological effects associated with them, then by all means, keep doing estimates.  But most people don't work in companies like that.  And even if your company is great, you still need to be careful of the psychological pressure created by the inherent biases and preconceptions of your developers.

In other words, estimate carefully.  Consider simpler, lower precision (but not necessarily lower accuracy) methods like Mr. Holub and the other #NoEstimates advocates describe.  Don't be fooled - those are still estimates, but they at least make an effort to work around the damaging side-effects associated with traditional estimation.  After all, it doesn't matter how good the estimates are if they destroy team morale.

Welcome to part three of my PSP series.   Now that the introductory material is out of the way, it's time to get to the good stuff!  This time, I'll discuss the details of the PSP and what you can actually get out of it.  Theoretically, that is.  The final post in this series will discuss my observations, results, and conclusions.

PSP Phases

As I alluded to in a previous post, there are several PSP phases that you go through as part of the learning process.  Levels 0 and 0.1 get you used to using a defined and measured process; levels 1 and 1.1 teach planning and estimation; and levels 2 and 2.1 focus on quality management and design.  There is also a "legacy" PSP 3 level which introduces a cyclical development process, but that's not covered in the book I used (though there is a template for it in Process Dashboard).  The phases are progressive in terms or process maturity.  So PSP 0 defines your baseline process and by the time you get to PSP 2.1 you're working with an industrial-strength process. 

For purposes of this post, my discussion will be at the level of the PSP 2.1.  Of course, there's no rule that says you can't use a lower level, but the book undoubtedly pushes you toward the higher ones.  In addition, while the out-of-the-box PSP 2.1 is probably too heavy for most people's taste, there is definitely useful material there that you can adapt to your needs.

Estimation

One of the big selling points for all that data collection that I talked about in part 1 is to use it for estimation.  The PSP uses an evidence-based estimation technique called Proxy-Base Estimation, or PROBE.  The idea is that by doing statistical analysis of past projects, you can project the size and duration of the current project.

The gist of the technique is that you create a "conceptual design" of the system you're building and define proxies for that functionality.  The conceptual design might just be a list of the proxies such that, "if I had these, I would know how to build the system."  A proxy is defined by its type/category, it's general size (from very small to very large), and how many items it will contain.  In general, you can use anything as a proxy, but in object-oriented languages, the most obvious proxy is a class.  So, for example, you might define a proxy by saying, "I'll need class X, which will be a medium-sized I/O class and will need methods for A, B, and C." 

By using historical project data, you can create relative size tables, i.e. tables that tell you how many lines of code a typical proxy should have.  So in the example above, you would be able to look up that a medium-sized I/O class has, on average, 15.2 lines of code per method, which means your class X will have about 46 lines of code.  You can repeat that process for all the proxies defined in your conceptual design to project the total system size.  Once you have the total estimated size, PROBE uses linear regression to determine the total development time for the system.  PROBE allows for several different ways to do the regression, depending on how much historical data you have and how good the correlation is.

Planning

As a complement to estimation, the PSP also shows you how to use that data to do project planning.  You can use your historical time data to estimate your actual hours-on-task per day and then use that to derive a schedule so that you can estimate exactly when your project will be done.  It also allows you to track progress towards completion using earned value.

Quality Management

For the higher PSP levels, the focus is on quality management.  That, in and of itself, is a concept worth thinking about, i.e. that quality is something that can be managed.  All developers are familiar with assuring quality through testing, but that is by no means the only method available.  The PSP goes into detail on other methods and also analyzes their efficiency compared to testing.

The main method espoused by the PSP for improving quality is review.  The PSP 2.1 calls for both a design review and a code review.  These are both guided by a customized checklist.  The idea is that you craft custom review checklists based on the kinds of errors you tend to make, and then review your design and code for each of those items in turn.  This typically means that you're making several passes through your work, which means you have that many opportunities to spot errors.

The review checklists are created based on an analysis of your defect data.  Since the PSP has you capture the defect type and injection phase for each defect you find, it is relatively easy to look at your data and figure out what kind of defects you typically introduce in code and design.  You can use that to prioritize the most "expensive" defect areas and develop review items to try to catch them.

As part of design review, the PSP also advocates using standard design formats and using organized design verification methods.  The book proposes a standard format and describes several verification methods.  Using these can help standardize your review process and more easily uncover errors.

Process Measurement

Another big win of the PSP is that it allows you to be objective about changes to your personal process.  Because you're capturing detailed data on time, size, and defects, you have a basis for before and after comparisons when adopting new practices. 

So, for instance, let's say you want to start doing TDD.  Does it really result it more reliable code?  Since you're using the PSP, you can measure whether or not it works for you.  You already have historical time, size, and defect data on your old process, so all you need to do is implement your new TDD-based process and keep measuring those things.  When you have sufficient data on the new process, you can look at the results and determine whether there's been any measurable improvement since you adopted TDD.

The same applies to any other possible change in your process.  You can leverage the data you collect as part of the PSP to analyze the effectiveness of a change.  So no more guessing or following the trends - you have a way to know if a process works for you.

Honesty

One of the least touted, but (at least for me) most advantageous aspects of using the PSP is simply that it keeps you honest.  You use a process support tool, configured with a defined series of steps, each of which has a script with defined entry and exit criteria.  It gives you a standard place in your process to check yourself.  That makes it harder to bypass the process by, say, skimping on testing or glossing over review.  It gives you a reminder that you need to do X, and if you don't want to do it, you have to make a conscious choice not to do it.  If you have a tendency to rush through the boring things, then this is a very good thing.

Next Up

So that's a list of some of the reasons to use the PSP.  There's lots of good stuff there.  Some of it you can use out-of-the-box, other parts of it offer some inspiration but will probably require customization for you to take advantage of.  Either way, I think it's at least worth learning about.

In the next and final installment in this series, I'll discuss my experiences so far using the PSP.  I'll tell you what worked for me, what didn't, and what to look out for.  I'll also give you some perspective on how the PSP fits in with the kind of work I do, which I suspect is very different from what Humphrey was envisioning when he wrote the PSP book.

This is part two of my evaluation of the Personal Software Process.  This time I'll be talking about the actual process of learning the PSP.  In case you haven't figured it out yet, it's not as simple as just reading the book.

Learning the PSP

The PSP is not the easiest thing to learn on your own.  It's a very different mindset than learning a new programming language or framework.  It's more of a "meta" thing.  It's about understanding your process: focusing not on what you're doing when you work, but rather how you're doing it.  It's also significantly less cut-and-dried than purely technical topics - not because the material is unclear, but simply because what constitutes the "best" process is inherently relative.

Given that, I suspect the best way to learn the PSP is though the SEI's two-part training course.  However, I did not do that.  Why not?  Because that two-part course takes two weeks and costs $6000.  If you can get your employer to give you the time and shell out the fee, then that would probably be great.  But in my case, that just wasn't gonna happen and the SAF (spouse acceptance factor) on that was too low to be viable.

Instead, I went the "teach yourself" route using the self-study materials from the SEI's TSP/PSP page.  You have to fill out a form with your contact information, but it's otherwise free.  These are essentially the materials from the SEI course - lecture slides, assignment kits, and other supplementary information.  It's designed to go along with the book, PSP: A Self-Improvement Process for Software Engineers, which serves as the basis for the course.  Thus my learning process was simply to read through the book and do the exercises as I went.

(As a side-note, remember that this is basically a college text-book, which means that it's not cheap.  Even the Kindle version is over $40.  I recommend just getting a used hardcover copy through Amazon.  Good quality ones can be had for around $25.)

Fair warning: the PSP course requires a meaningful investment of time.  There are a total of ten exercises - eight programming projects and two written reports (yes, I did those too, even though nobody else read them).  Apparently the course is designed around each day being half lecture, half lab, so you can count on the exercises taking in the area of four hours a piece, possibly much more.  So right there you're looking at a full work-week worth of exercises in addition to the time spent reading the book.

Personally, I spent a grand total of 55 hours on the exercises: 40 on the programming ones and 15 on the two reports (for the final report I analyzed not only my PSP project data, but data for some work projects I had done using the PSP).  While the earlier exercises were fairly straight-forward, I went catastrophically over my estimates on a couple of the later ones.  This was largely due partly to my misunderstanding of the assignment, and partly to confusion regarding parts of the process, both of which could easily have been averted if I'd had access to an instructor to answer questions.

Tools

 As I mentioned in the last post, you'll almost certainly want a support tool, even when you're just learning the PSP.  Again, there's a lot of information to track, and trying to do it on spreadsheets or (God forbid) paper is going to be very tedious.  Halfway decent tool support makes it manageable.

I ended up using Process Dashboard, because that seems to be the main (only?) open-source option. In fact, I don't think I even came across any other free options in my searches.  I understand other tools exist, but apparently they're not public, no longer supported, or just plain unpopular. 

One of the nice things that Process Dashboard offers is the ability to import canned scripts based on the PSP levels in the book.  To use that, you have to register with the SEI, just like when you download the PSP materials, which is annoying but not a big deal.  (The author got permission from the SEI to use their copyrighted PSP material and apparently that was their price.)  This is really handy for the learning process because it puts all the scripts right at your fingertips and handles all of the calculations for you at each of the different levels.

In terms of capabilities, Process Dashboard is actually pretty powerful.  The UI is extremely minimal - essentially just a status bar with some buttons to access scripts, log defects, pause the timer, and change phases.  Much of the interesting stuff happens in a web browser.  Process Dashboard runs Jetty, which means that it includes a number of web-based forms and reports that do most of the heavy lifting.  This includes generating estimates, entering size data, and displaying stock and ad hoc reports.

Process Dashboard has fairly extensive customization support, which is good, because one of the basic premises of the PSP is that you're going to need to customize it.  You can customize all the process scripts and web pages, the project plan summary report, the built-in line counter settings, the defect categories, etc.  And that's all great.  The one down side is that the configuration is usually done by editing XML files rather than using a graphical tool. 

Since this is an open-source developer tool, I guess that's sort of fine.  And at least the documentation is good.  But it's important to realize that you will need to spend some time reading the documentation.  It's a powerful tool and probably has just about everything you need, but it's not always easy to use.  On the up side, it's the sort of thing that you can do once (or occasionally) and not have to worry about again.  Just don't expect that you'll be able to fire up Process Dashboard from scratch and have all the grunt work just done for you.  There's still some learning curve and some work to do.  But if you end up using the PSP, it's worth it.

As I mentioned in a couple of previous posts, I've been studying the PSP.  As promised, this is my first report on it.  After several failed attempts to write a single summary of the process, I've decided to divide my assessment up into a series of posts.  This first entry will be a basic overview of what the PSP actually is and how it's intended to work.

What is the PSP?

PSP stands for Personal Software Process.  It was developed by Watts Humphrey of the Software Engineering Institute (SEI) at Carnegie-Mellon University (which you may know as the home of CERT). Humphrey is also the guy who came up with the Capability Maturity Model, which is the CMM in CMMI.  His intent was to take the industrial-strength process methodologies he worked on for the CMM and scale them down to something that could be useful to an individual developer.  The result is the PSP.

Despite the name, at its core the PSP isn't actually a development process itself.  Rather, it is a process improvement process.  To put it another way, the goal is not to tell you how to develop software, but rather to give you the intellectual tools to figure out the development process that works best for you and help you improve that process to make yourself more effective.

Now, to be clear, Humphrey actually does describe a particular development process in his PSP books.  And yes, that process is kind of "waterfally" and extremely heavy.  And if you look for PSP information online, you'll no doubt find a lot of people who take issue with that process.  And they're right to.

But try not to get hung up on that.

As Humphrey puts it, the process he describes is his process.  That doesn't mean it should be your process.  The process he describes is for pedagogical purposes as much as anything else.  It is intended to demonstrate the kinds of methods that apply to large-scale industrial systems, so that you have those tools in your toolbox if you need them.  If this process doesn't really work for you (and it almost certainly won't), then you should modify it or replace it with something that will work for you.

This is something that bears dwelling on.  Although pretty much all of the PSP focuses on Humphrey's process, he notes several times that it is important to pick a process that's sustainable for you.  To put it another way, it's important to be realistic.  Sure, his process might be objectively "better" by some measure than whatever you're doing, but that doesn't matter if you aren't going to follow his process.  Maybe his process conflicts with your company's process; maybe it's not effective for the kind of project you usually do; maybe you just plain don't have the will power to stick to it.  It doesn't matter.  The process you define needs to be in line with what you actually do, otherwise it's not going to help you.  So while you can learn from Humphrey's process, don't take it as gospel and don't think "this is what I need to be doing."

For me, the real take-away of studying the PSP is to be mindful of your development process and constantly try to improve it.  The goal is simple: quality and consistency.  Monitor your process and tune it such that you can consistently build a quality product in a predictable time-frame.  Ground yourself in reality and be guided by the data, not the latest fads.

The PSP Approach

The PSP is all about data-driven process changes.  As you work, you collect data on what you produce. The general idea is that you analyze that data and make changes to your process based on that analysis.  You then repeat this process, evaluating the results of your changes and making further tweaks as necessary.  So it's basically a plan-do-check-act cycle.  This is the same sort of thing that agile methods advocate (e.g. Scrum retrospectives).

Project Structure

The PSP structures a task or project into phases.  The exact phases will obviously depend on your particular process.  The "standard" (i.e. from Humphrey's process) phases are Planning, Development, and Postmortem.  In the Planning phase, you define what is to be built and construct estimates of the effort involved.  The Development phase is where you actually construct the software.  In the standard PSP process, this is divided into a number of sub-phases, depending on which level of PSP you're using.  Lastly, the Postmortem phase is where you record your final process measurements and document any problems and potential process improvements that came up in the course of the project.

The PSP levels I mentioned above are essentially maturity phases.  In the process of learning the PSP, you go through six of them: PSP0, PSP0.1, PSP1, PSP1.1, PSP2, and PSP2.1.  You start with PSP0, which is basically just "whatever you do now, but with measurements" and progress up to PSP2.1, which includes five sub-phases: Design (with standard design templates), Design Review, Code, Code Review, and Test.  I'll analyze this structure and the contents of the phases in a later post. 

Data Collection

Data collection in the PSP happens at a very fine-grained level.  The idea is that you do light-weight data capture as you work in order to minimize inaccuracies.  Data is collected on three axes: how much time is spent in each phase of the project, size of the product, and what defects were found.  In the standard process, you would track how many minutes you spend in each phase or sub-phase, how many lines of code you add or modify, and the number and type of bugs you find in in each phase.

The general idea behind this is simply that the more detailed the data, the more opportunities you have to use it later on.  The PSP suggests a number of ways to use this data to drive and measure process change.  Two of the most visible are the use of historical size and time data to drive the estimation process and the use of defect phase, type, and time data to measure the efficiency of defect removal activities such as code review and testing.

Interestingly, when Humphrey started developing and teaching the PSP, students tracked this stuff on paper.  He later introduced spreadsheets to track it, which is better, but still a little clumsy.  These days, there are PSP support tools you can use.  The main open-source one seems to be Process Dashboard, which I used from day one.  It will help you track PSP data and do most of the calculations you need for data analysis.  Frankly, I can't imagine trying to track this by hand or with Excel spreadsheets - it just seems like it would be painfully tedious.  But with decent tool support its really not that bad.

Quality Management

One of the interesting things to note about the standard PSP process is that it explicitly takes a position on quality management.  This is something you don't tend to hear a lot about - in fact, it might not even have occurred to you that quality is something you can manage.

The position Humphrey takes is simple: the right way is always the fastest way.  And the right way is to get things right the first time to the maximum extent feasible.  That means finding defects as early as possible and taking steps to stop them from being injected in the first place.

I'll get into the details of this in a later post.  I mention it now simply Humphrey devotes a lot of ink to it.  In fact, much of the new material in the higher PSP levels is devoted to quality control.  Quality management is one of the big uses of the defect data that you track as part of the PSP, so it's important to realize at the outset that this is a core part of the process.

Should I Care?

I think you should.  You may not want to actually use the PSP, but it's still an interesting thing to learn about.  If you're like me, you probably never thought about how you build software in terms of hard data, so it's interesting to see how it can be done.  It's a very different and empowering way of understanding how you work - it makes programming feel more like a science and less like a black art.  So even if you don't want to go all the way and actually try to use the PSP, it's enlightening to read about the various statistical analysis techniques and quality improvement recommendations.  If nothing else, it broadens your perspective and gives you more intellectual tools to draw on.

In part two, I'll talk a little about what I did to learn about the PSP, including the resources and tools that I used.

So as I mentioned in a previous post, I've been taking a look at the Personal Software Process, or PSP (not to be confused with Sony's portable gaming console, which makes Googling for it loads of fun).  It's been an eye-opening experience and I'll write more about it once I've officially finished the course.

At the same time, I've been reading about the #NoEstimates hashtag/movement/discussion/whatever-the-hell-it-is.  I actually first heard about it around the time I was starting the PSP book.  This makes for an interesting contrast, because one of the first things you learn about when studying the PSP is...how to do estimates.

The thing is, up until a few months ago, I would have completely agreed with the #NoEstimates people.  In fact, I wrote about my feelings on estimates a couple of years ago.  My feelings were pretty much in line with Woody Zuill's comments in .NET Rocks! episode 1160 - that estimates are useless because they're just made-up, grossly inaccurate numbers which can only be used for evil or to make even more inaccurate plans.  So why bother?  Just decide if something is important enough to do, and then either do it or not.  Stop pretending.

Then I started learning about the PSP.  I started accumulating data on my actual productivity and learning how to leverage it.  And you know what?  It turns out it actually is possible to do estimates with a reasonable degree of accuracy - and it doesn't even take huge amounts of up-front planning.  Granted, there are caveats - you need historical data that's representative of the technology and problem domain you're working in.  But once you've got a few projects under your belt to establish a baseline, it really does work.

So now I have some cognitive dissonance.  Was I wrong to take the time to learn this estimating stuff?  Or was I wrong before when I agreed with the #NoEstimates people?  Where do I go with this?

What got me thinking about this is an article by Henri Karhatsu in Methods and Tools.  Towards the end of the article, he discusses the #NoEstimates alternative to providing an estimate: forecasting.  This is what you use when you really do need to figure out when something will be done (e.g. so that you can prepare marketing materials for a new product or feature).  While this may sound like estimating, according to Karhatsu, there is a key difference: forecasts are based on data, while estimates are not.

I had several reactions to this revelation.  The first was, "Well, that's a crock - 'forecasting' is just estimating by a different name."  Then I realized, "Hey, 'forecasting' is exactly what you're doing in the PSP!  Cognitive dissonance resolved!"  Which prompted the question, "If PSP 'estimating' is the same thing as #NoEstimates 'forecasting', then what the heck does #NoEstimates even mean?"

And that's when I read this article analyzing the #NoEstimates movement.  It's an interesting read and pretty well summarizes my impressions of #NoEstimates based on what I've heard and read.  The short version is that #NoEstimates doesn't actually mean "no estimates ever".  It doesn't even mean that estimates are inherently bad in all cases.  Rather, it seems to be a reaction against people using estimates in stupid an fruitless ways.  This came up a number of times in the .NET Rocks! interview with Woody Zuill.  You have managers using estimates as a weapon against developers, or clueless project managers demanding estimates because they need to plug the numbers into Microsoft Project so they can finish their plan, even though nobody really thinks the plan reflects reality.  This ends in the demoralizing and unproductive result of developers fabricating numbers, without any clue if they're even vaguely realistic, for no other purpose than to keep the managers happy.

I don't think anybody could reasonably argue that such a situation is a good thing.

But on the other hand, when you're being paid to do a project, whether it's building software or building a house, the questions "How long is it going to take?" and "How much is it going to cost?" are not inherently unreasonable things for the customer to ask.  And while it's sometimes true that you're working on an unprecedented project an have no clue how long it will take, it's more often true that you're on reasonably familiar ground and have a pretty good idea of the effort involved in a project.  And let's be honest - while a big-deal consultant like Woody Zuill might be able to get away with telling the customer that they don't really need an estimate, that's not going to fly in every organization.  So #NoEstimates as a general recommendation is probably not realistic.

And that brings me back around to the first chapter of the PSP book.  In it, Watts Humphrey relates a story from one of his first TSP teams (that's "Team Software Process" - the team equivalent of the PSP).  They had just come off a two-year death-march project and now management had a new project for them - with a nine-month timeline.  The dev team was sure they couldn't make that schedule.  They didn't really know how long the project would take, but figured it would probably be closer to two years again. 

So what normally happens in that situation?  Well, the dev team says that the schedule is impossible to meet, management insists that the date is firm, and the developers eventually cave and say they'll "do their best".  I've been in that situation, as I'm sure a lot of other people have.  Why do we do that?  Because, well, what else can we do?  We don't really know how long the project will take, and even if we have an idea, we usually don't have anything concrete to back up our estimate.  And as Humphrey puts it, when we don't know and management doesn't know, they'll win every time.

But rather than go the #NoEstimates route and try to convince people that coming up with an estimate isn't a useful exercise, Humphrey goes the opposite direction.  He uses the power of data to give estimates teeth. Remember - even if management wants the new project done yesterday, it's still important for them to know when it's really going to be done.  And while it's easy to write off a guess-work estimate as "schedule padding", it's not as simple when you have a detailed plan based on historical data.  When you reach that point, estimates can become a weapon for the dev team rather than one used against them.

So where does this leave me?  Pretty much back where I started.  Estimates are good when they're needed and accurate.  They're bad when they're unneeded or inaccurate.  They should be based on data, not guess-work, and should be used responsibly.  So no earth-shattering news.  Just an interesting detour.

As I mentioned in my a previous post, I've been doing some reading around process.  A development process is one of those things that you always have, by definition.  It might be ill-defined and chaotic, and you might not even thingk about it, but eventually you go through some process to develop code.  But when you think about it, this process is going to have a not insubstantial impact on the quality of the software you produce.  After all, how could it not?

However, as developers, we often think of "process" as a dirty word.  It's one of those bureaucratic things imposed on us by pointy-haired project managers, with their Gantt charts and their Microsoft Project.  Sure, those highfalutin "processes" sold by over-priced consultants might be useful for those huge, soulless "enterprise" shops that have thousands of know-nothing drones cranking out Java ports of mainframe COBOL systems.  But we don't need those!  We're "code poets", "10x developers", "coding ninjas", etc.  We can code circles around a dozen of those enterprisey jokers in our sleep! 

Except it doesn't really work that way.  Even if you accept the notion that the best developers are ten or more times more effective than average ones, they still have to deal with the complications of ill-defined features, requirements changes, integration problems, performance problems, communication between stake-holders, and the myriad other issues that emerge and magnify as more people become involved in a project.  If you expect to make it through any decent sized project and not have it turn into a hot mess, you have to have some way of managing all the things that come up. Some organizations, such as the aforementioned Java-drone shop, do that by implementing processes that require mountains of paperwork that serve primarily to cover everyone's butt.  Others take the agile approach.

If you've been living in a cave for the past decade or so, the "agile" movement is a response to the stereotypical "big, up-front" processes of old.  The values of the agile community are summed up in the Agile Manifesto, but the basic idea is to focus less on the bureaucratic paper-pushing and more on delivering what the customer needs.  This generally sounds good to developers because it means less time spent on TPS reports and more time actually building software.  And it sounds good to managers because the iterative nature of agile development means you get the software faster (for certain definitions of "faster").  And it sounds very good to consultants who can offer agile training and certification.  So, naturally, everyone is talking about it.

That's where Bertrand Meyer's book Agile!: The Good, the Hype, and the Ugly comes in.  Its stated purpose is to separate the wheat from the chaff; that is, to pick out what's good about agile methods and what can be safely disregarded.  Meyer says it is intended as an even-handed and fact-based assessment to offset the hyperbole that often surrounds discussions of agile.

Part of what attracted me to this book was simply the fact that it's by Bertrand Meyer.  I first became aware of his work when researching for my Masters thesis.  He's probably best known as the originator of the idea of "design by contract" and Eiffel, and has a strong footing in formal methods, which is not usually the type of thing you'd associate with agile.  In fact, some people might consider it the opposite of agile.

Overall, I found the book not only enlightening, but also highly entertaining.  Meyer writes in an informal and even playful style.  While he does provide plenty of evidence and citations, he is also not light on the sarcasm - he's not shy about calling ideas stupid and harmful if that's what he thinks.  Sadly, this last point somewhat undercuts his claim to impartiality.  The "opinion" sections are helpfully marked with a marginal icon and they're just all over the place.  And while I don't think that Meyer has any anti-agile ax to grind, I was definitely left with the impression that the preponderance of his commentary was critical of agile.

Now, to be fair, criticizing agile is not a bad thing - every movement needs critics to keep it honest.  If the agile advocates you find on the web are any indication, much of the material out there is very "rah, rah agile" and this book is a bit of a reality-check.  While Meyer does give the agile movement credit for a number of good and even brilliant ideas, advocating for the greatness of agile is not the point.  The important thing is to critically examine it and not just take the press releases at face value.

One of the nice features of this book's structure is that it has book-ended assessments.  That is, Meyer ends the first chapter with an initial assessment of agile and then finishes the book with his final assessment of the good, hype, and ugly.  Though only a couple of pages, I found the initial assessment actually quite good as far as putting things in perspective.  He begins with this quote (apocryphally attributed to Samuel Johnson):

Your work, Sir, is both new and good, but what is new is not good and what is good is not new.

To me, this sets the stage, highlighting that, regardless of the hype, not all of the "innovations" in agile are actually new.  In fact, some have been around in different forms for years or even decades.  Of course, not all the good ideas are old and not all the new ideas are bad, but the point is that agile isn't quite as ground-breaking as some of the hype would have you believe. 

I won't get into the details of Meyer's analysis, but there are a few interesting items I'd like to point out.   First is his point that agile proponents (and many others in the industry) tend to work on the assumption that there are only two development models: agile and waterfall.  They use "waterfall" as a synonym for "anything that's not agile," which is both inaccurate and unfair (page 31).  Innaccurate because  "waterfall" is a specific lifecycle model proposed by Royce in the 1970's.  Unfair because there are many so-called "predictive" lifecycle models which are not the same as waterfall.  For example, the idea of iterative development has been around for a long time, except under other names like "spiral model".

Related to this, Meyer makes the interesting point that not all of the predictive processes out there actually preclude agile.  His big example of this is CMMI (page 44), which many (most?) of us think of as a super-heavy process that's only used by government defense contractors, and only because the government requires it.  However, he argues, if you really look at what CMMI is, there's no inherent contradiction with agile.  Of course, the marriage of the two is far from a no-brainer, but it is possible, as evidenced by a report on a Scrum team that was certified at CMMI level 5.

This is just part of what seems to be Meyer's biggest problem with agile methods - the deprecation of "big, up-front" tasks, which seems to be generalized to all up-front tasks.  He seems this as emblematic of a tendency in the agile movement to take a good idea and take it to the extreme.  It's invariably true that you can't get all the requirements right the first time, that the initial archtecture won't be perfect, that the initial plan won't work out perfectly, etc.  Nobody's perfect and we can't know everything - that's just life.  But does that really mean that we should just not plan out any of that out ahead of time at all?  Isn't that just as good of an argument for taking some more time to figure out a better plan up front?

My own experience the last few years has led me increasingly to sympathize with Meyer's views on this.  Like him, there are a lot of agile practices I think are great and helpful.  Short, time-boxed iterations with frequent deliveries are a great way to work and collect feedback; having a good product owner is priceless; and I don't think anybody really objects to heavy use of unit test suites and continuous integration anymore.  Let's keep those around.

But the more I think about and work with various "agile" processes, the clearer it becomes that there's a reason so many people seem to say that their organization "does Scrum," but not by the book - they have this and that modification, sometimes to the point that it no longer sounds like Scrum.  Some people interpret that as evidence that agile is nothing but hype.  After all, if nobody implements that actual process, then in what sense is it successful? 

Of course, that's not entirely fair because every team and organization is different so no process is going to be a perfect fit for everyone.  But on the other hand, is it entirely unfair either?  If hardly anyone seems to do things "by the book," could that be because the book is wrong?  Without more of the type of researched analysis that Meyer offers, we really have no way to know.

If you're interested in development processes, then Agile! The Good, the Hype, and the Ugly is definitely worth checking out.  I have to say that it really changed the way I think about agile processes.  For me, it really helped to put things in a wider perspective than the "good/new agile vs. bad/old waterfall" story that seems to dominate discussions.  And if nothing else, it's at least good to see some questioning and critical analysis of the agile mantra.

(Note: for a shorter read, Jim Bird has a very interesting take on agile in a similar vein to Meyer.)

Slashdot recently ran a story on why so many tech workers dislike their jobs.  The article links to a survey by TinyPulse, a company that measures employee engagement.  The thrust of the article was that people in IT are less happy and engaged in their jobs than their fellow employees in other departments. 

The article itself was somewhat interesting.  They showed a number of graphs that showed IT pros reporting reduced engagement as compared to people in other departments.  However, in looking at the graphs and the numbers, I got the distinct impression that somebody at TinyPulse was trying to pump up business by manufacturing a crisis.  For example, consider this screenshow of the graph (captured 2015-09-21):

This is the first graph in the article and the way it's shown is literally straight out How to Lie with Statistics.  No, seriously - read the book.  One of the ways to lie that it discusses is to use a bar graph with a discontinuous scale.  In this case, the horizantal axis appears to start at about 17%, which is why the bar on the right is about three times as tall as the one on the left, even though it only represents a 3% difference.

To their credit, TinyPulse didn't do that with any of the other graphs in the report, so maybe it was just a garden-variety screw-up.  It fould just be that somebody pushed the wrong button in the graph-generation tool and the result got pushed out.  Who knows?  You'd think a company that does statistics as part of its core business would be a little more careful, but hey, things like that happen.

The problem is that when you notice something like that, it immediately puts you on your guard and makes you suspicious of the rest of the data. Of course, that's assuming you know to look for it, which most people probably don't.  And that's the beauty of lying with statistics - you don't have to actually lie.  Why risk an out-and-out lie when you can just be careful in your presentation and trust that the vast majority of the people will draw the conclusion you want them to draw rathen than the one that's actually supported by the presented data?

Last summer I learned a new acronym, courtesy of Chinmay. He asked for a developer to put some estimates on the outstanding technical tasks needed to make a demo production ready. So I fabricated a few numbers, slapped them on the wiki page, and dropped a note in the project chat room saying that I'd put up some grossly inaccurate estimates. Chinmay replied by thanking me for the SWAG - Sophisticated, Wild-Ass Guess.

The reason I like this acronym so much is that it pretty much sums up all the software estimates I've ever seen. I mean, it's not like there's no methodology to it - you look at the task, break it down into pieces, and apply some heuristic to attach a number of hours to each piece. But really, at the end of the day, it's all just guess work. The "heuristic" usually just comes down to a gut feeling of how long something ought to take based on your knowledge of the system. In other words, a guess.

And it's not like there aren't more organized ways of doing this. We could use COCOMO models to come up with our estimates, for example. The problem is, approaches like that require you to have actual data to use as a basis for comparison, and most teams don't have that. Or they do have it, but the data is useless - corrupt, fabricated, poorly defined, etc.

The reason it's hard to get good data on developer productivity is obvious to anyone who has ever worked in a pathological development shop - because it can be dangerous. In a good organization, such data would be used to estimate costs and schedule and to help make better decisions about what projects to do and where to allocate resources. In a pathological organization...not so much.

In a pathological shop, measurement is used against you. If the measurement is number of hours needed to complete a task, then you'll be judged to be incompetent or slacking off if a "simple" task takes you "too long" to complete - where what counts as "simple" and "too long" are determined by a non-technical manager who really has no idea what you actually do. Or perhaps you'll be required to account for all eight hours of your day, down to five-minute increments. Because heaven forbid you should "cheat" the company by taking a bathroom break.

When I think of pathological examples of measurement, I always think about my last job with an online advertising startup. At one staff meeting, our VP of Engineering announced that he new expectation was that engineering would deliver on its estimates with 90% accuracy based on 80% definition from the product manager. So we were required to be 90% sure how long a task was going to take, even though we were only 80% sure exactly what it was we were building. And based on previous experience with the product manager, calling the specs he gave us "80% definition" was really generous - I'd say it was more like 50%.

In that case, the productivity measurement was being used as a tool to force the VP of Engineering out of the company (and it worked - he put in his notice the next week), but the principle applies to anyone - if a measurement can be used against you - even if that use is unfair and out of context - assume it will be used against you. That's a very sad reaction to have, but it's natural if your experience is primarily with unhealthy shops. And while I don't endorese such a paranoid attitude, having been in such a shop more than once in the past I can certainly understand it.

One of these day, I'd really like to work in a shop where performace metrics were used as a real tool in project planning. Even in healthy shops, this sort of thing seems to be uncommon. That seems to be something of a comment on the state of "software engineering" as a discipline. The differentiator between quality and pathological shops seems to be the absence of bad practices as opposed to the precense of good ones. Even in good shops, building software seems to be less engineering and more art or black magic.

Personally, I'm not convinced that that is a necessary or desireable state of affairs. I'd just like some first-hand evidence to support that suspicion.

Author's note: This one from the archives was written back in March 2008. This is something I've seen a couple of more times since then. A new developer, usually fairly young, comes in and wants to rewrite everything using whatever technology or method he's most familiar with. It seems to be not so much a problem with learning the new codebase as it is developer hubris. "My chosen way is obviously the right way, so clearly that's how we should be doing things." The people who do this don't seem to stop and think about whether their chosen method or tool might not be the best choice in the current situation, or whether implementing it might just be more trouble than it's worth. Perhaps it's just a lack of experience - they haven't had enough rewrites go south to realize that this isn't always a good idea. Either way, enjoy the rant!

Here's a little tid-bit for you young, aspiring developers out there: When you start a new job, it's usually a good idea to wait until your second week before you suggest rewriting the entire existing codebase.

I say this because we recently hired a new developer. He's obviously very bright and knows his stuff, but his first suggestion, on his first day, was that we should rewrite our entire codebase from a custom PHP framework to Symfony. After all, it would probably only take a couple of months.

Now, I don't know much about Symfony, but I have nothing against against it. And, in all fairness, our current framework is a bit of a mess. So I'm not dead-set against the very notion of switching. However, the current framework has been tailored to our needs and it is perfectly adequate. I'm sure that Symfony is very nice, but it's not immediately obvious what compelling advantages it would offer. After all, we already have an application framework that does what we need and which all three of the established developers are very familiar with. Given that we already have a full plate as far as projects go (hence hiring a new developer), it's hard to imagine how you could make a business case for spending 2 or 3 months learning Symfony and converting the whole site to use it. And if the new guy things that we're going to convert in parallel with adding features to the existing codebase (which he actually mentioned), he's out of his mind.

The thing that really bothers me about this, though, is that the new guy suggested this on his very first day. As far as I know, he hasn't even written a line of code yet and he's already writing up the business case for switching to Symfony. Even though he's barely looked at our codebase and so couldn't possibly have any idea what kind of effort this would entail.

So, to all the developers out there, next time you start a job on an established development team, at least take the time to learn what technology they're using and why before you suggest replacing it. Do a project or two with the current stack. Talk to the other developers about the history of the codebase and the company culture. Pushing to replace major technologies before you've even written a single line of code just makes you look like a tool.

You know, I've been making my a living writing software for over 10 years now. If my resume is any indication, I'm pretty good at it. But every now and then I still read something that gives me that. just for a minute, like maybe I've just been fooling myself all these years, like I'm actually completely inadequate as a software developer.

This Slashdot article gave me such a feeling. It links to a Google case study that involved porting MAME to the Chrome Native Client. The summary ends with this quote from the article: "The port of MAME was relatively challenging; combined with figuring out how to port SDL-based games and load resources in Native Client, the overall effort took us about 4 days to complete."

Now, I don't follow Chrome development at all, so when I read this, I had absolutely no idea what the Native Client actually was. My initial guess was that it was probably some API to access native libraries or hardware from JavaScript, or something like that. So it sounded like "porting MAME" would entail porting the code to a different programming language, or a different set of APIs for audio and video, or some similar.

That sounds like a pretty huge undertaking to me. And they did it in four days?!? Wow! I know Google is supposed to be all about hiring the best and the brightest, but that's just ridiculous. I mean, that would take me months to finish! And even if I was already familiar with the internal workings of MAME and Chrome, it would still be a matter of weeks, not days. How did they do it? Are they really that good? Am I just some third-rate hack by comparison?

Well...turns out they're not actually that good. This comment on the article sums it up nicely. Turns out that the Native Client isn't actually a totally different API, but presents a POSIX-like API with support for compilers and build tools. So really, this was less like porting MAME to something totally new and more like making it run on a different operating system. And they didn't even do a real production-quality port. Instead, they simly removed several problematic parts of the code and just sort of "made it work." Granted, that's still a pretty impressive amount to accomplish in only 4 days, but it's hardly the super-human feat it seemed a first.

This sort of story is one of the things that's always bothered me about the culture of software development - it's full of tall tales. Listening to the stories people tell, you'd think everyone was building big, impressive programs in a couple of days. It's not until you pry for details that you realize that the impressive sounding thing is actually little more than a prototype. Sure, Bob may have built a working C compiler over the weekend, but he doesn't mention that it can only reliably compile "Hello, world" so far.

It's almost a lie by omission - you report a truncated version of your accomplishment and rely on an implicit comparison to something much more involved to make you sound super-human. And I say "almost" because it's not just self-aggrandizers doing this. In many cases, the tale just grows in the telling. This case is an excellent example - Slashdot took an impressive sounding quote, stuck it in a brief summary, and made the whole thing sound bigger than it was.

I sometimes wonder what effect this sort of rhetoric has on beginning programmers. Do they find it inspirational? Does it make them want to become the "programming god" who accomplished this sounds-more-impressive-than-it-is feat? Or is it discouraging? Do they hear these stories and think, "I'd never be able to do something like that."

Looking back, I think that it was kind of a progression for me. When I was first learning to code, those stories sounded cool - I could be the guy telling them one day. Then, after a few years in the industry, they started to be discouraging. I mean, I'd been doing this for years and I still wasn't the guy in those stories. Now I'm over ten years in and I'm just skeptical. Whenever I hear one of those stories, my first thought is, "So what's the catch?" (Because, let's face it, there's always a catch.)

And the worst part is, I don't even know if that story should be inspiring or just sad.

Well, it's been a while since I've posted anything here. That's because I've been busy with a new job.

The new job is doing LAMP development for a major online community site. True to form, I'm not (yet) going to name the company. That policy has served me well so far. However, I will share some details of my new position and some of the challenges I've faced with it, both expected and unexpected.

The first innovation with regard to this job is it's location - my house. Yes, it's a 100% telecommute, work-from-home position. In fact, it's a totally distributed team, so everybody on the development works from home. And while that's great, so far I've actually found it to be one of the biggest challenges with the job. The challenge isn't motivating myself to actually work, which my mother thought it would be (seriously, she was concerned about that). In fact, quite the opposite. Until recently, I've had more problems with spending too much time working.

There are a three main reasons for this. Part of it is because I was still learning our system and felt like I had to "catch up" to the rest of the team. Another part is simply schedule diversity. We communicate primarily via Skype chatrooms, so we can when people are online. And since we have developers literally from California in the west to Russia in the east, many of them work non-standard schedules, there's always somebody "on", which adds some subconscious pressure. And lastly, of course, part of it is probably emotional damage from my last job. At that (highly dysfunctional) company, we were allowed to work from home on occasion, but if we did it too often, the CEO would make needling comments to the VP of Engineering suggesting that "working form home" was a code-word for "taking the day off". So I think I've partly been stuck in the mindset of trying to prove that, "No, really, I am actually working!"

The other new feature of this position is the sheer size of the operation. And I mean that in pretty much every conceivable way - the amount of traffic, the infrastructure, the size of the code-base, the size of the development team. In fact, pretty much the only thing that isn't big (to me) is the size of the overall company. By way of comparison:

• The largest site I'd work on was when I was at ebaumsworld.com, where we had something like 30 servers in our environment and were ranked about 1200 in the world by Alexa. The current job has nearly 100 web servers alone and is ranked in the top 150 in the world by Alexa.
• The largest code-base I'd worked on was at my last job - a little over 200K total LOC when you include unit tests. The new job has over 500K lines of just PHP code (that's not counting CSS or JavaScript) in just the web portion of the tree.
• The largest group I'd worked in was when I was with the county IT department - about 30 people total, about 9 of them programmers, no collaborative projects to speak of. My last two teams have been very tight 4-man operations. At the new job, we have about 30 people on the tech team, all of them programmers and all of them very good.

So, needless to say, I've been very busy figuring things out in my new position. I've been there for about three months now and I think I'm finally getting comfortable. I'm still learning the code-base, of course (nobody understands the entire thing - it's just too big), but I'm getting more confident and feel like I'm making progress. So in other words, so far, so good.

And most importantly, I'm much happier than I was at my last job. It's nice to feel like you can trust your colleagues and that the work you're doing is helpful. Amazing how increased morale boosts productivity.

At my current company, we've been trying to hire a new back-end PHP developer for some time. As the senior engineer on staff, and the only back-end person, it's my job to handle the technical portion of the screening process, including resume screening and technical interview. Unfortunately, I have lots of other work to do, so I never had time to come up with a really good procedure to test candidates. In addition, most of the interviews are over the phone with people in other cities, which makes it difficult to have them write code without some advanced preparation - which, again, I haven't had time to do.

So, as a compromise, I came up with a standard list of questions that I ask candidates. I originally wrote this list with the intention of using it for junior developers (1 to 3 years of experience), thinking that anyone with 5+ years of experience would breeze through it. However, I've found that a disturbingly large percentage of more experienced developers fumble more than one of the questions. Unfortunately, I'm not yet sure if that's just because we're getting poor quality applicants, or if it's because my standards are too high.

Therefore, in the interests of helping my fellow developers - both those looking for new opportunities and those who are interviewing the first group - as well as in the hopes of getting some good feedback, I present my list of interview questions. Note that I do not include answers. There are two reasons for this:
1) There is not always a single "right" answer - it is perfectly valid for the question to turn into a conversation.
2) If you're an interviewer and you don't already know the answer, you shouldn't be asking the question; if you're an interviewee and you don't know the answer, you should look it up and learn something new.
With each question I've included some notes on the question's intent. Some of these are conceptual, some are simply tests of knowledge, and some are spring-boards to further discussion.

1. Solve the FizzBuzz problem. How would you solve it in an object-oriented fashion? Using recursion?
OK, I hardly ever ask this question; you really need to be in the same room to make it work. However, it's still useful to see if the candidate can program and to see how they think. For the object-oriented and recursive versions, I'm not so much looking for the right answer as I am for the approach - and I tell them as much. In particular, I'm looking to see where the candidate goes with the open-ended requirement for object-orientation. Do they go for procedural code that just happens to use some objects? Or do they go straight to polymorphic classes, maybe with a factory pattern thrown in?

2. Define "inheritance" and "polymorphism". Give an example of when you might use each.
This is your "OOP 101" question. Any half-competent developer should be able to describe inheritance and give the stock "shape" or "animal" example. Polymorphism is a bit more slippery - I often don't get a good answer to that part of the question. In fact, a lot of people seem to think that polymorphism is basically the same thing as method overloading and don't really make the connection to inheritance.

3. What is the difference between an "abstract class" and an "interface"?
I actually ask this because one of our products is built primarily in FLEX, and interfaces abound in our ActionScript code. With this, I'm really interested in seeing if the candidates understands the idea of explicitly specifying a contract in the code, as opposed to relying on, e.g., PHP's loose typing to make things "just work".

4. What is "SQL injection" and how might you protect against it? What about XSS/CSRF?
This is, of course, the basic security question. On the SQL injection side, I'm specifically looking for awareness of prepared statements, e.g. in MySQLi or PDO. More generallly, I'm looking for an awareness of filtering an validation. On the script injection side, I like to see what candidates say they use to filter input. An awareness of standard escaping and sanitizing function (strip_tags(), htmlentities(), etc.) is always good - knowledge of the filter extension gets bonus points.

5. In JavaScript, what does it mean to say that functions are "first class" or "first order" objects?
This is a lead-in to determine how much, if anything, the candidate knows about the more functional aspects of JavaScript - anonymous functions, closures, etc. Surprisingly (at least to me), hardly anyone seems to be familiar with the terminology of "first class" entities. So far I've only had two people know what I was talking about right off the bat.

6. What is the difference between NULL and the empty string in SQL?
This one is more of a trivia question to test the depth of knowledge of SQL. Bonus points if the candidate mentions SQL's three-valued logic in the follow-up question: "What is the result of the comparison NULL = ''"?

7. What is an "index" in a database and how would you decide what columns to index?
This one is just to check that the candidate has some basic knowledge of database performance considerations.

8. Given a sample HR database that contains the following tables:
Employees (employee_id INTEGER PRIMARY KEY, employee_name TEXT, ...)
Departments (department_id INTEGER PRIMARY KEY, department_name TEXT, ...)
What would you have to add to the schema to associate employees with departments?
This is the basic SQL competency question. The typical answer is to add a foreign key from Employees to Departments, at which point I spring the "what if an employee needs to be in two departments" scenario and see if they come up with a join table. Also, it gives a good opportunity to see if they understand when to use inner versus outer joins. It's surprising how many people struggle with this question.

9. Describe a project you worked on that was particularly enjoyable or interesting. What did you like about it?
I stole this from an interview I did once where I was the applicant. It can be enlightening to see what catches a person's interest. If nothing else, it can help you get an idea of where their head is; are they a hard-core CS person, a touchy-feely UI person, or something else entirely?

Personally, I consider this list to be inadequate at best. But as I said, I haven't had time to develop an adequate test and this is at least better than asking "So how well do you know X?", if for no other reason than it's harder to BS an acceptable answer. Any thoughts or opinions are welcome.

Yesterday I upgraded FeedDemon to version 4. In the process, I did something new - I paid for a license.

For the last year or two that I've been using it, FeedDemon has been ad supported. The full feature set was available for free, but there was a little ad in the lower left-hand corner. If you wanted to get rid of that, you could buy a license key, but frankly, I never really cared that much.

Now, however, the business model has changed. There's now a free "Lite" version with some missing features and a paid "Pro" version with all the features and no ads. One of the annoying parts of this was that the features in the "pro" version are not all new - the for-pay features include some things that were previously free.

This bothered me a little for a couple of minutes. Then I read the FeedDemon author's article about why he changed the business model. Apparently, FeedDemon is no longer run by NewsGator. It's still branded for them, but they let the author, Nick Bradbury, go and returned control of the program to him. Apparently he's now doing the independent developer thing and making his living selling FeedDemon.

Well, that pretty much sealed the deal. I've tried a number of RSS readers - Linux, Windows, and web - and FeedDemon is the only one I've ever really liked. This is a program I use every day, I want some of the "pro" features, and buying it would help out a fellow developer. And to top it all off, he's only asking a lousy $20 for it. That's a no-brainer. So I bought a license and I'd encourage any other FeedDemon users to do the same.

Incidentally, some of the comments on Nick's article were really depressing. With users like some of those, I don't envy him trying to make his living off FeedDemon. Supporting a one-man project is tough enough when you release it as open-source and you don't owe anything to anyone. It must be hard trying to do right by people who paid you money when they feel your work is lacking - even (especially?) if they're clearly being unreasonable. Working 9 to 5 (well, officially 8 to 5 in my case, but you get the idea) might not be sexy, but at least it's stable and they pay someone else to deal with unhappy customers.

One thing I did notice in the new version that I found a little disappointing was the change in some of the newspaper themes. FeedDemon basically displays feeds as an HTML document and has a number of themes that control the look of the feed list and allows you to set themes on a per-feed basis. Changing the theme is now a "pro" feature, and I really wanted this, as I have several feeds that I have set to use the Outland theme, which is a nice, clean theme that displays full entries rather than just summaries. However, this theme is gone in FeedDemon 4.

The up side is that, while the theme format may have changed in FD4, it hasn't changed much. The "styles" are still just that - CSS and XSLT. In fact, I still had the Outland.fdxsl2 theme file sitting in my "C:\Program Files (x86)\FeedDemon\Data\Styles" directory, so I just tried renaming it to Outland.fdxsl4, restarting FD, and guess what? It worked! The only real problem was that the border on the feed was a little off, so I commented out the border on the HTML element (line 24), and it was just fine.

Last night I spent some time browsing Ron Jeffries web site. While I've seen a few of the items on Ron's site before, I never really stopped to look through - and boy was that a mistake! It turns out that Ron has a number of very interesting and entertaining articles on his site. I particularly recommend his classics section.

Included in the "classics" is an article entitled We Tried Baseball and It Didn't Work. Unless you're lucky enough to have only ever worked in high-quality organizations, which really, really had their act together, this article will probably ring true on some level. It also nicely sums up a number of Ron's other articles on the use and abuse of the term "Agile".

This kind of reminded me of when my current team first started to "do Scrum". For the first few months, the entire team agreed that it just wasn't working. Of course, we weren't exactly following the book in our application of Scrum: our "stand-up" meetings often lasted 30 to 60 minutes; we didn't always deliver "done-done" software at the end of a sprint; our sprints started at a month long, and when we failed to finish our tasks by then, we pushed the date. So "Scrum wasn't working" for us, if by Scrum you mean "some process that involves a daily meeting." But we had to do Scrum, or come up with some other "Agile" process, because if it's not Agile, then it's not good. I mean, how could all those industry pundits be wrong?

Fortunately, we've figured things out a bit better now. We're much closer to a real Agile process and it's working better for us. Of course, we're not really there yet, but we're making progress. And, really, that's what matters.

While catching up on Jason Gorman's blog today, I came across this entry, which linked to the Half-Arsed Agile Manifesto. In a word, it's awesome.

That manifesto made me think back to when I started my current job. I was reading through the process documentation for our parent company (whose processes the spin-off I work for inherited). The documents referred to themselves as describing the "agile" process the company employed. But as I read through them, with the descriptions of requirements documents and formal hand-off procedures, I just kept thinking, "This sounds a lot more like waterfall than any agile process I've ever heard of." It's always nice to know you're not alone in having experiences like that.

Sadly, while my team's current Scrum-like process is considerably more agile, we still have some vestiges of that. For instance, the stated expectation that our initial estimates will be 90% accurate based on what is (very) generously described as "80% definition" by product management. So we're only mostly sure what we're building, but we're almost positive how long it will take. I'm still not sure how the math is supposed to work out on that. Fortunately, we don't take that expectation too seriously.

Thanks to Carl and Richard from DNR for pointing out the best Twitter stream even. They mentioned it on show 379. You can find it at: http://twitter.com/thisispointless.

There are 3 things I like about this stream. First, it's actually kind of funny. Second, I think the username pretty well sums up everything about Twitter. And third, I just love the open mockery when there are people out there who are actually trying to communicate over Twitter. Because apparently nothing worth saying could possibly take more than 140 characters.

I'm sorry, that was a cheap shot. I was misrepresenting the true purpose of Twitter - to send pointless messages that people really shouldn't care about.

Leon Bambrick had a really great blog post today. You know the "one laptop per child" program? Well, he thinks it's great that the IT industry is doing this and that other industries should have similar programs, such as "one handgun per child." My favorite, though, was the program for government:

Governments could offer "One soul-destroying job in a miasma of bureaucratic inconsequentiality Per Child"

If you're a regular reader of this blog, or a member of my family, then you probably know that my last job was as a Systems Analyst in the IT department of a local government, so that quote really spoke to me. I know you hear people say things like that about public sector jobs all the time, but based on 6 years of experience, I can say that it's all true. There's a general atmosphere of hopelessness, where most things don't make any sense, lots of them don't really work, and everything is more complicated than it needs to be, yet nobody seems to think this is a problem. The only way to keep your sanity in such an environment is to simply stop caring. After all, nobody else has.

Of course, the private sector is no panacea. But at least there's always the possibility that you can improve your organization or you lot within it. Or at least change something.

The one thing I will say for government work is that if gives you a different angle on conspiracy theories. After a job in public service, it's impossible to waste even a second seriously considering any of those grand, sweeping government conspiracy theories you see floating around the internet. I mean, most government offices can just barely manage the job they're supposed to be doing. How are they going to manage some sort of vast cover-up? That would require motivation and initiative, and those aren't allowed in public service.

More live-blogging today. Unfortunately, it turns out that's the only kind I'll be doing until the class is over. I'm staying at the Crowne Plaza hotel, and while they offer WiFi service, they charge $10/day for it! That's actually worse than the $4/2-hour block that they charge for the AT&T WiFi at the Barnes & Noble back home. At that price, I could get 5 hours of service, and there's no way I'm going to be online that long here. Fortunately, the PCs in the training room for the class have web access, so I can at least check my e-mail.

Yesterday afternoon and this morning we got more into the details of managing ArcGIS geodatabases. Things like connection methods, authentication, data loading, management tools, "gotchas", and so forth. Basically, the stuff I will probably never need to know.

I'm actually a little ambivalent about this class so far. On the one hand, it's absolutely mind-numbing at times. It's not that the class is bad, it's just that it's getting into details that have absolutely no relevance for me.

On the other hand, it's actually very interesting in an academic sense. After all, we're talking about a system that scales up to clustered, multi-terabyte databases. The ArcGIS server runs under Windows, UNIX, and Linux and supports pretty much all the major DBMSs - Oracle, SQL Server, DB2, Informix, and there was even some talk of people using Postgress and Sybase. So we're really getting a closer look at the architecture of a very complex, high-end system. Plus our instructor has been around long enough that he can talk about how things have evolved over the years and the direction the architecture has taken. We're not just getting the tedious technical details, but some insight into the layers of the system, the APIs involved, and how everything interacts on various levels.

So as a case study of a major major information system, this class is actually quite interesting. However, it's really a class on managing geodatabases, not a case study on ArcGIS. So while the concrete details are putting me to sleep, the high-level stuff was definitely worth hearing. As a programmer, you tend to look at and read about things on more of a code-level. It's good to see how the "big boys" handle complicated design issues.

Well, here I am at Constitution Plaza in Hartford, Connecticut. I'm actually here on a business trip. My employer sent me to a training course with ESRI entitled Data Management in the Multiuser Geodatabase.

There are a few things to note about the very fact that I'm here. In the 6 years I've been with my current organization (which is far too long), this is the first time I've been on an out-of-state trip. In fact, it's the first time I've been on a trip that lasted longer than a day. It's also the first off-site paid training class I've been sent to. Kind of seems like a waste, given that if things things go well, I'll be able to get the hell out of here before I have a chance to put any of this to use.

As for the class itself, I have no real complaints so far. It's actually hosted by the City of Hartford IT Department, which has 2 people attending. The instructor, Jim, has been in the GIS business forever and has been working for ESRI for 14 years. He's really nice and seems pretty knowledgeable, so I'm actually enjoying the class so far. In particular, learning about the system architecture was kind of cool. Of course, I'll probably never put any of the stuff he's teaching us to use, but at least this gets me out of the office for a week.

One cool thing I've discovered is that Python is apparently one of the favored languages for automating ESRI's system. In fact, the lap machines we're using for the class actually have Python 2.4 interpreters installed on them. I don't know if we'll do any scripting in the class at all, but I just found that to be really cool.

There was an interesting article over on the Typical Programmer the other day. It consisted of a criticism of an anti-agile development paper entitled Agile Methods & Other Fairy Tales, by David Longstreet. While the criticism was interesting in its own right, I found it even more-so after reading the actual paper.

The main criticism the Typical Programmer leveled against the paper was that it was attacking a straw man. Specifically, he claims that many of the practices Longstreet describes are not actually agile, but simply broken. They just look agile because they don't involve much in the way of design, documentation, and so forth. Real agile methods, however, do not scorn those things when they are necessary.

I don't claim to be an expert on agile methods. However, I found Longstreet's paper to be good in some places. It certainly wasn't as intellectually bankrupt as the critique would lead us to believe.

Thinking about all this raised a few questions in my mind.
1) Couldn't the same complaint be leveled at the agile people? That is, might one not claim that they're not reacting against waterfall-style methods, but rather a caricature of them? After all, you can dress some broken methods up as waterfall just as easily as you can dress others up as agile.
2) Just what are real agile methods anyways? How do you differentiate it from "fake" agile? Viewed from the outside, it seems like agile methods could very easily be mistaken for cowboy coding. How do you decide where the line is?
3) Is it just my imagination, or do agile methods get less controversial every time I read about them? For instance, I remember when everyone was talking about the pair programming advocated by eXtreme Programming. Now, it seems like nobody cares about that anymore. (Actually,it almost seems like people have come to a silent agreement that it was kind of a silly idea.) These days, I read about the de-emphasis on documentation and other non-code artifacts. But then, when pressed, the agile people say they're only against the useless documentation.
4) Isn't "requires producing useless documentation" almost the definition of a broken development method? Yeah, sometimes it's demanded by the customer, but in other situations, does anyone ever think this is a good idea? It seems to me that if you're writing design documents that you never use, then you're missing the whole point of writing them in the first place.

Brain dump completed. Time to do some more reading on agile methods.

We've all heard the phrase "software engineering." The question is, does such a thing actually exist? Many programmers say it doesn't. I disagree.

Steve McConnell recently wrote a couple of very interesting blog posts on this very topic. The second was in response to this post. Eric echoes many of the typical "software development isn't engineering, it's a craft/art/form of black magic" arguments. Steve provides what I think is a good rebuttal.

I have a few problems with the typical anti-software engineering arguments. For starters, the "building software isn't like building bridges" line really misses the point. The idea is that building bridges involves repeatable and predictable patterns that allow engineers to accurately estimate project progress, whereas in software, every project is different and hardly anyone ever knows for sure how long things are going to take.

There may be some truth to that, but there's no reason to assume it's due to any inherent difference between civil engineering and software development. Part of it is simply that most programmers don't develop a lot of very similar systems because they don't have to. When you need a new bridge, you have no choice but to build it, but with software, you can just copy and customize.

But really, the reason most software estimates are just wild guesses is not because of the nature of software development, but because most development organizations are miserably primitive and unorganized. To form reasonable estimates, you need historical data. That means you need to track how long each phase of a project takes. Then you can run the numbers for similar projects and come up with a reasonable estimate for a new project. It's not rocket science. It's just that many organizations simply don't bother to do it.

Of course, as Eric mentioned, there is the issue of varying user requirements to consider. The whole reason we have custom software is that users have differing requirements, and so every application is unique in some sense. But if we're going to be honest, that only goes so far. Every custom business application may be different, but there are still a lot of similarities. You have your database layer, your business rules, your user interface, your reporting, and so on. They may never be exactly the same from application to application, but they're usually not fundamentally different. Think steel bridge over a river compared to a concrete bridge over a ravine. They're not exactly the same, but they're still comparable on some level.

Bottom line, software engineering, like "regular" engineering, is about process, not product. Just because you're building something doesn't mean you're doing engineering. A rigorous software development process is the exception in our industry, not the norm. So it's not that software engineering doesn't exist, it's just that the vast majority of the people writing software are not doing it. If you want engineering, head to someplace like Praxis Critical Systems, not your run of the mill .NET contractor.

I know there are those in the Linux community who will regard this as equivalent to spitting on the cross while blaspheming the holy spirit, but sometimes I feel sorry for the people at Microsoft. They have a tough job and they are frequently blamed for things that aren't their fault.

What made me think of this was Jeff Atwood's follow-up to his Windows spyware post.

I understand the pressure to be backwards compatible. There's no end of Vista blowback based on minor driver compatibility issues. The "if it doesn't work, it's automatically Microsoft's fault, even if the software or hardware vendor is clearly to blame" mentality is sadly all too common. But given the massive ongoing Windows security epidemic, was defaulting regular users to Administrator accounts-- exactly like Windows XP, Windows 2000, and Windows NT before it-- really the right decision to make?

In many ways, Microsoft is in a perpetual no-win situation. On the security front, for example, they are besieged by the evil forces of malware. However, every time they try to improve the situation, end users scream blood murder. For example, was the weird "quasi-administrator with prompting" a good idea? Probably not, but it's better for the average user than silently allowing the installation of spyware, and yet everyone seems to hate it. But what's the alternative? To make accounts regular users by default? How would the average Windows user would feel about that? I don't know, but I have read a many comments by recent converts to Linux who seem to think that entering a password just to install some software is completely stupid and unreasonable, so I can't imagine it would be universally hailed as a great improvement.

And, of course, there's always the breakage that accompanies any OS upgrade. For example, remember Windows XP Service Pack 2? I seem to recall widespread complaints about things breaking when that was rolled out. And we're seeing the same thing now with Vista. And who do people blame? Is it the ISV who are still coding with Windows 98 in mind? No - they blamed Microsoft.

What really kills me about this situation is when I read Raymond Chen's accounts of the efforts of the Windows App Compatibility team. For example, consider this sample chapter from his book. From the cases he mentions there, it is completely obvious that Microsoft takes backward-compatibility seriously. Very seriously. In fact, you might even say they take it too seriously.

Think of it this way. On Linux you're lucky if you can get source compatibility for an application that's more than 5 years old. Microsoft has binary compatibility with a large range of programs that are 10 or 15 years old. They're working with third-party binaries, diagnosing obscure bugs, and implementing fixes to keep the applications working, even though it's by sheer luck that they ever worked in the first place. As a programmer, it's hard to overstate how impressive this is. And yet all anyone ever focuses on is the problems they didn't fix.

Then there's the political angle. There are lots of people out there who seem to think that Microsoft can do no good. Everything they do is viewed with suspicion. Anyone who works for Microsoft has to contend with accusations that he is either in on the conspiracy or is bowing down to "the man" every time he says something they MS-haters don't like. That's got to be at least a little demoralizing. And while a certain degree of animosity is certainly warranted (as it is with practically any large business), it's not like Microsoft has been running child sweatshops or dumping toxic waste in the local drinking water. It just seems way out of proportion.

So no matter what the people in Redmond do, it seems like there's always somebody pissed off at them. And it's a shame, because they really do do some good work. The .NET world has lots of neat technologies and is a very cool place for developers. Even though OpenOffice may be good enough for many, MS Office is by far the best office suite available. And, last but not least, Windows, despite it's flaws (and they are legion) is a very impressive feat of software engineering. Not to mention that Microsoft employs a number of very bright people.

So, to the Linux community, I say, let's give Microsoft a chance. I'm not asking you to like MS, or to start using any of their products. Let's just be honest and realistic. Most people in the community aren't shy about placing the blame on them, but give credit where credit is due. We rightly object when people blame free software for not being a panacea, what with hardware incompatibilities and the lack of certain software. We should at least hold MS to the same standard and not judge them for failing to satisfy everyone.

I saw something on TV last night that really struck me. In fact, for a second I thought I was dreaming, as I had dozed off for a while. But I think it was actually real: an iPhone commercial that gave the ability to watch YouTube videos as a selling point.

Being the curmudgeon that I am, my initial reaction to this was, "Just what we need, another overly-expensive way to waste time." Now you can spend $500 to watch crappy home movies and/or blatantly pirated TV shows on a 3.5-inch screen, rather than paying $400 to watch them on a 17-inch screen. Personally, the only way I can tolerate watching videos on 19-inch wide-screen is if they're full-screen. The little embedded Flash viewer drives me crazy. Plus I was never able to get too excited about YouTube in general. It's a nice, well-designed site, but I'd usually rather have a podcast in the background while I work than stare at my monitor for half an hour.

But after thinking about it a little longer, what I found really interesting about this commercial is how the cell phone is dying out. In fact, I predict that by 2015, perhaps even 2010, there will no longer be such a thing as a cellular telephone. The mobile phone will be a thing of the past. People will look at them like they look at rotary telephones now, as a quaint reminder of how backwards we used to be.

Instead, the "phone" of the future will be like the iPhone, i.e. very small multimedia computers that just happen to have voice communication features. We're already part way there when you think about it. People are using their cell phones extensively for things like playing games, text messages (a.k.a. the poor-man's IM), taking and trading pictures, and listening to music. They've turned into PIM/multimedia appliances. Actual voice communication has become almost secondary.

The best part about this is the rate at which the technology to price ratio is increasing. When I got my first pre-paid cell phone seven years ago, it was about the size of a regular phone handset, rather expensive, and didn't do anything except make and receive calls. Then, four years ago, I signed up with Verizon and got two stick phones with biggish black and green screens, text messaging, calendars, and other simple tools for less than the one pre-paid. Two years later, for only slightly more than the two stick phones, I upgraded to two Samsung SCH-A670 camera phones with color screens, plenty of bells and whistles, and the ability to run BREW programs. Currently, I'm using an LG VX8300, which features, among other things, a 1.2 megapixel camera, an MP3 player, surprisingly good speakers, and a 1GB MicroSD slot to hold all that media. And once again, I didn't pay any more for this upgrade than for the last one.

In a few years, I'd love to have a phone powerful enough that I could use it to do actual work. Maybe something I could plug a roll-up keyboard into. Or maybe I could get something with a USB or FireWire port and an operating system with some degree of flexibility. Or maybe they can just invent a 4-inch laptop an put a phone on it! After all, that's practically what the high-end smart phones are. Now it's just a matter waiting for the prices to come down.

There were a few interesting threads on the Joel on Software discussion forum last week. They proved to be interesting reading on just how wrong a developer can go, with or without realizing it.

The short version is that poster Brice Richard took issue with the recommendations of other posters that applications should be written as, in his words, "all mini-functions." He then went on to describe his coding methods and argue that his way was right; if not for everyone, than at least for his circumstances.

Now, the title of that second thread was actually a little misleading. I actually read that one first, and I initially thought he was referring to the arbitrary guidelines on subroutine size that you often hear thrown about without any real supporting evidence. For example, some people say that a function should never be longer than a dozen lines, or that all functions should fit on one screen in you editor. That last one always sounds nice - unless you have poor eyesight and a small monitor.

But that's not what Mr. Richard was talking about. He only believes in writing separate functions when:

1) there is a bona-fide reason to believe that the code will be re-usable in other apps
2) when you have duplicative functionality within your application that results in the need to create a function that defines that functionality through either value or referential arguments allowing your code to execute variably when used.

And apparently he doesn't view 300+ line functions as a cause of concern. And all this from someone who has been in the consulting business for 8 years.

Part of the reason for this is probably that Mr. Richard is, in his words, a self-taught programmer who always works alone. Of course, there's nothing wrong with being self-taught. In this business, we're all self-taught to various extents. To quote Dr. Kelso from an episode of Scrubs, "If you've been out of college for more than five years, half of what you know is obsolete." There are always new technologies to learn and most schools don't even come close to covering all of what you need to be a professional developer.

However, as Jeff Atwood pointed out today, working alone can be dangerous. And it's not just the project management issues described in the essay he cites. When you don't have other people to learn from, you end up having to work harder to educate yourself. You need to read more, experiment by yourself, and try to validate your results based on just your research rather than peer input. You have no one to practice with or bounce ideas off.

It's important to remember that this is 2007. Software development is still a young field, but it's no longer in its infancy. Translation: you don't need to go about rediscovering how to do things. When people talk about "doing things your own way" and the like, that's usually what they mean. It's much better to study the recommendations of the experts and use or adapt them as appropriate.

When you're working alone and aren't vigilant in your studies, it's easy to fall into bad habits. It's also easy to get stuck in your ways and not learn new techniques or technologies. The forum threads I mentioned above are a good example of this. Working alone isn't necessarily detrimental to your skills, but it's much better to have smart people around to give you an occasional reality check.

Of course, you can still end up working with other developers who have no concern for modern practices or procedures, and hence don't help much. But that's a different story....

This week I finally decided to take the plunge: I started working on my résumé. That's right! After six years I have finally decided that it's time to get my career moving and so I have officially entered the job market.

Ah, job hunting! It's quite the experience, isn't it? I'd almost forgotten what it was like. There really is nothing like a good job search to make you feel like a useless, incompetent sack of crap!

I don't know about other industries, but this is definitely the case in the IT world. If you've ever looked for a job in software development, you know what I'm talking about. For every reasonable job listing you see, there are twelve that absolutely require a 10 years using laundry-list of excruciatingly specific technologies, strong interpersonal skills, a Mensa membership, and a strong track record of miraculous healing. And that's for an entry-level position. With a typical career path, if you start early, you should be ready for their grunt-work jobs by about the time your kids are graduating from college and moving back in with you.

The listings that have really been killing me, though, are the absurdly specialized ones. Not the ones that require 5 years experience with ASP.NET, C#, Java, Oracle, SQL Server, SAP, Netware, Active Directory, LDAP, UNIX, SONY, Twix, and iPod - they're just asking for the kitchen sink and hoping they get lucky. I'm talking about listings like the one I saw that required advanced training in computer science, a doctorate in medical imaging, and 10 years of experience developing imaging software. Or, perhaps, all those listings for a particular defense contractor that required experience with technologies I had never even heard of. I mean, I couldn't even begin to guess what these abbreviations were supposed to stand for, and I'm pretty up on my technology! When you come across a lot of listings like that at once, it can be a little depressing. "How am I ever going to find a job? I don't even know about grombulating with FR/ZQ5 and Fizzizle Crapulence GammaVY5477 or how to do basic testing of quantum microcircuits using radiation harmonics with frequency-oscillating nano-tubes on a neural net. Every idiot understands that!"

But the real killer for me is location. I'm in the southern tier of New York state, which is not exactly a hotbed of tech startups. I like the area and don't really want to move, but there's practically nothing here in terms of software development. The best possibility I found was a local consulting company 10 minutes form home. However, when I sent them a résumé, I got a message back saying that they were currently unable to add new positions due to the fact that they were going out of business. I've applied for a couple of other semi-local positions, but of all the possibilities I've found, the closest is about 50 miles from my house. Workable, but not a situation I'm crazy about.

I'm now starting to think seriously about relocating. I don't really want to move to the west coast, both because of the cost of living and on general principle, so I'm thinking of looking either downstate (i.e. New York City) or south to the Washington, D.C. or Atlanta metropolitan areas. All three of those seem to have a fair number of positions in software development.

However, I'm faced with something of a moral dilemma. You see, having been born and raised in upstate New York, it is my patriotic duty to hate New York City. But as a New Yorker, it is also my patriotic duty to look down on the South and New Jersey. That leaves me wondering whether I'm forced in to choosing Washington, or whether it counts as "the South" too and I'm just out of luck.

In the end, I guess I'm just not that patriotic. All three of those cities sound good to me. But New Jersey is another story.

I think I've put my finger on the reason that, despite being a rookie in it's practice, I'm so enamoured of Test-Driven Development. It's because TDD, despite being an "agile" programming practice, is the very inverse of agile methods. It's good, old-fashioned design. The only difference is that it's dressed up to look like programming.

At least that's what I get from reading Dave Astel's thoughts on TDD. He's an advocate of Behavior-Driven Development. This is an evolution of Test-Driven Development where the emphasis is shifted from testing units of code to describing the desired behaviour of the system. The idea is that rather than concentrate on writing a test for each class method, you figure out what the system should do and design your test cases around that rather than around the program structure. (Apologies if I mangled that explanation - I only learned about BDD today.)

This is somewhat similar to the other expansion of the initialism "TDD" that I discovered early in my research - Test-Driven Design. Note that the emphasis here is on using the unit tests as a design tool, i.e. specifying the behavior of the system. In other words, the unit tests are not written to do testing per se, but rather as a vehicle for expressing the low-level system specification.

Having studied formal development methods, it seems to me that, at a fundamental level, practitioners of TDD and formal methods are really doing pretty much the same thing. They're both trying to describe how a program is supposed to function. The TDD people do it by creating a test suite that asserts what the program should and shouldn't do. The formal methods people do it by writing Z specifications and SPARK verification conditions. Both are creating artifacts that describe how the system is supposed to work. The formal methods people do it with verifiable mathematical models, while the TDD people do it with executable test. They're just two different ways of trying to get to the same place.

In a way, this seems to be the opposite of what agile methods promote. Most of the agile propoganda I've read advocates doing away with specifications and documentaiton to the maximum extent possible. In fact, I've even read some statements that the amount of non-code output in a project should approach zero. And yet TDD can be viewed primarily as a design activity of the type that, using a different notation, would generate reams of paper. The only thing that saves it is the technicallity that the design language is also a programming language. It just seems a little ironic to me.

I suppose that's really the genius of TDD. It's the type of specification activity programmers are always told they should do, but repackaged into a code-based format that's palatable to them. It's sort of a compromise between traditional "heavy-weight" design methods and the more common "make it up as we go" design. I don't want to say it's the best of both worlds, but it's certainly better than the undisciplined, half-assed approach to design that's so common in our industry.

Things at work have been slow lately. Really slow. The fact is we're severely over-staffed and there just isn't enough work to go around right now. This has left me with an amount of down-time that I find...uncomfortable.

You'd think this might be a good position to be in. After all, it leaves me plenty of time to read up on development practices, the latest technologies, and keep up on my tech blogs. I have spare time to experiment with Ruby on Rails and play with Mono. I can browse through Code Complete or Refactoring at my leisure. What more could a programmer want?

As I've been discovering, this isn't quite as nice in practice as it seemed at first. For starters, there are still the miscellaneous menial tasks that need to be done - software installations, security and configuration changes, and just general support calls. These are excellent for knocking you out of the zone. Furthermore, the constant threat of them makes it a bit harder to concentrate in the first place.

Second, while reading and experimenting to build your skill-set is great, you need the right environment to make it truly profitable. You need some degree of freedom and some goal to work towards. Or at least I do. I find I learn the most when I have something I'm trying to accomplish. I also need periodic breaks to process and assimilate what I'm studying. It just doesn't seem to work as well when my "goal" is to stave off boredom and my breaks are scheduled.

Last, I've found that it's becoming a challenge just to stay sharp at times like this. You see, I need some sense of purpose to stay sharp. I feel like I'm chained in a room for 8 hours a day being forced to do nothing but pour water back and forth from one bucket into another. It feels like my brain is starting to rust.

This is exactly the opposite of how to motivate programmers. We need crave interesting progblems to solve. Or, at the very least, some problems to solve. Playing the point-and-click Windows monkey won't do it and I can only stand to read so many hours a day.

The problem is, the more I try to spend my time improving my knowledge and skills, the more unbearable I find my condition. I feel like I should be "out there" putting what I learn to use, but instead I have to sit at my desk for...no particular purpose. And what's worse, it's sapping my mental and emotional energy. After being stuck in the office all day, trying to keep from going stir-crazy, I feel like I've got nothing left when I get home. It's turning into a vicious cycle.

Have other people been in a situation like this? How do you deal with it? I mean, short of quitting (which I'd do if not for that damn mortgage). Are their any coping strategies to hold me over until I can get out?

Yesterday, I was telling you about the ridiculous install procedure for CRAPS. Today I'd like to continue the discussion.

As you remember from last time, the recommended procedure for getting the MS Access databases CRAPS uses on your network is to just install all the software directly onto the C: drive of your file server. If you're in IT in any capacity, and are at least marginally competent, I shouldn't have to explain why this is really stupid.

The customization and configuration process is similarly disjointed. Some of this you do in a crowded graphical configuration dialog. Some of it you do by hand-editing INI files. I'd say it's about a 60/40 split. And, as an added bonus, for a few settings you actually have to hand-edit rows in one of the database tables. Unless you don't have Access 2000. Then you just live with the defaults.

From the care and attention they put into the installation and configuration process, it should go without saying that the CRAPS developers take security very seriously. Thus they decided to use "binary files (also known as flat files)" to store certain data on field units. After all, "Binary files use a format that is specific to the application that creates them," which means that other programs are "unable to interpret binary files that they did not create. This makes [CRAPS] data secure, as only [CRAPS] can interpret and use its binary files." And if you don't believe me, that comes directly from the CRAPS administration manual. Seriously. The only thing I changed was the name of the system. The manual also claims speed and smaller file size as benefits, despite the fact that field units are single-user laptops with Pentium IV processors and 80GB hard drives.

It's always a bad sign when the developers feel the need to justify their choice of data format in the user manual. So it probably comes as no surprise that when you actually look at one of these "binary files," it contains mostly serialized text data. It's definitely not encrypted and most of it isn't even really binary. With a little intelligence, it's not even too hard to figure out the format just by looking at it. It sure is a good thing that opening sequential files in Vim is such a well-guarded secret, or else the security benefits might not seem so compelling.

And even worse, some of the "binary files" referred to in the manual are just standard formats. For example, guess what the "binary" import and export files are? They're ZIP archives! With a different file extension!

The really sad part is that, for all its fauilts, CRAPS actually works fairly well. Yes, it's slow (despite the speed benefits of binary files), the configuration is tedious, the user interface is arcane, and the whole system has a feel of being held together with duct tape and bailing wire, but it does work. It truly is worse than failure.

Today I start a new series: Tales of IT. In this series, I will be discussing some of the more humorous things I come across in my work in the IT world. Some are funny ha-ha, some are funny-weird, and some are just funny-sad.

Let me start by telling you a little about what I do. I'm a "Systems Analyst" for a local governement agency. In my organization, that means that I do software development (that's my favorite part), analysis, some server administration, some help desk stuff - whatever comes up. So at this point, I can basically do some of everything.

Now let me tell you about the organization I work for. Being the government, it should go without saying that it's extremely disfunctional. One of the more annoying disfunctions of the IT department is what I call the "you touch it, you own it" theory of tech support. By that, I mean that, for any software system (especially big, complicated, or expensive ones), there is an analyst who "owns" that system. Ownership is determined on a first-come, first-served basis, so if you're the first person to touch a new system, you will own that system until you quit, retire, or die. The natural result of this is that nobody will ever volunteer for anything.

Now, you may be thinking that having a "local expert" on every system we use is a good idea. And you'd be right. It is a good idea to have somebody with a degree of in-depth knowledge around. That isn't the issue.

The problem is simply that the whole situation is completely demoralizing. When you own a system, you become the goto guy for all problems related to that system. That includes everything from the hardish analysis work of determining the proper system configuration all the way down to the grunt work of clicking next through client installations on a couple dozen workstations. If somebody calls the help desk with a problem related to your system, it gets passed straight back to you - the help desk doesn't even try to resolve it or even get more information.

This brings me to a system I own, the Crazy and Ridiculously Attrocious Police System, or CRAPS for short. CRAPS is the system used by the police force to write tickets, take accident reports, and so forth. I got stuck with it because it runs on the terminals in the police cars, and since I already owned the 911 dispatching software on those terminals, and my boss apparently didn't think I had enough pain in my life, I was the obvious choice.

I've got plenty of stories about CRAPS, but today I'll just tell you about the system itself. For starters, CRAPS has a "client-server" design. By that, they mean the system stores data in Access 2000 databases that sit on a file "server" and that the "clients" get pointed to the mapped drive or UNC path to those databases. Sort of.

You see, the CRAPS manuals and installation program seem a little confused about just what a server is. The installer has options to perform a "server install" and a "client and server install." What this choice really comes down to is "install just the .MDB files on this machine or install the .MDB files and the program files on this machine."

The important thing to note here is that both options are for this machine. That means that if you want to do what we did and put just the databases on your file server, the installed expects you to log in on the file server to run the installation. You can't just run from another machine and point the installer to the network location. I know, I tried. It fouls up things in the next stages of installation. It turns out that there's a reason the manual "strongly recommends" that you keep the default installation path of C:\CRAPS - things tend to get messed up if you don't.

Part of the reason things get messed up is that a basic CRAPS install is a 2-step process. First, you install the base system. Second, you install an add-on package with all the databases you need to actually get things done. The databases from both packages need to be on the server, but there is no "server install" option for the add-on databases. In fact, you can't even change the installation path. They get installed to the same path as your base system. Unless you changed the default path. In that case, some of the files get installed to the right place and some of them disapear into the ether, never to be heard from again. In fact, I've found that it's more reliable to just install everything on a workstation and then copy manually copy the data directory to the file server. It really shouldn't be, but it is.

At this point, CRAPS is installed on the file server, but is not yet functional. We'll explore how to make it work tomorrow.

This week, .NET Rocks had a great interview with Jeff Atwood. Jeff is a really insightful guy and listening to him was as much fun as reading his blog. In fact, this interview inspired me to start listening to other episodes of .NET Rocks. Well, that and the fact that Carl Franklin co-hosts Hanselminutes, which I also enjoy.

One the topics the interview touch on was Jeff's Programmer's Bill of Rights. It enumerates six things a programmer should expect if he is to be productive.

I found this both depressing and comforting. It's depressing because, as Jeff pointed out in the interview, these things are neither unreasonable nor hard to fix. You can basically just throw money at them without putting in any real effort. These conditions should not be widespread enough that anyone needed to bring them up.

As for comfort...well, it's just nice to know you're not alone. I'm currently one of those poor schleps Jeff talked about who's still working on a single 17" CRT monitor, a three year old PC, and sitting in a cubicle right next to one of the network printers. I'm not even within sight of a window and my cube is literally just barely big enough to fit my desk. I write my code in SharpDevelop because my boss won't spring for a visual studio upgrade. Two years ago it was, "Well, we'll wait for the 2005 version to come out instead of buying 2003." This year it was, "We'll wait for the 2007 version to come out instead of buying 2007." And last but not least, despite the fact that we write mostly reporting-heavy information systems, I use the version of Crystal Reports that came bundled with VS.NET because, as crappy as it is, it's the best thing available to me.

I have to agree with Jeff, Richard, and Carl. The message you get from a setup like this is clear: you are not important. We don't value you as a person, we don't value your work, and we're certainly not going to waste money on making your life easier. The net effect is that morale, especially among the more clueful people in my office, is in the gutter. There's misery galore and productivity is next to nothing. But fortunately we work for the government, so nobody notices. And no, that was not a joke.

Sometimes it seems like our environment is tailored specifically to sabotage productivity. It's kind of like the keyboard they put on the laptops that the police use. I'm the happless IT guy who has to do configuration and maintenance on these laptops, and I can tell you that the only explanation for those keyboards is that I did something really, really terrible in a past life. They're ruggedized keyboards made of semi-hard plastic. The problem is that they're so rugged that it's completely impossible to type on them. You have to use the two-finger method because the keys are too hard to press with your little fingers. Trying to type with any speed at all is completely futile. And yet the cops are somehow expected to type up tickets and accident reports on these things. It's a wonder they even give out tickets anymore. Actually, maybe that was the idea....

I suppose this is what I get for taking an IT job when I really wanted to be in software development. In retrospect, maybe I should have stayed a full-time student that extra semester or two, finished my damned thesis and looked for a job with a real software company. But I thought I needed some experience and this was the best offer I got, so I took it. Unfortunately, I was too inexperienced to know that crappy experience isn't necessarily better than no experience.

Though on the up side, when I took this job is when I moved in with my (now) wife. It also provided the money that paid for that engagement ring. So in some ways this was the right decision. It's just the professional advancement wasn't one of them.

Now I just need to finish my damned Master's thesis and get the hell out of here.

Last week, Jeff Atwood posted an anecdote demonstrating yet again that bloggers aren't real journalists. I know this meme has been floating around for some years, but I'm still surprised when people bring it up. In fact, I'm still surprised that it ever got any traction at all.

I'm going to let you in on a little "open secret" here: blogging in 2007 is no different than having a Geocities site in 1996. "Blogging" is really just a fancy word for having a news page on your website.

Oh, sure, we have fancy services and self-hosted blog servers (like this one); there's Pingback, TrackBack, and anti-comment spam services; everybody has RSS or Atom feeds, and support for them now built into browsers. But all that is just gravy. All you really need to have a blog is web hosting, an FTP client, and Windows Notepad.

That's the reason why bloggers in general are not, and never will be, journalists. A "blog" is just a website and, by extension, a "blogger" is just some guy with a web site. There's nothing special about it. A blogger doesn't need to study investigative techniques, learn a code of ethics, or practice dispassionate analysis of the facts. He just needs an internet connection.

That's not to say that a blogger can't practice journalism or that a journalist can't blog. Of course they can. It's just that there's no necessary relationship. A blogger might be doing legitimate journalism. But he could just as easily be engaging in speculation or rumor mongering. There's just no way to say which other than on a case-by-case basis.

Like everything else, blogging, social media, and all the other Web 2.0 hype is subject to Sturgeon's law. The more blogs there are out there total, the more low-quality blogs there are. And the lower the barrier to entry, the higher the lower the average quality is. And since blogs have gotten insanely easy to start, it should come as no surprise that every clueless Tom, Dick, and Harry has started one.

I think George Carlin put it best:

Just think of how stupid the average person is. Then realize that half of them are stupider than that!

Any average person can be a blogger. Thus the quality of those blogs will follow a standard distribution. For every Raymond Chen, Jeff Atwood, and Roger Johansson, there are a thousand angst-ridden teenagers sharing bad poetry and talking about not conforming in exactly the same way. They're definitely bloggers, but if we're going to compare them to journalists, then I think society is pretty much done for a blog.

I came across a weird article today. It's entitled Why Bugs should not be Tracked and the basic premise is that it's a bad idea to use bug tracking software.

What struck me as really strange about this article is the final conclusion. After telling us how bug tracking is bad and unnecessary, the author offers an alternative: write your priorities down on a piece of paper. In other words, do the extremely low-tech version of what bug tracking software is for. Say what?

The real problem that this article addresses is that bug tracking systems are easily abused. It's easy to enter bugs and just let them sit there until the end of time. However, if you're using something like a piece of paper or a spreadsheet, you have to dispose of the "stale" issues in one way or another, otherwise the whole thing becomes completely unmanageable.

There's an easy response to that, though: "Don't abuse your bug tracking system." Don't let bugs pile up. Categorize sanely. Do some house-cleaning when the number of outstanding bugs gets too big for comfort. It's not rocket science, just use the tool productively. Agile development is all well and good, but sometimes it seems like they're throwing the baby out with the bath water.

You often hear Mac and Linux people going on about how their operating system doesn't suffer from viruses and malware. They claim it's because their OS is inherently more secure. The Windows people then retort that the real reason is that hardly anybody uses MacOS or Linux, so the hackers just don't put much effort into it. A huge flamewar then ensues, in which very few of the participants actually know what they're talking about.

I read an InfoWorld blog on this very topic today. While I found it largely unremarkable, I did take issue with one passage.

The difference isn't market share, it's the foundation of the operating systems. Given that most virus authors and hackers are in it for the ego, don't you think that there would be a huge incentive to be the first one to write a widespread OS X, Linux, or FreeBSD virus?

There are two problems with this passage. First, there's the claim that "most virus authors and hackers are in it for the ego." That may have been true 10 years ago, but not anymore. These days, many hackers and malware writers are in it for the money. Some of them are even in bed with organized crime. It's not about learning how systems work anymore. Now it's big business.

In light of this, it's just absurd to dismiss the possibility that market share is not an issue. Just look at the numbers. On desktop PCs, Windows has well over 80% market share - probably more like 90%. So if you're trying to build a big botnet, what are you going to target? Windows is generally less secure by default, has more non-technical users, and if you get just 10% of them, that's more systems than if you got every Mac out there. With numbers like that, targeting anything other than Windows is just a waste of time.

Of course, the underlying operating system may have something to do with why Mac and Linux users have fewer security worries. However, it's certainly not the only reason. The default configuration of each system is another big reason - the out-of-the-box Windows configuration has historically been wide-open, while MacOS X and Linux are fairly secure by default. But if we're going to be honest, we can't ignore market share. It may or may not be the primary reason, but to claim it's not an issue is just wishful thinking.

Roger Johansson over at 456 Berea Street has posted a really great rant on lame excuses for not being a web professional. It's a great read for any developer (of any type) who takes pride in his work.

My personal favorite excuse is the "HTML-challenged IDEs and frameworks." It always seems odd to me that back-end developers can look down on the front-end web designers as "not real programmers" and yet be utterly incapable of writing anything even close to valid markup. Sometimes they don't even seem to aware that there are standards for HTML. They have the attitude that "it's just HTML," that it's so simple as to not even be worth worrying about. "Hey, FrontPage will generate it for me, so I'll just concentrate on the important work."

This is fed by the "the real world" and "it gets the job done" excuses. The "target audience" excuse even comes into play a bit. After all, nobody in the real world worries about writing HTML by hand. Especially when FrontPage gets the job done. And since our target audience all uses Internet Explorer anyway, it's all good.

This sort of thinking is especially widespread in the "corporate IT" internal development world. For example, if Roger ever saw the homepage to my employer's "intranet," he would fall over dead, having choked on vomit induced by the sickeningly low quality of the HTML. The index.html page contains an ASP language declaration (but no server-side script in, oddly enough - probably a relic of a Visual Studio template) and a client-side VBScript block before the opening HTML element. Several of the "links" on the page are not actually links at all, but TD elements with JavaScript onclick events to open new pages. For that matter, it uses tables despite the fact that it's laid out as a couple of nested lists of links. Needless to say the markup doesn't even come close to validating against any DOCTYPE. And this was written by a senior programmer who fancies herself the "local expert" on web development.

I guess it's just a problem of mindset. Some people just want to get the project finished and move on. They don't really care about code quality, maintainability, interoperability, or any of those other little things that make for really good software. As long as the customer accepts the final product, everything is fine.

While I don't share that attitude, I can sort of understand it. I'm interested in software development for its own sake. I care about the elegance and purity of my work and enjoy trying new technologies and learning new theories and techniques. But to some people, programming (or web design) is just a job. It's not a hobby or part of their identity, but simply a way to pay the mortgage. For those people, it's probably hard to get excited about semantic HTML or the efficacy of object oriented programming. As long as they get the project done and get paid, that's really all that matters.

Of course, that's just an explanation. It's no excuse for professional incompetence or unwillingness to learn. If you're going to call yourself a professional, I think you have an obligation to at least try to keep up with the current technologies and best practices in your niche. Not everybody has to be an über geek or an expert on the latest trend, but it would be nice if all web developers were at least aware of the basics of web standards, all CRUD application developers knew the basics of relational theory and that XML is more than just angle brackets, and all desktop developers had some basic grasp of object orientation. Yeah, that would be really nice....

Here's an odd one. A recent Slashdot story last week linked to a summary of Mark Shuttleworth's comments on Dell and Linux. The Slashdot summary said the article contained a response to Shuttleworth that they described as "amusing and telling at the same time."

I found that link odd because I'm not sure if that was meant to be ironic or not. The "amusing and telling" response was, and I quote:

"i'm getting so sick and tired of hearing excuses and rationalizations. just put the cd in the cupholder, install it and sell it. period. there's no need to analyze or certify. what is so hard about this?"

As far as I can tell, the only thing that tells us is that I was right and the community really is full of people who need to grow the hell up.

You almost have to feel sorry for Dell when people say things like that. It's a classic case of "damned if you do, damned if you don't." First, people complain that they don't pre-install Linux because, after all, you just have to stick in the CD and wait. But if they did that, people would complain that the system came unconfigured and they had to just start over from scratch anyway.

It reminds me of an interview with someone from Loki Games that I read years ago. (Wish I could find the link....) For those who don't remember, Loki was in the business of porting popular PC games to Linux. They went out of business in late 2001.

Apparently, they got complaints from some people that they couldn't download their games for free. Sure, the game engines were free and ran on Linux, but the media were all licensed from the original publisher, so Loki couldn't give them away even if they wanted to. But people didn't care about that. They said they wanted games that were open-source and ran natively on Linux, but not enough to pay for them. And yet some still felt entitled to complain about it. I guess it's just human nature....

Last week, HConsumer published an article entitled 30 Days with Linux, in which the reviewer spent a month using Ubuntu. I have to say that this is probably the single best Linux review I've ever read.

Thre were two things thing I really loved about this article. First, I thought it was extremely fair. The reviewer seems to understand and accept that when 90% of the world uses Windows, there is a price associated with choosing to use anything else. In particular, he didn't blame Linux or Ubuntu for things like not supporting his favorite software or doing things differently than Windows, which you see that far too often in this kind of review. That little bit of understanding really made it clear to me that the reviewer was trying to give an honest assesment, as opposed to just cranking out enough words to fill a column.

The second thing I loved about this article was the length of the review period: 30 days. I think it's difficult, if not impossible, to fairly evaluate an operating system/environment in much less than that. Every system is different, and it takes time to adjust when switching from one to another. You need time to figure out how things are done in the new system, to find the shortcuts, and to really appreciate the more advanced features. A few days or a week might suffice for a shallow overview, but those aren't really useful to anybody.

I hope we see more articles like this in the future. The computing trade press has a nasty tendancy to be shallow and trashy, so it's nice to have some thoughtful, open-minded criticism.

Linux has spoiled me. I've gotten so used to things "just working" that I sometimes forget just how miserable the Windows world can be.

That probably sounds weird to some people, but it's true. Granted, lots of hardware vendors and ISVs don't even pretend to support Linux, game companies largely ignore it, and many online services don't seem to be aware of its existence. But there is an up side to that - no malware or crapware.

This was driven home to me he other day when I had to "fix" a user's flash drive. The flash drive in question was a SanDisk Cruzer Micro which the user had purchased to replace his "official" one which broke. The short version of the background is that this user needed a flash drive to work with a system I'll call TCSFOUBTS (pronounced 'ticks-fouts'), which stands for The Crappy Software Foisted On Us By The State (not its real name). Since the "official" keeper of the flash drives had no extras and it would take days or weeks to get a replacement (welcome to civil service), he just bought one on his own.

As with all such cases, this didn't work out well. TCSFOUBTS, being a crappy Widnows application, requires an administrator to set drive letter at which the flash drive will be mounted. However, SanDisk saw fit to include something called U3 on the drive, which adds an annoying systray icon that allows you to install and run software from the flash drive.. This apparently adds a second partition to the drive, which Windows sees as a CD-ROM. Naturally, this is the first partition, so instead of the writable portion showing up as D:, it shows up as E:. Needless to say, TCSFOUBTS was configured to use D:.

The up side is that SanDisk was considerate enough to provide a removal utility for this crapware. It would be better if they didn't include it in the first place, but as crapware purveyors go, this makes them a good citizen. Most of the companies that push this stuff go out of their way to make it difficult to remove. Funny how even annoying things can look good by comparison.

I'm with Clinton Forbes: I don't buy computer books either. Almost never. In fact, I can count the number of computing books I've bought in the last five years on my fingers. If I need information on a technical topic, I go to the web, not the bookstore.

It's not that I don't like books. I love books. My Saturday morning routine is to drop my wife off at work and then spend an hour or two sitting in the café at the local Barnes & Noble, reading and drinking coffee. (There are actually about a half-dozen regulars in the café on Saturday mornings, and I'm the only one under 40. I'm not sure what that says about me.)

It's just that technical books, like those 800 page volumes on C++ or VB.NET, are generally a huge waste of time, space, and money. They're almost universally boring, often poorly written, and usually cost upwards of $40 a copy. On top of that, the information they contain will be usually be outdated in five years and most of it is available for free on the web.

There's also the space issue. I have never in my life had enough shelf space for all my books. And if it comes down to choosing between my leather-bound volumes of Nietzsche and Plato or Teach Yourself C++ in 21 Days, it's easy to pick the winner. Heck, even my 30-year-old, second-hand Bertrand Russell paperbacks, which I bought for $1.50 each in 1998, would beat that out.

But ultimately, the real reason is that I'm a competent professional. I can read online documentation, apply concepts acquired from past experience, and follow articles and blogs written by the experts. I don't need the hand-holding, walk-through kind of books you see in the stores. If I'm going to buy a book, I want it to open my mind to new ideas, not show me yet another iteration on "Hello, world."

Slashdot carried a story today linking to a blog post about Dell deleting a negative Linux-related post from their IdeaStorm site. There was angst, cries of censorship, and so forth. Too bad the /. editors didn't actually read the deleted post first.

For the record, I think Dell did absolutely nothing wrong here. If you don't believe me, go to the blog post linked above and read their screenshot of the removed post. It's a single-paragraph, semi-coherent diatribe with no useful or constructive content whatsoever. It's just one step above, "Hey Dell, you suck!"

Frankly, I'm glad Dell deleted this, because it makes the Linux community look like it's populated entirely by petulant teenagers with an overdeveloped sense of entitlement. "What do you mean I can't have the moon on a stick?!? Screw you!"

As far as I can tell, Dell hasn't said they won't ship systems with Linux pre-installed. They've just said they're not going to do it right away. And this is a problem? I mean, what did people expect? Dell is a huge corporate bureaucracy with established business practices. It's not like they're the local computer store and they can just hire a couple of college kids to start installing Linux tomorrow. There's analysis to do, tests to perform, procedures to revise, employees to train. Organizations like Dell don't just change overnight and expecting them to is foolish and unreasonable.

The bottom line is that Dell is no longer going to pretend that Linux doesn't exist. This is unquestionably a good thing. Dell has given us an inch. The appropriate response is to push for more, not spit in their face because they didn't give us a mile.

It's finally happened: on Tuesday, Steve Jobs came out against DMR-infested music. He basically blames the need for DRM entirely on the music studios and says that Apple would whole-heartedly support the distribution of unencumbered MP3 files.

This is great news, if it's true. But as Jon Lech Johansen, a.k.a. "DVD Jon", points out in several interesting blog entries, Steve's reasoning sounds a little fishy. The Gizmodo people are also a bit skeptical about Steve's sincerity on this.

I suspect they're probably right. Steve may blame DRM entirely on the studios, but as Jon points out, Apple never gave a second thought to putting DRM restrictions on all tracks, whether the studio demanded it or not. Maybe it's just me, but that makes Steve's "man of the people" schtik a little harder to swallow.

I guess time will tell. Good ole' Steve wasn't much concerned about empowering his customers when it came to the iPhone. Maybe he's actually serious about it this time.

Dealing with other IT organizations is always interesting. Not only is it fun to talk shop with the techs, but you can get some humor value out of the business and bureaucracy end.

Today, for example, I was at one of our public safety sites supporting a vendor tech installing a voice recording system. I say "supporting," but I actually spent half the time pacing around and playing sudoku on my cell phone, because the tech pretty much had everything under control. I was just there so nobody could complain that our IT department wasn't involved.

Anyway, while the on-site tech was pretty on the ball, the rest of his company was a little...off. My first indication was when they sent us the install files for the recorder's client access application. They e-mailed me their FTP server installation and the file name to download: "our_org clien installer.zip". No, that's not a typo, the file name actually had "clien" instead of "client." What really kills me is that not only did someone mistype the file name, but they sent us the misspelled file name rather than fixing it. That seemed just a tiny bit unprofessional.

The second sign was the contents of this ZIP archive. It contained one setup file and three pieces of documentation, none of which were even tangentially related to the client application. However, if I had been responsible for the ANI and ALI circuits coming into the PBX, it would have been very helpful. And it turns out they only sent me one of the three client apps we needed. Makes you wonder if the sales guy was even paying attention.

You know what I noticed in my hosting company research? Nobody's home page is valid. Of the few dozen I've looked at so far, not a single one has had a home page that goes throught the W3C's HTML validator cleanly.

To me, that's a real turn-off. I know that the vast majority of web sites in existence are, frankly, complete and total garbage from a coding standpoint. I can accept that. I can even accept that many of them have a good reason for being complete and total garbage, like a sickeningly expensive legacy CMS. But for tech companies that exist almost entirely on the web, and many of which offer web design services, this is just disgraceful. To me, it indicates that they just don't take their web design work seriously.

And you know what? It's not the fact that some of them have crappy markup that bothers me. It's not even the fact that most of them have crappy markup. It's that all of them do. Out of all those sites, I would have expected at least a couple of them to have valid markup.

Is this really too much to ask? I mean, it's almost 2007, for crying out loud! Get with the program, people! At the very least, take a little damn pride in your work and put up a site with half-decent code. Remember, just because it looks good doesn't mean it is good.

For the last few weeks, I've been looking into new web hosting. My current host is, frankly, a bit crappy and I'm looking to trade up. The service hasn't been bad, but it hasn't been great either, and they don't really offer much in the way of features. In particular, I'm looking for subdomains and imap e-mail, which they don't offer.

So now I'm in the middle of the daunting task of picking a new host. I'm finding it quite difficult to locate reliable-sounding information on how good any of these companies are. After all, half of them promise you the moon on a stick for $5 a month with 99.9% uptime and the Dalai Lama working the support line, so it's not like you can take their word.

There are lots of hosting review sites, but many of them appear to be full of shills, with short, useless positive "reviews" that inspire no confidence. So far, Web Hosting Jury is one of the more credible looking review sites I've come across. The reviews tend to be a bit longer and there are lots of highly negative reviews (which means they're not shills). They also have dates and IP addresses attached to the posted reviews, which adds a bit of confidence.

Right now, I'm leaning toward the Lunarpages basic hosting plan. It includes basically everything I want and is pertty well rated. I'm going to be doing a little more research, though, as the last three hosts I thought I'd decided on ended up looking not so great on closer inspection.

Dennis Forbes posted a good article on the topic of programmer compensation the other day. It's an interesting counter-balance out the latest incarnation of the "programmers are unique and special people" meme, which is that programmers really aren't all that interested in money.

Am I the only one who's getting sick of this meme? I mean, do you see this sort of thing in other professions too, or is it just in the computer industry? Do doctors and lawyers sit around talking about how they're fundamentally different from people who do other types of work? It's kind of hard to imagine that.

If you subscribe to the same RSS feeds I do, you've probably come across this ZDnet blog by now. It was plastered all over the net a couple of days ago.

I don't know why everyone picked it up. It's really not that interesting or insightful. It's not blindingly stupid, but it doesn't say anything that hasn't been well known for years.

What I find interesting is that, for 90% of the desktop Linux problems the author mentions, there's not a damned thing anybody in the community can do to solve them. Believe it or not, this is actually a very, very good thing. Why? Because it means desktop Linux has come so far that we've solved all of the big problems that we can.

Just consider the big three problems Kingsley-Hughes mentions: software, hardware, and games. Aside from half-solutions like emulation and virtualization, there's not a thing we in the community can do about support for proprietary software and games. If the vendors don't want to support Linux, we're out of luck.

This is less true of hardware, as we can always write our own drivers. However, there's no way the community can keep up with every exotic, no-name device on the market (though we're doing a good job on the more common devices). The vendors are the ones with all the knowledge of these devices, and if we don't have their support, we can't help but spend our time playing catch-up.

Of course, the lack of hardware support extends to the point about hardware being difficult to install. I'm sure configuring hardware could be easier, but when it comes to devices that are only experimentally supported, what can you expect? When the device itself is just barely working, it's not reasonable to expect that the developers will have a nice, polished, newbie-friendly configuration GUI. Again, this wouldn't be a problem if we had vendor support.

The last two "problems" listed are simply "grin and bear it" type problems. You can't shut up the zealots and you can't make everyone agree on a single distribution (nor can you make people stop coming out with new ones, what with that whole "free software" thing). It's the price of freedom. It's unfortunate that some people find that a turn-off, but we'll just have to show them other reasons to join the community. If we keep making progress like we have been since I started using Linux about seven years ago, that shouldn't be hard.

Here's some Friday humor for you: If Architects Had To Work Like Web Designers.... Quite the humorous little piece. It comes from a web developer's point of view, but much of it applies to any kind of custom development.

It wasn't complete, though. One thing I missed was the requirement for psychic ability.
Customer: "Why doesn't it do X? It absolutely, positively has to do X or it's useless to us!"
Developer: "X? What's X? Nobody ever even mentioned X until just now!"
I guess that wouldn't fit into that piece, though, since it's set at the beginning of a project and the psychic ability conversation doesn't happen until you're in the middle of delivering.

I just saw something I've never seen before: a TV commercial attacking net neutrality. Apparently they've started appealing directly to voters now.

I must admit to having some degree of ambivalence about net neutrality. Let's face it: it's just one set of big companies fighting another. Who should pay for the internet: Google or Time Warner? Does it really matter to me which one it is?

Not that I have no opinions on the issue. I'm definitely not a fan of the "tiered" service concept. Discriminating against certain kinds of data doesn't appeal to me either. When you take the money out of the calculation, none of this is good for the end user.

On the other hand, you really can't take the money out of the calculation. And it's not like the service providers don't have a point. Somebody has to pay for the cost of providing bandwidth, and a non-neutral scheme might very well result in lower overall costs and/or lower costs for end-users. At least, that's the claim. I don't claim to know enough about the business to evaluate its truth.

I think the only thing I'm really sure of in this discussion is that getting the government involved is a bad idea. In fact, as a public servant, I take it as a general rule that getting the government involved is nearly always a bad idea. And what with the DMCA and software patents, it's not like the US government has the best track record on technical issues.

So for now, I'm more inclined to let the market decide this issue. Who knows, the non-neutral net might not even be really feasible. We can only hope....

Periodically, you see stories making the rounds about how online applications are going to completely supplant traditional desktop applications. In fact, these ideas have recently been extended to encompass the entire desktop, with the rise of web-based "operating systems."

It sounds great, doesn't it? All your data and all your applications would be available from a central server. You could use any computer, anywhere in the world, to access your personal, customized desktop, and it would always be exactly the same.

However, over the last month or so, and this week in particular, I've experienced the perfect proof that such ideas are, well, over-rated. That proof is internet outage.

Yes, Time Warner's Road Runner cable internet service has been very unreliable the last month or so. It's normally pretty good, but I've been experiencing frequent outages, usually for several hours at a time.

With wide broad-band availability, many of us have started to take high-speed, always-on connections for granted. Putting your entire desktop online is great when you know you will always be able to access it. But if everything is online, then when your connection goes down, your computer is completely useless.

The everything online philosophy also seriously limits the usefulness of laptops. I know that may sound shocking to some people, but the truth is that you can't get free WiFi access everywhere. In fact, there are many places where you can't even get paid WiFi access. For instance, my wife sometimes takes the laptop to work on slow days, where they have no web connection (though I'm not sure why) and no wireless access points nearby. On those days, it's nice that OpenOffice and Klickity (her new addiction) are desktop, rather than web, applications.

Not that I have anything against web applications. They're great! It's just that, like everything else in the world of computing, they've been over-hyped. Not everything needs to - or should - be a web application. Not every web application has to use AJAX. Not every program will benefit from using the latest trendy technology. And, finally, one that seems to have finally sunk in: not every application has to incorporate XML in some way, whether it makes sense or not.

It seems that the MEPIS people have finally decided to comply with the GNU GPL. You may remember this issue from the Newsforge story a month ago. Basically, the SimplyMEPIS Linux distribution, which is a derivative of Ubuntu (and was formerly a derivative of Debian) got in trouble with the FSF for not complying with the terms of the GNU GPL. Basically, while they were providing the source code for the Ubuntu packages they modified, they were not providing code for the packages they copied unmodified from Ubuntu. Apparently they figured that as long as the source is "out there," that was good enough.

However, it doesn't work that way. The fact that the source is "out there" is not enough to satisfy the terms of the GPL and it never has been. And if they'd bothered to read and understand the license, they would have known that. The GNU GPL cleary states that if you distribute binaries of GPL-licensed software, you must either include a copy of the corresponding source, whether in the same box or from the same web site, or include a written offer to distribute the source on demand. There's nothing in there that says, or even suggests, that this only applies if you make changes to the code.

The main argument from MEPIS and others seems to be that this provision of the GPL is onerus and stifles innovation. Managing all that extra source code implies a lot more work, and hosting and distribution for that code implies more expense. The idea seems to be that since the code is "out there," it's not reasonable to force all this duplicate effort on small-time Linux distributors. Why, it just might be enough to discourage many from even building their own distribution! In fact, it's even a burden on people giving out Linux CDs to their friends, since they are techically required to give out the source too! And, really, who even cares if they have the source? Just those communist GPL police!

Of course, to be honest, they have a point. Distributing source and binaries is definitely harder than distributing just the binaries. Likewise, hosting and CD replication for both source and binaries is more expensive than for just binaries. I'm sure there are some people who would be put off starting a distribution because of this.

But on the other hand, so what? The stated purpose of the GNU GPL is to make sure that anyone who receives a program licensed under it is able to change and share the program, not to make things easy for Linux distributors. Requiring that the person who distributes the binaries also distribute the source is the simplest way to accomplish this. Sure, it's more trouble for the distributor, but using the "the source is out there" approach would leave the user open to a never-ending game of "pass the buck." He'd be entitled to a copy the source, but no particular person would be obligated to actually give him a copy of it. And if that happened, then the program would be free software in name only.

I just find the whole line of reasoning put forth by MEPIS really annoying. Maybe complying with the GPL would be burdensome. Maybe their users really don't care about the source. That's not the point. The point is that they don't get to make that decision. The GPL gives you license to redistribute the software it covers under a specific set of conditions. You can abide by those terms or you can violate them and infringe on the owner's copyright. Just don't try to argue that you ought to be able to pick and choose which terms you want to follow. It doesn't work like that. A proprietary software company certainly wouldn't put up with such nonsense and I don't see any reason why the owner of a free software package should be expected to.

You know what the best part about free software is? Not having to worry about licenses. Oh, I know you still have to think about them when you're modifying or redistributing the program, but if you're simply using it, there are absolutely no worries.

In the commercial world, this is not the case, as I was reminded the other day.

You see, I'm the poor schlep in the IT department that got stuck taking care of the AutoCAD licenses for our engineering department. We have a subscription to a couple of AutoCAD products, which includes support and version upgrades. Every year, something goes wrong with our AutoCAD subscription. At first, it was the fact that we had four seats of AutoCAD on three separate subscription contracts. Then we got them partially consolidated onto two contracts, and then finally down to a single one.

This week, it was a license transfer from one of our other departments to engineering. In addition to the transfer, we needed to add subscription support, because the other department was no longer using AutoCAD and had let it lapse. I didn't handle the paperwork for this, so I called up to see if everything had gone through. Well, this time, they messed up by quoting us a new contract rather than adding the new license to our existing subscription contract. They also didn't process our order until I called - 2 months after the purchase order was written. What's up with that? They don't want our thousand dollars?

Of course, I've been referring to the people involved as just "they" so far. There are actually three companies involved in our AutoCAD dealings. Of course, there is Autodesk, who owns the software and apparently never deals with customers directly. Beneath them is DLT Solutions, who are apparently the exclusive government reseller (at least in our area). And below them is our local reseller.

The problem seems to lie primarily with DLT. Our local reseller is very good. They're friendly, knowledgeable, and know how all the licensing works. It's DLT who keeps adding a new support contract for every purchase (keeping track of multiple contracts is a big and unnecessary pain) and who didn't bother to process our order for two months. Unfortunately, since they're the government reseller, it's not like we can just go to somebody else. Not unless we want to pay retail, which is 20% more taxpayer dollars.

Of course, free software wouldn't save me from the horrors of dealing with support contracts. However, it can and does save me from the horror of begging for money for half-way decent tools. It also saves me from having to worry about whether or not I'm going to get us audited by the BSA.

Another nice thing about free software is that, if the users like it, I don't have to hear complaints about too few licenses. In particular, I'm thinking of this one guy in engineering (he fancies himself the local "computer guru") who is constantly asking me to install proprietary software on multiple PCs. He always brings up that the licenses for several programs he used to use said that you could install them on any number of PCs, so long as you only used one copy at a time. Of course, that scheme still exists - in the form of software that requires a license server. But when it comes to things like Paint Shop Pro, that's not exactly the typical scenario. And let's face it - acting based on what the license for an old piece of software used to say isn't exactly a sound legal strategy. Especially when you haven't actually read the license to the software you're using.

Let's continue my "bad argument" theme from yesterday with a news item from Slashdot. This one is regarding a ZDNet report that Microsoft representative Alan Yates claimed the MS Office XML format is preferable to the Open Document Format (ODF) for performance reasons. For support he cites this ZDNet blog by George Ou, which compares MS Office to OpenOffice.

Now, it may or may not be true that the Office XML format can be implemented more efficiently, in terms of space and manipulation speed, than ODF. I really don't know enough about the issue to say one way or the other. However, I do know that the ZDNet blog entry Mr. Yates cited doesn't say much of anything about it either. In other words, the argument is bad because the cited evidence doesn't really support the conclusion.

So what does the blog entry say about Office XML and ODF? Well, it gives an example of two 16 sheet spreadsheet files with the same data, one in Office XML format and one in ODF format. The author found that OpenOffice Calc used around three times as much memory and 100 times the CPU time as Excel.

Does this mean that Office XML is faster than ODF? No. It just means that Excel 2003 is faster than OpenOffice Calc 2.0. Given that OpenOffice has a history of being slow and bloated (version 2 is better, but still not great), this should come as a surprise to no one. I'm sure that if you compared the two programs using the same file format, Excel would still be faster.

So as far as the file formats go, this entry is comparing apples to oranges. And why shouldn't it? After all, Mr. Ou's original point was to compare the speed of OpenOffice to MS Office, which is exactly what he did. He's just had his words taken out of context by a Microsoft spokesman to support an argument he never tried to make.

Well, I finally did it.  I bit the bullet and bought a laptop.  I just finished ordering an Inspiron B120 from Dell.  It's basically the cheapest laptop they sell, but they were offering $150 "instant savings" plus discounted shipping and no-charge upgrades to half a gigabyte of RAM and an integrated wireless card, so I figured I'd go for it.  

Eventually, I'll be putting Kubuntu on this machine, but I did shell out the extra $10 for the Windows CDs anyway.  If nothing else, they may come in handy for use in VMware.  I'm also not sure how Linux-friendly the WiFi card is, so having Windows around might be useful.  From the research I've done, I'm lead to believe the card is a Broadcom chipset that can be made to work with ndiswrapper (yuck), but I'm not entirely confident in my ability to get that working in a timely fashion.  It doesn't help that the laptop doesn't have any PCMCIA slots, so it's not like I can just get a card to stick in it.  I guess I'll see how it goes.

I have to say that, during the checkout process, I saw a few things that made me uneasy.  First, there was an option on the payment page for paying with two credit cards.  And, when I paid with a single card, there was an option for the daily charge limit.  Now, maybe I'm just cynical (which I am), but does it sound to anyone else like Dell's business model is to sell over-priced computers to people who can't really afford them?  It kind of seems that way to me.

And just to be fair, it's not that Dell's systems are over-priced.  They're actually really cheap.  The base systems are cheap, that is.  It's just the "upgrades" and add-ons that are over-priced.  Then again, it's not like any of the other vendors are any better in that regard.