LinLog

The other day I listened to a very interesting talk by Allen Holub on the #NoEstimates movement, which I've posted about before.  It's definitely worth a look if you have some time.  Mr. Holub makes the slightly controversial claim that all estimates are evil and we should all just stop doing them immediately.  His rationale for this is essentially that estimates are made up and harmful.  And, of course, he's not wrong.

I've always been a little skeptical of the #NoEstimates movement, but the thing it clearly get right is that estimates are frequently misused and misapplied.  Just about every developer has a story about estimates being interpreted as promises, or having to work late just to "meet your sprint commitments", or skimping on quality because the sprint is about to end and you need to close out your tasks.  And we won't even get into managers who try to use estimates as a performance evaluation tool - the dysfunction of this should be obvious to anyone with any experience.

Is it the estimates?

So estimates can be used for bad things.  That's not controversial.  But that's also not a problem with estimation per se.  I guess you could think of it as the software equivalent of "guns don't kill people, people kill people."  It's not that estimates are inherently bad, it's just that they're seldom used for good.  That doesn't mean that you should abolish estimates, but it does mean that you should use them with caution.

The thing about estimates is that, while they're often just guesses, they don't have to be.  In my study of the Personal Software Process (PSP), I learned about doing estimates based on historical data.  I've even seen it work - in my last job, I reached the point where my time estimates were within about 15% of actual task completion time, which I consider to be pretty good given that the estimates were in minutes.  Granted, it doesn't always work - sometimes you have tasks that are truly unprecedented or have too many unknowns - but it can be done.

The main problem with data-based estimation is that most organizations and developers are not equipped to do it.  For one thing, it requires data, and a lot of shops don't actually collect the kind of data that would be useful.  Most developers certainly don't collect the kind of data that the PSP estimation method uses.  It also takes more time than many people seem to want to devote to estimates.  It's not that much time, but still a lot longer than saying, "well, I guess that's about eight story points".

What's the point?

The real question here is: why do you want an estimate?  What are you trying to accomplish?

When I do PSP estimates, I'm using them to gauge my performance.  I'm looking at my historical data and using that to project out time, code volume, and defect counts.  I can compare these projections to my actual results, see if my performance is consistent, and if not, analyze the reasons and look for ways to improve in the future.

Yes, these are "estimates", but not in the same sense as most teams do them.  My manager never sees these "estimates"; I make no commitments based on them; nobody depends on their being accurate; in fact, nobody but me even knows what they are.  They're a part of my task planning, so they're short-term estimates - they go out two or three weeks at most, usually more like a few days.  I start with some requirements analysis, sketch out a conceptual design, and then do the estimate based on that.  So the process of coming up with the estimate actually helps me flesh out and validate my plan, which is valuable in and of itself.

The important thing to note in the above description is that my estimates are not targets.  If I'm working on a task and it takes 500% longer than my estimate, I'm the only one who knows and I don't care.  Sure, I'll try to figure out why that task took so much longer than I thought, but the fact that "I missed my estimates" has no relevance and no consequences.

But that's usually not how it works when your boss or your project manager (or even your scrum master) asks you for an estimate.  They want you to tell them when the task will be done.  And they'll be very disappointed if it's not done when you said it would be.  And that's where the dysfunction begins....  Because even in the best of circumstances, we don't really know exactly how long s development task will take.  And we seldom work in the best of circumstances.

So...no estimates?

The interesting thing about doing PSP estimates is that, until relatively recently, I had trouble not thinking of the estimates as targets.  Process Dashboard, my PSP support tool, has a progress bar that counts down the time remaining for your estimate and then turns red when you go over the estimate.  And I would actually get upset when that bar turned red!  Why?!?  There was no reason for that - there was literally nothing at stake!  I was just psychologically stuck in the wrong mindset.  And that's the same mindset that most people get stuck in when they talk about estimates.

So I'm starting to come around to this "no estimates" thing.  I still believe that estimates can be useful and productive, but they usually aren't.  And if your company is particularly enlightened and can do estimates in a way that's productive and avoids the pathological effects associated with them, then by all means, keep doing estimates.  But most people don't work in companies like that.  And even if your company is great, you still need to be careful of the psychological pressure created by the inherent biases and preconceptions of your developers.

In other words, estimate carefully.  Consider simpler, lower precision (but not necessarily lower accuracy) methods like Mr. Holub and the other #NoEstimates advocates describe.  Don't be fooled - those are still estimates, but they at least make an effort to work around the damaging side-effects associated with traditional estimation.  After all, it doesn't matter how good the estimates are if they destroy team morale.

For the last six months or so, I've spent a lot of time reading about the SOLID principles and listening to lectures about agile development practices by the likes of Bob Martin, Dave Thomas, and Martin Fowler.  The result has been a new and different understanding of what "agile" really means.

One of the agile practices I've really warmed up to is test-driven development. Uncle Bob advocates very strongly for it, and it was one of his talks that got me to finally try it again.  I'm too lazy to find the exact quote, but it was something like:

What if you had a button on your keyboard that you could press and it would tell you if your code was working?  How would that change your life?  Test driven development gives you that button.

Of course, that's a bit hyperbolic, but the point still stands.  The idea that you could have a tool that could tell you, "yup, this code works as intended" is really powerful.  Even if building such a tool is a lot of work, it seems like it would be worth it.

But the thing is, I've tried TDD before.  In fact, I've tried it for about a week or so every year for about the last five years.  And you know what?  It never really worked for me.  It was awkward, the tests were brittle and hard to read, and it just didn't seem to help me much at all.  I came away thinking that it was a stupid fad that didn't help and didn't really make sense. 

But things were different this time.  I'd spent months immersing myself in the SOLID principles, object-oriented design, and techniques for writing good tests.  This time I was ready.  It turns out that knowing how to structure your code and your tests makes the entire process infinitely smoother and simpler.  Heck, it's even kind of fun.

TDD Results

So how did it go this time?  Well, I'd say it was fairly successful.  I've been able to actually use TDD and stick with it without feeling like I was wasting my time.  That alone is a big improvement from my previous attempts.  But I can go deeper than that.  Thanks to the data I've collected using the PSP, I can actually do some analysis of just how well TDD is working for me.

About the Process

Before getting into the details, let's talk about what my process looked like before and what it looks like now.  That will help give some context to the numbers and make it more obvious why they look the way they do.

My process is based on the (now deprecated) PSP 3.  For each task, I start with a quick high-level design sketch, and then one or more development cycles, each of which consist of design, design review, code, code review, and then test.  The "design" phase consisted of conventional design and some skeleton coding (defining interfaces, defining UI components, outlining algorithms, etc.), the "code" phase consisted of fleshing out those outlines and implementing the rest of the design, and the "test" phase was writing unit tests followed by manual system-level testing.

The new TDD-based process changes that a bit.  The "design" phase is very similar, but I've been going into less detail in the algorithms involved.  The idea was to let the detailed design "emerge" in the process of writing tests, though I've started moving away from that a little.  The "code" phase is now a series of TDD cycles.  That means that it encompasses both actual coding and unit testing.  The "test" phase has become just manual testing.  I did this because, realistically, trying to track test vs. code time when I'm switching back and forth every few minutes was going to be tedious and error prone and I didn't want to do it. 

Unfortunately, this makes the analysis much harder because we're not comparing apples to apples.  Both processes did try to achieve comprehensive unit test coverage, so they're not completely incomparable, but the change in definition of the "testing" and "coding" phases does make certain comparisons moot.

About the Data

I've been collecting PSP data on the coding tasks I've had over the last six months at my current job.  Currently I have 25 data points - 15 using the old process and 10 using TDD.  The data is collected on a per-task basis, and the task sizes vary wildly, from about an hour and a couple dozen lines of code, to two weeks and a couple thousand lines of code.  The tasks are all webdev work and include a combination of back-end PHP, front-end functionality in JavaScript, and HTML/CSS display stuff in varying proportions.

Of course, the variety in the individual tasks undermines the data a bit, but that can't really be helped.  I'm just working with the tasks I'm given - I don't have the time to do a genuinely scientific comparison.I mean, I do actually need to get some work done at some point - I'm not getting paid to analyze my own performance.  

The Results - Summary

Overall, the TDD-based process seems to be working as well as, if not better than, what I'll call the "conventional" process.  Below is a summary table of overall metrics:

For the two data sets, the total time and code volume were substantial and in the same neighborhood, so the data sets are at least comparable.  We can see the TDD data shows higher productivity (in LOC/hour), fewer defects, better review yield, and lower cost of quality. 

So this means that TDD is definitively better, right?

Well...no.  The change in definition of the "code" phase means that I'm logging fewer defects in general.  Part of my defect standard is that I don't record a defect if it's introduced in the same phase where it's fixed.  So with TDD, little things like typos and obvious logic errors are caught by unit tests in the code phase, so they don't show up at all in these numbers.  In the conventional process, they wouldn't have been picked up until code review or testing.  The same reasoning applies to the failure cost-of-quality - since writing unit tests is now part of the code phase rather than the test phase, less time would naturally be spent in testing because there's simply less work that needs to be done in that phase now.

However, the productivity difference is still interesting, as is the improvement in percent yield. Those do suggest that there's some improvement from using TDD.

Drilling Down - Time

So since the change in phase definition makes defect numbers incomparable, let's look at time data instead.  This table shows the percentage of time spent in each phase.

 The results here are largely unsurprising.  The percentages are roughly equivalent for most of the phases, with the notable exceptions of code and test. 

However, it is interesting to note that there's still a discrepancy between the total code+test time for the two processes.  Theoretically, based on the phase definitions, that combination should encompass the same work for both processes.  But the TDD process spent 67.2% of the time on coding and testing, but it was only 50.1% for the conventional process.  It appears that the single largest chunk of that disparity comes out of design time.  This makes sense because there was less detailed design in the TDD process.

Note that the review times were roughly equivalent for each process - in fact, they were slightly lower with TDD.  Yet the percent yield for reviews was higher with TDD.  This gives us some evidence that the difference in percent yield can be attributed is likely due to the use of TDD.

It's not entirely clear why this would be the case.  The most obvious outcome would be lower yield, as the use of TDD means that fewer bugs escape to be found.  My working hypothesis, however, is that using TDD lowers the cognitive load of code review by removing most of the noise.  With TDD, you already know the code works.  You don't have to review for things like typos, syntax errors, or misused functions because you've already tested for those things.  That leaves you more time and energy to dig into more substantive issues of design, requirements coverage, and usability.

Back to Defects

Hmm....  Maybe we should take another look at the defect data after all.  We've already established that number of defects isn't going to be useful due to the change in phase definition.  But what about average fix times?  If TDD is weeding out the simpler bugs so that they don't escape the coding phase and don't get recorded, we'd expect the average fix time to be higher.

The above table shows the break-down of defect count and fix time by injection phase.  So what does this tell us?  Well, my hypothesis that TDD would produce higher average fix times seems to be correct.  If we look at the fix times for defects injected in code and found in test, the average fix time is about 30% higher. 

However, it's worth noting that the TDD data has higher average fix times across the board.  It's not entirely clear why this should be.  One possible explanation is that the use of TDD means that unit tests are introduced earlier in the process, so defects fixed in code and code review would require changes to the test suite, whereas in the conventional process that would only happen for defects found in test.  That's something I'll have to watch in the future.

Conclusion

So far, my new TDD-based process seems to be working out pretty well.  The results are somewhat ambiguous, but the numbers suggest that TDD is resulting in higher LOC/hour productivity and more efficient defect removal. 

From a more human perspective, it has the benefit of making unit testing much less tedious and painful.  It gets you earlier feedback on whether your code is working and gives a nice sense of satisfaction as you watch that list of test cases grow.

But ore importantly, TDD is a great way to test your test cases.  When you're writing your unit tests after the fact, it's very easy to get stuck in a situation where a test is unexpectedly failing (or, worse, passing) and you can't tell whether the problem is with your test case or the code under test.  With TDD, you're changing things in very small increments, so it's easier to pinpoint the source of problems like that.

But the big lesson here is that TDD isn't something you can just jump into.  You have to understand the context and the design techniques first.  If you do, then it's pretty great.  If not, then you're going to have a hard time and end up wondering who came up with this hair-brained idea.

Welcome to the end of my series of PSP posts.  In part one, I started with an overview of the Personal Software Process.  Part two covered the process of learning about the PSP and how to apply it.  Part three was the sales pitch of nice things that the PSP is supposed to enable you to do.  Now, in part four, we'll get down to brass tacks and talk about how well the PSP actually works for me.

I'll start with a quick overview of the good, bad, easy, and hard parts.  Then I'll dive into a deeper discussion of my experiences and the details of what did and didn't work.

PSP at a Glance

This table gives you a nice little overview of my experience.  It rates various parts of using the PSP on two axes - whether I judge them to be useful or not (good/bad) and how difficult they are to apply in practice (easy/hard).

As you can see, I find the benefits of the PSP to outweigh the drawbacks.  For me, it's useful enough that I plan to keep using it, in some form, for the foreseeable future. 

Using the PSP

Despite what you might read about how cumbersome the PSP is to use, I actually didn't find it that difficult at all.  Granted, it takes a little getting used to - every process change does.  But once I had the basics down, actually using and sticking to the process wasn't that hard.  The use of written scripts with well defined entry and exit criteria helps keep you honest and disciplined.

Likewise, with the help of Process Dashboard, I found the entire data collection process to be relatively painless.  There are a few pain-points, of course.  For me, the biggest one was simply properly configuring a line counting tool so that you can measure project size.  This is actually more annoying than you'd think.  I use the integrated line-counter in Process Dashboard to count change size and cloc to measure initial size for estimation purposes, mostly because I've integrated it into my IDE.  The Process Dashboard tool has some nice features, but will require you to write custom language definitions for pretty much anything that doesn't use C-style syntax.  It uses a fairly easy to follow XML format for configuration, but still....  Cloc has much better language support, but is harder to customize.  As a further annoyance, while Process Dashboard does have VCS diff support, it currently only supports Subversion.  So if you're using Git, Mercurial, or anything else reasonably modern, you'll have to set up two copies of your local repo for before and after comparison.

Code Review

Personal pre-commit code review is one of those things that every responsible developer does, but hardly anyone seems to talk about.  I know I've been doing an informal version of it for years.  It simply consisted of looking over the diff of my changes before hitting the "commit" button and making sure that I didn't make any obvious mistakes, that I'm not checking in any changes I don't mean to, etc.  It's something you quickly learn to do after making a few embarrassing mistakes.

Doing an organize code review with a checklist really takes this to the next level.  Instead of being a CYA thing, code review becomes a way to preemptively find problems in your code.  And at its best, it can be hugely effective.  As any experienced developer knows, there are classes of problem that are hard to find in testing, but stick out like a sore thumb when you actually stop and read the code.

The only hard thing about code review is actually customizing your review checklist.  I find that it's difficult for me to do that simply by looking at the defect categories in my data because those seldom tell you anything actionable.  There are a few defect categories that readily translate into checklist items, but there are many defects that are more subtle and are either difficult to categorization or difficult to generalize into checklist items.

The one thing I really didn't like about the PSP code review process was the recommendation to do it on paper.  Using paper does have the benefit that you can get more code in front of you at a time, and it is easier to make annotations.  However, it also bypasses the navigational and analysis power baked into modern IDEs.  For instance, Komodo let's me easily navigate between functions, access standard library documentation at a click, and search for uses of an identifier.  Those things are much more tedious to check on paper. 

But the big kicker for me was that trying to review a diff on paper is just painful.  It's hard enough on screen with color highlighting, but it really sucks on paper.  And on paper I don't have things like the Komodo diff viewer's feature to jump from a diff item to that location in the file to view the context.  It might work well to review new code on paper, but for changes to existing code it feels really clunky.

Design Reviews

While we're on the topic of reviews, let's talk about design reviews for a minute.  Again, this is a really good idea, for the same reasons that code review is a good idea.  And the PSP does offer some productive advice in the recommendation to adopt a design standard and a checklist for common errors.

However, the specific methods the PSP recommends just don't work for me.  The four standard design templates are a nice idea, but they feel very repetitive and clunky to work with.  And even if I switched to the UML equivalent, the recommended list of artifacts is just too much for a lot of the things I do.  And at the risk of having my CS degree rescinded, I have to admit that I have trouble constructing a state machine for many of the projects I do - at least, one that's even remotely enlightening.  In general, I just find them painful to work with and biased toward "new program" development rather than incremental enhancement.

And the design verification techniques are even worse!  They're time-consuming and tedious - you're basically executing your program on paper.  It's a nice idea, and might come in handy occasionally, but frankly, I have less confidence in my ability to perform those verification exercises correctly than I do in my code being correct in the first place.  And, again, they're just way too heavy for most of the projects I do.

I'm still working on finding a design and design review approach that's sustainable for me.  Since my last few shops have used agile methodologies, most of the "projects" I do are fairly small enhancements to existing code - usually just two or three days, seldom more than a week.  So a heavy-weight design process with lots of templates or UML diagrams just isn't going to work. 

My current approach is as follows (note that I'm using a variant on the PSP3 process in Process Dashboard):

1. Sketch out a high-level design in a word processor.  This is a refinement of the conceptual design used for planning, usually in the form of plain prose and bullet lists.
2. Review that primarily for feasibility, completeness, and requirements coverage.
3. For each component I've broken out, do a more detailed "transient design" (I'll describe that in a moment).
4. Review the transient design primarily for completeness, correctness, and requirements coverage.

I refer to the detailed designs as "transient designs" because I don't actually create separate documents for them.  I blend implementation and design and actually do the design right in the source code.  I generally stub out things like classes and methods and fill in the details either with actual code (for simpler items) or with "design annotations", which are just comments that use a special formatting to mark them as design artifacts.  Sometimes they're pseudo-code, other times they're just descriptions of what needs to be done, whatever seems appropriate.  Then, in the code phase, I simply replace those annotations with the actual implementation.  It's certainly not perfect, but it seems to be working well enough for me so far.  As a next step, I'm going to look at a TDD-like approach and try incorporating unit test definitions as one of the design artifacts.

Estimation

In my experience so far, PSP estimation using PROBE actually works remarkably well. In the data from my last job, I was eventually able to get to the point where my actual development times were generally within about 15% of my estimates.  I consider that to be pretty good, especially when you consider that the estimates were done in minutes and based on fairly sketchy user-stories.

Of course, estimation is a learned skill, and PROBE doesn't change that.  You still need to be able to accurately account for the possible changes when constructing the conceptual design.  And as with any data-driven approach, the results are going to be sensitive to the quality of your data.  So if your relative size tables are just made up rather than being based on your past work, then don't expect your estimates to be too accurate.

It's also important to note that there's a bias against project diversity here.  For example, line counts can differ wildly for different programming languages, different problem domains, etc.  So if you tend to work on projects that are generally very similar to each other, then PROBE will work much better than if all your projects are widely divergent.  My data from my last job is based largely on a single code-base, so while the purposes of the individual projects varied wildly, the technology stack was consistent.

The hardest part about estimation, at least for me, is coming up with those relative size tables.  It's one of those things that sounds easy, but actually isn't.  For one thing, I don't have a tool to automatically count lines and methods in classes - much less one that works across a diversity of languages.  For another, my data largely comes from work, which is a problem because I do mostly web development on a team, which means I need to extract my method and class size data from a code base written in five different languages by five people.  When you wrote a quarter of the methods in one class, half of a third of the methods in another class, etc., how do you count all that?  You can just forget that and count the files you have, but then it's not really your data, so it's not clear how useful it will be. 

I've also found it challenging to come up with useful categorizations for my relative size tables.  Perhaps it's just the products I've been working on, but I end up with a whole lot of database-related classes and a smattering of other categories.  That's fine for those products, it's hard to figure out how to extrapolate that to other kinds of projects.  My suspicion is that that's a result of sub-optimal system design.  I'm currently trying to adhere religiously to the SOLID principles, which should result in more and more targeted classes, which should solve that problem.

Other Bad Things

There are a few other annoying things about the PSP as Humphrey describes it.  While the general focus on templates and checklists is not bad in and of itself, their value tends to vary.  The test report template, for instance, is one that I've not found particularly valuable.  While it is useful to sketch out your testing strategy, or maybe make a quick checklist of your test cases, the test report template is more like something that you'd give to a QA team to do manual testing.  It has a bias towards verbosity that makes it seem like more effort than it's worth.

Likewise with the focus on coding standards.  We can all agree that having coding standards, and following them consistently, is a very good thing.  Everyone should do it.  However, I've been working as a software developer for a long time now and my "coding standard" is something I've long since internalized.  I don't need to spend time formalizing it or checking it in my code reviews.  Ditto the size counting standard.  You can get really fancy if you want to, but to I'm not convinced that anything much more complicated than counting physical lines is likely to be helpful.  I suspect that, at least for my purposes, any elaborate counting standard would just serve to complicate measurement.

And, of course, there's the simple fact of process overhead.  It's really not too bad when you use Process Dashboard, but it's still there.  For example, there's a non-trivial amount of work that goes into configuring Process Dashboard itself.  It's a useful and powerful tool, but its not always simple to use.  There's also the analysis time use to assess and correct your process.  This is unquestionably valuable, but it's still some additional time that you need to plan for.  And, of course, there's just the time to actually follow the process.  This isn't actually that much, but for very small tasks (e.g. one or two hours), your estimates might be thrown off by the fact that your standard phase break-down results in a phase that's one or two minutes, and it takes you longer than that just to type in the data you need.

And Some Good Things to End On

Last but not least, I wanted to highlight two more "good but hard" things from the table above: the planning and postmortem stages.  At first, these seemed like silly, pro forma phases to me, but they're actually quite valuable.  And the most valuable thing about them is something that doesn't show up in the script: they make you stop and reflect.

The planning phase forces you to think about what you're doing.  To construct an estimate, you have to think about the project you're trying to do, break it down, and define its scope.  Even if you don't believe there's value in the estimate itself, simply going through the process gives you lots of good insight into just what you're trying to do, which reduces the number of surprises later in the development cycle and makes everything go smoother in general.

Likewise, the postmortem stage prompts you to reflect on how you're doing and figure out how you can improve.  It's like a one-man sprint retrospective.  And like the sprint retrospective, it's actually the most important part of the process.  The simple task of looking at your statistics and filling out a Process Improvement Proposal forces you to stop and focus on your performance and what you can do to improve your work.  I find that simply looking at that PIP line in the postmortem exit criteria keeps me honest and makes me stop and think of something I could improve.  And if you can't thing of at least one thing you could do better, you're just lying to yourself.

Conclusion

So there you have it.  In many ways, the PSP is working well for me.  Some of Humphrey's suggestions work well out of the box, some don't, and some need tweaking.  I do find that my process is evolving away from the "standard" PSP, but that's neither bad nor unexpected.  The basic techniques and ideas are still useful to me, and that's what matters.

Welcome to part three of my PSP series.   Now that the introductory material is out of the way, it's time to get to the good stuff!  This time, I'll discuss the details of the PSP and what you can actually get out of it.  Theoretically, that is.  The final post in this series will discuss my observations, results, and conclusions.

PSP Phases

As I alluded to in a previous post, there are several PSP phases that you go through as part of the learning process.  Levels 0 and 0.1 get you used to using a defined and measured process; levels 1 and 1.1 teach planning and estimation; and levels 2 and 2.1 focus on quality management and design.  There is also a "legacy" PSP 3 level which introduces a cyclical development process, but that's not covered in the book I used (though there is a template for it in Process Dashboard).  The phases are progressive in terms or process maturity.  So PSP 0 defines your baseline process and by the time you get to PSP 2.1 you're working with an industrial-strength process. 

For purposes of this post, my discussion will be at the level of the PSP 2.1.  Of course, there's no rule that says you can't use a lower level, but the book undoubtedly pushes you toward the higher ones.  In addition, while the out-of-the-box PSP 2.1 is probably too heavy for most people's taste, there is definitely useful material there that you can adapt to your needs.

Estimation

One of the big selling points for all that data collection that I talked about in part 1 is to use it for estimation.  The PSP uses an evidence-based estimation technique called Proxy-Base Estimation, or PROBE.  The idea is that by doing statistical analysis of past projects, you can project the size and duration of the current project.

The gist of the technique is that you create a "conceptual design" of the system you're building and define proxies for that functionality.  The conceptual design might just be a list of the proxies such that, "if I had these, I would know how to build the system."  A proxy is defined by its type/category, it's general size (from very small to very large), and how many items it will contain.  In general, you can use anything as a proxy, but in object-oriented languages, the most obvious proxy is a class.  So, for example, you might define a proxy by saying, "I'll need class X, which will be a medium-sized I/O class and will need methods for A, B, and C." 

By using historical project data, you can create relative size tables, i.e. tables that tell you how many lines of code a typical proxy should have.  So in the example above, you would be able to look up that a medium-sized I/O class has, on average, 15.2 lines of code per method, which means your class X will have about 46 lines of code.  You can repeat that process for all the proxies defined in your conceptual design to project the total system size.  Once you have the total estimated size, PROBE uses linear regression to determine the total development time for the system.  PROBE allows for several different ways to do the regression, depending on how much historical data you have and how good the correlation is.

Planning

As a complement to estimation, the PSP also shows you how to use that data to do project planning.  You can use your historical time data to estimate your actual hours-on-task per day and then use that to derive a schedule so that you can estimate exactly when your project will be done.  It also allows you to track progress towards completion using earned value.

Quality Management

For the higher PSP levels, the focus is on quality management.  That, in and of itself, is a concept worth thinking about, i.e. that quality is something that can be managed.  All developers are familiar with assuring quality through testing, but that is by no means the only method available.  The PSP goes into detail on other methods and also analyzes their efficiency compared to testing.

The main method espoused by the PSP for improving quality is review.  The PSP 2.1 calls for both a design review and a code review.  These are both guided by a customized checklist.  The idea is that you craft custom review checklists based on the kinds of errors you tend to make, and then review your design and code for each of those items in turn.  This typically means that you're making several passes through your work, which means you have that many opportunities to spot errors.

The review checklists are created based on an analysis of your defect data.  Since the PSP has you capture the defect type and injection phase for each defect you find, it is relatively easy to look at your data and figure out what kind of defects you typically introduce in code and design.  You can use that to prioritize the most "expensive" defect areas and develop review items to try to catch them.

As part of design review, the PSP also advocates using standard design formats and using organized design verification methods.  The book proposes a standard format and describes several verification methods.  Using these can help standardize your review process and more easily uncover errors.

Process Measurement

Another big win of the PSP is that it allows you to be objective about changes to your personal process.  Because you're capturing detailed data on time, size, and defects, you have a basis for before and after comparisons when adopting new practices. 

So, for instance, let's say you want to start doing TDD.  Does it really result it more reliable code?  Since you're using the PSP, you can measure whether or not it works for you.  You already have historical time, size, and defect data on your old process, so all you need to do is implement your new TDD-based process and keep measuring those things.  When you have sufficient data on the new process, you can look at the results and determine whether there's been any measurable improvement since you adopted TDD.

The same applies to any other possible change in your process.  You can leverage the data you collect as part of the PSP to analyze the effectiveness of a change.  So no more guessing or following the trends - you have a way to know if a process works for you.

Honesty

One of the least touted, but (at least for me) most advantageous aspects of using the PSP is simply that it keeps you honest.  You use a process support tool, configured with a defined series of steps, each of which has a script with defined entry and exit criteria.  It gives you a standard place in your process to check yourself.  That makes it harder to bypass the process by, say, skimping on testing or glossing over review.  It gives you a reminder that you need to do X, and if you don't want to do it, you have to make a conscious choice not to do it.  If you have a tendency to rush through the boring things, then this is a very good thing.

Next Up

So that's a list of some of the reasons to use the PSP.  There's lots of good stuff there.  Some of it you can use out-of-the-box, other parts of it offer some inspiration but will probably require customization for you to take advantage of.  Either way, I think it's at least worth learning about.

In the next and final installment in this series, I'll discuss my experiences so far using the PSP.  I'll tell you what worked for me, what didn't, and what to look out for.  I'll also give you some perspective on how the PSP fits in with the kind of work I do, which I suspect is very different from what Humphrey was envisioning when he wrote the PSP book.

This is part two of my evaluation of the Personal Software Process.  This time I'll be talking about the actual process of learning the PSP.  In case you haven't figured it out yet, it's not as simple as just reading the book.

Learning the PSP

The PSP is not the easiest thing to learn on your own.  It's a very different mindset than learning a new programming language or framework.  It's more of a "meta" thing.  It's about understanding your process: focusing not on what you're doing when you work, but rather how you're doing it.  It's also significantly less cut-and-dried than purely technical topics - not because the material is unclear, but simply because what constitutes the "best" process is inherently relative.

Given that, I suspect the best way to learn the PSP is though the SEI's two-part training course.  However, I did not do that.  Why not?  Because that two-part course takes two weeks and costs $6000.  If you can get your employer to give you the time and shell out the fee, then that would probably be great.  But in my case, that just wasn't gonna happen and the SAF (spouse acceptance factor) on that was too low to be viable.

Instead, I went the "teach yourself" route using the self-study materials from the SEI's TSP/PSP page.  You have to fill out a form with your contact information, but it's otherwise free.  These are essentially the materials from the SEI course - lecture slides, assignment kits, and other supplementary information.  It's designed to go along with the book, PSP: A Self-Improvement Process for Software Engineers, which serves as the basis for the course.  Thus my learning process was simply to read through the book and do the exercises as I went.

(As a side-note, remember that this is basically a college text-book, which means that it's not cheap.  Even the Kindle version is over $40.  I recommend just getting a used hardcover copy through Amazon.  Good quality ones can be had for around $25.)

Fair warning: the PSP course requires a meaningful investment of time.  There are a total of ten exercises - eight programming projects and two written reports (yes, I did those too, even though nobody else read them).  Apparently the course is designed around each day being half lecture, half lab, so you can count on the exercises taking in the area of four hours a piece, possibly much more.  So right there you're looking at a full work-week worth of exercises in addition to the time spent reading the book.

Personally, I spent a grand total of 55 hours on the exercises: 40 on the programming ones and 15 on the two reports (for the final report I analyzed not only my PSP project data, but data for some work projects I had done using the PSP).  While the earlier exercises were fairly straight-forward, I went catastrophically over my estimates on a couple of the later ones.  This was largely due partly to my misunderstanding of the assignment, and partly to confusion regarding parts of the process, both of which could easily have been averted if I'd had access to an instructor to answer questions.

Tools

 As I mentioned in the last post, you'll almost certainly want a support tool, even when you're just learning the PSP.  Again, there's a lot of information to track, and trying to do it on spreadsheets or (God forbid) paper is going to be very tedious.  Halfway decent tool support makes it manageable.

I ended up using Process Dashboard, because that seems to be the main (only?) open-source option. In fact, I don't think I even came across any other free options in my searches.  I understand other tools exist, but apparently they're not public, no longer supported, or just plain unpopular. 

One of the nice things that Process Dashboard offers is the ability to import canned scripts based on the PSP levels in the book.  To use that, you have to register with the SEI, just like when you download the PSP materials, which is annoying but not a big deal.  (The author got permission from the SEI to use their copyrighted PSP material and apparently that was their price.)  This is really handy for the learning process because it puts all the scripts right at your fingertips and handles all of the calculations for you at each of the different levels.

In terms of capabilities, Process Dashboard is actually pretty powerful.  The UI is extremely minimal - essentially just a status bar with some buttons to access scripts, log defects, pause the timer, and change phases.  Much of the interesting stuff happens in a web browser.  Process Dashboard runs Jetty, which means that it includes a number of web-based forms and reports that do most of the heavy lifting.  This includes generating estimates, entering size data, and displaying stock and ad hoc reports.

Process Dashboard has fairly extensive customization support, which is good, because one of the basic premises of the PSP is that you're going to need to customize it.  You can customize all the process scripts and web pages, the project plan summary report, the built-in line counter settings, the defect categories, etc.  And that's all great.  The one down side is that the configuration is usually done by editing XML files rather than using a graphical tool. 

Since this is an open-source developer tool, I guess that's sort of fine.  And at least the documentation is good.  But it's important to realize that you will need to spend some time reading the documentation.  It's a powerful tool and probably has just about everything you need, but it's not always easy to use.  On the up side, it's the sort of thing that you can do once (or occasionally) and not have to worry about again.  Just don't expect that you'll be able to fire up Process Dashboard from scratch and have all the grunt work just done for you.  There's still some learning curve and some work to do.  But if you end up using the PSP, it's worth it.

As I mentioned in a couple of previous posts, I've been studying the PSP.  As promised, this is my first report on it.  After several failed attempts to write a single summary of the process, I've decided to divide my assessment up into a series of posts.  This first entry will be a basic overview of what the PSP actually is and how it's intended to work.

What is the PSP?

PSP stands for Personal Software Process.  It was developed by Watts Humphrey of the Software Engineering Institute (SEI) at Carnegie-Mellon University (which you may know as the home of CERT). Humphrey is also the guy who came up with the Capability Maturity Model, which is the CMM in CMMI.  His intent was to take the industrial-strength process methodologies he worked on for the CMM and scale them down to something that could be useful to an individual developer.  The result is the PSP.

Despite the name, at its core the PSP isn't actually a development process itself.  Rather, it is a process improvement process.  To put it another way, the goal is not to tell you how to develop software, but rather to give you the intellectual tools to figure out the development process that works best for you and help you improve that process to make yourself more effective.

Now, to be clear, Humphrey actually does describe a particular development process in his PSP books.  And yes, that process is kind of "waterfally" and extremely heavy.  And if you look for PSP information online, you'll no doubt find a lot of people who take issue with that process.  And they're right to.

But try not to get hung up on that.

As Humphrey puts it, the process he describes is his process.  That doesn't mean it should be your process.  The process he describes is for pedagogical purposes as much as anything else.  It is intended to demonstrate the kinds of methods that apply to large-scale industrial systems, so that you have those tools in your toolbox if you need them.  If this process doesn't really work for you (and it almost certainly won't), then you should modify it or replace it with something that will work for you.

This is something that bears dwelling on.  Although pretty much all of the PSP focuses on Humphrey's process, he notes several times that it is important to pick a process that's sustainable for you.  To put it another way, it's important to be realistic.  Sure, his process might be objectively "better" by some measure than whatever you're doing, but that doesn't matter if you aren't going to follow his process.  Maybe his process conflicts with your company's process; maybe it's not effective for the kind of project you usually do; maybe you just plain don't have the will power to stick to it.  It doesn't matter.  The process you define needs to be in line with what you actually do, otherwise it's not going to help you.  So while you can learn from Humphrey's process, don't take it as gospel and don't think "this is what I need to be doing."

For me, the real take-away of studying the PSP is to be mindful of your development process and constantly try to improve it.  The goal is simple: quality and consistency.  Monitor your process and tune it such that you can consistently build a quality product in a predictable time-frame.  Ground yourself in reality and be guided by the data, not the latest fads.

The PSP Approach

The PSP is all about data-driven process changes.  As you work, you collect data on what you produce. The general idea is that you analyze that data and make changes to your process based on that analysis.  You then repeat this process, evaluating the results of your changes and making further tweaks as necessary.  So it's basically a plan-do-check-act cycle.  This is the same sort of thing that agile methods advocate (e.g. Scrum retrospectives).

Project Structure

The PSP structures a task or project into phases.  The exact phases will obviously depend on your particular process.  The "standard" (i.e. from Humphrey's process) phases are Planning, Development, and Postmortem.  In the Planning phase, you define what is to be built and construct estimates of the effort involved.  The Development phase is where you actually construct the software.  In the standard PSP process, this is divided into a number of sub-phases, depending on which level of PSP you're using.  Lastly, the Postmortem phase is where you record your final process measurements and document any problems and potential process improvements that came up in the course of the project.

The PSP levels I mentioned above are essentially maturity phases.  In the process of learning the PSP, you go through six of them: PSP0, PSP0.1, PSP1, PSP1.1, PSP2, and PSP2.1.  You start with PSP0, which is basically just "whatever you do now, but with measurements" and progress up to PSP2.1, which includes five sub-phases: Design (with standard design templates), Design Review, Code, Code Review, and Test.  I'll analyze this structure and the contents of the phases in a later post. 

Data Collection

Data collection in the PSP happens at a very fine-grained level.  The idea is that you do light-weight data capture as you work in order to minimize inaccuracies.  Data is collected on three axes: how much time is spent in each phase of the project, size of the product, and what defects were found.  In the standard process, you would track how many minutes you spend in each phase or sub-phase, how many lines of code you add or modify, and the number and type of bugs you find in in each phase.

The general idea behind this is simply that the more detailed the data, the more opportunities you have to use it later on.  The PSP suggests a number of ways to use this data to drive and measure process change.  Two of the most visible are the use of historical size and time data to drive the estimation process and the use of defect phase, type, and time data to measure the efficiency of defect removal activities such as code review and testing.

Interestingly, when Humphrey started developing and teaching the PSP, students tracked this stuff on paper.  He later introduced spreadsheets to track it, which is better, but still a little clumsy.  These days, there are PSP support tools you can use.  The main open-source one seems to be Process Dashboard, which I used from day one.  It will help you track PSP data and do most of the calculations you need for data analysis.  Frankly, I can't imagine trying to track this by hand or with Excel spreadsheets - it just seems like it would be painfully tedious.  But with decent tool support its really not that bad.

Quality Management

One of the interesting things to note about the standard PSP process is that it explicitly takes a position on quality management.  This is something you don't tend to hear a lot about - in fact, it might not even have occurred to you that quality is something you can manage.

The position Humphrey takes is simple: the right way is always the fastest way.  And the right way is to get things right the first time to the maximum extent feasible.  That means finding defects as early as possible and taking steps to stop them from being injected in the first place.

I'll get into the details of this in a later post.  I mention it now simply Humphrey devotes a lot of ink to it.  In fact, much of the new material in the higher PSP levels is devoted to quality control.  Quality management is one of the big uses of the defect data that you track as part of the PSP, so it's important to realize at the outset that this is a core part of the process.

Should I Care?

I think you should.  You may not want to actually use the PSP, but it's still an interesting thing to learn about.  If you're like me, you probably never thought about how you build software in terms of hard data, so it's interesting to see how it can be done.  It's a very different and empowering way of understanding how you work - it makes programming feel more like a science and less like a black art.  So even if you don't want to go all the way and actually try to use the PSP, it's enlightening to read about the various statistical analysis techniques and quality improvement recommendations.  If nothing else, it broadens your perspective and gives you more intellectual tools to draw on.

In part two, I'll talk a little about what I did to learn about the PSP, including the resources and tools that I used.

So as I mentioned in a previous post, I've been taking a look at the Personal Software Process, or PSP (not to be confused with Sony's portable gaming console, which makes Googling for it loads of fun).  It's been an eye-opening experience and I'll write more about it once I've officially finished the course.

At the same time, I've been reading about the #NoEstimates hashtag/movement/discussion/whatever-the-hell-it-is.  I actually first heard about it around the time I was starting the PSP book.  This makes for an interesting contrast, because one of the first things you learn about when studying the PSP is...how to do estimates.

The thing is, up until a few months ago, I would have completely agreed with the #NoEstimates people.  In fact, I wrote about my feelings on estimates a couple of years ago.  My feelings were pretty much in line with Woody Zuill's comments in .NET Rocks! episode 1160 - that estimates are useless because they're just made-up, grossly inaccurate numbers which can only be used for evil or to make even more inaccurate plans.  So why bother?  Just decide if something is important enough to do, and then either do it or not.  Stop pretending.

Then I started learning about the PSP.  I started accumulating data on my actual productivity and learning how to leverage it.  And you know what?  It turns out it actually is possible to do estimates with a reasonable degree of accuracy - and it doesn't even take huge amounts of up-front planning.  Granted, there are caveats - you need historical data that's representative of the technology and problem domain you're working in.  But once you've got a few projects under your belt to establish a baseline, it really does work.

So now I have some cognitive dissonance.  Was I wrong to take the time to learn this estimating stuff?  Or was I wrong before when I agreed with the #NoEstimates people?  Where do I go with this?

What got me thinking about this is an article by Henri Karhatsu in Methods and Tools.  Towards the end of the article, he discusses the #NoEstimates alternative to providing an estimate: forecasting.  This is what you use when you really do need to figure out when something will be done (e.g. so that you can prepare marketing materials for a new product or feature).  While this may sound like estimating, according to Karhatsu, there is a key difference: forecasts are based on data, while estimates are not.

I had several reactions to this revelation.  The first was, "Well, that's a crock - 'forecasting' is just estimating by a different name."  Then I realized, "Hey, 'forecasting' is exactly what you're doing in the PSP!  Cognitive dissonance resolved!"  Which prompted the question, "If PSP 'estimating' is the same thing as #NoEstimates 'forecasting', then what the heck does #NoEstimates even mean?"

And that's when I read this article analyzing the #NoEstimates movement.  It's an interesting read and pretty well summarizes my impressions of #NoEstimates based on what I've heard and read.  The short version is that #NoEstimates doesn't actually mean "no estimates ever".  It doesn't even mean that estimates are inherently bad in all cases.  Rather, it seems to be a reaction against people using estimates in stupid an fruitless ways.  This came up a number of times in the .NET Rocks! interview with Woody Zuill.  You have managers using estimates as a weapon against developers, or clueless project managers demanding estimates because they need to plug the numbers into Microsoft Project so they can finish their plan, even though nobody really thinks the plan reflects reality.  This ends in the demoralizing and unproductive result of developers fabricating numbers, without any clue if they're even vaguely realistic, for no other purpose than to keep the managers happy.

I don't think anybody could reasonably argue that such a situation is a good thing.

But on the other hand, when you're being paid to do a project, whether it's building software or building a house, the questions "How long is it going to take?" and "How much is it going to cost?" are not inherently unreasonable things for the customer to ask.  And while it's sometimes true that you're working on an unprecedented project an have no clue how long it will take, it's more often true that you're on reasonably familiar ground and have a pretty good idea of the effort involved in a project.  And let's be honest - while a big-deal consultant like Woody Zuill might be able to get away with telling the customer that they don't really need an estimate, that's not going to fly in every organization.  So #NoEstimates as a general recommendation is probably not realistic.

And that brings me back around to the first chapter of the PSP book.  In it, Watts Humphrey relates a story from one of his first TSP teams (that's "Team Software Process" - the team equivalent of the PSP).  They had just come off a two-year death-march project and now management had a new project for them - with a nine-month timeline.  The dev team was sure they couldn't make that schedule.  They didn't really know how long the project would take, but figured it would probably be closer to two years again. 

So what normally happens in that situation?  Well, the dev team says that the schedule is impossible to meet, management insists that the date is firm, and the developers eventually cave and say they'll "do their best".  I've been in that situation, as I'm sure a lot of other people have.  Why do we do that?  Because, well, what else can we do?  We don't really know how long the project will take, and even if we have an idea, we usually don't have anything concrete to back up our estimate.  And as Humphrey puts it, when we don't know and management doesn't know, they'll win every time.

But rather than go the #NoEstimates route and try to convince people that coming up with an estimate isn't a useful exercise, Humphrey goes the opposite direction.  He uses the power of data to give estimates teeth. Remember - even if management wants the new project done yesterday, it's still important for them to know when it's really going to be done.  And while it's easy to write off a guess-work estimate as "schedule padding", it's not as simple when you have a detailed plan based on historical data.  When you reach that point, estimates can become a weapon for the dev team rather than one used against them.

So where does this leave me?  Pretty much back where I started.  Estimates are good when they're needed and accurate.  They're bad when they're unneeded or inaccurate.  They should be based on data, not guess-work, and should be used responsibly.  So no earth-shattering news.  Just an interesting detour.

So as part of my ongoing professional improvement program, I've been working my way through PSP: A Self-Improvement Process for Software Engineers.  And by "working", I mean I'm doing the exercises from the SEI website and everything.  So far it's actually quite an interesting process - I'd definitely recommend it to any professional software developer, even if you're not interested in using the process, just as "food for thought".  You can pick up a relatively cheap used copy of the book on Amazon (it is technically a textbook, so new ones are a little pricey).  I'll have to write a post on it when I'm finished.

Anyway, the PSP uses line-of-code counting to estimate program size, defect densities, etc.  So I thought it would be nice to be able to run line count reports right from within Komodo.  Fortunately, it turned out to be pretty easy.  I just lifted some code from Nathan Rijksen's "Open Terminal Here" macro as a starting point and went from there.  The macro just takes the selected files or directories in you "places" pane and appends them to a custom command.  I've used cloc as my line-counting tool of choice for a number of years, but you can change the command to be whatever you want (even "wc -l" if you really want).  I also added in a little code to calculate and run from a common base directory, so that you wouldn't get a full, absolute path for every item in the report.  The command output is sent to the Komodo output pane.

The code for the macro is below.  Or, if you're feeling lazy, you can just download the tool here and drop it in your toolbox.

/**
 * Adds a "Count LOC" menu item to items in the Places widget.
 * This will run a line-counter on the selected paths and display the command output.
 * Based on the "Open Terminal Here" macro by Nathan Rijksen.
 *
 * Usage: Update the "command" variable to contain whatever command you want to run.  The paths to the files selected
 * in the places pane will be appended to this command.
 *
 * If the "use_common_directory" variable is true, then the macro will calculate the deepest common directory of all
 * the selected files and will run the command from that directory, passing relative paths.  For example, if you
 * select files /foo/bar/baz/buzz.js and /foo/bar/fizz/fuzz.css, the command will be run from /foo/bar and will be
 * passed the paths baz/buzz.js and fizz/fuzz.css.
 * If this variable is set to false, then the command will be passed the full, absolute paths to all files and no
 * working directory will be specified for it.
 *
 * @author Peter Geer
 * @contributor Nathan Rijksen
 * @contributor Mathieu Strauch
 * @version 0.1
 */
/*global ko, extensions:true */

// Register namespace
if ((typeof extensions) == 'undefined') {
    extensions = {};
}
extensions.CountLOC = {};

(function() {
    
    var command = "cloc --by-file --force-lang=PHP,phtml",
        use_common_directory = true,
        label = 'Count LOC',
        id = 'contextCountLOC',
        sibling,
        d,
        mi,
        callback,
        longest_common_path;
    
    longest_common_path = function(list) {
        var longest_item = list[0].substr(0, list[0].lastIndexOf('/')),
            i = 0;
        for (i = 0; i < list.length; i++) {
            while (longest_item !== '' && list[i].indexOf(longest_item) < 0) {
                longest_item = longest_item.substr(0, longest_item.lastIndexOf('/'));
            }
            if (longest_item === '') {
                break;
            }
        }
        return longest_item;
    };
    
    callback = function(e) {
        var i = 0,
            cmd = command,
            curr_dir = null,
            uris = ko.places.viewMgr.getSelectedURIs();
        
        if (uris.length === 0) {
            return;
        }
        
        // Clean up the URIs.
        for (i = 0; i < uris.length; i++) {
            uris[i] = uris[i].replace(/^[a-zA-Z]+:\/\//,'');
            if (uris[i].match(/^\/[a-zA-Z]:\//)) {
                uris[i] = uris[i].substr(1);
            }
        }
        
        // If set, turn the absolute paths into relative paths.
        if (use_common_directory) {
            curr_dir = longest_common_path(uris);
            if (curr_dir !== '') {
                for (i = 0; i < uris.length; i++) {
                    uris[i] = uris[i].substr(curr_dir.length + 1);
                }
            } else {
                curr_dir = null;
            }
        }

        // Prepare command for each platform
        for (i = 0; i < uris.length; i++) {
            cmd += ' ' + uris[i];
        }

        // Run command, show output in bottom pane
        ko.run.command(cmd, {cwd: curr_dir, runIn: 'command-output-window'});
    };
    
    // Get places pane document object
    d = document.getElementById('placesViewbox').contentDocument,

    // Remove existing menu entry if it exists
    mi = d.getElementById(id);
    if (mi) {
        mi.parentNode.removeChild(mi);
    }

    // Get the sibling element which we want to insert our menu item after
    sibling = d.getElementById('placesContextMenu_rename');
    
    // Create our menu item
    mi = document.createElement("menuitem");
    mi.setAttribute("id", id);
    mi.setAttribute("label", label);

    // Add event listener for when the menu item is used
    mi.addEventListener('command', callback);

    // Append menu item to popupmenu
    if (sibling && sibling.parentNode) {
        sibling.parentNode.insertBefore(mi, sibling);
    }

}.apply(extensions.CountLOC));