Review - Agile!: The Good, the Hype, and the Ugly

As I mentioned in my a previous post, I've been doing some reading around process.  A development process is one of those things that you always have, by definition.  It might be ill-defined and chaotic, and you might not even thingk about it, but eventually you go through some process to develop code.  But when you think about it, this process is going to have a not insubstantial impact on the quality of the software you produce.  After all, how could it not?

However, as developers, we often think of "process" as a dirty word.  It's one of those bureaucratic things imposed on us by pointy-haired project managers, with their Gantt charts and their Microsoft Project.  Sure, those highfalutin "processes" sold by over-priced consultants might be useful for those huge, soulless "enterprise" shops that have thousands of know-nothing drones cranking out Java ports of mainframe COBOL systems.  But we don't need those!  We're "code poets", "10x developers", "coding ninjas", etc.  We can code circles around a dozen of those enterprisey jokers in our sleep! 

Except it doesn't really work that way.  Even if you accept the notion that the best developers are ten or more times more effective than average ones, they still have to deal with the complications of ill-defined features, requirements changes, integration problems, performance problems, communication between stake-holders, and the myriad other issues that emerge and magnify as more people become involved in a project.  If you expect to make it through any decent sized project and not have it turn into a hot mess, you have to have some way of managing all the things that come up. Some organizations, such as the aforementioned Java-drone shop, do that by implementing processes that require mountains of paperwork that serve primarily to cover everyone's butt.  Others take the agile approach.

Cover of "Agile!: The Good, the Hype, and the Ugly"If you've been living in a cave for the past decade or so, the "agile" movement is a response to the stereotypical "big, up-front" processes of old.  The values of the agile community are summed up in the Agile Manifesto, but the basic idea is to focus less on the bureaucratic paper-pushing and more on delivering what the customer needs.  This generally sounds good to developers because it means less time spent on TPS reports and more time actually building software.  And it sounds good to managers because the iterative nature of agile development means you get the software faster (for certain definitions of "faster").  And it sounds very good to consultants who can offer agile training and certification.  So, naturally, everyone is talking about it.

That's where Bertrand Meyer's book Agile!: The Good, the Hype, and the Ugly comes in.  Its stated purpose is to separate the wheat from the chaff; that is, to pick out what's good about agile methods and what can be safely disregarded.  Meyer says it is intended as an even-handed and fact-based assessment to offset the hyperbole that often surrounds discussions of agile.

Part of what attracted me to this book was simply the fact that it's by Bertrand Meyer.  I first became aware of his work when researching for my Masters thesis.  He's probably best known as the originator of the idea of "design by contract" and Eiffel, and has a strong footing in formal methods, which is not usually the type of thing you'd associate with agile.  In fact, some people might consider it the opposite of agile.

Overall, I found the book not only enlightening, but also highly entertaining.  Meyer writes in an informal and even playful style.  While he does provide plenty of evidence and citations, he is also not light on the sarcasm - he's not shy about calling ideas stupid and harmful if that's what he thinks.  Sadly, this last point somewhat undercuts his claim to impartiality.  The "opinion" sections are helpfully marked with a marginal icon and they're just all over the place.  And while I don't think that Meyer has any anti-agile ax to grind, I was definitely left with the impression that the preponderance of his commentary was critical of agile.

Now, to be fair, criticizing agile is not a bad thing - every movement needs critics to keep it honest.  If the agile advocates you find on the web are any indication, much of the material out there is very "rah, rah agile" and this book is a bit of a reality-check.  While Meyer does give the agile movement credit for a number of good and even brilliant ideas, advocating for the greatness of agile is not the point.  The important thing is to critically examine it and not just take the press releases at face value.

One of the nice features of this book's structure is that it has book-ended assessments.  That is, Meyer ends the first chapter with an initial assessment of agile and then finishes the book with his final assessment of the good, hype, and ugly.  Though only a couple of pages, I found the initial assessment actually quite good as far as putting things in perspective.  He begins with this quote (apocryphally attributed to Samuel Johnson):

Your work, Sir, is both new and good, but what is new is not good and what is good is not new.

To me, this sets the stage, highlighting that, regardless of the hype, not all of the "innovations" in agile are actually new.  In fact, some have been around in different forms for years or even decades.  Of course, not all the good ideas are old and not all the new ideas are bad, but the point is that agile isn't quite as ground-breaking as some of the hype would have you believe. 

I won't get into the details of Meyer's analysis, but there are a few interesting items I'd like to point out.   First is his point that agile proponents (and many others in the industry) tend to work on the assumption that there are only two development models: agile and waterfall.  They use "waterfall" as a synonym for "anything that's not agile," which is both inaccurate and unfair (page 31).  Innaccurate because  "waterfall" is a specific lifecycle model proposed by Royce in the 1970's.  Unfair because there are many so-called "predictive" lifecycle models which are not the same as waterfall.  For example, the idea of iterative development has been around for a long time, except under other names like "spiral model".

Related to this, Meyer makes the interesting point that not all of the predictive processes out there actually preclude agile.  His big example of this is CMMI (page 44), which many (most?) of us think of as a super-heavy process that's only used by government defense contractors, and only because the government requires it.  However, he argues, if you really look at what CMMI is, there's no inherent contradiction with agile.  Of course, the marriage of the two is far from a no-brainer, but it is possible, as evidenced by a report on a Scrum team that was certified at CMMI level 5.

This is just part of what seems to be Meyer's biggest problem with agile methods - the deprecation of "big, up-front" tasks, which seems to be generalized to all up-front tasks.  He seems this as emblematic of a tendency in the agile movement to take a good idea and take it to the extreme.  It's invariably true that you can't get all the requirements right the first time, that the initial archtecture won't be perfect, that the initial plan won't work out perfectly, etc.  Nobody's perfect and we can't know everything - that's just life.  But does that really mean that we should just not plan out any of that out ahead of time at all?  Isn't that just as good of an argument for taking some more time to figure out a better plan up front?

My own experience the last few years has led me increasingly to sympathize with Meyer's views on this.  Like him, there are a lot of agile practices I think are great and helpful.  Short, time-boxed iterations with frequent deliveries are a great way to work and collect feedback; having a good product owner is priceless; and I don't think anybody really objects to heavy use of unit test suites and continuous integration anymore.  Let's keep those around.

But the more I think about and work with various "agile" processes, the clearer it becomes that there's a reason so many people seem to say that their organization "does Scrum," but not by the book - they have this and that modification, sometimes to the point that it no longer sounds like Scrum.  Some people interpret that as evidence that agile is nothing but hype.  After all, if nobody implements that actual process, then in what sense is it successful? 

Of course, that's not entirely fair because every team and organization is different so no process is going to be a perfect fit for everyone.  But on the other hand, is it entirely unfair either?  If hardly anyone seems to do things "by the book," could that be because the book is wrong?  Without more of the type of researched analysis that Meyer offers, we really have no way to know.

If you're interested in development processes, then Agile! The Good, the Hype, and the Ugly is definitely worth checking out.  I have to say that it really changed the way I think about agile processes.  For me, it really helped to put things in a wider perspective than the "good/new agile vs. bad/old waterfall" story that seems to dominate discussions.  And if nothing else, it's at least good to see some questioning and critical analysis of the agile mantra.

(Note: for a shorter read, Jim Bird has a very interesting take on agile in a similar vein to Meyer.)

The Mythical Man-Month

I've decided it's time to do a little professional development.  In other words, stop putzing around with whatever new technology is "cool" these days (grumble, grumble kids with their Snapchats, and their Node.js, and their Dan Fogelberg) and get back to fundamentals.  That means design and architecture, development process, project management, quality controlThe Mythical Man-Month, 20th anniversary edition, documentation, and all those other things that are important, but not even remotely "hip".

To that end, I'll be doing some reading.  The first book on my list has been on my bookshelf for about 13 years - The Mythical Man-Month, by Fred Brooks.  I bought and started reading the 20th anniversary edition over a decade ago, but never got around to finishing it. 

In retrospect, I think I was just too inexperienced at the time really get the significance of the book.  After all, it might be a classic, but its's an old book (written in 1975) about an old project (much of it relates to developing OS/360).  How relevant could it be?  Especially once you get past the punch line of why the man-month is mythical - which is revealed in chapter two.

Now, after almost 14 years in tech, it's easy to see just how brilliant a book it is.  While there is some discussion of obselete methods and technologies, that's beside the point.  The Mythical Man-Month is still relevant because it isn't really about specific development techniques or technologies.  It's about the complications of managing and communicating with lots of people on a large project.  And while we've come a long way as an industry in the last 40 years, the "human nature" part of the equation hasn't really changed appreciably.

One of the nice things about this book is that Brooks clearly gets what it means to be a developer.  He starts the book with an essay on the joys and woes of the craft of programming, all of which are as relevant today as they were 40 years ago.  There are a number of other such insights sprinkled throughout the book.  My personal favorite is his statement that all programmers are optimists.  When you think about the process of estimation and how that usually turns out, this is the sort of insight that seems so blindingly obvious that you're suprised you didn't think of it yourself.

The third chapter, "The Surgical Team," was particularly interesting to me.  The proposal is essentially to treat building software less like building a bridge and more like performing surgery.  So, in other words, you'd have one senior developer act as the "surgeon" and he and a core junior assistants perform all of the deliverable work.  The rest of the team supports him, doing the project management work, building supporting tools, serving as expert consultants, etc.  Since one top-notch programmer is supposedly ten times as productive as a mediocre programmer, having one of them do the work and nine other people support them gives you the same total productivity while reducing communication problems and maintaining design consistency.

One of Brooks' themes throughout the book is that the way to manage complexity is to maintain consistency of architectural vision.  That is, the overall system architecture should reflect as few viewpoints as possible, rather than being a mish-mash of the viewpoints of everyone who worked on it.  This plays into another major issue Brooks discusses: communication cost.  This is part of the reason the man-month is mythical - because not only does adding new people to a project add ramp-up time, it also increases the communication cost because you now have more people to keep in the loop.

I think one of the best things about the 20th anniversary edition is the added chapters.  For starters, they include Brooks' classic paper No Silver Bullet - Essence and Accident in Software Engineering, which is a good read.  It also includes a chapter of analysis and reflection, in which Brooks discusses not only the legacy of the book, but also what he got right and wrong.  The things he got wrong are particularly interesting.  They include the classic "build one to throw away" quote, which Brooks says is wrong "not because it is too radical, but because it is too simplistic," being rooted in the waterfall model that was popular in the 1970's, as well as his notion of communicating the full project workbook to everyone.

Overall, The Mythical Man-Month is a very engaging and surprisingly easy read, especially given the volume of references.  While it contains some details that now seem like quaint digressions into the history of computing, the majority of the material is still relevant.  It definitely contains useful insights for anyone who is interested in the dynamics of running a large project.  I'm very glad I came back to it.

Flask on ICDSoft

A year or two ago, I decided to update my skill set a little and brush up on my Python. In particular, I wanted to do a little web development in Python, which I hadn't done in nearly five years. Since I wanted to start with something fairly basic, I decided to go with the Flask micro-framework. I ended up using that to build Lnto, which I've mentioned before and I swear will get its own page someday.

One big problem with this project was was that my hosting is kind of bare-bones. Don't get me wrong - the service is fine. I use ICDSoft, whom I've been with for several years. I have no complaints about the service I've received - in fact I'm quite happy with it. However, it's mostly focused on PHP and MySQL and there's no shell access. But on the other hand, I'm paying less than $5 a month with only a one-year commitment, so I don't have much to complain about.

Anyway, the problem with running Flask apps, or pretty much anything other than PHP, is that they have no documentation for that whatsoever. There's a FAQ, but it says absolutely nothing about Python other than that they "support" it. As far as I can tell, that just means that they have a Python interpreter installed and Apache CGI handler is configured to run *.py files. There's certainly no mention of running things using WSGI, which seems to be the recommended method for running most Python frameworks.

Another problem is actually installing the Flask libraries. The documentation for, well, pretty much every Python framework or app out there tells you the best way to install it is with pip or easy_install. But, of course, you need shell access to run those, assuming they're even installed on the server. (And I did check - they're not installed.)

Fortunately, getting around these problems was relatively easy using using virtualenv, which I'd nearly forgotten existe). This creates a virtual Python environment which is isolated from the rest of the system. A side-benefit of this is that virtualenv creates a copy of pip in your virtual environment.

You can use virtualenv directly from the source distribution. This was required in my scenario, since I lack any shell access, let alone root privileges. I simply extracted the virtualenv source archive, uploaded it to my host, and ran the following command (I used PHPsh, a web-based shell emulator, but copying them into a PHP script would have worked just as well):
python /path/to/virtualenv-1.11.4/virtualenv.py /path/to/myapp/venv

This create a virutal environment in the /path/to/venv directory. You can then install packages into that environment using the "activate" script to configure the shell, like this:
. /path/to/venv/bin/activate && pip install Flask

That was easy. I now have a Python environment with Flask installed. All I need do at this point is configure my application code to use it. That's accomplished with a few lines to initialized the virtualen environment and start up Flask as a CGI app:
activate_this = '/path/to/venv/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from myapp import app
from wsgiref.handlers import CGIHandler
CGIHandler().run(app)

I just re-ran that entire process using the latest version of virtualenv and it's actually quite painless.

And as a side-note, the reason I did that was because I noticed the other day that Lnto had suddenly stopped working - the server was just returning 500 errors. Which was odd because I hadn't changed anything with the app or the database in weeks. However, the answer was found on the virtualenv PyPi page:

Python bugfix releases 2.6.8, 2.7.3, 3.1.5 and 3.2.3 include a change that will cause "import random" to fail with "cannot import name urandom" on any virtualenv created on a Unix host with an earlier release of Python 2.6/2.7/3.1/3.2, if the underlying system Python is upgraded.

When I created that virtualenv environment, the server was running Python 2.6. But when I checked yesterday, the Python version was 2.7. So apparently ICDSoft upgraded their servers at some point. No big deal - just recreated the environment and I was good to go!

New job - at home

Well, it's been a while since I've posted anything here. That's because I've been busy with a new job.

The new job is doing LAMP development for a major online community site. True to form, I'm not (yet) going to name the company. That policy has served me well so far. However, I will share some details of my new position and some of the challenges I've faced with it, both expected and unexpected.

The first innovation with regard to this job is it's location - my house. Yes, it's a 100% telecommute, work-from-home position. In fact, it's a totally distributed team, so everybody on the development works from home. And while that's great, so far I've actually found it to be one of the biggest challenges with the job. The challenge isn't motivating myself to actually work, which my mother thought it would be (seriously, she was concerned about that). In fact, quite the opposite. Until recently, I've had more problems with spending too much time working.

There are a three main reasons for this. Part of it is because I was still learning our system and felt like I had to "catch up" to the rest of the team. Another part is simply schedule diversity. We communicate primarily via Skype chatrooms, so we can when people are online. And since we have developers literally from California in the west to Russia in the east, many of them work non-standard schedules, there's always somebody "on", which adds some subconscious pressure. And lastly, of course, part of it is probably emotional damage from my last job. At that (highly dysfunctional) company, we were allowed to work from home on occasion, but if we did it too often, the CEO would make needling comments to the VP of Engineering suggesting that "working form home" was a code-word for "taking the day off". So I think I've partly been stuck in the mindset of trying to prove that, "No, really, I am actually working!"

The other new feature of this position is the sheer size of the operation. And I mean that in pretty much every conceivable way - the amount of traffic, the infrastructure, the size of the code-base, the size of the development team. In fact, pretty much the only thing that isn't big (to me) is the size of the overall company. By way of comparison:

  • The largest site I'd work on was when I was at ebaumsworld.com, where we had something like 30 servers in our environment and were ranked about 1200 in the world by Alexa. The current job has nearly 100 web servers alone and is ranked in the top 150 in the world by Alexa.
  • The largest code-base I'd worked on was at my last job - a little over 200K total LOC when you include unit tests. The new job has over 500K lines of just PHP code (that's not counting CSS or JavaScript) in just the web portion of the tree.
  • The largest group I'd worked in was when I was with the county IT department - about 30 people total, about 9 of them programmers, no collaborative projects to speak of. My last two teams have been very tight 4-man operations. At the new job, we have about 30 people on the tech team, all of them programmers and all of them very good.

So, needless to say, I've been very busy figuring things out in my new position. I've been there for about three months now and I think I'm finally getting comfortable. I'm still learning the code-base, of course (nobody understands the entire thing - it's just too big), but I'm getting more confident and feel like I'm making progress. So in other words, so far, so good.

And most importantly, I'm much happier than I was at my last job. It's nice to feel like you can trust your colleagues and that the work you're doing is helpful. Amazing how increased morale boosts productivity.