LinLog

Author's note: This episode of From The Archives is the stub of an article I wrote on December 10, 2007.  At the time, I was working for eBaum's World (this was back around the time ebaum sold it, but before he was forced out).  It was kind of a weird time because it was my first experience working for an actual tech company.  (It was also a weird place to work, for other reasons, but that's a different story.)  Previously, I'd been stuck in the world of public-sector internal IT which is...not great, by comparison.

This particular post was expressing my annoyance over how our development environment was broken.  Our dev environment, at the time, was literally just a shared server that we all pushed code to. And yes, that means we had plenty of opportunity to step all over each other and things could easily break for non-obvious reasons.

This was obviously terrible, but it's what we had to work with.  We had a four-man development and system administration team with not a huge budget or set of internal resources.  And, of course, there was always something more important to do than improving our dev environment setup.

These days, I still have dev environment issues, but for completely different reasons.  My company has an entire infrastructure for spinning up development VMs, including dedicated host clusters, custom tooling to setup environments, and teams responsible for managing that stuff.  So now I have my own virtual instance of every server I need (which, for reference, is currently about eight VMs).  However, there are still some holes in that infrastructure.

Part of the issue is that we have a lot of teams that share or depend on the same infrastructure and/or services.  For instance, one of the services that my team maintains is also worked on by at least three other teams.  And we all share the same database schema, which doesn't automatically get updated in dev environments when changes are made in production, so when you pull in the develop branch, you frequently get breaking changes that you may not have ever heard of, usually in the form of config or database changes that aren't set in  your environment.  Sure, everything "just works" if you start fresh, but nobody ever starts fresh because it takes too long to spin up all the required pieces and set up the test data.  (However, we are moving toward a Docker/Kubernetes setup for the pieces that don't need test data, so things are moving in the right direction.)

Not that I can complain too much.  Even for my personal projects, which are much simpler, my dev environment is frequently in disarray.  Things get out of date, permissions or services don't get correctly configured after an update, or my setup is out of sync with the production environment.    In fact, I had that problem just the other day - I pushed some code for this blog that worked fine locally, but broke on the live server because it was running a different version of PHP.  Granted, the code in my dev environment was wrong (and I really should have caught it there), but that's not the point.

The point is that environment maintenance is hard and dev environment maintenance doubly so because it changes frequently and is of much lower priority than production.  That's something I hadn't really had to worry about in my internal IT job, because I was writing desktop apps and everybody was running identical Windows desktops.  It was a a simpler time....

This week's installment of "Pete's list of things that suck" features broken development environments.  I spent the last couple of days at work wrestling with one and it really, really sucks.

The thing is, our development environment is...really screwed.  Things went really bad during the production server upgrade a couple of months ago and we had to hack the hell out of the code just to keep the site running.  As a result, we had all kinds of ugly, server-specific hackery in production which just plain broke when we moved it back into devel.  And since, of course, we have a huge backlog of projects that management wants implemented, we've had no time to go back and reconfigure the development servers.

This week, I've been trying to test some changes to our user upload process.  However, due to some NFS misconfiguration and some code problems, uploads just plain don't work in our main development environment.  We do have a new, additional set of development servers (well, one box with a half-dozen virtual machines, but you get the idea), but I couldn't get those to work either.  Part of it was that the configuration on those servers was incomplete, and part of it was that Firefox sucks.  (Note from the present: I no longer remember why that was.)