First, let me start by describing the workflow I was used to at my last job.  We had a private GitLab instance and used a forking workflow, so getting a change into production went like this:

1. Developer forks the main repo to their GitLab account.
2. Developer does their thing and makes a bunch of commits.
3. Developer opens a merge request from their fork's branch to the main repo's main branch.  Code review ensues.
4. When review is complete, the QA team pulls down the code from the developer's fork and tests in a QA environment.  Obviously testing needs differ, but a "QA environment" is generally exactly the same thing as a developer environment (in this case, a set of disposable OpenStack VMs).
5. When testing is complete, the merge request gets merged and the code will go out in the next release (whenever that is - we didn't do continuous deployment).
6. Every night, a set of system-level tests runs against a production-like setup that uses the main branches of all the relevant repos.  Any failures get investigated by developers and QA the next morning.

I'm sure many people would quibble with various parts of this process, and I'm not going to claim there weren't problems, but it worked well enough.  But the key feature to note here is the simplicity of the branching setup.  It's basically a two-step process: you fork from X and then merge back to X.  You might have to pull in new changes along the way, but everything gets reconciled sooner or later.

The new team's process is not like that.  We use GitHub, and instead of one main branch, there are three branches to deal with: dev, test, and master, with deployment jobs linked to dev and test.  And in this case, the merging only goes in one direction.  So a typical workflow would go like this:

1. Developer creates a branch off of master, call if "feature-X"
2. Developer does their thing and makes a bunch of commits.
3. Developer opens a pull request from feature-X to dev and code review ensues.
4. When the pull request is approved, the developer merges it and the dev branch code is automatically deployed to a shared development environment where the developer can test it.  (This might not be necessary in all cases, e.g. if local testing is sufficient.)
5. When the developer is ready to hand the code off to QA, they open a pull request from feature-X to test.  Again, review ensues.
6. When review is done, the pull request gets merged and the test branch code is automatically deployed to test, where QA pokes at it.
7. When QA is done, the developer opens a pull request from feature-X to master and (drum roll) review ensues.
8. When the master pull request is approved, the code is merged and is ready to be deployed in the next release, which is a manual (but pretty frequent) process.

You might notice something odd here - we're only ever merging to dev and test, never from them.  There are occasionally merges from master to those branches, but never the other way around.  Now, in theory this should be fine, right?  As long as everything gets merged to all three branches in the same order, they'll end up with the same code.  Granted, it's three times as many pull requests to review as you really need, but other than that it should work.

Unfortunately, theory rarely matches practice.  In fact, the three branches end up diverging - sometimes wildly.  On a large team, this is easy to do by accident - Bob and Joe are both working on features, Bob gets his code merged to test first, but testing takes a long time, so Joe's code gets out of QA and into master first.  So if there are any conflicts, you have the potential for things like inconsistent resolutions.  But in our case, I found a bunch of code that was committed to the dev branch and just never made it out to test or master.  In some cases, it even looks like this was intentional.

So this creates an obvious procedural issue: the code you test in QA is not necessarily the same as what ends up in production.  This may be fine, or it may not - it depends on how the code diverges.  But it still creates an obvious risk because you don't know if the code your releasing is actually the same as what you validated.

But it gets worse.  This also creates issues with the GitHub pipeline, which is where we get to the next part of the story.

Our GitHub pipelines are set up to run on both "push" and "pull_request" actions.  We ended up having to do both in order to avoid spurious error reporting from CodeQL, but that's a different story.  The key thing to notice here is that, by default, GitHub "pull_request" actions don't run against the source branch of your pull request, they run against a merge of the source and target branches.  Which, when you think about it, is probably what you want.  That way you can be confident that the merged code will pass your checks.

If you're following closely, the problem may be evident at this point - the original code is based on master, but it needs to be merged to dev and test, which diverge from master.  That means that you can get into a situation where a change introduces breakage in code from the target branch that isn't even present in the source branch.  This makes it very hard to fix the pipeline.  Your only real choice at that point is to make another branch of the target branch, merge your code into that, and then re-create the pull request with the new merged branch.  This is annoying and awkward at best.

But it gets worse than that, because it turns out that your pipeline might report success, even if the merge result would be broken!  This appears to be a GitHub issue and it can be triggered simply by creating pull requests.  

The easiest way to explain is probably by describing the situation I actually ran into.  I had a change in my feature-X branch and wanted to go through our normal process, which involves creating three pull requests.  But in my case, this was just a pipeline change (specifically, adding PHPStan analysis), so it didn't require any testing in dev or test.  Once it was approved, it could be merged immediately.  So here's what I did:

1. First, I created a pull request against dev.  The "pull_request" pipeline here actually failed, because there was a bunch of code in the dev branch that violated the PHPStan rules and wasn't in master, so I couldn't even fix it.  Crud.
2. After messing around with dev for a while, I decided to come back to that and just create the pull requests for test and master.
3. So I created the pull request for test.  That failed due to drift from master as well.  Double crud.
4. Then I created the pull request for master.  That succeeded, as expected, since it was branched from master.  So at least one of them was reviewable.
5. Then I went back and looked at the dev pull request and discovered that the "pull_request" pipeline job now reported as passing!

Let me say that more explicitly: the "pull_request" job on my pipeline went from "fail" to "pass" because I created a different pull request for the same branch.  There was no code change or additional commits involved.

Needless to say, this is very bad.  The point of running the pipeline on a pull request is to verify that it's safe to merge.  But if just doing things in the wrong order can change a "fail" to a "pass", that means that I can't trust the results of my GitHub pipeline - which defeats the entire purpose of having it!

As for why this happens, I'm not really certain.  But from my testing, it looks like GitHub ties the results of the "pull_request" job to the last commit on the source branch.  So when I created the pull request to dev, GitHub checked out a merge of my code and dev, ran the pipeline, and it failed.  It then stores that as part of the results for the last commit on the branch.  Then I created the master pull request.  This time GitHub runs the pipeline jobs against a merge of my code with master and the jobs pass.  But it still associates that result with the last commit on the branch.  Since it's the same commit and branch are for both pull requests, this success clobbers the failure on the dev pull request and they both report a "pass".  (And in case you're wondering, re-running the failed job doesn't help - it just runs whatever the last branch it tested was, so the result doesn't change.)

The good news is that this only seems to affect pull requests with the same source branch.  If you create a new branch with the same commits and use that for one pull request and the original for the other, they don't seem to step on each other.  In my case, I actually had to do that anyway to resolve the pipeline failures.

So what's the bottom line?  Don't manage your Git branches like this!  There are any number of valid approaches to branch management, but this one just doesn't work well.  It introduces extra work in the form of extra pull requests and merge issues; it actually creates risk by allowing divergence between what's tested and what's released; and it just really doesn't work properly with GitHub.  So find a different approach that works for you - the simpler, the better.  And remember that your workflow tools are supposed to make things easier.  If you find yourself fighting with them, then you're probably doing something wrong.

This particular post was expressing my annoyance over how our development environment was broken.  Our dev environment, at the time, was literally just a shared server that we all pushed code to. And yes, that means we had plenty of opportunity to step all over each other and things could easily break for non-obvious reasons.

This was obviously terrible, but it's what we had to work with.  We had a four-man development and system administration team with not a huge budget or set of internal resources.  And, of course, there was always something more important to do than improving our dev environment setup.

These days, I still have dev environment issues, but for completely different reasons.  My company has an entire infrastructure for spinning up development VMs, including dedicated host clusters, custom tooling to setup environments, and teams responsible for managing that stuff.  So now I have my own virtual instance of every server I need (which, for reference, is currently about eight VMs).  However, there are still some holes in that infrastructure.

Part of the issue is that we have a lot of teams that share or depend on the same infrastructure and/or services.  For instance, one of the services that my team maintains is also worked on by at least three other teams.  And we all share the same database schema, which doesn't automatically get updated in dev environments when changes are made in production, so when you pull in the develop branch, you frequently get breaking changes that you may not have ever heard of, usually in the form of config or database changes that aren't set in  your environment.  Sure, everything "just works" if you start fresh, but nobody ever starts fresh because it takes too long to spin up all the required pieces and set up the test data.  (However, we are moving toward a Docker/Kubernetes setup for the pieces that don't need test data, so things are moving in the right direction.)

Not that I can complain too much.  Even for my personal projects, which are much simpler, my dev environment is frequently in disarray.  Things get out of date, permissions or services don't get correctly configured after an update, or my setup is out of sync with the production environment.    In fact, I had that problem just the other day - I pushed some code for this blog that worked fine locally, but broke on the live server because it was running a different version of PHP.  Granted, the code in my dev environment was wrong (and I really should have caught it there), but that's not the point.

The point is that environment maintenance is hard and dev environment maintenance doubly so because it changes frequently and is of much lower priority than production.  That's something I hadn't really had to worry about in my internal IT job, because I was writing desktop apps and everybody was running identical Windows desktops.  It was a a simpler time....

This week's installment of "Pete's list of things that suck" features broken development environments.  I spent the last couple of days at work wrestling with one and it really, really sucks.

The thing is, our development environment is...really screwed.  Things went really bad during the production server upgrade a couple of months ago and we had to hack the hell out of the code just to keep the site running.  As a result, we had all kinds of ugly, server-specific hackery in production which just plain broke when we moved it back into devel.  And since, of course, we have a huge backlog of projects that management wants implemented, we've had no time to go back and reconfigure the development servers.

This week, I've been trying to test some changes to our user upload process.  However, due to some NFS misconfiguration and some code problems, uploads just plain don't work in our main development environment.  We do have a new, additional set of development servers (well, one box with a half-dozen virtual machines, but you get the idea), but I couldn't get those to work either.  Part of it was that the configuration on those servers was incomplete, and part of it was that Firefox sucks.  (Note from the present: I no longer remember why that was.)

Well, they're wrong.  PHP's documentation sucks.  And if you disagree, you're just plain wrong.

Actually, let me add some nuance to that.  It's not that the documentation sucks per se, it's that it sucks as documentation. 

You see, a lot of PHP's documentation is written with an eye to beginners.  It has lots of examples and it actually does a very good job of showing you what's available and giving you a general idea of how to use it.  So in terms of a tutorial on how to use the language, the documentation is actually quite good.

The problem is that, sometimes, you don't need a tutorial.  You need actual documentation.  By that, I mean that sometimes you care less about the generalities and more about the particulars.  For instance, you might want to know exactly what a function returns in specific circumstances, or exactly what the behavior is when you pass a particular argument.  Software is about details, and these details matter.  However, PHP frequently elides these details in favor of a more tutorial-like format.  And while that might pass muster for a rookie developer, it's decidedly not OK from the perspective of a seasoned professional.

Case in point: the socket_read() function.  I had to deal with this function the other day.  The documentation page is rather short and I was less than pleased with what I found on it. 

By way of context, I was trying to talk to the OpenVPN management console, which runs on a UNIX domain socket.  We had a small class (lifted from another project) that basically provided a nice facade over the socket communication functions.  I'd noticed that, for some reason, the socket communication was slow.  And I mean really slow.  Like, a couple of seconds per call slow.  Remember, this is not a network call - this is to a domain socket on the same box.  It might not be the fastest way to do IPC, but it should still be reasonably quick.

So I did some experimentation.  Nothing fancy - just injecting some microtime() and var_dump() calls to get a general idea of how long things were taking.  Turns out that's all I needed.  It quickly became obvious that each call to the method to read from the was taking about 1 second, which is completely absurd.

For context, the code in that method was doing something like this (simplified for illustration):

$timeoutTime = time() + 30;
$message = '';
while (time() < $timeoutTime) {
    $character = socket_read($this->socket, 1);
    if ($character === '' || $character === false) {
        break;  // We're done reading
    }
    $message .= $character;
}

Looks reasonable, right?  After all, the documentation says that socket_read() will return the number of characters requested (in this case one), or false on error, or the empty string if there's no more data.  So this seems like it should work just fine. 

Well...not so much.

The problem is with the last read.  It turns out that the documentation is wrong - socket_read() doesn't return the empty string when there's no more data.  In fact, I couldn't get it to return an empty string ever.  What actually happens is that it goes along happily until it exhausts the available data, and then it waits for more data.  So the last call just hangs until it reaches a timeout that's set on the connection (in our case, it was configured to 1 second) and then returns false.

So because we were relying on that "empty string on empty buffer" behavior to detect the end of input, calling that method always resulted in a one-second hang.  This was fairly easily fixed by just reading the data in much larger chunks and checking how much was actually returned to determine if we needed another read call.  But that's not the point.  The point is that we relied on what was in the documentation, and it was just totally wrong!

And it's not like this is the first time I've been bitten by the PHP docs.  Historically, PHP has been very bad about documenting edge cases.  For example, what happens if a particular parameter is null?  What's the exact behavior if the parameters do not match the expected preconditions?  Or what about that "flags" parameter that a bunch of functions take?  Sometimes the available flags are well documented, but sometimes it's just an opaque one-line description that doesn't really tell you what the flag actually does.  It's a crap shoot.

To be fair, the PHP documentation is not the worst I've ever seen.  Not even close.  And it really is very good about providing helpful examples.  It's just that it errs on the side of being light on details, and software is details.

This particular afternoon I was listening to some talks from the GOTOpia Europe 2020 virtual conference.  This one had an especially good lineup, including talks from Dave Thomas, Kevlin Henney, and Allen Holub, who are some of my favorite presenters.  Cat Swetel, who I'd never seen speak before, also had a very interesting presentation on metrics that I would heartily recommend.

It might not be the same vibe as the live conferences, but it's at least nice to be reminded that the entire world hasn't been canceled due to the pandemic.  And there are always some interesting discussions at GOTO, so it's worth a watch.

You see, this post is on LnBlog, and specifically what is (or was) wrong with it.  If you don't know, LnBlog is the software that runs this website - I wrote it as a "teach yourself PHP" project starting back around 2005.  I've been improving it, on and off, ever since.  So in this post, I'm going to show you what I thought of it back in 2013 and then discuss what I think now and what has changed.  Hopefully it will be somewhat enlightening.  Enjoy!

The year is (relatively) new and it's time for some reflection. In this case, reflection on past code - namely LnBlog, the software that runs this site.

I've come a long way from LnBlog, which as my first "teach yourself PHP" project. I've now been doing full-time professional PHP development since 2007 and can reasonably claim to have some expertise in it. And looking back, while the LnBlog codebase is surprisingly not horrifying for someone who had a whopping two months of web development experience going into it, it's still a mess. So it's time to start slowly refactoring it. And who knows? Blogging my thought process might be useful or interesting to others.

(Back to now: I actually did blog some of this stuff, but not until 2017 or so.  And I still agree with that initial assessment.  The code had plenty of problems then and it still does.  If I were starting fresh today, I'd probably do almost everything differently.  But on the other hand, I've seen much worse in much newer code.  And in the last three years or so I've been making slow and steady improvements.)

The Issues

There are lot of things about LnBlog that need changing. A few of them are functional, but it's mostly maintenance issues. By that I mean that the code is not amenable to change. It's not well organized, it's too hard to understand, and it's too difficult to make updates. So let's go over a few of the obvious difficulties.

1. The plugin system

I have to face it - the plugin system is an unholy mess. The entire design is poorly thought out. It's built on the premise that a "plugin" will be a single PHP file, which makes things...painful. Any plugin with significant functionality or a decent amount markup starts to get messy very quickly. The "single file" limitation makes adding styles and JavaScript ugly as well.

On the up side, the event-driven aspect works reasonably well. The code for it is a bit nasty, but it works. The main problem is that there aren't really enough extension points. It needs a bit more granularity, I think. Or perhaps it just needs to be better organized.

(Back to now: I still agree with most of this, except perhaps the thing about extension points.  So far, the only place where that's been a real problem is when it comes to inserting markup mid-page.  But yeah, the whole "a plugin is one file" thing was ill-conceived.  The good news is that it's totally fixable - I just need to figure out some design conventions around splitting things out, which hasn't been a priority so far.)

2. The templating system

This one is also an unholy mess. The idea isn't bad - allow any file in a theme to be over-ridden. However, I tried to abstract the template files too much. The files are too big and contain too much logic. Also, the simple template library I'm using is more a hindrance than a help. I'd be better off just ditching it.

I've also been thinking of getting rid of the translation support. Let's face it - I'm the only person using this software. And I'm only fluent in one language. Granted, the translation markers don't cause any harm, but they don't really do anything for me either, and accounting for them in JS is a bit of a pain.

(Back to now: The only thing I still agree with here is that the existing templates are a mess.  But that has nothing to do with the template system - I just did a bad job of implementing the template logic.  I'm working on fixing that - for instance, I added some Jinja-like block functionality to the template library.  I had considered re-writing the templates in Twig or something, but it quickly became obvious that that would be a huge amount of work, that it would be difficult to do in a piece-wise fashion, and it's not clear that the payoff would be worth it.  Likewise with the translation markers - taking them out would be a bunch of work for almost zero payoff and the JS thing isn't really that big a deal.  Besides, if I ever changed my mind again it's WAY more work to put them back in.)

3. The UI sucks

Yeah, my client-side skills have come a long way since I built LnBlog. The UI is very Web 1.0. The JavaScript is poorly written, the style sheets are a mess, the markup is badly done, and it's generally "serviceable" at best.

As I realized the other day, the style sheets and markup are probably the worst part. Trying to update them is difficult at best, which is exactly the opposite of what you want in a theme system. In retrospect, my idea to replace files wholesale rather than overriding seems misguided. They're too fragmented. When it comes to the style sheets and JavaScript, this also hurts performance, because there are a lot of files and everything is loaded in the page head.

(Back to now: This is pretty much still accurate.  I've been slowly improving the UI, but it's still not looking particularly "modern".  That's not such a big deal, but the templates and CSS are still a pain-point.  Really, what I need to do is rework the theme system so that I can easily make lighter-weight themes, i.e. I should be able to just create one override CSS file and call it good.  I have the framework for that in place, but I have yet to actually go through the existing themes and make that work.)

4. Too much compatibility

When I first started writing LnBlog, I had a really crappy shared web hosting account. And by "really crappy", I mean it offered no database server and had safe-mode and the various other half-baked PHP "security measures" enabled by default. So I actually built LnBlog to be maximally compatible with such an environment.

These days, you can get decent hosting pretty cheap. So unless you can't afford to pay anything, there's no need to settle for such crappy hosting. And again, let's be honest here - I don't even know anyone other than me who's using this software. So supporting such crappy, hypothetical configurations is a waste of my time.

In addition, I really put an absolutely ridiculous number of configuration settings into LnBlog. The main config file is extensively documented and comes to over 700 lines. That's completely nuts and a pain to deal with. It contains a lot of settings that are pointless and that hardly anyone would ever want to override. And most of those could be moved into a GUI rather than having to edit a file. There's absolutely no reason for many of those settings.

(Back to now: This is also still true.  I've been looking at redoing the config system, but that's another one of those things that is a big change because it has tendrils all through the code.  I have been moving some stuff out of the main blogconfig.php file, and I've been avoiding adding to it, but there's still a lot there.  For the most part, it's not a huge issue, since most of the things you would want to configure are through the UI, but still....)

5. No real controller structure

I knew nothing of MVC or design patterns when I first wrote LnBlog. As a result, the "glue" code is in the form of old-style procedural pages. They're messy, poorly organized, and hard to maintain. A more modern approach would make things much easier to deal with.

(Back to now: The old "pages" are dead in all but name.  A handful of them still exist, but they're three-liners that just delegate to a controller class.  The bad news is that it's pretty much just two monolithic controller classes with all the old logic dumped into them.  So that sucks.  But they have dependency injection and some unit test coverage, so this is still an improvement.  And I've at least got a little routing groundwork laid so that I could start breaking off pieces of functionality into other classes in the future.)

The Problem

While I'd like to fix all this stuff in one shot, there are three big problems here:

1. That's a lot of stuff, both in terms of the number of tasks and the amount of code involved.
2. I no longer have the kind of free time I did when I first wrote this.
3. I'm actually using this software.

Of course, those are two sides of the same coin.  LnBlog isn't huge, but it isn't tiny either - the codebase is upwards of 20,000 lines.  That wouldn't be a big deal if I were working on it as my full-time job, but this is a side-project and I can devote maybe a couple hours a day to it sometimes.  So major surgery is pretty much out.  And the third factor means that I need to be careful about breaking changes - not only do I not want to break my own website, but I also want to avoid having to do a lot of migration work because writing migration scripts is not my idea of a fun way to spend my free time.

(Back to now: This is always a problem with open-source and side projects.  Nothing has changed here except, perhaps, my development process.  After that year I spent learning about the Personal Software Process, I started using some of those methods for my personal projects.  The main change was that, when making any kind of a big change or feature addition, I actual do a semi-formal process with a requirements and design phase and review phases.  It sounds kind of silly for a personal project, but it's actually extremely useful.  The main benefit is just in having my thoughts documented.  Since I might be going a week or more between coding sessions on any particular feature, it's insanely helpful to have documentation to refer back to.  That way I don't have to remember or waste time figuring things out again.  And by having design- and code-review phases as part of my development process, I have a built-in reminder to go back and check that I actually implemented all those things I documented.  Having the whole thing written out just makes it much easier when you have long gaps in between work sessions.)

General commentary from the present: So as you can see from the above comments, I've fixed or am fixing a lot of the things that bothered me about LnBlog eight years ago.  In the last two or three years I've put a lot of work into this project again.  Part of it is because I actually use it and want it to be better, but part of it is also "sharpening the saw".  I've been using LnBlog as an exercise in building my development skills.  It's not just coding new features, like the flurry of development in the first two years or so that I worked on LnBlog, it's cleaning up my past messes, adding quality assurance (in the form of tests and static analysis), updating the documentation and figuring out how to balance responsible project management with limited resources).  It's an exercise in managing legacy code.

To me, this is a useful and important thing to practice.  As a professional developer, you will have to deal with legacy code.  In my day job, I've had to deal with code that was written by our CEO 10+ years ago when he started the company.  Software is a weird combination of things that live a week and things that live forever, and there's seldom any good way to tell which group the code will be in when you're writing it.  So while it's important to know how to write code correctly the first time, it's also important to know how to deal with the reality of the code you have.  And no, "let's rewrite it" is not dealing with reality.  And when you have a code-base that's 15 years old, that you're actively using, and that you originally wrote, it's a great opportunity to experiment and build your skills in terms of modernizing legacy code.

And that's just what I'm doing.  Slowly but surely, LnBlog is getting better.  I've implemented a bunch of new features, and in the process I've worked on my design and analysis skills, both at a product level and at a technical level.  I've fixed a bunch of bugs, which makes my life easier.  I've implemented additional tests and static analysis, which also makes my life easier by finding bugs faster and giving me more confidence in my code.  I've improved the design of the system, which again makes my life easier because I can now do more with less effort.  Sure, there's still plenty do to, but I've made lots of progress, and things are only getting better.

Of course, this is obviously not true of all developers (like Lauren, for example), but if we apply it to the general zeitgeist of the community, at least as you see it online, then there does seem to be something to this.  As a group, developers are very focused on "the coming thing", as Brisco County Jr. would say.  We all want to be ahead of the curve, working with the cool new technology that's going to take over the world.  We want to be on greenfield projects that are setting the standard for how to do things.  That's why otherwise intelligent programmers do or suggest crazy things like rewriting their conventional LAMP-based site in Go and ReactJS.  Of course, it's long been established that rewriting from scratch is almost always stupid and wasteful, but the fact is that while PHP might pay the bills, it isn't cool.

Of course, it isn't just because they want to be cool that developers like newer technologies.  There are plenty of other reasons.  Intellectual curiosity, for one.  Many of us got into this line of work because we enjoy learning new things, and there are always interesting new technologies coming out to learn.  Learning old things can be interesting as well, but there are a few problems with that:

1. Older technologies are less marketable.  Learning new tech takes a lot of time and effort, and if the tech is already on the way out, the odds of seeing a return on that investment of time, whether financial or just in terms of re-using that knowledge, are significantly lower.
2. Older tech involves more grunt work.  In other words, older programming technologies tend to work at a lower level.  Not always, but the trend is to increasing levels of abstraction.  That means that it will likely take more effort and/or code to do the same thing that you might get more or less for free with newer tech.
3. The problems are less fun.  This particularly applies to things like "supporting Internet Explorer", which Lauren mentions.  When you have to support both the old stuff and the new stuff, you generally have lots of problems with platform-specific quirks, things that are supposed to be compatible but really aren't, and just generally trying to work around limitations of the older tech.  These are the kind of problems that can be difficult, but not in a good way.  They're less like "build a better mousetrap" and more like "find a needle in this haystack".

So, in general, developers aren't usually super enthusiastic about working with or supporting old tech.  It's not really as bad as some people make it sound, but it's not really where most of us want to be.

Another factor is the way websites are developed.  The ideal is that you'd have somebody who is trained and experienced in designing user experiences and who is capable of considering all the use-cases and evaluating the site based on them.  That person could communicate that information to the designers and developers, who could incorporate it into their work and produce sites that are easy to use, compatible with assistive technologies, degrade gracefully when using less capable hardware or software, etc.  The reality is that this rarely happens.  In my experience:

1. Many teams (at least the ones I have experience with) have no UX designer.  If you're lucky, you'll have a graphic designer who has some knowledge or awareness of UX concerns.  More likely, it will be left up to the developers, who are typically not experts.  And if you're very unlucky, you'll have to work with a graphic designer who is fixated on pixel-perfect fidelity to the design and is completely indifferent to the user experience.
2. Most developers are on the young side.  (There are plenty of older developers out there, but the field has been growing for years and the new people coming in are almost all young.)  They're also able-bodied, so they really have any conception of the physical challenges that older people can have.  And it's hard to design for a limitation that you didn't think of and don't really understand.
3. While it's easy in principle, progressive enhancement and graceful degradation can be very tricky to actually pull off.  The main reason is that it's extremely easy to accidentally introduce a change that doesn't play well with some browser, doesn't display properly at some resolution, or what have you.
4. And let's not forget testing.  Even if you can build a site with progressive enhancement, proper accessibility, and attention to the needs of less technical users with little support available, you still need to test it.  And the more considerations, use-cases, and supported configurations you have, the more your testing space expands.  That makes it much harder and more time-consuming to make sure that all these things are actually present and working as intended for all users.

So what am I trying to say here?  I do agree with Lauren that supporting elderly users, disabled users, etc. is an important thing.  It's a thing that, as an industry, we should do.  But it's hard.  And expensive (at least compared to the way most shops work now).  That's not an excuse for not doing it - more like an explanation.

Every shop needs to find a balance between supporting a diversity of users and doing what they need to do within a reasonable budget of time and money.  While it's important to think about security and support new standards, I think that in recent years the industry has probably been a little too quick to abandon old, but still widely used technologies.  If nothing else, we should at least think more about our target user base and whether we're actually serving them or ourselves by introducing $COOL_JS_FRAMEWORK or dropping support for Internet Explorer.  I'm sure that in many (but not all) cases, dropping the old stuff is the right choice, but that shouldn't be the default assumption.

Of course, approximately 90% of those startups will be out of business in less than five years, but hey, you never know, right?  After all, the entire business model is "get big so you can IPO or get bought out".  Sure, you could try to build a profitable, medium-sized business, but that's basically the same thing as failure.  Besides, if you can attract venture capital, it's not your money you're blowing through anyway, so who cares?

The good news is that the industry seems to be moving away from that mindset a little bit in the last few years.  Or, at least, things seem to be moving more toward business-to-business sales, as opposed mass-market products that have no clear business plan beyond "get huge".  So maybe that's a good sign and this idea of copying every crazy/innovative thing that the big boys are doing will be a thing of the past soon.

Or maybe not.  But I can dream.

As might be clear from my review of Bertrand Meyer's Agile!: The Good, the Hype, and the Ugly, I've been rethinking the whole agile development craze that has swept the industry.

There are a number of good presentations online questioning the "agile" movement.  For a more provocative point of view, I recommend Erik Miejer's One Hacker Way talk.  Dave Thomas, one of the "pragmatic programmers", also has a good talk on the topic. There's also a good one by Fred George on the hidden assumptions of agile.

My current thought (note: we're back to 2019 now) is that "agile" has become pretty much a meaningless buzz word.  Pretty much everybody is "doing agile" now - or at least claiming they do.  It's come to mean "anything that's not waterfall".  And we all know that "waterfall" doesn't work, which is why everyone is doing "agile".  (Side note: Winston Royce, in his paper that initially described the waterfall process, actually says that it doesn't really work.  But, of course, that didn't stop people from trying.)

Not that agility is a bad concept.  It isn't - being flexible is a good thing.  Responding to change is almost a requirement in most shops.  The values in the Agile Manifesto are all good things to emphasize.  But none of that amounts to a process.  It's just a loose collection of principles and ideas that are useful, but aren't a road-map for how to build software.

That's why, in practice, most shops that "do agile" are using some variation on Scrum.  And while I have no problem with Scrum per se, it's hardly the be-all and end-all of development processes.  In fact, the main problem with Scrum is probably that it's not a software development process - it's more of a project management framework.  It doesn't have a whole lot to say about the details of how to code, how to test, how to manage defects and other quality issues, how to manage releases, etc.  It's up to each shop to figure that stuff out for themselves.

Of course, that's not bad.  Every business is different and you should expect that you'll have to adapt any process to a certain extent.  Scrum is useful in that it gives you a framework for tracking what needs to be done and creating a feedback loop to improve your process.  But you still have to actually use that feedback loop to improve your process, i.e. you have to do the hard work of self-improvement.  Simply going through the motions of what the "agile consultant" says you should do it's going to cut it.  As with everything else in life, there are no shortcuts.

The short-short version of the article is that DropBox originally had the idea to share code between their iOS and Android mobile applications.  After all, why write the same app twice when you can do it once?  Well, it turned out to be a really bad idea.  Because they were doing things for both platforms in a custom, non-standard way, they ended up taking on a lot of extra work just for the sake of re-using the code.  And not just in the amount of extra code they had to write in terms of custom libraries and tools - there's the extra effort of bringing new developers up to speed.  In fact, they eventually realized that it would actually be less work to just write everything twice and ended up abandoning their code-sharing approach.

Of course, this is an extreme case.  Trying to share code between two radically different platforms with different native implementation languages is bound to generate some issues.  A more prosaic example is the left-pad debacle.  You know, the one where a developer removed an 11-line JavaScript "library" from NPM and broke half the internet's deployment scripts.  But in that case, the scale was reversed - instead of trying to share a significant amount of code between a couple of projects, the left-pad package shared a completely trivial amount of code between an absurdly huge number of packages.  But the underlying problem is similar.  By trying to share that 11 lines of JavaScript "the right way", you end up taking on an external dependency that now has to be managed and probably generates more long-term maintenance effort than you saved from not writing the code yourself.  (I mean, seriously, why use a library for something that any half-competent programmer could write in less than 15 minutes?)

On the same theme, I've seen similar problems in client-side JavaScript.  I've worked on a codebase or two where large portions of the front-end were essentially cobbled together out of jQuery plugins that sorta-kinda work together, but not really.  Clearly the intention was to save effort by re-using existing components, but it didn't quite work out.  Sometimes it was because they didn't play well together, other times it was just that one of the components was not very well written.  In a couple of cases, after examining the plugins and the desired behavior, it became apparent that I could just write a version that did work cleanly in an hour or so - less time than it would take to debug the third-party plugin.

Of course, that's not to say that re-using code, whether your own or someone else's, is a bad idea.  It's clearly not.  Sometimes you find yourself writing nearly identical components for multiple projects, in which case it probably makes sense to abstract that into a shared library.  Or maybe there's a third-party library that already does more or less exactly what you need, in which case it probably makes sense to just use it.  That's all fine.

My point is just that "re-use some existing code," like any other idea in engineering, comes with trade-offs.  Sometimes those trade-offs are minor, such as the cost of managing a dependency that buys you a really big chunk of functionality.  Other times, they're pretty major, like the opportunity cost to DropBox of breaking all of the standard platform tooling.  Either way, the important thing is to think about those trade-offs and evaluate them honestly, not just jump to an "easy" answer.

If you read back through this blog, you might get the idea that I'm a bit of a contrarian.  I'm generally not the type to jump on the latest popular thing.  I'd rather go my own way and do what I think is best than go along with the crowd.  But at the same time, I know a lost cause when I see it and I can recognize when it's time to cut my losses.

For many years, I ran my own Mercurial repository on my web host, including the web viewer interface, as well as my own issue tracker (originally MantisBT, more recently The Bug Genie).  However, I've reached the point where I can't justify doing that anymore.  So I'm giving up and switching over to GitHub like everybody else.

I take no real pleasure in this.  I've been using Git professionally for many years, but I've never been a big fan of it.  I mean, I can't say it's bad - it's not.  But I think it's hard to use and more complicated than it needs to be.  As a comment I once saw put it, Git "isn't a revision control system, it's more of a workflow tool that you can use to do version control."  And I still think the only reason Git got popular is because it was created by programming celebrity Linus Torvalds.  If it had been created by Joe Nobody I suspect it would probably be in the same boat as Bazaar today.

That said, at this point it's clear that Git has won the distributed VCS war, and done so decisively.  Everything supports Git, and nothing supports Mercurial.  Heck, even BitBucket, the original cloud Mercurial host, is now dropping Mercurial support.  For me, that was kind of the final nail in the coffin.  

That's not the only reason for my switch, though.  There are a bunch of smaller things that have been adding up over time:

• There's just more tool support for Git.  These days, if a development tool has any VCS integration, it's for Git.  Mercurial is left out in the cold.
• While running my own Mercurial and bug tracker installations isn't a huge maintenance burden, it is a burden.  Every now and then they break because of my host changing some configuration, or they need to be upgraded.  These days my time is scarce and it's no longer fun or interesting to do that work.
• There are some niggling bugs in my existing environment.  The one that really annoys me is that my last Mercurial upgrade broke the script that integrates it with The Bug Genie.  I could probably fix it if I really wanted to, but the script is larger than you'd expect and it's not enough of an annoyance to dedicate the time it would take to become familiar with it.
• My web host actually now provides support for Git hosting.  So I can actually still have my own repo on my own hosting (in addition to GitHub) without having to do any extra work.
• Honestly, at this point I've got ore experience with Git than Mercurial, to the point that I find myself trying to run Git commands in my Mercurial repos.  So by using Mercurial at home I'm kind of fighting my own instincts, which is counterproductive.

So there you have it.  I'm currently in the process of converting all my Mercurial repos to Git.  After that, I'll look at moving my issue tracking in to GitHub.  In the long run, it's gonna be less work to just go with the flow.