LinLog

Note to self: just because Powershell defines aliases that mimic many of the standard UNIX commands does not mean they function the same way.

Last night, I was trying to migrate my company's Subversion repository to Mercurial - not for production use (yet), just as an experiment. After eventually getting the latest Mercurial installed on the Ubuntu 8.04 VM that hosts our Subversion repository, I tried running hg convert -s svn /path/to/svn/repo /path/to/hg/repo. As expected, the conversion process took some time, but chugged along nicely...for a while. Eventually, it hit an error and came back with:
svn: In file '/build/buildd/subversion-1.6.9dfsg/subversion/libsvn_ra/ra_loader.c' line 595: assertion failed (*path != '/')

I Googled around a bit, but still have no idea what that error message means or how to fix it. My best guess is that something is borked in our repo - not broken enough to break SVN, but maybeSo I tried a different tack - take the repository dump I had, import it into a fresh repository, and try again. That didn't go so well....

Since the SVN VM has a very small drive, I decided to load the dumpfile on my local Windows box. As you may know, svnadmin load reads streams, so you have to either pipe the dump file in or redirect standard input. Well, my first instinct was to do something like this:
svnadmin load newrepos < dumpfile.repo
One problem with that: the "<" character that you normally use for redirecting STDIN is reserved in Powershell. Drat! So I figured I'd just use a pipe instead:
cat dumpfile.repo | svnadmin load newrepos
So I ran that and waited. And shortly after I started waiting, I noticed my system slowing down. And then things started grinding to a halt - it was just barely responding. When I finally managed to get resmon up, I noticed that Powershell was eating nearly all of my system's RAM! And the command still hadn't produced a single line of output!

I'm not sure exactly what Powershell was doing, but it must have something to do with the Get-Content commandlet (for which "cat" is an alias) not liking the 1.4 GB dump file. Why it would use up more than twice the size of the file in memory, I'm not sure.

Anyway, I just switched to cmd.exe and did the input redirection method, which didn't eat huge amounts of memory. However, it didn't work either. The import died shortly after starting with an error about a bad transaction. Looks like the gods of revision control are not smiling on me today.

Last time I mentioned that I'd set up Mantis with a Subversion hook. Well, I've been rethinking the Subversion part.

I was listened to a back episode of Hanselminutes last week in which Scott was interviewing SourceGear's interviewing Eric Sink about DVCS and it was a very interesting conversation. Source control is one of those things you don't usually think about as a developer. You learn how to use a source control system that's "good enough", and after that, just sort of take it for granted. You certainly don't spend a lot of time jumping back and forth between systems and trying out the possibilities.

I really liked the way Eric explained distributed version control. As he pointed out, most of the arguments for DVCS that you hear seem pretty pointless, such as "There's no central server," or "You can commit on an airplane." Well...so what? Why wouldn't I want a central server - how else do you determine the authoritative repository? And honestly, who does important work on an airplane?

Instead, Eric framed the discussion in terms of flexibility. The flip-side of the above stupid arguments is actually pretty compelling. With DVCS, there's no central server, but that just means that your central server is determined by convention, not by the software, and it's much easier to set up "slave" repositories for remote sites. Likewise, while being able to commit on a plane is pointless, there's definite value in not needing to have a connection to the server to do core version control operations. Or, to put it another way, I may not code on a plane, but I do often code in coffee shops with slow WiFi connections.

So, in light of that, I decided to give Mercurial a try. According to Eric, it's fairly user-friendly, especially for Subversion users, and it's fairly Windows-friendly (it even has a TortoiseHg client, for whatever that's worth). So, since I was thinking of resurrecting LnBlog, I decided to use that as my test-case.

That leads me back into the Subversion hooks tie-in. Since I went to all the trouble of setting up an issue tracker and hooking it to SVN, I figured I'd do the same with Mercurial. Fortunately, it wasn't too bad once I figured out what I needed to do. I was able to fairly easily adapt the Powershell script I wrote for Subversion to work with Mercurial. In fact, the Mercurial version was actually shorter.

Actually adding a commit hook in Mercurial is pretty simple. You just add a line to your .hgrc file:
[hooks]
commit = powershell \path\to\hg-commit-hook.ps1
It seems that Mercurial preserves the environment for hooks, so you don't seem to have to worry about absolute paths and such like you do in Subversion.

The changes to the script itself were fairly small. This wiki page had a few good examples that got me started. The two big things were the passing of data and generating the message to pass to Mantis. Mercurial actually passes data into hook scripts through environment variables rather than command-line parameters, which is nice in that you actually get meaningful variable names coming in. As for the message generation, Mercurial's log command allows you to specify a template for its output, including substitution variables and some simple formatting functions. The result is a nice, short script with only a couple of calls to hg:

[System.Reflection.Assembly]::LoadWithPartialName("System.Web") | Out-Null

$issue = hg log -vr $env:HG_NODE

# If we don't have an issue number, just exit.
if ($issue -match "\b(?:bug|issue)\s*[#]{0,1}(\d+)\b") {
   
   $style_file = @'
changeset = "Changeset {branches}{rev}:{node|short} by {author|person}, {date|date}\n{desc}\n{files}"
file = " {file}\n"
'@
   Write-Output $style_file | Out-File -Encoding ASCII -Width 1000 hgstyle

   $data = hg log -vr $env:HG_NODE --style hgstyle
   rm hgstyle

   # Keep the cast to string from clobbering line breaks.
   $data = [String]::Join("`n", $data)
   $postData = "secret=somePassword&message="
   $postData += [System.Web.HttpUtility]::UrlEncode($data)

   C:\Cygwin\bin\wget.exe -O req.out --post-data=$postData http://yourhost.com/path/to/mantis/scripts/do_checkin.php
   rm req.out
}

That's it - a measly two calls to Mercurial, including one to see if we even need to run the hook. By specifying the right template, we can get the entire message in one command. That template gives me something like this:
Changeset 70:cbdb625298ca by Peter Geer, Mon Dec 13 16:45:53 2010 -0500
Got rid of unneeded requires now that we have an autoloader (issue #0000005).
   lib/article.php
   lib/creators.php
Perhaps not quite as pretty as the Subversion version, but close enough for now. Perhaps if I feel like getting really fancy, I'll look at the Python API for Mercurial and try my hand at writing some in-process hook code.

Edit: Updated the script to put each of the listed file on a new, indented line.

I came across an interesting article on Task Driven Development the other day. I came to it via the recent Linus on branching story on Slashdot.

One of the first things mentioned in that article is that you need an issue tracking system and it needs to be integrated with your source control system. After a moment's consideration, I said to myself, "That's a good idea." I've been thinking lately about getting back into doing some open-source stuff as well as some educational projects for myself, and something like that might be helpful. In the past, I usually didn't bother to track bugs and features in my personal projects, unless you count post-it notes and unorganized text files, because, well, they were personal projects and it just didn't seem important. However, now I'm used to living and dying by Jira and am fully aware of how crazy it is to think I'm just going to remember all the bugs I find and potential features I think up.

So, to that end, I installed a copy of Mantis on my site. I settled on Mantis, after a good 5 minutes of careful research, based largely on the facts that the article recommended it and that the requirements were low enough that I could run it on my cheap-ass shared hosting account. Fortunately, it seems nice enough so far - not radically different from Bugzilla or Jira, but OK.

Anyway, I came across a slight snag in setting up the Mantis integration with my Subversion repository. You see, Mantis supplies a script that can be called from Subversion's post-commit hook that will update any issues referenced in the commit message. However, the script is written on the assumption that Mantis and SVN live on the same box. In fact, that is not the case - SVN lives on my local Windows 7 box, while Mantis lives on my hosting provider's box, to which I do not have shell access (yes, I'm that cheap).

With a little Googling, I was able to turn up a few useful resources to help me get this set up. This article gave a nice overview of SVn hooks and Mantis integration, while this one provided a nice perspective for SVN on Windows. However, I had to cobble these together to make it work on my system.

My main problem with the documentation I had was my lack of SSH access to the box on which I'm running Mantis - all the articles I found assumed you had it. So the first order of business was a simple gateway PHP script to wrap the command-line Mantis script that would normally be run via SSH. Therefore I quickly banged out this do_checkin.php script and dropped it in a web-accessible directory.

<?php
# Very cheap security measure, simply so that just *anybody* can't post issues.
if ($_POST['secret'] !== 'somePasswordHere') {
   exit;
}

$cmd = 'php checkin.php <<< ' . escapeshellarg($_POST['message']);
system($cmd);


Next was the Subversion hook. I'm using SlikSVN, and it turns out that this will only execute hook scripts that are .exe or .bat files. I know this because I tried to write a Powershell hook script and it didn't work. But no worries - it's easy enough to write a wrapper batch post-commit.bat file such as this:

@ECHO OFF

SET REPOS=%1
SET REV=%2

CD %REPOS%\hooks
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe %REPOS%\hooks\post-commit.ps1 "%REPOS%" "%REV%"


For the Powershell script, it was easy enough to just port the BASH script posted in the previously linkd article. However, instead of calling the Mantis checkin.php script by SSH, I actually just used wget to send a POST to the do_checkin.php posted above.

[System.Reflection.Assembly]::LoadWithPartialName("System.Web") | Out-Null

$REPOS = $args[0]
$REV = $args[1]

"$REPOS $REV"

$svnlook = 'C:\Program Files\SlikSvn\bin\svnlook.exe'

$auth = & $svnlook author -r $REV $REPOS
$dt = & $svnlook date -r $REV $REPOS
$changed = & $svnlook changed -r $REV $REPOS
$changed = $changed -join "`n"
$log = & $svnlook log -r $REV $REPOS
$log = $log -join "`n"
$data = "Changeset [${REV}] by $auth, $dt
$log
$changed"

$postData = "secret=somePasswordHere&message="
$postData += [System.Web.HttpUtility]::UrlEncode($data)

C:\Cygwin\bin\wget.exe -O req.out --post-data=$postData http://yoursite.com/path/to/mantis/scripts/do_checkin.php
rm req.out


This seems to work fine in my testing. We'll see if that changes once I start adding in bugs and features for my projects and then writing the code to complete them. Hopefully that process will be somewhat easier for being more organized.

As previously mentioned, since switching my desktop to Windows, I've set up a Subversion service using SlikSVN and an SSH service using Cygwin. So this week, I figured I'd try getting them to play together so that I can do Subversion over SSH and not have to open up another port in my firewall. It eventually worked out, but wasn't as painless and it should have been. And it was totally my fault.

As you know if you've ever used Subversion over SSH, it's pretty simple. You just change your protocol, point SVN to the repository on the remote server, and supply your OS login credentials. So, since I already had an account set up on the server and svnserve running, I figured this should work:
svn ls svn+ssh://myuser@myserver/myproject
But no dice - I got the message "svn: No repository found in 'svn+ssh://myuser@myserver/myproject'"

Hmm..... Time to Google.... Oh, that's right - you can't use the root directory supplied to svnserve over SSH! Instead, you have to supply the full path to the repository and project. But wait - my repository is on the D: drive...so how do you reference that? Well, SSH is running on Cygwin, so we can use Cygwin's drive mapping. So change that command to:
svn ls svn+ssh://myuser@myserver/cygdrive/d/myrepos/myproject
That should work, right?

Yeah...not so much. That definitely should work, but I'm still getting that "no repository found" message. So what's the deal?

A little searching revealed that, behind the scenes, the svn+ssh:// protocol runs a command similar to this:
ssh myserver svnserve -t
Turns out that the problem was in that command.

See, the svnserve portion runs on the Subversion server, which, in this case, is inside Cygwin on a Windows box. However, I have two copies of svnserve - one from Cygwin and one from SlikSVN, and they don't both work the same way.

For SVN+SSH to work, I need to pass the repository path in with the Cygwin path mapping, and SlikSVN doesn't understand that. Thus the need for Cygwin's SVN. However, SlikSVN is first in my path when I connect via SSH, so it's SlikSVN's svnserve that's getting run inside Cygwin. Hence the "no repository found" message.

After a bit of experimentation, it turns out that this is really easy to fix. All you need to do is set the PATH in your Cygwin .bashrc file to explicitly put the Cygwin binaries first. Just add the following line to the end of the file:
export PATH=/bin/:/usr/bin/:/usr/local/bin/:$PATH

So, problem solved. Unfortunately, it took a lot longer than I would have thought, mainly because I couldn't find anyone else who had the same problem. So hopefully anyone else who's crazy enough to set things up this way will come across this post if they have any problems.

Well, I did my Windows 7 install the other day. One day later, I'm doing pretty well. Ran into some problems, but so far the results are not bad.

Unsurprisingly, the actual install of Windows 7 was pretty uneventful. Pretty much the same as a typical Ubuntu installation - selecting partition, entering user info, clicking "next" a few times, etc. Nothing to report there.

The initial installation of my core programs was pretty easy too, thanks to Ninite. They have a nifty little service that allows you to download a customized installer that will do a silent install of any of a selected list of free (as in beer) programs. So I was able to go to a web page, check off Opera, Thunderbird, Media Monkey, the GIMP, Open Office, etc., download a single installer, and just wait while it downloaded and installed each program. Not quite apt-get, but pretty nice.

My first hang-up occurred when installing the Ext2IFS. Turns out that the installer won't run in Windows 7. You need to set it to run in Windows Server 2008 compatibility mode. And even after that, it was a little dodgy. It didn't correctly map my media drive to a letter on boot. It worked when I manually assigned a drive letter in the configuration dialog, but didn't happen automatically. It was also doing weird things when I tried to copy some backed-up data from my external EXT3-formatted USB drive back to my new NTFS partition. Apparently something between Ext2IFS and Win7 doesn't like it when you try to copy a few dozen GB of data in 20K files from EXT3 to NTFS over USB. (Actually, now that I write that, it seems less surprising.) The copy would start analyzing/counting the files, and then just die - no error, no nothing. I finally had to just boot from the Ubuntu live CD and copy the data from Linux. Still not sure why that was necessary.

I also had some interesting issues trying to install an SSH server. I initially tried FreeSSHD, which seemed to be the best reviewed free server. The installation was easy and the configuration tool was nice. The only problem was, I couldn't get it to work. And I mean, at all. Oh, sure, the telnet server worked, but not the SSH server. When set to listen on all interfaces, it kept complaining that the interface and/or port was already in use when I tried to start the SSH server. When bound to a specific IP, it gave me a generic access error (literally - the error message said it was a generic error).

After messing around fruitlessly with that for an hour or so, I gave up and switched to the MobaSSH server. This one is based on Cygwin. It's a commercial product with a limited home version and didn't have quite as nice an admin interface, but seems to work sell enough so far. The one caveat was that I did need to manually open port 22 in the Windows firewall for this to work.

The biggest problem so far was with setting up Subversion. Oh, installing SlikSVN was dead simple. The problem was setting up svnserve to run as a service. There were some good instructions in the TortiseSVN docs, but the only worked on the local host. I could do an svn ls <URL> on the local machine, but when I tried it from my laptop, the connection was denied. So I tried messing with the firewall settings, but to no effect. I even turned off the Windows firewall altogether, but it still didn't work - the connection was still actively denied.

I started looking for alternative explanations when I ran netstat -anp tcp and realized that nothing was listening on port 3690. After a little alternative Googling, I stumbled on to this page which gave me my solution. Apparently, the default mode for svnserve on Windows, starting with Vista, is to listen for IPv6 connections. If you want IPv4, you have to explicitly start svnserve with the option --listen-host 0.0.0.0. Adding that to the command for the svnserve service did the trick.

Since I haven't posted anything in a really long time, with the exception of the Press Your Luck post of a couple weeks ago, I thought I'd do a little list to get back into the swing of things. So this is just a list of thing technical/computery that have made me happy, to some extent, over the course of this year.

1. Windows 7. I've preferred Linux for a long time, but I've got to say, Win7 is awfully nice. I fact, it's so good I'm actually thinking of dumping Linux on my main desktop at my next upgrade. And for reference, I haven't had Windows on my home desktop since 2002 - and I was dual-booting then.
2. Subversion. Not that Subversion is that great in and of itself (although it is a perfectly good revision control system), but this year my team finally migrated from CVS to Subversion, and it sucks so much less! Yes, you read that right - my company was actually using CVS for new projects until the beginning of 2010. It's nice to finally step forward into 2005!
3. The new versions of Opera. I've been an Opera user for a very long time, and version 10.50+ has gotten really nice. The updated UI is pretty nifty, Opera Link is a very handy feature I've always wanted, and I actually find Opera Unite really cool. I use it to access my home PC fairly regularly. I don't care how fast Chrome is, Opera's still better in my book.
4. Not coding in JavaScript and HTML. Our new product at work is implemented in Adobe's FLEX. Yeah, FLEX is based on Flash, which sucks, but it's actually a pretty nice framework. Way more pleasant to work with than JavaScript and HTML. Even if you include jQuery, which takes most of the suck out of JavaScript, FLEX is still nicer. Now if only Flash didn't suck so much.... Too bad there's no viable alternative. (Silverlight doesn't have the market penetration and nobody with half a brain thinks HTML 5 is comparable. And yes, that implies Steve Jobs doesn't have half a brain.)
5. Not using Web TimeSheet. Last year, our company was using Web TimeSheet for our time tracking. If you've never used it, suffice it to say it sucks beyond all rational belief. Seriously - I can't describe how bad it was. It took damn near 30 seconds to load the time entry form. And that wasn't even waiting for an HTTP response - it was actively loading a bajillion things during that time. Now, we're doing our time/task tracking in Jira instead. Sadly, Jira still sucks, but not nearly as bad as Web TimeSheet.
6. I have a window. For the first time ever, I have a desk by the window. The president of our company left for another job a couple weeks ago, and since we're not replacing him, the CEO gave our four-man dev team permission to move into his office. It's just big enough for all four of us (only slightly smaller than big-ass the cube we all shared), and two of the four walls are all window. And I got the corner where the two windows meet. This is a far cry from when I was in public service and had the "cube" that used to be a printer nook and was exactly large enough to hold my desk - and nothing else.
7. Whipping out SQL For Smarties. A couple months ago I actually had an excuse at work to whip out my copy of Joe Celko's SQL For Smarties and apply one of the chapters. I needed to store a nested folder structure in the database, so I turned to the trees and hierarchies section, did a little reading, and settled on the path enumeration method. One of my colleagues had suggested adjacency lists, which Celko calls "the most commonly used and worst possible" model for trees. I love that book - it always makes me happy when I have an excuse to use something from it.

Seven seems like a good number. Time to call it a night. I get up early enough to go to the gym and still make it to the morning scrum. Perhaps I can continue the list later.

After six months of silence, here's a random "I'm writing this so I don't have to keep Googling it" post.

The "no whammy" game show was called "Press Your Luck". According to Wikipedia, it was hosted by Peter Tomarken and featured announcer Rod Roddy, in case anybody cares that much.

That just jumped into my head again as I was merging my private dev branch in Subversion back to trunk. For some reason - probably a version mismatch between clients and the server, which is only on 1.4 - we've been getting a lot of tree conflicts on merge. So this morning I found myself watching the merge progress saying, "No conflict, no conflict, no conflict, STOP!"