LinLog

Another "note to my future self", recoded here for posterity.

Today I finally got around to making SSH agent work in Powershell with Git.  For the last year or so, I haven't had to deal with because my work mostly involved writing PHP code inside of WSL.  In that scenario, you're essentially using Linux, so ssh-agent works just fine.

But on native Windows with Powershell...not so much.

I mean, sure, I could just use Git BASH.  But why would I want to do that?  Powershell is awesome, and even if I probably know BASH better, I prefer Powershell.

But it turns out it wasn't all that difficult to get things working in Powershell.  There were two pieces to it:

1. Use the right SSH agent (i.e. the one that comes with Git).
2. Write a Powershell adapter for it.

It turns out that there's a Windows service for "OpenSSH Authentication Agent".  I'm not entirely sure what installs that, but apparently it's different from the ssh-agent that's installed with the Windows Git package and the ssh-add command from that doesn't seem to talk to it properly.

My solution was to just disable that service and use the plain-old ssh-agent that comes with Git.  The only problem with that is that the traditional invocation of eval `ssh-agent` doesn't work in Powershell because it outputs a BASH-formatted script.  But that's easily fixed with a couple of regular expressions.  So I added this to my Powershell $profile:

Set-Alias ssh-agent "$env:ProgramFiles\git\usr\bin\ssh-agent.exe"
Set-Alias ssh-add "$env:ProgramFiles\git\usr\bin\ssh-add.exe"

# Need to turn off Open SSH Authentication Agent system service,
# then run the ssh-agent from Git.
function Start-SshAgent {
    $output = (ssh-agent)
    $sock_match = ($output | Select-String -Pattern 'SSH_AUTH_SOCK=(\S+);')
    $sock_path = $sock_match[0].Matches.Groups[1].Value
    $pid_match = ($output | Select-String -Pattern 'SSH_AGENT_PID=(\d+);')
    $agent_pid = $pid_match[0].Matches.Groups[1].Value
    $env:SSH_AUTH_SOCK = $sock_path
    $env:SSH_AGENT_PID = $agent_pid
    Write-Output "Agent pid $agent_pid"
}

Start-SshAgent

And there we go!  Now I can just run ssh-add to add my key and Git picks it up as expected.

Another quick note to my future self: setting up Git under Windows to use SSH key authentication is pretty easy...once you know what to do.

At work, we have some Composer and Bower packages that we fetch from our internal GitHub Enterprise server.  And, of course, all the source lines in the composer.json and bower.json files use SSH references.  I just use HTTP for my code checkouts, so I finally had to figure out how to make Git for Windows authenticate with my SSH key.

Turns out it's pretty easy.  I have a key pair I created with PuTTYgen.  All I had to do was export my private key to OpenSSH format and copy that file to C:\Users\<my_username>\.ssh\id_rsa.  Git picks it up with no further configuration.  Then I just added my key to GitHub and I was good to go.

This is yet another one of those "post this so I can refer back to it later" things.  So if you're not a Windows user, or if you don't use ConEmu, then I suggest you go get a cup of coffee or something.

So for a while now I've been using ConEmu as my Windows console app.  It supports multiple tabs, transparency (ooooh), customizable hotkeys, customizable sessions, Far manager integration and a whole bunch of other nifty stuff.  

A couple of months ago, I saw that it was possible to embed PuTTY, the popular Windows-based SSH client, directly in a ConEmu tab.  So I tried it out and found it to be pretty slick.  The only down side was some key binding weirdness.

First, there's the general putty issue that if you accidentally press ctrl+S - you know, the key combination that means "save" in just about every editor in existence - it effectively locks the terminal and it's not obvious how to get control back.  The second issue is that, when you embed an application like PuTTY in ConEmu, it steals most of your keyboard input, so the standard key bindings for switching between window tabs don't work.

Luckily, these problems are easily fixed.  The fixes are just hard to remember, which is why I'm writing it down.  For the ctrl+S iissue, you can just hit ctrl+Q to turn input back on.  For the tab-switching issue, you can use the global key bindings for ConEmu - namely Win+Q, Win+Shift+Q, and Win+<Number> to switch consoles, as well as Win+Z to toggle focus between ConEmu and PuTTY.

As previously mentioned, since switching my desktop to Windows, I've set up a Subversion service using SlikSVN and an SSH service using Cygwin. So this week, I figured I'd try getting them to play together so that I can do Subversion over SSH and not have to open up another port in my firewall. It eventually worked out, but wasn't as painless and it should have been. And it was totally my fault.

As you know if you've ever used Subversion over SSH, it's pretty simple. You just change your protocol, point SVN to the repository on the remote server, and supply your OS login credentials. So, since I already had an account set up on the server and svnserve running, I figured this should work:
svn ls svn+ssh://myuser@myserver/myproject
But no dice - I got the message "svn: No repository found in 'svn+ssh://myuser@myserver/myproject'"

Hmm..... Time to Google.... Oh, that's right - you can't use the root directory supplied to svnserve over SSH! Instead, you have to supply the full path to the repository and project. But wait - my repository is on the D: drive...so how do you reference that? Well, SSH is running on Cygwin, so we can use Cygwin's drive mapping. So change that command to:
svn ls svn+ssh://myuser@myserver/cygdrive/d/myrepos/myproject
That should work, right?

Yeah...not so much. That definitely should work, but I'm still getting that "no repository found" message. So what's the deal?

A little searching revealed that, behind the scenes, the svn+ssh:// protocol runs a command similar to this:
ssh myserver svnserve -t
Turns out that the problem was in that command.

See, the svnserve portion runs on the Subversion server, which, in this case, is inside Cygwin on a Windows box. However, I have two copies of svnserve - one from Cygwin and one from SlikSVN, and they don't both work the same way.

For SVN+SSH to work, I need to pass the repository path in with the Cygwin path mapping, and SlikSVN doesn't understand that. Thus the need for Cygwin's SVN. However, SlikSVN is first in my path when I connect via SSH, so it's SlikSVN's svnserve that's getting run inside Cygwin. Hence the "no repository found" message.

After a bit of experimentation, it turns out that this is really easy to fix. All you need to do is set the PATH in your Cygwin .bashrc file to explicitly put the Cygwin binaries first. Just add the following line to the end of the file:
export PATH=/bin/:/usr/bin/:/usr/local/bin/:$PATH

So, problem solved. Unfortunately, it took a lot longer than I would have thought, mainly because I couldn't find anyone else who had the same problem. So hopefully anyone else who's crazy enough to set things up this way will come across this post if they have any problems.

There was one annoying side-effect of installing the Cygwin SSH server - I had to create an account for it. Not that this is inherently a problem, but I noticed that the new "Privileged service" account showed up on the login screen. That needed to go.

Turns out that hiding an account in Windows 7 is actually pretty easy. Just a simple registry entry - create a DWORD with the name of the user to hide and value zero under "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\SpecialAccounts\UserList" - and you're all set. Nice to know.

Yeah.... So remember how I was messing with freeSSHd last weekend? I'm thinking that was a waste of time.

While the native shell thing in freeSSHd was cool, I had two big issues with it. First, the terminal was buggy. I got lots of weird screen artifacts when I connected to it - the screen not clearing properly, weird random characters hanging around after a lot of output, etc. Not the rock-solid terminal experience I'm used to.

The second thing was something that occurred to me after trying out a couple of SSH sessions. If freeSSHd is using a native shell, with native file paths, how is that going to work with SCP and SFTP? How does it account for drive letters and path delimiters? Turns out the answer (at least for SFTP) was restricting access to a certain pre-defined directory ($HOME by default). For SCP...well, I still haven't figured out how that's supposed to work. But either way, this is decidedly not the behavior I was looking for.

So instead, I decided to go the old-fashioned way and just install Cygwin and use their OpenSSH server. And you know what? It was completely painless. I just used this handy LifeHacker entry to get the auto-config command and the necessary settings, and I was done. I started the service, connected to the SSH server, and all my stuff was right where it was supposed to be. I have a solid, familiar terminal, access to my whole system via SCP and SFTP, and NT authentication worked out of the box. Heck, the hardest part was figuring out the funky non-standard UI of the installer's package selection dialog. I should have just used Cygwin from the beginning and saved myself some effort.

Well, after my dismal failure yesterday, I mostly got FreeSSHd working. Not sure if it was worth the effort or not, but we'll see.

Turns out the problem was with my expectations. Or maybe I can blame it on the (extremely) minimal documentation. You see, I said "yes" when asked if I wanted to run freeSSHd as a Windows service. Now, I figured that starting the freeSSHd program in the start menu while the service was running would allow me to configure that service and view its status. Yeah...not so much. Apparently running the config program actually tries to start another instance of freeSSHd. Hence why I was getting errors that the the port was in use. It was already being used by the service and I was unwittingly trying to spawn another instance.

It turns out that to configure the service, I need to start the config program, do my stuff, close the config program, and restart the service. At least, that seemed to do the trick - I'm not actually sure how much of it was necessary. After that, freeSSHd ran quite nicely.

My one remaining problem was authentication. I wanted to use NT authentication, which freeSSHd gives as an option. The problem was, it didn't quite work. After creating a user and setting it to NT authentication, I was able to log in...and that's it. I connected with Putty, entered my password, and freeSSHd immediately disconnected me. No warnings, no errors, nothing in the logs - just immediate disconnection.

The really odd thing was that NT authentication worked just fine if I ran freeSSHd by starting the config program rather than as a service. Running it as service, though, disconnected every time. The only time it didn't was when I disabled the "new console" option. Then the session would just hang and not accept input, which wasn't an improvement. I tried various settings and Googled fruitlessly, but no luck. I still have no idea what was going on. After mucking about with this for probably an hour and a half, I finally gave up and changed my freeSSHd user to use a SHA1 hashed password. That worked just fine, but feels like defeat.

The one thing I do like about freeSSHd so far is that it allows you to select your command shell. You actually get a native Windows shell instead of being forced into Cygwin weirdness. I changed mine from the default cmd.exe to PowerShell. That should make for a more pleasant experience.

Well, I did my Windows 7 install the other day. One day later, I'm doing pretty well. Ran into some problems, but so far the results are not bad.

Unsurprisingly, the actual install of Windows 7 was pretty uneventful. Pretty much the same as a typical Ubuntu installation - selecting partition, entering user info, clicking "next" a few times, etc. Nothing to report there.

The initial installation of my core programs was pretty easy too, thanks to Ninite. They have a nifty little service that allows you to download a customized installer that will do a silent install of any of a selected list of free (as in beer) programs. So I was able to go to a web page, check off Opera, Thunderbird, Media Monkey, the GIMP, Open Office, etc., download a single installer, and just wait while it downloaded and installed each program. Not quite apt-get, but pretty nice.

My first hang-up occurred when installing the Ext2IFS. Turns out that the installer won't run in Windows 7. You need to set it to run in Windows Server 2008 compatibility mode. And even after that, it was a little dodgy. It didn't correctly map my media drive to a letter on boot. It worked when I manually assigned a drive letter in the configuration dialog, but didn't happen automatically. It was also doing weird things when I tried to copy some backed-up data from my external EXT3-formatted USB drive back to my new NTFS partition. Apparently something between Ext2IFS and Win7 doesn't like it when you try to copy a few dozen GB of data in 20K files from EXT3 to NTFS over USB. (Actually, now that I write that, it seems less surprising.) The copy would start analyzing/counting the files, and then just die - no error, no nothing. I finally had to just boot from the Ubuntu live CD and copy the data from Linux. Still not sure why that was necessary.

I also had some interesting issues trying to install an SSH server. I initially tried FreeSSHD, which seemed to be the best reviewed free server. The installation was easy and the configuration tool was nice. The only problem was, I couldn't get it to work. And I mean, at all. Oh, sure, the telnet server worked, but not the SSH server. When set to listen on all interfaces, it kept complaining that the interface and/or port was already in use when I tried to start the SSH server. When bound to a specific IP, it gave me a generic access error (literally - the error message said it was a generic error).

After messing around fruitlessly with that for an hour or so, I gave up and switched to the MobaSSH server. This one is based on Cygwin. It's a commercial product with a limited home version and didn't have quite as nice an admin interface, but seems to work sell enough so far. The one caveat was that I did need to manually open port 22 in the Windows firewall for this to work.

The biggest problem so far was with setting up Subversion. Oh, installing SlikSVN was dead simple. The problem was setting up svnserve to run as a service. There were some good instructions in the TortiseSVN docs, but the only worked on the local host. I could do an svn ls <URL> on the local machine, but when I tried it from my laptop, the connection was denied. So I tried messing with the firewall settings, but to no effect. I even turned off the Windows firewall altogether, but it still didn't work - the connection was still actively denied.

I started looking for alternative explanations when I ran netstat -anp tcp and realized that nothing was listening on port 3690. After a little alternative Googling, I stumbled on to this page which gave me my solution. Apparently, the default mode for svnserve on Windows, starting with Vista, is to listen for IPv6 connections. If you want IPv4, you have to explicitly start svnserve with the option --listen-host 0.0.0.0. Adding that to the command for the svnserve service did the trick.

Here's a quick code post from my Powershell profile. I keep copying this around every time I set up a new system, so maybe it will be useful to others. And to me, since I can copy it off the web in the future rather than having to dig it up or recreate it.

Being an old-school UNIX guy, I get used to just typing ssh hostname to access my servers. So the Putty syntax of loading a profile annoys me. To that end, I wrote a few little functions to add to my Powershell $profile to address this. They just let me load a session by typing the familiar commands and don't require me to change syntax when using a profile versus connecting directly to a hostname.


# Suppress the assembly loading output
[System.Reflection.Assembly]::LoadWithPartialName("System.Web") | Out-Null

function _puttyHasProfile {
   $profile_name = $args[0].ToLower()
   $profiles = Get-ChildItem HKCU:\Software\SimonTatham\PuTTY\Sessions\
   foreach ($p in $profiles) {
      $name = [System.IO.Path]::GetFileName($p.Name)
      # Putty stores spaces and other special characters URL encoded.
      $name = [System.Web.HttpUtility]::UrlDecode($name).ToLower()
      if ($name -eq $profile_name) {
         return 1
      }
   }
   return 0
}

function ssh {
   $profile_name = $args[0]
   if (_puttyHasProfile($profile_name)) {
      putty -load $profile_name
   } else {
      putty $args
   }
}

function sftp {
   $profile_name = $args[0]
   if (_puttyHasProfile($profile_name)) {
      psftp -load $profile_name
   } else {
      psftp $args
   }
}


Here's a little problem I ran across at work the other day. I was writing a little script to SSH into our 2 app servers and restart Memcached. Simple and easy, right?

Well, not quite. When I ran the script, the restart was hanging the SSH session. The service restarted properly, but my SSH connection never closed, so the script couldn't move on to the second server. What the heck?!?

Turns out that this is an issue with the way OpenSSH handles I/O streams. Apparently the init script for starting memcached didn't close standard output, so OpenSSH stuck around waiting for it. The solution: redirect STDOUT and STDERR on the server to /dev/null. That closes the streams and keeps the session from hanging. Of course, that could be a problem if you actually need to see any of the output, but in this case, I didn't.