LinLog

Another "note to my future self", recoded here for posterity.

Today I finally got around to making SSH agent work in Powershell with Git.  For the last year or so, I haven't had to deal with because my work mostly involved writing PHP code inside of WSL.  In that scenario, you're essentially using Linux, so ssh-agent works just fine.

But on native Windows with Powershell...not so much.

I mean, sure, I could just use Git BASH.  But why would I want to do that?  Powershell is awesome, and even if I probably know BASH better, I prefer Powershell.

But it turns out it wasn't all that difficult to get things working in Powershell.  There were two pieces to it:

1. Use the right SSH agent (i.e. the one that comes with Git).
2. Write a Powershell adapter for it.

It turns out that there's a Windows service for "OpenSSH Authentication Agent".  I'm not entirely sure what installs that, but apparently it's different from the ssh-agent that's installed with the Windows Git package and the ssh-add command from that doesn't seem to talk to it properly.

My solution was to just disable that service and use the plain-old ssh-agent that comes with Git.  The only problem with that is that the traditional invocation of eval `ssh-agent` doesn't work in Powershell because it outputs a BASH-formatted script.  But that's easily fixed with a couple of regular expressions.  So I added this to my Powershell $profile:

Set-Alias ssh-agent "$env:ProgramFiles\git\usr\bin\ssh-agent.exe"
Set-Alias ssh-add "$env:ProgramFiles\git\usr\bin\ssh-add.exe"

# Need to turn off Open SSH Authentication Agent system service,
# then run the ssh-agent from Git.
function Start-SshAgent {
    $output = (ssh-agent)
    $sock_match = ($output | Select-String -Pattern 'SSH_AUTH_SOCK=(\S+);')
    $sock_path = $sock_match[0].Matches.Groups[1].Value
    $pid_match = ($output | Select-String -Pattern 'SSH_AGENT_PID=(\d+);')
    $agent_pid = $pid_match[0].Matches.Groups[1].Value
    $env:SSH_AUTH_SOCK = $sock_path
    $env:SSH_AGENT_PID = $agent_pid
    Write-Output "Agent pid $agent_pid"
}

Start-SshAgent

And there we go!  Now I can just run ssh-add to add my key and Git picks it up as expected.

I discovered something enlightening and annoying this morning: PowerShell uses different execution policy settings for the 32-bit and 64-bit versions.  This is something I did not know and did not expect.

I'd seen this problem before, but could never figure out what was happening.  I'd try to run a command through PowerShell using another program, in this case Vim, and it would fail with the standard error about execution of my profile.ps1 being prohibited by the execution policy setting.  Yet when I bring up Powershell on the command line, everything works and the execution policy looks correct:

PS pgeer> Get-ExecutionPolicy -List Scope ExecutionPolicy ----- --------------- MachinePolicy Undefined UserPolicy Undefined Process Undefined CurrentUser Undefined LocalMachine RemoteSigned

Well, it turns out we're talking about two different versions of Powershell.  When I run it from the terminal, I'm using the 64-bit version.  But my Vim build is 32-bit, which apparently means that it's running the 32-bit version of Powershell.  And that version uses its own execution policy setting.  You can confirm that by explicitly opening "Windows Powershell (x86)" from the start menu and running Get-ExecutionPolicy.  

As for the fix, it's simple: just run Set-ExecutionPolicy RemoteSigned as admin, but do it in both the x86 and x64 versions of Powershell.

As for why Microsoft chose to make the different builds use different execution policies, I couldn't say.  It doesn't make much sense to me.  Although, to be honest, I'm not sure why I even need both versions on the same system.  Maybe to support interop between commandlets and unmanaged code?  Who knows?

As yet another note to myself (because I keep having to look this up), the equivalent of the UNIX which command in PowerShell is Get-Command.  That commandlet will let you pass in a name, with no extension, and return you the executable in your path that corresponds to that name.  This is really handy because it works with any executable, including native binaries and batch and PowerShell scripts.  So, for example:

PS pgeer> Get-Command rsync

CommandType     Name                 Version    Source
-----------     ----                 -------    ------
Application     rsync.cmd            0.0.0.0    C:\Users\pgeer\bin\rsync.cmd

I came across an interesting little program the other day. It's called Go. It's a Python script for quickly navigating directories via what are essentially command-line shortcuts. I discovered it while perusing the settings in my Komodo Edit preferences - it was mentioned in the settings for the Fast Open extension, which I believe is included by default as of Komodo 5.1.

The beautiful thing about Go is how simple it is. You run a simple command to set an alias and from there on out, you can just type go alias and it will change you to that directory. You can also add paths after the alias, such as go alias/dir1/dir2 to switch to a subdirectory of the alias. Great for switching between deep hierarchies, like how Windows programs like to bury things three levels under your "Documents" directory.

However, as I played around with Go, I did come across a few annoyances. The biggest one, or course, was that it didn't work under Powershell. The go.bat wrapper script would run...and do nothing. The current directory stayed the same. Turns out this was because go uses a driver-based system for changing directory which is based on your current shell. The Windows driver was using batch file syntax and running cmd.exe. Powershell does this in a new process, so naturally the current directory wasn't changing.

So, in the spirit of open-source, I decided to fix that problem. And while I was at it, I fixed a couple of other things and implemented some of the feature requests posted on the Google Code issue tracker. Here's the quick list:

• Added support for Powershell.
• Added built-in shortcut "-" pointing to the OLDPWD environment variable. On Windows, Go handles setting this variable (on UNIX, "cd" takes care of that).
• When invoked without any argument, change to home directory.
• Resolve unique prefixes of shortcut, e.g "pr" resolves to "projects" if there's no other shortcut starting with "pr".
• Made -o option work without having the win32api bindings installed.

Below are a patch file for the official Go 1.2.1 release (apply with patch -p2 < go-posh.patch) as well as a fully patched setup archive for those who just want to get up and running. Note that, to enable Powershell support, prior to running the Go setup, you'll need to set the SHELL environment variable so that Go knows to use Powershell. You can do this by adding $env:SHELL = "powershell" to your Powershell profile script.

Patch file: go-posh-1.2.1.patch
Full Go archive: go-posh-1.2.1.zip

Edit: Fixed support for "-" so that it actually works like it does in UNIX.

Edit again: I've added a few more small features and also created a project for this patch in my bug tracker.

Note to self: just because Powershell defines aliases that mimic many of the standard UNIX commands does not mean they function the same way.

Last night, I was trying to migrate my company's Subversion repository to Mercurial - not for production use (yet), just as an experiment. After eventually getting the latest Mercurial installed on the Ubuntu 8.04 VM that hosts our Subversion repository, I tried running hg convert -s svn /path/to/svn/repo /path/to/hg/repo. As expected, the conversion process took some time, but chugged along nicely...for a while. Eventually, it hit an error and came back with:
svn: In file '/build/buildd/subversion-1.6.9dfsg/subversion/libsvn_ra/ra_loader.c' line 595: assertion failed (*path != '/')

I Googled around a bit, but still have no idea what that error message means or how to fix it. My best guess is that something is borked in our repo - not broken enough to break SVN, but maybeSo I tried a different tack - take the repository dump I had, import it into a fresh repository, and try again. That didn't go so well....

Since the SVN VM has a very small drive, I decided to load the dumpfile on my local Windows box. As you may know, svnadmin load reads streams, so you have to either pipe the dump file in or redirect standard input. Well, my first instinct was to do something like this:
svnadmin load newrepos < dumpfile.repo
One problem with that: the "<" character that you normally use for redirecting STDIN is reserved in Powershell. Drat! So I figured I'd just use a pipe instead:
cat dumpfile.repo | svnadmin load newrepos
So I ran that and waited. And shortly after I started waiting, I noticed my system slowing down. And then things started grinding to a halt - it was just barely responding. When I finally managed to get resmon up, I noticed that Powershell was eating nearly all of my system's RAM! And the command still hadn't produced a single line of output!

I'm not sure exactly what Powershell was doing, but it must have something to do with the Get-Content commandlet (for which "cat" is an alias) not liking the 1.4 GB dump file. Why it would use up more than twice the size of the file in memory, I'm not sure.

Anyway, I just switched to cmd.exe and did the input redirection method, which didn't eat huge amounts of memory. However, it didn't work either. The import died shortly after starting with an error about a bad transaction. Looks like the gods of revision control are not smiling on me today.

Yesterday I tried creating a playlist in MediaMonkey and synchronizing it with my Sansa e280 running RockBox. I never really bothered with trying to sync playlists to my MP3 player before, because syncing, well, anything that's not an iPod using third-party software is a little iffy. Needless to say, my fears were well founded.

While MediaMonkey's syncing feature is nice, I ran into several problems. First, and most obvious, MediaMonkey didn't like the layout of my media directory. You see, I have a main "Music" folder which contains a bunch of artist and album sub-folders, but also a bunch of miscellaneous MP3s at the top-level. MediaMonkey didn't see those MP3s for some reason. After the sync, they just weren't there - I had to copy them over manually.

My other problem was a combination of the playlists themselves, how RockBox on the Sansa works, and where I'm putting my music on the Sansa. A couple of weeks ago, I "upgraded" my Sansa by buying a 16GB microSDHC card for it to complement the 8GB of internal flash storage. This resulted in a change of organization - I now keep all my actual music on the microSD card and put podcasts and other spoken material on the internal storage. Turns out that this messes with MediaMonkey's assumptions about playlists.

There are several issues here. The first is that when MediaMonkey syncs to the microSD card, it sees it as a separate device and creates the playlists accordingly. The result is that the playlists use absolute paths within the microSD card. However, the real absolute path for files on the microSD card starts with the device path, i.e. <microSD1>. So, basically, the paths in the playlist are wrong for RockBox.

The second issue is that the playlists get synced onto the microSD card. This isn't really a problem, just now what I want. I want the files synced to the main RockBox playlist folder on the Sansa's internal storage.

The third issue was with those assorted MP3s in my root "Music" directory. Even though I set the sync path to <Path:2> in MediaMonkey's sync configuration, which should mirror the directory structure on my hard drive, when I ran the sync, the assorted files on my playlist got copied into a "Music" folder on the microSD card. I'm not sure why. But as long as that's the case, I decided to "solve" that problem by just moving the rest of the assorted files there too. They weren't being synced anyway, so why not.

I decided to solve the playlist path and location problems with a simple Powershell script. You can download a copy here. It simply reads your playlists, adjusts the paths, and moves them to the main device storage. The code is below:

$source = "MUSICSD"
$device = "Sansa e280"
$listDir = "Playlists"

function findDriveByLabel {
   $drives = [System.Io.DriveInfo]::GetDrives()
   foreach ($drv in $drives) {
      if ($args[0] -eq $drv.VolumeLabel) {
         return $drv.RootDirectory
      }
   }
}

$sd = findDriveByLabel $source
$dev = findDriveByLabel $device
$sdDir = [System.Io.Path]::Combine($sd, $listDir)
$devDir = [System.Io.Path]::Combine($dev, $listDir)

$lists = ls $sdDir

foreach ($lst in $lists) {
   $outFile = [System.Io.Path]::Combine($devDir, $lst.Name)
   $outFile
   Get-Content $lst.FullName | ForEach-Object {
      "/<microSD1>" + $_.Replace("\", "/")
   } | Out-File -Encoding utf8 -Width 1000 $outFile
   rm $lst.FullName
}

$remLists = ls $sdDir
if ($remLists -eq $Null) {
   rmdir $sdDir
}

So far, this part is working fine. I'm still not 100% happy with the sync experience, though. The missing files is the part that really bugs me. I'll have to do a little experimenting at some point and see if I can make it work. Only problem is that it takes a while to write 16GB of data to the device....

Here's a quick code post from my Powershell profile. I keep copying this around every time I set up a new system, so maybe it will be useful to others. And to me, since I can copy it off the web in the future rather than having to dig it up or recreate it.

Being an old-school UNIX guy, I get used to just typing ssh hostname to access my servers. So the Putty syntax of loading a profile annoys me. To that end, I wrote a few little functions to add to my Powershell $profile to address this. They just let me load a session by typing the familiar commands and don't require me to change syntax when using a profile versus connecting directly to a hostname.


# Suppress the assembly loading output
[System.Reflection.Assembly]::LoadWithPartialName("System.Web") | Out-Null

function _puttyHasProfile {
   $profile_name = $args[0].ToLower()
   $profiles = Get-ChildItem HKCU:\Software\SimonTatham\PuTTY\Sessions\
   foreach ($p in $profiles) {
      $name = [System.IO.Path]::GetFileName($p.Name)
      # Putty stores spaces and other special characters URL encoded.
      $name = [System.Web.HttpUtility]::UrlDecode($name).ToLower()
      if ($name -eq $profile_name) {
         return 1
      }
   }
   return 0
}

function ssh {
   $profile_name = $args[0]
   if (_puttyHasProfile($profile_name)) {
      putty -load $profile_name
   } else {
      putty $args
   }
}

function sftp {
   $profile_name = $args[0]
   if (_puttyHasProfile($profile_name)) {
      psftp -load $profile_name
   } else {
      psftp $args
   }
}


Well, I feel stupid. I just finally got syntax highlighting for PowerShell to work in Vim. I did a little Googling and found my solution in the PDF document linked form this article. Why you need an 8 page PDF to explain this is a completely different issue.

Turns out I was just missing some lines in my filetype.vim file. I already had the syntax, indent, and file type plugins, I just didn't think to add the code to activate them (for some reason I thought that stuff was loaded dynamically). However, page 6 of that PDF gave the answer:

" Powershell
au BufNewFile,BufRead *.ps1,*.psc1 setf ps1

I just added those lines to my filetype.vim, below the corresponding entries for Povray files, and voilà! Syntax highlighting now works.