Bitwise OR in Oracle

This is another one of those "for my own reference" posts, because I have virtually no experience with Oracle and I'll likely forget about this in a couple of months. 

Here's a handy link on creating a bitwise OR function in Oracle.  There's a built-in BITAND function that does a bitwise AND, but no corresponding OR.  Fortunately, it turns out that you can do an OR in terms of AND using the following:

function bitor(p_dec1 number, p_dec2 number) return number is
begin
  return p_dec1-bitand(p_dec1,p_dec2)+p_dec2;
end;

In my case, I actually just needed the formula, not the actual function, as it was for a one-off data migration query.  However, I did have an extra wrinkle in that I needed to do the bitwise OR on a CLOB column.  But that turned out to not be a big deal either, as simply running the column through the TO_NUMBER() function was all I needed.  So it actually worked out pretty well.

Fixing fonts in XUL apps

Today I finally got fed up enough to fix the problem with fonts in XUL apps on my work desktop.

The problem and its solution are described in this post.  Basically, in XUL-based apps like Firefox and Komodo, the text would randomly distort.  It would usually happen when scrolling through a page or something like that that causes window redraws.  If I scrolled some more, or selected some text, it would go away.  And it only seems to happen on this particular system - my laptop doesn't seem affected.

Well, turns out it's a common issue.  In my case, I was able to fix it by going into the "about:config" page and changing the gfx.direct2d.disabled setting to true and restarting the app.  Turns out that while this page is well-known in Firefox, it's also available in Komodo.  You just need use the "preview buffer in browser" tool from the toolbar.  Just choose to preview the item in a Komodo tab and then enter "about:config" as the URL.  That will take you to the exact same config page you see in Firefox.

Finally upgraded to Komodo IDE

Well, I finally did it - I shelled out for a license of Komodo IDE.  I've been using Komodo Edit on and off since about 2007, but had never needed the extra features of the IDE to justify the $300 price tag.  But last week ActiveState had a one-week $100 off sale, so I decided to try it out again and ended up deciding to make the purchase.

My Komodo IDE setup

Part of what motivated me to give Komodo IDE another shot after getting along fine with Komodo Edit for so many years (aside from the price, of course) is my new-ish job.  It turns out that Eclipse is fairly popular among the Pictometry engineers, dispite the fact that they're a pretty smart bunch (I'm not a fan of Eclipse).  Thus many of them take integrated debugging for granted, so on a few occasions I've been told to debug an issue by "just stepping through the code."  And to be fair, there have been a few bugs I've worked on where integrated debugging genuinely would have been useful.  Perhaps this is due to the fact that our internal back-end framework is significantly more Java-like than most code-bases I've worked on (but that's another post entirely).

Of course, that's not all there is to Komodo IDE.  There are plenty of other nifty features, such as the regex testing tool, structure browser, integrated source control, and a number of other things.  So when you put them all together, it's a pretty good value, especially at the sale price of $195.  And with the cool new stuff in Komodo 8.5, I have to say I've pretty much lost interest in trying out new editors.  The ActiveState team did a really nice job on the latest version.

Flask on ICDSoft

A year or two ago, I decided to update my skill set a little and brush up on my Python. In particular, I wanted to do a little web development in Python, which I hadn't done in nearly five years. Since I wanted to start with something fairly basic, I decided to go with the Flask micro-framework. I ended up using that to build Lnto, which I've mentioned before and I swear will get its own page someday.

One big problem with this project was was that my hosting is kind of bare-bones. Don't get me wrong - the service is fine. I use ICDSoft, whom I've been with for several years. I have no complaints about the service I've received - in fact I'm quite happy with it. However, it's mostly focused on PHP and MySQL and there's no shell access. But on the other hand, I'm paying less than $5 a month with only a one-year commitment, so I don't have much to complain about.

Anyway, the problem with running Flask apps, or pretty much anything other than PHP, is that they have no documentation for that whatsoever. There's a FAQ, but it says absolutely nothing about Python other than that they "support" it. As far as I can tell, that just means that they have a Python interpreter installed and Apache CGI handler is configured to run *.py files. There's certainly no mention of running things using WSGI, which seems to be the recommended method for running most Python frameworks.

Another problem is actually installing the Flask libraries. The documentation for, well, pretty much every Python framework or app out there tells you the best way to install it is with pip or easy_install. But, of course, you need shell access to run those, assuming they're even installed on the server. (And I did check - they're not installed.)

Fortunately, getting around these problems was relatively easy using using virtualenv, which I'd nearly forgotten existe). This creates a virtual Python environment which is isolated from the rest of the system. A side-benefit of this is that virtualenv creates a copy of pip in your virtual environment.

You can use virtualenv directly from the source distribution. This was required in my scenario, since I lack any shell access, let alone root privileges. I simply extracted the virtualenv source archive, uploaded it to my host, and ran the following command (I used PHPsh, a web-based shell emulator, but copying them into a PHP script would have worked just as well):
python /path/to/virtualenv-1.11.4/virtualenv.py /path/to/myapp/venv

This create a virutal environment in the /path/to/venv directory. You can then install packages into that environment using the "activate" script to configure the shell, like this:
. /path/to/venv/bin/activate && pip install Flask

That was easy. I now have a Python environment with Flask installed. All I need do at this point is configure my application code to use it. That's accomplished with a few lines to initialized the virtualen environment and start up Flask as a CGI app:
activate_this = '/path/to/venv/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
from myapp import app
from wsgiref.handlers import CGIHandler
CGIHandler().run(app)

I just re-ran that entire process using the latest version of virtualenv and it's actually quite painless.

And as a side-note, the reason I did that was because I noticed the other day that Lnto had suddenly stopped working - the server was just returning 500 errors. Which was odd because I hadn't changed anything with the app or the database in weeks. However, the answer was found on the virtualenv PyPi page:

Python bugfix releases 2.6.8, 2.7.3, 3.1.5 and 3.2.3 include a change that will cause "import random" to fail with "cannot import name urandom" on any virtualenv created on a Unix host with an earlier release of Python 2.6/2.7/3.1/3.2, if the underlying system Python is upgraded.

When I created that virtualenv environment, the server was running Python 2.6. But when I checked yesterday, the Python version was 2.7. So apparently ICDSoft upgraded their servers at some point. No big deal - just recreated the environment and I was good to go!

Going WYSIWYG

I must be getting old or something.  I finally went and did it - I implemented a WYSIWYG post editor for LnBlog (that's the software that runs the blog you're reading right now).

I've been holding out on doing that for years.  Well, for the most part.  At one point I did implement two different WYSIWYG plugins, but I never actually used them myself.  They were just sort of there for anybody else who might be interested in running LnBlog.  I, on the other hand, maintained my markup purity by writing posts in a plain textarea using either my own bastardized version of BBCode or good, old-fashioned HTML.  That way I could be sure that the markup in my blog was valid and semantically correct and all was well in the world.

The LnBlog post editor using the new TinyMCE plugin.

If that sounds a little naive, I should probably mention that I came to that conclusion some time in 2005.  I had only been doing web development for a few months and only on a handful of one-man projects.  So I really didn't know what I was talking about.

Now it's 2014.  I've been doing end-to-end LAMP development as my full-time, "I get paid for this shit" job for almost seven years.  I've worked for a couple of very old and very large UGC sites.  I now have a totally different appreciation for just how difficult it is to maintain good markup and how high it generally does and should rank on the priority scale.

In other words, I just don't care anymore.

Don't get me wrong - I certainly try not to write bad markup when I can avoid it.  I still wince at carelessly unterminated tags, or multiple uses of the same ID attribute on the same page.  But if the markup is generally clean, that's good enough for me these days.  I don't get all verklempt if it doesn't validate and I'm not especially concerned if it isn't strictly semantic.

I mean, let's face it - writing good markup is hard enough when you're just building a static page.  But if you're talking about user-generated content, forget it.  Trying to enforce correct markup while giving the user sufficient flexibility and keeping the interface user-friendly is just more trouble than it's worth.  You inevitably end up just recreating HTML, but with an attempt at idiot-proofing that end up limiting the user's flexibility in an unacceptable way.  And since all the user really cares about is what a post looks like in the browser, you end up either needing an option to fall back to raw HTML for those edge-cases your idiot-proof system can't handle, which completely defeats the point of building it in the first place, or just having to tell the user, " Sorry, I can't let you do that."

"But Pete," you might argue, "you're a web developer.  You know how to write valid, semantic HTML.  So that argument doesn't really apply here."  And you'd be right.  Except there's one other issue - writing HTML is a pain in the butt when you're trying to write English.  That is, when I'm writing a blog post, I want to be concentrating on the content or the post, not the markup.  In fact, I don't really want to think about the markup at all if I can help it.  It's just a distraction from the real task at hand.

Hence the idea to add a WYSIWYG editor.  My bastardized BBCode implementation was a bit of a pain, I didn't want to fix it (because all BBCode implementations are a pain to use), and I didn't want to write straight HTML.  So my solution was simply to resurrect my old TinyMCE plugin and update it for the latest version.  Turned out to be pretty easy, too.  TinyMCE even has a public CDN now, so I didn't even have to host my own copy.

So there you have it - another blow stricken against tech purity.  And you know what?  I'm happier for it.  I've found that "purity" in software is tends not to be a helpful concept.  As often as not, it seems to be a cause or excuse for not actually accomplishing anything.  These days I tend to lean toward the side of "actually getting shit done."

The local Red Baron

Author's note: Here's another random "from the archives" post.  This is from May 6, 2007.  It's more of a "personal reminiscience" type post, more in keeping with the "personal journal" nature of a blog than my usual subject matter.  But hey, it's my website, so I can post whatever the heck I want.

I live in Corning. It's a small city in central New York, about 30 minutes from the Pennsylvania border. As locations for software developers and other technologists go, it's not quite in the middle of nowhere, but it's close.

However, there are still advantages to be had. For example, there are the air shows. It just so happens that the Elmira/Corning regional airport is also the location of the National Warplane Museum (note: that has changed - now it's the Wings of Eagles), and the airport just happens to be across the street from the local mall. So when we went shopping on Sunday, we got a nice view of the day's airshow while we were driving around. The part we saw was a bright red biplane doing loops, barrel rolls, flying in low over the highway exit. It was actually very cool. I wish I'd had a camera....

Commentary: I remember watching this plane as my wife and I walk though the parking lot of the Tops supermarket in the plaza across the street from the mall.  I don't remember exactly why we were there (it wasn't our normal market) or what we'd done before, but I do remember that it was a sunny afternoon and I kept looking up at the red biplane doing loops and barrel rolls overhead.  It was a symbol of freedom - rolling unfettered through the blue sky, no cares about what was going on below.  I was a little jealous - I'd been feeling trapped in a job that I increasingly disliked and I wanted to be like that plane.

It's funny how images like that will stick with you sometimes.  It's been seven years, but I still remember that plane, even though I don't remember any of the context.  In fact, the only other thing I remember doing that day was working in my garden and thinking about that plane.  Perhaps it's not a coincidence that my garden was one of my main sources of escapism....

Unhelp desk

Author's Note: This is another entry in my "from the archives" series.  I've decided it's time to clear some of those out of my queue.  This was an unfinished stub entry I last edited on April 20, 2007, when I was still working in county government.  I actually still have some recollection of the incident that inspired this entry, nearly seven years later.  There wasn't much to the original entry itself (as I said, it was kind of a stub), so this post will feature some commentary on my own comments.  Yeah, I could try to actually finish the post, but I only vaguely recall what my original point was and I'd rather make a different one now.

Question: what is the purpose of having a "help desk" when all they do is forward support requests to the programmers and analysts? Isn't that the opposite of help?

I admit that's not entirely fair. Our help desk does all sorts of things and often solves small problems on their own. So it's not like they're just a switchboard.

It just annoys me when they pass on a request without even trying to get any more details. For instance, when they get e-mail requests and just copy and paste the body of the e-mail into our help desk software. Then they print out the ticket on a sheet of blue paper and lay it on the desk of the programmer or analyst who last worked with the department or system in question.  There's no indication that they even asked the user any questions.  It seems like I'd be just as well off is the user just e-mailed me directly.  At least we wouldn't be killing trees that way.

Commentary: I don't even really remember what the partiuclar issue that prompted this was anymore.  I just remember coming back to my desk one day, seeing a blue print out of a help desk ticket, and realizing from what it contained that the "help" desk had done no analysis or trouble-shooting on it whatsoever.

This was one of those moments that was a combination of anger and shaking my head in resignation.  I had a lot of those in 2007.  I guess it was just part of the realization that government IT just wasn't for me.  In fact, it was that August that I took the job with eBaum's World  and moved into full-time development, which is really what I wanted to be doing all along.  So really, it all worked out in the end.

Happy birthday, Grandma

Today is a special day - it's my grandma's 100th birthday. My parternal grandmother, Althea Geer, was born a century ago today. I'm not much on the sappy personal posts, but I figured this deserved mention. Birthdays happen every year, but it's not often you get to celebrate a milestone like that in your family.

My grandmother visiting with her my son.

My grandmother still lives by herself in a tiny village in upstate New York, in the same house where she raised my father. She's lived there most of her life - I remember playing in the back yard as a child. We'll be going over there to visit her for an open-house celebration on Saturday. I'm sure we'll have a pretty good crowd coming through. And then on Sunday the plan is to have a family celebration at my parents house.

I'm hoping to get some nice pictures of the occasion. In particular, some of grandma with my son. He's her only great-grandchild and she adores him. Unfortunately, it's a pretty long trip, so she doesn't get to see him very often. So it'll be nice for him to be there for such a special occasion. Of course, he's still too young to remember it when he's grown up, but I hope to get some picture to commemorate the occasion.

So even though you're never going to see this entry (I don't think she's ever even used a computer), happy birthday, Grandma!

Edit: A correspondent from the Walton Reporter had a conversation with my grandmother for the occasion and wrote a very nice article about it.

Getting a password manager

After shamelessly reusing passwords for far too long, I finally decided to get myself a decent password manager. After a few false starts, I ended up going with KeePass. In retrospect, I probably should have started with that, but my thought process didn't work out that way.

Originally, my thought was that I wanted to use a web-based password manager. I figured that would work best as I'd be able to access it from any device. But I didn't want to use a third-party service, as I wasn't sure how much I wanted to trust them. So I was looking for something self-hosted.

PPMA

I started off with PPMA, a little Yii-based application. It had the virtue of being pretty easy to use and install. There were a few down sides, though. The main one was that it wasn't especially mobile-friendly, so there were parts of the app that actually didn't work on my phone, which defeats the whole "works on any device" plan. Also, it really only supported a single user, so it's not something I could easily set my wife up on as well. (To be fair, the multi-user support was sort of there, but it was only half-implemented. I was able to get it basically working on my own, but still.)

More importantly, I wasn't entirely confident in the overall security of PPMA. For starters, the only data it actually encrypted was the password. Granted, that's the most important piece, that's sort of a minimalist approach to account security. And even worse, I wasn't 100% convinced that that was secure - it's not clear to me that it doesn't store a password or key in session data that could be snooped on a shared server. Of course, I haven't done an extensive analysis, so I don't know if it has any problems, but the possibility was enough to make me wary and I didn't really want to do an extensive audit of the code (there was no documentation to speak of, and certainly nothing on the crypto scheme).

The next package I tried was Clipperz. This is actually a service, but their code is open-source, so you could conceivably self-host it. I had a bit more confidence in this one because they actually had some documentation with a decent discussion of how their security worked.

Clipperz - beta UI

The only problem I had with Clipperz was that I couldn't actually get it to work. Their build script had some weird dependencies and was a pain to deal with (it looked like it was trying to check their source control repository for changes before running, for some reason). And once I got it installed, it just flat-out didn't work. I was able to create a new account, but after that every request just returned an error out. And to make things worse, it turns out their PHP backend is ancient and not recommended - it's still using the old-school MySQL database extension. The only other option was the AppEngine Python backend, which wasn't gonna work on my hosting provider. So that was a bust.

It was at that point that I started to think using a web-based solution might not be the best idea. Part of this is simply the nature of the web - you're working over a stateless protocol and probably using an RDBMS for persistence. So if you want to encrypt all the user's data and avoid storing their password, then you're already fighting with the medium. A desktop app doesn't have that problem, though - you can encrypt the entire data file and just hold the data in memory when you decrypt it.

It also occurred to me that accessing my passwords from any computer might not be as valuable as I'd originally thought. For one thing, I probably can't trust other people's computers. God alone knows what kind of malware or keyloggers might be installed on a random PC I would use to access my passwords. Besides, there's no need to trust a random system when I always have a trusted one with me - namely, my phone.

Great! So all I really need is a password manager than runs on Android.

Well...no, that won't do it. I don't really want to have to look up passwords on my phone and manually type them into a window on my desktop. So I need something that produces password databases that I can use on both Android and Windows.

Luckily, KeePass 2 fits the bill. It has a good feature set, seems to have a good reputation, and the documentation had enough info on how it works to inspire some confidence. The official application is only Windows-based, but there are a number of unofficial ports, including several to iOS and Android. It's even supported by the Ninite installer, so I can easily work it into my standard installation.

KeePass2

For me, the key feature that made KeePass viable was that it supports synchronization with a URL. There are extensions that add support for SSH and cloud services, if you're into that sort of thing, but synchronizing via standard FTP or WebDAV is built right in. KeePass also supports triggers that allow you to automatically synchronize your local database with the remote URL on certain events, e.g. opening or saving the database.

For the mobile side, I decided to go with Keepass2Android. There are several options out there, but I chose this one because it supports reading and writing the KeePass 2.x database format (which not all of them do) and can directly read and write files to FTP and WebDAV. It's also available as an APK download from the developer's site, as opposed to being available exclusively through the Google Play store, which means I can easily install it on my Kindle Fire.

Keepass2Android also has a handy little feature called "QuickUnlock", which allows you one chance to unlock your database by typing just the last few characters of your passphrase. If you get it wrong the database is locked and you need to enter the full passphrase. This addresses one of my main complaints about smart phones - the virtual keyboards work to actively discourage good passwords because they're so damned hard to type. I chose a long passphrase which takes several seconds to type on a full keyboard - on a virtual keyboard, it's absolutely excruciating. This way, I don't have to massively compromise security for usability.

So, in the end, my setup ended up being fairly straight-forward.

  1. I install KeePass on all my computers.
  2. I copy my KeePass database to the WebDAV server I have set up on my web hosting.
  3. I set up all my computers with a trigger to sync with the remote URL.
  4. I install Keepass2Android on my phone and tablet.
  5. I configure them to open the database directly from the URL. Keepass2Android caches remote databases, so this is effectively the same as the desktop sync setup.
  6. Profit! I now get my password database synchronized among all my computers and devices.

I've been using this setup for close to a month now, and it works pretty darn well. Good encryption, good usability, plenty of backups, and I didn't even have to involve a third-party service.

New toy - Sandisk Wireless Flash Drive

I got a new toy the other week - a Sandisk Wireless Flash Drive. This was not normally something I would have bought, but it showed up as a Kindle-exclusive special offer on my new Kindle Fire HDX (post on that coming later) - the 32GB model was only $20, which is about 66% off the normal retail price. I didn't really know anything about the device or how it worked, but for only $20, I figured, why not?

Sandisk Wireless Flash Drive(Photo by Pierre Lecourt, licensed under Creative Commons BY-NC-SA 2.0.)

It turns out that this is an interesting little "flash drive". First, to be clear, it's not really what I would normally consider a "flash drive". For starters, it doesn't actually have any built-in storage - it has a MicroSD slot with a 32GB card in it. So the "flash drive" itself is more like a MicroSD card reader that's also a networking appliance.

The networking portion is actually kind of cool. In it's most basic configuration, the Wireless Flash Drive acts as a WiFi access point. You associate to it and it supplies you an IP and can serve out content. However, it also allows you to configure it to associate to an internet-connected access point. So you can tell it the SSID and WPA credentials of your network and anything on the LAN will be able to access it.

The device is built to work with a mobile app (available for both iOS and Android) which allows you to not only access the data on it, but also configure it. However, while the app is actually not bad, it turns out you don't need it. The device provides a web interface that lets you configure it as well as browse content. And on further research, it turns out that the Wireless Flash Drive serves it's content out over WebDAV! So forget the mobile app - you can actually access it from any PC with a WebDAV client.

Though I haven't had opportunity to use it heavily yet, I have to say the Wireless Flash Drive is actually a pretty cool little deice. I probably wouldn't pay $60 for it, simply because I don't need it that much. But if you happen to have a distinct use for such a device, it's pretty cool and more open to tinkering than I'd expected.