LinLog

You know that weird jump in auto-incremented IDs we saw in our database at work the other day? Well, we discovered the cause: apparently we were "hacked".

I use the work "hacked" in the loosest sense of the word. The culprit contacted us the next day, making a number of demands and issuing veiled threats. I may get into the details in another post (it's a long story), but the point is that he appeared to have, at best, a script kiddie level understanding of security. He pointed out a number of flaws in the site, but the ID incrementing was the only one that could actually have caused us any real trouble - and he didn't even mention it! We assume that he actually did that by accident and didn't realize the potential implications if we ran out of ID numbers.

The flaw allowed the ID incrementing was actually quite simple. No SQL injection or cross-site scripting. It was a simple case of passing too much data into a function.

Here's how it works. Our site, which runs on PHP and MySQL, is built on a custom MVC framework. It uses a custom Active Record-style ORM, which is where the flaw lies. The ORM performs database updates and inserts by validating the object's fields against the database schema. Basically, it reads the schema from the database, compares that with the data to be updated, and constructs the SQL accordingly. So when you execute the save() method on an object, it will save the values for the fields that appear in the table schema, but ignore any other fields in the object. The ORM also has static update() and insert() methods that take an associative array, mapping the indexes to field names and performing this same validation. So if you have an array of data, only some of which maps to actual columns in the underlying table, you can just pass the whole thing and not have to go through and separate out the fields you need to save.

That last point is where we got in trouble. We have a method that adds items to our media table. It takes an array of data, does some sanitizing and validation, calls the insert() method to add it to the media table, and adds appropriate records to other tables. The problem was that, in the place where this was called most frequently, we were passing in $_POST as the data array. And while this method did sanitize the fields that we wanted to add to the database, it didn't check for extra fields that just happen to be valid fields in the media table. So, to make a long story short, if you were to put an "id" field in the POST and assign it an integer value, our ORM would happily add that field and value to the INSERT statement it sent to the database.

Of course, this was easily fixed. In fact, it wasn't even hard to find. I did a fair amount of work on our ORM at the beginning of the year, so once I made the connection between the "hacker" and the ID number jump, the source of the bug was immediately obvious. It's just one of those things that nobody ever thought to check until it became a problem.

So the moral of the story is: security is all about attention to detail. Following the "rules" is all well and good, but it's not enough. In our case, we were sanitizing data to protect against XSS attacks and using PDO prepared statements to protect against SQL injection attacks, but it wasn't enough. By forgetting to check for unexpected additional input, we left ourselves open to a completely different type of attack. Of course, it's a significantly less serious class of attack - maxing out our auto-incrementing IDs is recoverable, if annoying - but it's still an issue.

With any luck some good will come out of this. I think we've all learned to be a little more mindful of such issues. And perhaps this will act as a cue to management that maybe - just maybe - it would be better to do some actual testing and review of new code before it's released, rather than just pushing things into production and hoping they work.

We had one of those weird experiences at work today. The kind where something strange happens, and nobody knows how or why, and when you look into it, it doesn't even make sense.

Basically, the auto-numbered ID field on the table we use for media items (videos, pictures, etc.) jump up noticably this morning. And by "noticably" I mean it went from about 1 million to about 80 million. And when we looked at the timestamps, the jump happened in about 20 seconds. The IDs created before 8:38:12 AM were in teh 1 million range, and the ones createdafter 8:38:32 AM were in the 80 million range.

So the obvious question is: how did this happen? It doesn't look like it was caused by actually adding 79 million records to the table. There were only about 800,000 records total, and no indication in our moderation logs of any mass deletions. We didn't get any indication of increased server load either. In fact, the only reason we even noticed it is because the media IDs are in our URLs. I kind of doubt a bot could have created 79 million new media items in 20 seconds without at least generating a Nagios warning. In fact, I doubt our master database server could handle 79 million writes in 20 seconds.

So what does that leave? User error? Nobody with access to the database server was even working at 8:30 in the morning. Random MySQL screw up? Maybe, though that's a really wierd random error. Something else? Who knows...?

After sniffing around the server and tossing out ideas for 30 or 40 minutes, we ultimately gave up. It's a little disquieting that we don't know what happened, but we really can't justify spending all that much time on this. It's a very weird problem, but nothing is broken and we all have more important things to worry about at the moment.

Edit: Found the cause. Apparently we were hacked.

Well, that sucked.

I upgraded my Kubuntu box at work from 8.04 to 8.10 on Monday morning. It did not go well. Not only did the experience waste several hours of my time getting my system back to a state where I could actually do some work, it left me feeling bitter and fed-up.

Not that the upgrade failed or anything - on the contrary. The upgrade process itself was relatiely fast and painless. So, in contrast to some of my previous upgrade experiences - which have left systems completely inoperable - this wasn't that bad. It's just that, once the upgrade was done, nearly every customization I'd made to my desktop was broken.

Broken Stuff

As for the breakages, they were legion - at least it felt that way. The 2 most annoying were the scrolling on my Logitech Marble Mouse trackball and KHotKeys. It turns out the mouse scrolling was fixable by adding a line to my xorg.conf to disable some new half-working auto-configuration feature.

KHotKeys, on the other hand, was a lost cause. From what I've read, it just plain doesn't work right in KDE 4. So, since key bindings are an absolute must-have feature for me, I worked around it by installing xbindkeys. This works well enough, but it's a huge pain in the neck. Now, not only do I have to recreate all my key bindings, but I have to look up the DBUS commands for all those built-in KDE functions rather than just picking them from a list.

Another annoying one was that the upgrade somehow broke the init scripts for my MySQL server. I don't know how the heck that happened. I tried uninstalling it, wiping the broken init scripts, and reinstalling, but they weren't recreated, which seemed odd to me. I eventually ended up just doing a dpgk --extract on the MySQL package and manually copying the scripts into place.

On another weird note, KDE and/or X11 has been randomly killing the buttons on my mouse. I'll be working along fine and suddenly clicking a mouse button will no longer do anything. It still moves, and the keyboard still responds, but clicking does nothing. Restarting the X server resolves the problem, but that's cold comfort. It seems to happen randomly - except for when I try to run Virtual Box, in which case it happens every time the VM loses focus. Fortunately I'm more of a VMware person, so that's not a big deal, but it's still disquieting.

KDE4 In General

The other big pain-point is KDE 4. To be perfectly blunt, I don't like it. It has a few neat new features, but so far it doesn't seem worth the effort to upgrade.

The good parts that I've noticed so far seem to be small. For instance, Dolphin has a couple of nice enhancements. The one that sticks out is the graphical item-by-item highlighting. It allows you to click a little plus/minus icon to select/deselect an item, so that you no longer need to hold the control key to do arbitrary muliple selects. The media manager panel applet is nice too. It pops up a list of inserted storage devices and allows you to mount and eject them. I have to admit that I also really like the new "run" dialog. It does program searching much like Katapult, but makes it easier to run arbitrary commands and select commands with similar names. While it doesn't have some of the cool features supplied by Katapult's plugins, it's still quite good.

On the other hand, there are a lot of things I don't like (not counting the breakage). For one, I think the new version of Konsole is a huge step backward. I can't access the menus with keyboard shortcuts, the "new tab from bookmark" feature is MIA, the session close buttons are gone, and generally everything I had gotten used to is missing.

And then there's the new "kickoff" application menu. I'm getting slightly more used to it, but I still don't like it. It just feels a lot slower to access items using it. This is only made worse by the "back" button for browsing sub-menus, which is extremely hard to click when you're in a hurry (hint: Fitt's law doesn't apply on multi-monitor setups).

As for the "cool" new look of KDE 4...I'm not a fan. Maybe it's just because I don't have any of the fancy desktop effects turned on on my system (a side-effect of the crappy integrated video card that's part of my tri-monitor setup), but I just don't think it looks good. Yeah, the bare desktop itself is kind of nice looking, but the window theme is ugly as sin. It's one of those "brushed metal" sort of looks, which I find even more depressing than Windows 95 gray. It's too dark for my taste and far too monochromatic. I also find the active window highlighting to be way too subtle to be helpful. The icons also leave something to be desired. They look nice, but they don't look distinct - even after a week, it takes me a second to figure out what some of them are supposed to represent. It kind of defeats the entire point of icons.

As for the much touted Plasma, I'll grant them this - it is pretty. The panel and desktop plasmoids do pretty much all look nice. Not that it matters to me, though, because I never see my desktop - it's always covered with work. And while the various applets and widgets may look pretty, approximately 90% of them are completely useless. That's the problem with all desktop widgets for any platform. I find that if a desktop widget actually provides enough valuable functionality to justify leaving a space open for it on the desktop, it's job is probably better served by a full-fledged applicaiton. And if it's not important enough to make constantly visible, then why bother to put it on the desktop at all? I'm never going to see it, so I might as well save the RAM and CPU cycles.

Conclusion

Overall, I guess Kubuntu 8.10 and KDE 4 aren't bad systems. But to be honest, I'm not impressed. For the first time, I think that the new Kubuntu is not an improvement. In fact, I have no plans to upgrade the 3 Kubuntu boxes I have at home any time in the forseeable future.

The thing that's most disappointing to me about the upgrade to KDE 4 is that it totally defeats my purpose in switching to KDE in the first place. When I switched from the ROX desktop to KDE back in 2005, my main reason was that I was tired of having to build my own desktop. ROX was great, but it was a small community and just didn't have the range of applications and degree of integration that KDE had. You see, I always had this crazy idea that I could just use all KDE applications and everything would be tightly integrated and work well together and there would be harmony throughout my desktop.

However, more and more I've been finding that that just isn't true. Part of the problem is that lots of KDE applications just aren't that good - many of them are missing functionality and have stability problems. I find myself using fewer KDE applications all the time. I dropped Quanta+ for Komodo Edit; I tried to like Konqueror, but it just doesn't hold a candle to Firefox or Opera; I recently tried to become a KPilot user, but was almost immediately forced to switch to JPilot; I finally got fed-up with Akregator and am just using the RSS reader in Opera's M2 mail client; I still use KMail, but not because I particularly like it - I just dislike it less than M2 or Thunderbird. In fact, I think the only KDE app I would actually miss is Amarok. (K3B is very good too, but I don't burn enough disks to care what program I use, just so long as it works.)

So now I'm starting to wonder: What's the point of using KDE? If I'm not using many KDE applications, and most of the ones I am using could be easily swapped out, it seems like there's nothing keeping me with it. Maybe I should just switch to GNOME. Or maybe Windows. I have been wanting to get more into .NET development, and my tollerance for things not working has been falling over the years, so Windows is sounding better all the time.

I think next weeek I'm going to have to reinstall my work machine. Maybe a fresh install and a fresh KDE profile will give me a better experience. Or perhaps I'll ditch Kubuntu and go for straight Ubuntu with GNOME. Or perhaps I could take another look at ROX. I don't know. And while I'm at it, I think I might reinstall that old Windows partition I still have on that machine. Maybe some time playing with a nice clean install of XP, or even Vista, if we have a spare copy, will give me a little perspective.

Here's an odd one. So I was trying to run a simple query for a data import I'm working on. I've got about 11,000 rows in the media table with a site_id of 2 and I wanted to see which media types were represented. So I ran this:

SELECT DISTINCT media_type FROM media WHERE site_id = 2;

To my surprise, this returned...nothing! No rows at all. So I took off the DISTINCT, just to check, and got back all 11,000 rows. What the heck?

Something was obviously wrong. This should never happen - if there are rows with the correct site_id, that query should always return something. So what now? Might as well look at the query plan:

mysql> explain extended select distinct media_type from media where site_id = 2 \G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: media
type: range
possible_keys: idx_media_site_media_type,idx_frontpage
key: idx_typesiteid
key_len: 2
ref: NULL
rows: 15
Extra: Using where; Using index for group-by
1 row in set, 1 warning (0.00 sec)

mysql> show warnings \G
*************************** 1. row ***************************
Level: Note
Code: 1003
Message: select distinct `cms`.`media`.`media_type` AS `media_type` from `cms`.`media` where (`cms`.`media`.`site_id` = 2)
1 row in set (0.00 sec)

Nothing out of the ordinary there. It's using the idx_typesiteid index, which is correct. Maybe that index is broken. Let's try another one:

select distinct media_type from media use index (idx_media_site_media_type) where site_id = 2;

Yup, it worked that time - I got 5 rows back, which is about what I was expecting. I get the same results if I use
IGNORE INDEX (idx_typesiteid) as the index hint. So idx_typesiteid must just be broken. I should probably fix that, but it was getting into peak hours for our site and I didn't want to run an OPTIMIZE TABLE or anything that would lock the media table.

But wait, there's one other weird thing. Just for the heck of it, I tried running the query with a few different index hints. And guess what: the query still works if I use FORCE INDEX (idx_typesiteid). Does that make sense to anybody? So if I ignore that index it works, and if I force the same index, it works, but if I give no hint, it claims to use that index and the query returns no data. I got nothin' on that one.

Well, it's official: the people who develop PHP are morons. Or, rather, the people responsible for adding namespaces to PHP 5.3 are.

Why do I say this? Because I just read an annoucnement on Slashdot that they've decided on the operator to use for separating namespace in PHP 5.3: the backslash (\).

Seriously? The friggin' backslash? What kind of choice is that? Last I knew they'd pretty much decided to go with the double colon (::), like C++, which at least makes sense. But the backslash?

What's worse, just look at the RFC listing the operators they were considering. In addition to the backslash, they had the double star (**), double caret (^^), double percent (%%), a shell prompt (:>), a smiley face (:)), and a triple colon (:::). For God's sake, it looks like they picked this list out of a hat. They might as well have just used the string NAMESPACESEPARATOR. It's no less absurd than any of those.

Now, let's be realistic for a minute. In terms of syntax, PHP is a highly derivative language. It's an amalgamation of Perl, C++, and Java, with a dash of a few other things thrown in.

Given that heritage, there's really only a handful of choices for namespace separators that even make sense. The first, and most natural, is the double colon (::). This is what C++ uses and it's already used for static methods and class members in PHP. So the semantics of this can naturally be extended to the generic "scope resolution operator." Keeps things clean and simple.

The second choice is the dot (.), which is what's used in Java, C#, Python, and many others. This is a bit unnatural in PHP, as dot is the string concatenation operator, but it at least offers consistency with other related languages.

Third is...actually, that's it. There are only 2 valid choices of namespace separator. And the PHP namespace team didn't pick either one. Nice work guys.

The Slashdot article also linked to an interesting consequence of the choice of backslash: it has the potential to mess up referencing classes in strings. So if your class starts with, say, the letter "t" or "n", you're going to have to be very careful about using namespaces in conjunction with functions that accept a class name as a string. Just what we needed. As if PHP isn't messed up enough, now the behaviour of a function is going to depend on the names of your classes and the type of quotes you use.

I guess I'm going to have to bone up on my C#, because PHP seems to be going even farther off the deep end that before. It was always a thrown-together language, but this is just silly. The backslash is just a stupid choice for this operator and there's just no excuse for it.