Three Sundays ago, my primary Mac OS X hard drive failed. Those of you who follow me on Twitter got somewhat of a play-by-play as I discovered the depth of my drive failure I got home to the Spinning Pinwheel of Death (SPOD), and discovered quickly that my computer would not wake from screensaver or boot. However, I didn’t panic. Why? Because I have what I believe to be a relatively robust backup system for home use.

I can’t stress enough how important regular backups are. Data loss is one of my personal nightmares (well, that, and Lego or Andrle loss), since most of my life (professional and personal) is on the computer. Among other things, I’d lose every picture I’ve ever taken since freshman year of college, every homework assignment I’ve written on the computer since late 6th grade (when we got our first Mac), not to mention substantial configuration work and those precious saved games.

I sit atop what I call the Backup Tripod: regular clones to an external disk stored off-site, hourly incremental backups to a local disk or local network storage, and as-needed on-save synchronization to cloud storage. I’m sure there are many other articles out there that recommend a particular strategy, but this is my solution for Macs. I even convinced my parental units to use a similar setup. I’ll go into detail on what solutions I use and why (as well as recovery strategy) for each below the cut.

I can’t emphasize enough how important data backup is for the typical modern power user.

Offsite Clones

Why do an offsite clone? First, this is primarily to solve the “your apartment burned down” problem. You want to be able to recover as much of your data as possible, all at once, but you might be okay in that situation with a little bit of recent data loss. The latter can be mitigated of course by using the other two legs of the tripod in combination

I’ve been using the excellent Carbon Copy Cloner for several years to make cloned backups of my primary hard drive. It’s a Mac-specific solution that has seen regular improvements over that time, including a migration to launchd as part of updating for Mac OS 10.5 that enables some features I use, such as “initiate backup on mount”, which triggers my standard clone job as soon as my external drive is plugged in. Cloning the actual disk is of course only half of the solution: you need to be responsible for storing your backup drive in a safe location, and bringing it home for regular incremental updates, to minimize the annoyance during a recovery.

I store my home hard drive in a locked drawer in my office at work; my parents keep an old iPod with important files in their safe deposit box. I should also admit at this point that, for this particular failure, my clone backup was almost 4 months old, therefore making recovery a fair bit more annoying than it needed to be had I been keeping my clone more up-to-date.

Incremental Local Backups

Why do incremental backups? This is primarily to smooth out the frequency of your backups: now instead of a backup only as recent as the last time you remembered to bring your clone drive home, you can have one as recent as a few minutes ago. The other major advantage is that, when configured properly, they are completely automated. For the most part, you shouldn’t even care that they’re set up; they’ll just run silently in the background, and you’ll only notice their existence when you need them to perform a recovery. Note that in this context, by “local” I mean “in the same building”, not necessarily on the same machine, since some network solutions are viable here.

Being a Mac junkie, I use Apple’s Time Machine. Given my experience with this recent near-loss of data, I can safely say that for me, this is now Apple’s most important product ever (no matter how magical and revolutionary the iPad may be, it hasn’t saved my bacon… yet). Since my primary computer is a Mac Pro desktop, I have Time Machine configured to back up to a second internal drive. My parental units, since they both have laptops, use a Time Capsule as their backup system (and primary wireless router).

I can’t from personal experience recommend a particular incremental backup solution for Windows or Linux, but TSOR suggests TimeVault for Ubuntu. At work they use a network version of IronMountain Connected Backup, which is at least the fourth Windows backup solution IT has tried since I’ve worked there. So far it seems to have staying power, and I assume it’s working, but a running backup causes a massive performance hit on my Windows XP development machine, making it essentially unusable due to I/O when the backup is running. (I should add that in my experience, Time Machine does not cause a performance hit, even when playing a processor-intensive 3D game like World of Warcraft. I believe this has to do with Apple’s implementation using FSEvents, although Siracusa’s review of Leopard explains the underpinnings of Time Machine better than I.)

One of the major caveats of an incremental backup, as I learned the hard way during this particular recovery, is that they are bad at dealing with single large files that change often. In other words, databases of various sorts, as well as writeable disk images. Examples include the iTunes Music Library file, and the Mac OS 9 disk image used by the SheepShaver PowerPC emulator. There are two problems that these files can cause: the first is that they may cause your backup to fail due to file locking (especially if the program using the particular file is currently running); the second is that, since most incremental backup systems operate at filesystem granularity, they have to back up the entire file even if only a small portion of its data has changed. The latter leads to your incremental backup disk getting filled with many version changes

This is where careful management of your incremental backup’s ignore list comes in: I had both of these files ignored in Time Machine, so I had to restore them from my clone drive. More on that in the Recovery section below.

Cloud Storage

Why use cloud storage? Or, in this case, a better question might be “Why not use a cloud backup service?”.

I’ll answer the second first: I don’t trust them. Maybe this is some bizarre geek control freak paranoia, but I just don’t believe that their service can actually deliver what they promise while simultaneously being sufficiently secure. There are at least three parts to this: the first is that I don’t believe that the service will necessarily be available when I need it (whether during backup or recovery), the second is that I don’t believe they’ll actually delete my data completely when I ask, and the third is that I don’t believe they can completely protect my data from an attack on their systems. To some extent, these factors are all true of existing cloud services I rely on (such as GMail), but that data is a lot less critical or private than the full contents of my hard drive (and, in the case of e-mail, I keep a full local cache). I haven’t used any of the major cloud backup services out there for full backups, so I don’t want to bash any of them by name, but I would argue that they all suffer from these potential problems.

That brings us to my (admittedly limited) use of the Dropbox service. I only use the free 2 GB level of service, and I use it primarily for temporary file transfer to my iPhone or iPad. However, for a handful of important files that I’m actively working on, I use it as a sort of temporary remote version control. That is, as I reach a point in a particular document that I want to make sure I don’t lose, I’ll save the file locally and then copy it to my Dropbox, generally not overwriting the previous copy. That means if I need to “restore”, I just retrieve the appropriate copy from my Dropbox. When I’m done, I delete that particular working folder.

Recovery

Since I refer to this setup as the Backup Tripod, then there is the platform being supported by the three legs described above: quick and relatively painless recovery of your lost data. How do you use these three levels of backup to restore your data after a disk failure? To a large extent this depends on the type of failure you experienced, and how broad it was.

If you’ve accidentally deleted a single file, or somehow produced a corrupted version, then pulling a previous copy from either your cloud storage or your hourly local backup is probably a sufficient level of recovery, particularly if the first iteration of the file was created since your last offsite backup. (Depending on what you’re working on, there may also be a place where a distributed version control system like git can play a role.) Deciding how many past versions to keep over the course of a work session is up to your discretion (and the importance of the file). Of course, if the “single file” is one of the files described above as being difficult to backup incrementally, you will probably have to recover the version from your clone drive. Again, regular backups here are critical, in order to make recovery possible.

If you’ve lost the entire drive, you have two options, assuming you were using Time Machine as your incremental backup solution: you can restore from the last successful hourly backup, or you can restore from your clone (obviously both require purchasing a new drive and replacing the old one). Most likely you need to do a combination of the two: restore from the incremental backup first, to get the most recent version of various files, and then manually restore any files you keep on the ignore list such as virtual machine images or databases, plus any “high priority” files you have stored “in the cloud”.

For my particular recovery, in the case of the aforementioned VM image, this was fine, since I hadn’t used it since my last complete clone; in the case of my music library, I had to reimport a number of apps from the backed up iTunes data folders and eliminate duplicates and stale versions. This was mostly a task of synchronizing the state of the iTunes Library with the fully backed up on-disk state. Annoying, but avoidable/minimized by keeping more recent clones than I had. This was also a case where I didn’t need the cloud storage leg at all, since the failure occurred while I wasn’t home, so I wasn’t actively working on an important file that I wouldn’t have wanted to lose if I had a local disaster.

Incidentally, my hard drives would probably be the first thing out of my apartment in the event of a fire. <.<

Conclusion

Those are the three legs to the Backup Tripod, blending a balance of manual vs. automatic, offsite vs. local, and complete vs. incremental. I claim you need all of these features to insure a complete, up-to-date backup under a variety of failure conditions.

If you follow the general gist of this guide, and keep good backup habits, there’s a reasonable chance that a complete hard drive failure will be nothing more than a minor inconvenience. You’ll have to order a new drive and wait for it to arrive (or pick one up at your favorite tech shop), and then most of the process is waiting several hours while your computer does the restore, during which time you can get outside or something. Maybe a little bit of manual file restoration. All in all, annoying, but not a massive tragedy that would be the loss of all your home data (and a hell of a lot cheaper than hiring a data recovery expert).

Seriously. Back. Up. DO EET NAOW!!!

(And while you’re at it, make sure the IT guys at your work are on top of this.)

Like every other critic and salivating fanboy, I feel compelled to chime in on yesterday’s religious experience in which The Steve descended from on high bearing a tablet. However, this is not a review, but merely a (lengthy) answer to a simple question: is the iPad for me? I won’t be talking about the market for digital content distribution, I won’t be whining about what software and hardware widgets weren’t included, I’m not going to rant about Apple’s closed ecosystem, I won’t be begging to lick someone’s boots for a chance just to touch one. Additionally, although hopefully this is obvious, this is heavy on speculation, since I have yet to actually hold the product, let alone use it for any length of time.

I’ll also take this opportunity to brag that I got 29.5 points on the prediction score card, with only one question as yet unanswered: will textbooks be available (I said yes, and I think this is eventually likely, based on the list of publishers involved). I was briefly unsure if my existing Apple Wireless Keyboard would be supported, but the Design page indicates that in will be, in spite of the existence of the iPad Dock. I got the name right, and most of the detailed features based on the rumorsphere. The substantive places I was wrong were the absence of a camera, the price point (cheaper than I expected), and the lack of any information on iPhone OS 4. I had a hope for an open development environment, but I knew that wasn’t going to be true, so that’s more a self-docking principle point. I failed to predict the dock, and I gave myself a half-point for saying no 3G when there are models both with and without.

Below the cut I’ll start off with a brief history of my personal electronics habits from college through today, and then consider where the iPad would fit into my little niche… and, if it does fit, whether it’s worth it. I’ll also look at what still-open questions about the device would affect my potential buying decision (not the least of which is that I need to try it out in an Apple Store to get a sense of the ergonomics). While I’m only speaking for myself, maybe my analysis will be useful to people similar to me.

Usage

For the purposes of this post, I’m discussing a subset of my uses for various electronics in rather broad categories. Over time, the breakdown of which device(s) I use for each task has changed, mostly due to the addition of new devices, but also to some extent due to changes in my tasks breakdown over the past 8 years.

• Programming
• Gaming
• Taking pictures
• Listening to music
• Reading news
• Making phone calls
• Remote access
• Online socializing
• Taking notes

I’m not going to get into how my time breaks down into those categories, but it’s safe to say what dominates given my employment (software engineer) and primary entertainment activity (video games). Taking notes is mostly only relevant for school, online social networking has increased since I got on Twitter, and I rarely take pictures or make phone calls.

In all four of the charts below, click to embiggen for a more detailed view (although they’re pretty broad generalizations, given that my percentages are rough estimates at best).

In the Beforetime

I jokingly refer to the period of my adult life before I owned an iPhone, which significantly changed my usual “loadout” when going anywhere by combining my mobile phone and music player into one device and also carving a few of the above tasks away from my (relatively much heftier) laptop. While it is reflected in the chart below, I’m not going into detail about how my habits changed as I accumulated new devices over the course of college.

Chart breakdown of my personal electronics from 2001 to 2005, during college.

At school, my laptop was my sole computer. I still have that PowerBook G4/400, and the only thing it’s needed over the years has been a new battery and a new power cord. Still going strong (more on how that affects my decision later). I got my first iPod at Christmas 2002, and my first mobile phone (a mid-range Siemens that had Mac Bluetooth support) at Christmas 2003. Before then, I had a Koss CD player for mobile music, and I was wedded to a landline. I still have my old Olympus digital camera, even though it’s only 2.1 megapixels and uses one of the losing memory formats, Smartmedia. I’m not a big picture-taker anyway.

The next big change was getting a home desktop computer once I was employed and out of school. My laptop, Chronos, became relegated to 2nd-class status, mostly taking on the roles of Remote Access and Note Taking, and being my primary computer when traveling for all other tasks. Gaming moved to the desktop almost exclusively, since newer games required newer and newer hardware, more than the Rage 128 card in the PowerBook could provide. (As you have read here previously, most of this gaming is World of Warcraft.)

Chart breakdown of my personal electronics from 2005 to 2007, as a young professional without an iPhone

The iPhone Cometh

The iPhone (I got a second-iteration EDGE one) significantly changed my electronics usage by replacing the iPod and phone in one fell swoop. Because I now had mobile data, it also reduced how often I brought my laptop with me, since I could do some limited forms of online interaction while out and about. Now instead of a laptop requiring a backpack (the titanium powerbooks are light, and beautiful machines, but still a lot heftier than a phone-class device), I could bring a device that just clipped to my belt.

Once Apple added the App Store, I suddenly had my first mobile gaming platform (I think my first purchase was Trism, downloaded over EDGE somewhere in Wisconsin while headed to my friend Gus’ wedding). These are all casual games, and mostly only occupy me while waiting somewhere or riding public transit, although I do occasionally play while sitting at home. The state of my electronics post-App Store is below.

Chart breakdown of my personal electronics from 2008 to present, covering two iPhone generations

This past summer, I upgraded to the iPhone 3GS, which meant I had a better camera (in fact, except for the lack of a flash, on par with my old Olympus), and one capable of video. It was otherwise a 1:1 replacement of my old iPhone, which I then promptly unlocked for use in Mali.

Room for one more?

So, how does the iPad likely fit into my usage patterns? In my case, it basically needs to be able to replace the few tasks still assigned to my PBG4. I don’t need a full-power laptop because I have my desktop at home, but I do need a device that has a bigger screen for graphical remote access (Microsoft RDP through work VPN, VNC tunneled through SSH to home Mac), and sufficient input capabilities for taking notes in class. It can also take over a few of the tasks that my iPhone had carved away from my laptop, such as Online Socializing while traveling. Pretty much everything else (larger screen web browsing, iBooks) would be an ease-of-use bonus or creating a use I don’t currently have.

Given my proposed usage breakdown in the chart below, the big questions are whether I can easily get a secure remote desktop connection to both my work and home computers, and how easy it is to take notes in class, possibly using the iWork app, particularly if mathematical notation is involved.

Chart breakdown of a possible fit for the iPad in my usage

I definitely do not see the need for two 3G-capable mobile devices for myself, so I’m glad there’s a WiFi-only version. The other advantage of not being tied to a mobile contract is that I’ll have far fewer qualms jailbreaking the device to run arbitrary 3rd-party apps, as I’ve done with my old iPhone, which might solve my remaining concerns.

Conclusion

At this point, I’d say I’m fairly likely to get one, although I have a hard time giving up on a perfectly functional (albeit old) laptop. It clearly fits in between my laptop and my iPhone in terms of functionality, with some overlap, and I think it might be just enough to retire the old TiBook. Another big advantage is that I won’t feel guilty about playing games while mobile as much, since I don’t have to worry about the necessity of the battery of the device-I-use-as-phone.

If, however, it is very lacking in editing capabilities (The Steve was certainly emphasizing it for content consumption for the most part), or cannot easily access my work and home desktops (possibly requiring jailbreaking), I probably won’t be interested except as a new shiny toy I can lust after until a second generation with improvements.

The verdict is still out until I feel it in my hands. What do you think?

Specifically, certain characters which much be escaped in the GSM 03.38 character encoding are getting treated as the wrong encoding when posted to Twitter from Verizon Wireless SMS, and showing up as ? in text messages sent by Twitter to Verizon Wireless customers via SMS.

I should add that I didn’t find this bug alone – @elliotreed asked why I used question marks to note something in a tweet when I had actually used square brackets around some text. Some quick investigation with him revealed the more specific nature of the problem, but it wasn’t until I actually found out that there was such a thing as GSM encoding that I came up with a hypothesis to explain the character weirdness.

As far as I can tell, Verizon’s HTTP/SMS gateway is now doing the GSM/UTF-8 mapping internally, but Twitter is assuming it still has to send GSM bytes to Verizon, so the encoding is happening twice, or at least attempting to happen twice. Verizon chokes on the GSM two-byte characters, since they’re not valid UTF-8, while Twitter receives certain ASCII-range one-byte UTF-8 characters but converts them as if they were GSM one-byte characters, resulting in a totally different UTF-8 character!

The GSM-to-UTF-8 encoding bug, shown here for square brackets, curly braces, tilde, backslash, and carat.

The GSM encoding doesn’t allow certain characters as single-byte characters; this appears to be a way to shove a number of European characters into a 7-bit mutant ASCII, with control characters and certain punctuation replaced by characters from the Latin-1 codepage. To some extent this makes sense, given that with the 160-byte length limit on SMS messages you want to avoid multibyte encodings while still supporting commonly used characters (UTF-16 is used for non-roman languages). Unfortunately, this leaves [, ], ~, {, }, \, |, and ^ out in the cold. As a programmer, I use these punctuation characters often as separators in various notations, so it is perhaps not surprising that one of my tweets revealed the problem. These characters can be sent as a two-byte sequence in the GSM encoding, but those start with an escape byte 0x1B, which since it starts with more than one initial bit high will always be invalid as the first byte of a UTF-8 character.

I would have thought that the Age of Unicode would have ended many of these non-standard application-specific encodings (and plus, given the way mobile carriers love to gouge on SMS, if they make your characters take more bytes, they get more money!). It looks like that’s exactly what Verizon is trying to do, in moving to exposing UTF-8 on the edge of their network… they just didn’t tell anyone that they had changed encodings, or if they have, Twitter hasn’t acted on the change yet.

Since Twitter disabled their help ticket creation (probably because too many stupid people were posting the same questions without reading the FAQs), I reported the bug using the Twitter API ticketing system on Google Code.

Short story: if you use any of the punctuation characters above in your tweets, expect texting Twitter users with Verizon to see ?, and expect to receive tweets from them with weird European characters, until this is fixed by one or both parties.

On a whim tonight, I whipped up my first Google Chrome extension in about 2 hours. A non-trivial amount of time was spent writing it up and making the icons. It’s obviously very simple, but it replicates one of my favorite features of Safari 3: SnapBack (the feature got eviscerated in Safari 4).

Basically all this extension does is add a button to the Chrome toolbar that you can click to jump back to the first page in a tab’s history. I realize the button and icons are ugly; I am not a design-type person.

The BackSnapper button once installed in Chrome 4

You can read a bit more about my BackSnapper extension, download it if you’re using the developer edition of Google Chrome (currently version 4), or view the code on github.

As Chrome rolls out the Extensions Gallery, I’ll deploy the extension out there. It could probably use some better options, and some smarter heuristics for determining where the beginning is, but for my purposes it gives me the magic button I want.

Installation

You can install the BackSnapper extension from the .zip file more or less by following Step 4 in these instructions. Note that at present this only works for the dev channel (version 4) of Google Chrome.

1. Download and unpack the .zip file
2. Select Extensions from the Tools menu.
3. Click “Developer Mode” on the right in the Extensions display.
4. Click “Load unpacked extension…” and select the unpacked BackSnapper folder

Development Tips

There were a few things I learned getting this working that weren’t immediately obvious from the documentation:

• The debug console is per tab
• You may need to select your injected content Javascript in the debug console to view logged messages
• For simple calls into content scripts, chrome.tabs.sendRequest() is sufficient, you don’t need to use the more complicated connect() message passing calls.

There were also a few things I couldn’t figure out:

• Why won’t the current developers-only Extensions Gallery accept my unsigned zip file?
• Why can’t I determine the current URL in the history after having called history.go()? location.href remains unchanged, and history.current is undefined.

Oh yeah, I have a blog! Lots has been going on in the intervening months (see my Twitter feed for short attention span details), but I figured a video game post during NaBloPoMo would be a good way to get back on the wagon, even if I’m not actually posting every day during November.

While visiting my Little Brother this weekend, I noticed a rather unusual magazine cover… a (very pixelated) monster from the original Doom. This turned out to be the latest issue of Game Informer, specifically Volume XIX, Number 12, Issue 200. In honor of this decimalist anniversary, they published their Top 200 Video Games of All Time list, which unsurprisingly is linkbait for any video game fan who likes to rant about what should and should not be included in such a list. I ran through my opinions quickly with my Little, mostly fixating on why so many recent games were already on the list, but decided a deeper analysis was in order.

Instead of complaining about the contents of the list, I thought I’d use it to track my personal video game history (much as my father has in the past used the Rolling Stone 500 Greatest Albums of All Time and Rolling Stone 500 Greatest Songs of All Time to guide his music purchases). I’ve also done some histogram breakdowns of what’s on the list. I would say that my guideline for inclusion on any such list would involve adjectives like “innovative” and “influential”, and explicitly avoid conditions like “critically acclaimed”, “popular”, or “best-selling”. This in turn means that inclusion must be viewed through a somewhat temporally distant lens, for sufficient perspective on a particular cultural artifact’s import.

How many of these have you played? Do you strongly agree/disagree with any of the rankings?

The columns are Game Informer rank, game title, platform(s), and year of publication from the original article; I believe using this data for commentary is covered by Fair Use. I added platforms in a few places to account for the particular port of a game that I played. I have also added columns for myself, for Played, Owned, and Completed. The full table and further analysis is below the cut.

Columns

A brief explanation before the table: I count a game as “Played” if I personally controlled the game for at least one significant play session (enough to see one full level, game, etc.); I count a game as “Owned” if I purchased the game or received it as a gift; and I count a game as “Completed” if I finished the primary plot levels for games with solo play, reached a reasonably high or maximum level in games with increasing difficulty level play, or played a simulation game to a goal threshold. Ownership and Completion are both subsets of Played, with some overlap, but not identical subsets.

Top 200 Video Games Table

Overall Breakdown

From browsing the list, I was expecting it to be skewed towards more recent offerings. A few quick histograms to some extent confirm this for the full list, but disprove it when you consider only the Top 20 games on the list. Click to embiggen any of the graphs.

Bar graph histogram of the Top 200 and Top 20 games by publication year

Bar graph histogram of the Top 200 and Top 20 games by console platform (including PC)

Bar graph histogram of the Top 200 and Top 20 games by console generation.

As you can see, the Top 200 list (in red) is skewed towards PC games, since that category is not further broken down by OS or era. It also seems to be skewed towards more recent consoles, particularly the Playstations 2 & 3 and both Xboxen.

However, if you consider only the Top 20 (in blue), a more historical perspective is available. Nintendo is more favored, and the breakdown is more even by by both year and generation. To me that implies that Game Informer recognizes Nintendo as an early innovator (particularly on the NES and SNES), but more lacking in recent offerings for the GameCube and Wii, especially compared to what’s available on the Xbox and Playstation.

My Breakdown

So, in total, of the 200 games, I’ve played 89 (44.5%), owned 25 (12.5%), and completed 26 (13%); however, I’ve only completed 11 of the games I’ve owned. There are three factors here: first, I’ve played games to completion on friends’ consoles/computers; second, I played several classic console games via emulation in high school, where I had downloaded the ROMs; third, I have purchased some newer games on the list that I’ve yet to finish.

Until the Wii, I never owned a console myself (with the exception of the short-lived Virtual Game Station PlayStation emulator for Mac), so all the other games I owned are listed under PC (which I mean to include DOS, Windows, Mac OS 9, and Mac OS X versions of games). PC is nearly impossible to split into “generation”, and since it includes multiple hardware/software combinations, I can’t easily get good stats for it. In spite of all of this, I feel like I’ve experienced most of the key points in video game history from the Nintendo revolution on. In that regard I’m in a historical sweet spot, I suppose. Click to embiggen any of the graphs.

Bar graph histogram of the Top 200 games broken down by my Played, Owned, and Completed lists organized by publication year

Bar graph histogram of the Top 200 games broken down by my Played, Owned, and Completed lists organized by console platform (including PC)

Bar graph histogram of the Top 200 games broken down by my Played, Owned, and Completed lists organized by console generation

As you can see, my own gameplay is heavily skewed towards the PC, although until recently that was almost exclusively games that were available on Mac. This makes sense, given that my mother didn’t allow game consoles in the house, but non-violent and arguably educational games my sister and I could sneak onto the computer.

There’s also a strong “bump” for games in the mid-’90s; this is when I would have been playing lots of console games at friends’ houses (for sleepovers and such), but it also includes the non-trivial number of SNES and Playstation games I played via emulation on my Mac in high school. (Incidentally, that very Mac is now functioning as the webserver displaying this blog entry.)

My ownership numbers have unsurprisingly gone up since 2005, given that I now have disposable income and can indulge my video game habits more easily than when I was living at home or in college. Additionally, I have added a Wii (my first console I can call my own!), a gaming-capable phone (first the original iPhone, and now a 3GS), and with the advent of Intel-based Macs, a drive booting into Windows 7 that I use to play various Windows-only games, such as Half Life 2 and Borderlands.

Conclusion

Overall, I’d say this list is skewed towards recent years, which I think is explainable largely by the target audience of the magazine: roughly, my age group (and a little younger) who probably started playing on SNES (4th-gen) consoles and are currently playing Xbox 360. The breakdowns for the Top 20 however makes me feel a little better, in that games that are more influential (or arguably started a genre) are reflected there.

There are definitely a few “classics” on here that I’d like to pick up and play for historical reasons, and a few I’d go back to for nostalgia, but most of the games I haven’t played aren’t in genres I particularly care about. A number are relatively obscure Japanese releases that only the hardest of hardcore video game fans would have seen in the US.

My biggest complaint about the list is a few items that had been out less than a week at the time of publication of this Game Informer issue! (I’m looking at you Modern Warfare 2…) I don’t care how good your opening sales weekend numbers are, or how good the previews are, you don’t deserve to be on this list yet if you came out in 2009. Give it time!

What do you think? How do you measure up?

Like many Unix geeks, I have software installed that I’ve built manually from source. A good example is my post on compiling django; a number of the relevant dependencies were built in /usr/local/src/ and installed in /usr/local/. I also like using package managers, because if I’m not doing any customization (and the package is common and not hard-to-find), I want to just get the latest version and slap it in the right place. The conflict between the two methodologies is when a managed package depends on software that is already installed on your system, either part of the default configuration (OS X ships with a fair bit of Unixy software included, especially if you install the Dev Tools, although not always a “standard” or particularly recent version) or custom-built.

I recently dumped Fink for MacPorts; while I’ve used Fink for a long time, since an early version was available for Mac OS 10.2 Jaguar in fact, it’s just gotten in a messy state maintenance-wise. I’ve been familiar with apt since using Debian-based systems at the SCCS, but the mish-mash of binary and source items, the preponderance of out-of-date packages, and the apparent need to install 70 metric boatloads of GNOME just to satisfy a few dependencies was frustrating. Of course, MacPorts has its own weaknesses, as do almost all package managers; in particular, none of them seem to be able to track whether a package was installed explicitly by the user or merely to satisfy a dependency. My opinion is that the latter should get uninstalled when all of its dependents are uninstalled, but no package manager seems to agree with me on that. A rant on that probably merits a separate post.

Below the cut is a rough step-by-step guide to creating a local portindex and creating portfiles for your manual dependencies. Note that most MacPorts users would tell you this is a terrible idea, and you should just install all the port dependencies, but I already put the effort into these custom from-source builds and I just want to use them without duplicates getting dropped all over my hard drive.

Create Local Portfile Repository

These instructions are based on Section 4.6 of the MacPorts Guide.

1. Create a directory that will serve as your portfile repository; I used /usr/local/ports/
2. Add this directory to your MacPorts configuration. For me, that meant editing /opt/local/etc/macports/sources.conf to add the line file:///usr/local/ports/ before the standard rsync line (for priority)
3. Update the port index. For me, I needed to run sudo portindex in /usr/local/ports/ because of the way I have permissions set.

Now, if you use any port command, it will look at your local source first.

Create a Placeholder Portfile

For this example, I’m focusing on installing the port phpmyadmin, but satisfying its runtime dependencies on MySQL and PHP using local builds.

1. Check the dependencies of your target port: port deps phpmyadmin

   phpmyadmin has runtime dependencies on:
   	mysql5
   	php5

2. For each dependency, look it up using MacPorts search, and then click on the link to view the checked-in portfile. In general this link will have the form http://trac.macports.org/browser/trunk/dports/CATEGORY/PORT/Portfile, where CATEGORY and PORT are the relevant names of the package you’re trying to replace
   • mysql5
   • php5
3. Now you need to create a local placeholder portfile for each of these packages; in the port repository you created above, you’ll need to create a subdirectory for the CATEGORY and then open a file named PORT in that directory in your favorite text editor. For example:
   • cd /usr/local/ports/
   • sudo mkdir -p databases/mysql5
   • sudo touch databases/mysql5/Portfile
   • sudo emacs databases/mysql5/Portfile
4. We only need a few variables in the portfile to trick MacPorts into accepting our dependency, some of which can be copied from the real portfile. Note that I added a mention of “placeholder” to the description, changed the version to match what I built locally (which is a newer version than what’s available in MacPorts), and added some information about the build. We also need to make MacPorts do a no-op for all port phases. Here’s what mine looks like:

   PortSystem              1.0
   name                    mysql5
   version                 5.1.30
   homepage                http://www.mysql.com/
   categories              databases
   platforms               darwin
   distname                mysql-${version}
   
   description \
       Placeholder for multithreaded SQL database server
   
   long_description \
       MySQL is an open-source, multi-threaded SQL database \
       with a command syntax very similar to mSQL. \
       \
       This is a placeholder built from source in /usr/local/src/mysql-5.1.30/ \
       on 2009.01.27 using the options:\
       \
       ./configure --prefix=/usr/local/mysql --with-extra-charsets=complex \
       --enable-thread-safe-client --enable-local-infile --disable-shared \
       --with-unix-socket-path=/usr/local/mysql/run/mysql_socket \
       --with-mysqld-user=_mysql --disable-dependency-tracking
   
   fetch {}
   checksum {}
   extract {}
   patch {}
   configure {}
   build {}
   test {}
   destroot {}
   install {}
   activate {}

5. After creating and editing the relevant portfiles(s), you need to regenerate the portindex as above.
6. If you run port search mysql5, you’ll see both the real port and your placeholder listed with the short description, like this:

   mysql5 @5.0.83 (databases)
       Multithreaded SQL database server
   
   mysql5 @5.1.30 (databases)
       Placeholder for multithreaded SQL database server

7. If you run port info mysql5, you’ll get a warning about multiple definitions, but see the long description of your custom placeholder:

   Warning: Found 2 port mysql5 definitions, displaying first one.
   mysql5 @5.1.30 (databases)
   Variants:    universal
   
   MySQL is an open-source, multi-threaded SQL database with a command syntax very
   similar to mSQL. This is a placeholder built from source in
   /usr/local/src/mysql-5.1.30/ on 2009.01.27 using the options: ./configure
   --prefix=/usr/local/mysql --with-extra-charsets=complex
   --enable-thread-safe-client --enable-local-infile --disable-shared
   --with-unix-socket-path=/usr/local/mysql/run/mysql_socket
   --with-mysqld-user=_mysql --disable-dependency-tracking
   Homepage:    http://www.mysql.com/
   
   Platforms:            darwin

Undoing the Mess

Pretty simple. Just uninstall the placeholder you created, remove its file from your local ports, rebuild the portindex, and reinstall the “real” package (or just leave it uninstalled, depending on what you’re trying to do). Alternatively, uninstall all the placeholders you created, and remove the offending line from your sources.conf before continuing.

Conclusion

I’ve only tested this with runtime dependencies so far, not build dependencies. I think the general idea is sound, but as I said, it’s probably a terrible idea. I’m curious if you can extend this process to get any other ports to install using system or local dependencies. I’d also welcome any suggestions for a better way to achieve or similar result. Feel free to just call me misguided and mad, as well!

Version 1.0.0b1 of the tweetworks package for Python 2.6 is now available. This package implements the web service API for Tweetworks, a Web 2.0 service that facilitates threaded conversations on top of Twitter.

This is definitely a beta, because while I’ve tested everything I can think of, I haven’t tried writing anything seriously complicated with it, although I certainly plan to. Comments and questions are welcome here, or find me in the Tweetworks Developers group or as @UltraNurd. I admit that the documentation is a little light at the moment.

If you’re interested in using Tweetworks programmatically from Python, or want to know more about the service, read on.

What is Tweetworks?

Tweetworks is a service and site built on top of Twitter allowing a user to create public and private groups (instead of hashtags) with threaded conversations and more.

Tweetworks is a Boston-area startup created by @MikeLangford. I first heard about it at a WBUR tweet-up hosted by @KenGeorge. Mike was there, promoting his then-brand-new site, and handing out wearable plastic toucan beaks (part of the Tweetworks logo).

Naturally, I checked it out right away, and ended up becoming Tweetworks user #15! I would not call myself a heavy user of the site, and this is not a review thereof, but it is a useful place to have Twitter-based discussions with threading and without having to rely on the sometimes unreliable Twitter hashtag searches.

Why did I make this?

As Tweetworks’ usage increased, my computational linguist side became interested in the threaded data about short-form online conversations that the site was accumulating. I was therefore very excited when they released a public API.

It’s my hope that creating this library will help other developers get started playing with Tweetworks from a service perspective, as well as facilitating my own site mash-up ideas.

Finally, there is also the totally selfish reason that I like coming up with small projects like this one to hone my programming skills, both in terms of design and learning new best practices for interacting with different systems and built-in libraries. In this case, I learned a lot about Python’s urllib, as well as setuptools.

Getting the Package

The easiest way to obtain the tweetworks package is using setuptools to grab it from PyPI. Assuming you have Python, with setuptools installed, you should just be able to run this command in your shell:

easy_install tweetworks

For alternative methods, or if you’re interested in contributing to the development of the API package, check out my Tweetworks page.

This software is released under the GNU General Public License.

Note on the name: all of my Mac devices are named after Star Trek animals, and all of my Windows devices are named after Star Wars animals. This is Spot, as in Data’s cat.

Does anyone know how to correct YouTube’s display to handle a vertical video orientation more cleanly? I just manually adjusted the <embed> size, but on youtube.com it is still horribly letterboxed. A good size appears to be 220×320, but that messes with the YouTube player’s control layout, even though it frames nicely.

I was inspired by a recent tweet by Doc Searls to play with Wolfram|Alpha and get some stats on how old I am. In particular, if you make a query of the form [<month name> <day of month>, <4-digit year> birthday] Wolfram|Alpha (why is there a pipe in the middle of the name?) will calculate your age in a number of differen units, and how many days until your next birthday. You will not be surprised to learn that I am a total statistics addict, so this got my brain juices flowing.

First, I’m 9,720 days old. I recently got a good laugh by pointing out that I was “twenty-six and a half!” after someone commented on how young I look; I think the “and a half” made me sound too much like a 4 year old being ever-so-proud of being closer to 5 than 4 :oD.

Quick aside: someone (I forget who) on a SWIL mailing list suggested the notation of writing search queries demarcated by square brackets, since this is not something that is typically in search strings. This was in response to people saying things like “You should just google “search string”", which leads to nasty quote-nesting problems and also induces some ambiguity as to whether to quote (and therefore group) the specified search keywords. I’m not going to post the full BNF notation for this, so I hope that [<search term> ...] is a clear enough explanation of the format.

Second, I realized that I’m pretty close to 10,000 days old. While I usually am not a big fan of celebrating arbitrary anniversaries, particularly those that favor Base 10 (that’s for all you decimalists out there), I missed 2¹³ days a while ago and I have a ways to go until 2¹⁴ :oD. Another quick Wolfram|Alpha search ([today + 280 days]). I guess this means I’ll celebrate my 10,000th day of existence on March 12th, 2010. A quick glance at Wikipedia doesn’t indicate anything particularly auspicious about that date, but hey.

Third…

I don’t have a problem

I am of course referring to World of Warcraft. Seeing my age in days reminded me of the /played command in WoW, which gives you how many days you’ve played a particular character. I have only one primary character, which I had a vague recollection of being at over 100 days; however, I have two other level 80 characters (as of Wrath of the Lich King, the level cap), plus a number of mid-level alts as well as some very low-level alts rolled on various other servers either as experiments

I didn’t feel like mathing this all myself, so I grabbed the addon AllPlayed to do the hard work for me; all I had to do was login on each of my characters so that it could accumulate the data. This addon also has some other nice features, including allowing me to see which of my characters are fully rested (giving them the optimal XP bonus while leveling). I enjoy all of the character classes, and while sometimes I’m in the mood for one class or another, I try to make leveling as fast as possible given my play time (which I’m sure is more than most casual players, but less than a lot of people I know and play with, primarily because I’m limited to a few hours in the evening, possibly with longer sessions on the weekends).

The cold, hard total is 179 days, 4 hours, the vast majority of which (61.2%) is on my main Kjallstrom (109 days, 15 hours). That means I’ve spent about 1.84% of my entire life playing World of Warcraft. I’m pretty sure I could have done some amazing things with that time (which probably does not include writing the great American novel), but on the other hand, it’s entirely after work entertainment which most people would spend watching TV (about 13.5% on average over a 65 year lifetime). I would argue that it’s a bit different, given that it’s interactive and social, but I know a lot of people disparage online socialization as “not as good as the real thing”. I contend that I’ve greatly expanded the diversity of my friends on a number of axes thanks to long-term involvement in the guild community.

I’ve been playing WoW since October 2005 (I don’t have an exact date), and I’ve been playing on Kirin Tor (and a member of Mellonea) since November 2005. I did play non-trivially on other servers with real-life friends well into 2006, and briefly convinced them to join me on Kirin Tor, although they eventually gave up on the game. If we isolate the percentage to just the time that I’ve owned WoW, the past 1300 days or so, that gives about 13.7%, which is on par with the typical American’s TV usage.

Even more absurd details below the cut.

I can quit any time I want

Here’s the full data table of my characters, and how long I’ve played each of them; where available, I’ve added a link to their Armory profile. You’ll note that I play almost exclusively on the roleplaying server Kirin Tor; this is where I am guildmaster of Mellonea, and where I also spend most of my WoW-based socialization time. The table is in descending time-played order.
 

 
A few things to note: the old abandoned characters on other servers took much longer to get to their level, for a number of factors: Blizzard has significantly sped up leveling; I’m a more experienced player (knowing where stuff is); and on PvP servers there’s more death while soloing. Another odd thing: Thrun the Hunter was the very first character I rolled (because I had heard hunters were easy, and a good way to learn the game, and I liked that Tauren felt more alien than other classic fantasy races), but I don’t have any hunters I’ve leveled with any attention after switching to Alliance on Kirin Tor. Also, the character Mellobanka is a guild bank mule with bags, so I only log into to buy and sell items for the guild.

As you can see, the play time falls off pretty quickly, down to “just trying the class” and “just checking on people on that server”. Another thing to keep in mind is that Death Knights, as a hero class, start at level 55, and are also insanely easy to level. I have deleted a few characters, so I don’t have data on them, but all were below level 20. The nubs on other servers basically don’t affect the total, compared to my mains who raid at the level cap. Also, these values are only valid at the time of this post; I’m still playing, after all :o).

Conclusion

I would say that I do play too much WoW, but I find it to be a fun source of entertainment, a social activity, and a good way to unwind after work. I probably do read less, although I always have a book that I read before bed; I certainly watch less TV (although I sometimes have shows or movies I don’t care about that much on in the background while playing, particularly if I’m soloing). I have forgone real-world social events to raid with my WoW friends; I don’t know what people think about that generally. I would guess the most significant problem is that between work, WoW, and assorted other computer uses, I spend nearly every waking hour in front of a screen. No doubt this is why I started going nearsighted near the end of college!

What is your dominant non-work/non-school and non-sleep activity? Feel free to judge me now :oP.

As you may have gathered, I have a… healthy… relationship with everyone’s favorite MMO, World of Warcraft. I forget who originally planted the idea in my head (there’s a good chance it was Lilboo, formerly of the Daring Blades on Kirin Tor), but I decided that I wanted a dedicated game controller that was more than just the keyboard; there’s just too much going on in WoW for an FPS-like control layout, in my opinion. I settled on a Nostromo, and after a few weeks of adjusting, and very few changes to my bindings, I have gotten very used to what may be a very unusual control style.

Verbose explanation of how I use the device below the cut.

Nostromo

First, the product itself. I have the Nostromo n52, which gives you a D-pad, 14 programmable keys, a scrollwheel, and two thumb buttons. I’ve only seriously tried configuring it for World of Warcraft (and also for Lord of the Rings Online, but I intentionally set that up to be almost identical to WoW).

Last year Belkin released the Nostromo n52te (Flash website), which as far as I can tell is the same product, costs twice as much, and has silly blue lights. Maybe they improved the “touch” of the keys, but the vanilla n52 works fine after hundreds of days of playtime (although the key labels have all worn off).

The Nostromo works with both PC and Mac, and the Mac software is one of the better ones for third-party peripherals; it’s implemented as a System Preferences pane, which is the “right” way of doing this sort of thing on a Mac (as opposed to a standalone application). Here’s what configuring it looks like on a Mac:

Configuring the Nostromo n52 on Mac OS X

Pretty straight-forward; you select the key on the Nostromo you want to configure, and then you can select a wide variety of actions (and add modifiers). I haven’t ever used any of the “turbo” functionality, I treat them all as normal keys, with the exception of the Shift buttons.

Here’s how configuring the Nostromo looks on Windows (and how it looked on the Mac before they wrote new drivers, as opposed to just porting the Windows ones):

Configuring the Nostromo n52 on Windows 7

The same functionality, but the “cool looking” but non-standard UI I find a bit confusing. Both the Mac and Windows versions support some of the other Nostromo series products. The Windows version is a bit better at saving profiles for export; as far as I know, the current Mac version can’t do this (unless you want to dig around in OS X plist files).

Shifting is one of the really cool capabilities of the Nostromo: while there are only a few physical buttons, if you assign 3 of them to Red Shift, Blue Shift, and Green Shift, you effectively quadruple your available buttons (I definitely don’t use that many), as long as you’re willing to press multiple keys at once. This is usually called “chording”, and at this point it’s very well embedded in my muscle memory, as you’ll see below. This is what really gives you “speed at your fingertips” or whatever they put in the marketing literature – with only a few movements of your left hand, you basically get as many keybindings as a standard keyboard.

n52 Keybindings

As you can see below, I use the D-pad for strafe-based movement, since I use mouse turning. The orange button near the D-pad is hard to use with the D-pad, so I just use it as an auto-run toggle. The six buttons in the upper right are associated with the active action bar; either 1-6 or 7-12, depending on whether or not I’m red shifted. The mousewheel is used for applying target markers (skull, X, etc.). Most of the keys on the left periphery are lower priority things, since they’re hit by my pinky or by an extension of one of my other fingers.

The main functionality is with that block of 6 keys. Between the red shift on my thumb (button 15, below the D-pad), and shift (the literal keyboard key) on Button 11, in the lower left for my pinky, I can select any of my character’s actions with a three-fingered cord. Since I organize my action buttons in-game carefully (see the next section), most of my fast in-combat actions can be achieved by mashing one key with the relevant finger, possibly while pressing my thumb. I can press my thumb on the red shift key while using the D-pad to move, although in highly mobile fights I’m likely to use a combination of autorun and mouse movement.

Also, these bindings should be perfectly valid for a newer Nostromo n52te.

The other keybindings are pretty boring: they open the character pane, my current favorite map mod, or my bag management mod. Green shift and Blue Shift are largely unused, although they focus on modifying the core 6 buttons; Green lets me target party members (function keys), including myself (I mostly rely on Grid and mouseover macros these days); Blue lets me activate pet/vehicle abilities (control + number keys).

Blue shift also modifies some other actions, such as making the D-pad turn instead of strafe, or making my flying mount go down instead of up (note that x and space are assigned to the same button, just shifted). There’s a lot of room for expansion, and the configuration is admittedly a bit haphazard… but I’ve been using it for over 2 years, so I’m very used to it. It’s all in muscle memory, and I can react very quickly to most in-game events without moving my hands at all (I use a Logitech Cordless TrackMan® Optical in my right hand).

WoW Keybindings

If it wasn’t obvious to most WoW players based on the bindings listed above, I keep the in-game keybindings set almost exclusively to the defaults (with the exception of a few additions, or tweaks to use particular mods in place of some default UI components). I also use only the default action bar; the shift-# bindings above select one of the six default action bar “pages”, and then 1 through = on the number row activate an action on the current bar. Thus my Nostromo configuration depends heavily on my bar configuration, since I memorize it

Without going into detail at the level of specific class abilities, here’s how I break down the six action bars:

1. Primary abilities: attacks, anything with a short cooldown; in the case of my healers, geared towards soloing.
2. Panic buttons: potions, trinkets, class abilities like Evasion and Desperate Prayer
3. Secondary abilities: longer cooldown combat abilities; in the case of my healers, mouseover macros for their various group healing and cleansing spells
4. Crafting: maybe silly to have a whole bar dedicated, but I don’t otherwise need the space
5. Transportation: mounts, pets, and usable quest items
6. Buffs: poison hotkeys on my rogue, blessings on my Paladin, etc.

I’m admittedly not 100% consistent in this, but that’s the general idea. Because of where 1 and 2 are bound on the Nostromo, the most important bars are under my pointer finger; similarly, within each bar (and related to the core 6 keys above when combined with Red Shift), the most important abilities are in slots 1, 2, 7, and 8, because these are again under my pointer finger.

Maybe a full description of my UI configuration, including button bar breakdowns by class, can come in a future post; the main takeaway from all of this is that I try very hard to keep things similar across characters, so my muscle memory doesn’t have to relearn much as I switch between the different roles on my various alts. Feel free to suggest any tweaks to this!