Classic AlpineInker Post #1: The Microsoft Research Codex

My old blog, the AlpineInker, is currently read-only (I can’t even access the blog stats any more) and it will go away completely by the end of the year, but there’s still a number of posts there that are near and dear to my heart, and which I occasionally like to direct people to, so I figured why not get this blog on its feet by moving some of the real classics over here.

This particular post was probably one of the most fun blogs to write that I ever put together, and the project was a blast. Since I did this project, OQO went bankrupt and the Model 02 fell into the dustbin of history with it, the Courier came and went like a glint in the eye of the girl who got away, and e-readers and slates have exploded and are on the verge of toppling traditional publishing like a loose boulder teetering on the brink of a nine thousand foot cliff.

But I’m still here and plugging away on my sane and not-so-sane visions of the near future, looking for that next idea that’s oh-so-obvious in retrospect. Sometimes they come easy, sometimes they’re like pulling a loose molar from your jaw with no anesthetic. And I ain’t never been to no dental school, folks, although I did have the fine honor of working with some incredibly gifted and hard-working neurosurgeons back when I was in grad school.

So I’ve worked side-by side with brain surgeons and with Turing award winners and I figure all I need to do is come up with some zany project involving rocket scientists and then I should be about set to call it a wild ride of a career.

I hear NASA might be looking for the next great thing too now that the Space Shuttle is history, so maybe there is hope for me yet on that front.

As for the Codex. Since we’re on WordPress here, I can even enter the 21st century and embed a YouTube video right in my post without pulling yet another molar by hand.  Below I’ve linked to a copy of my Codex Video that someone pirated and put up on YouTube on my behalf violating every law of copyright known to man, but heck, once something’s on the ‘net and everyone’s hyperlinks point to it, having them take it down is just counterproductive, so I just went with it. The video has almost 100,000 views so I guess someone must have checked it out. (This video, by the way, was one of the most difficult to shoot that I ever did because at the time we shot it the software kept crashing and the screens were dim and had a lot of glare and, and, and… so a belated apology that it looks a bit clunky in the video, but the thing was pretty damn cool to use and I wish I had the time and energy to resuscitate the software and do more with it).

Anyway, enough of my blathering. Here’s the Classic AlpineInker Post #1, The Microsoft Research Codex:

The Microsoft Research Codex: Are Dual Screens the Future of Mobile Devices?

Never buy one of anything. That’s advice you should stand by when you’re buying unusual gadgets. The advice was good when Randy Pausch offered it to me some 15 years ago, and it’s still good now.

Of course, with 18 month old twin girls at home, this has become second nature to me. Two boxes of diapers. Two gallons of milk. Two Elmo plush dolls.

Oh, and yes, of course. Two screens for my tablet computer.

Dual-screen devices have become the subject of increasingly elaborate internet fantasies.  Now, rumors about an Apple “Brick” device have stirred up dreams of future tablets, such as the alluring One Laptop Per Child v2.0 concept photos, which now orbit the internet once again.

My interest in dual-screen devices goes back a lot further, though, and was really spurred on by the University of Maryland dual-screen e-book reader project. That effort is led by Francois Guimbretiere, who is a long time collaborator and friend. I had some ideas to build on what his team had done, but also to take things in a different direction.

First of all, I wasn’t interested in an ebook reader. I wanted a device that was all about writing. Sure, reading and writing go hand in hand – you encounter cool ideas and search out reference material on the web-but what I wanted to build was a tool for thought. To me that means a tool with writing, sketching, and annotating as the core of the experience.

I’d been thinking for a long time about picking up an OQO Model 02 computer. My team has an extensive code base for pen-and-tablet functionality resulting from InkSeine, and the OQO runs it out of the box. It’s got an active digitizer for high fidelity pen input, and it’s the smallest slate Tablet PC that money can buy. That’s a pretty good start on a small form-factor tool for thought.

So when it came down to it, just how many of those OQO Model 02’s do you think I purchased?

Project “Codex” was born.

Project Codex Logo

Twice the Screen at Half the Size

Here’s the Codex packed up for mobility. It folds up quite nicely and has a moleskine-style knitted elastic strap to hold it securely shut. There’s a loop for the pen and a mesh pocket so you won’t lose small accessories, business cards, or receipts that you collect in your travels.

Codex packed for mobility

Sure, it’s a bit of a brick at present. The whole thing weighs just over 2 pounds. The OQO’s are considerably thicker than I’d like. But my goal is to prototype the future as quickly as possible and start living it. The OQO offers a handsome, jet black time-travel machine in that regard.

Don’t underestimate the ability to quickly pack up with the screens protected. Folding the Codex in half makes it comfortable to carry and easier to stuff into my gadget bag. It’s a self contained kit for ultra-mobility.

The Codex is Not a Container for Dead Trees

A codex is just an archaic term for a bound book. But this Codex is unlike any book that you’ve ever read. It’s not a long linear text that you flip through. To me, there’s no use in going to all the trouble to build a dual-screen tablet prototype and write elaborate software just to mimic a traditional book. This is the 21st century and it’s about time we moved past containers for dead trees.

A book is a container for dead trees. Codex is much more.

We have lots of information sources. We have multi-tasking. We have hyperlinks. We have split attention. We have a left brain and a right brain and we rarely do one thing at a time any more.

A Separation of Concerns

The Codex has two screens, it’s designed to be used that way, and you won’t find any half-apologetic demos that try to mash them back together into one big screen. Instead, it’s all about the intelligent partitioning of tasks and interface elements across the screens.

Codex notes side-by-side with the web

Above is one example where I’m working on a blog post that I’ve had planned for a while. On the left I have a whole bunch of cool photos that I found tagged with moleskine on Flickr. I was browsing through these as inspiration for our InkSeine digital note-taking software. On the right I’m organizing bits and pieces from these photos along various themes.  So I just take a snapshot from the collection on the left screen and it appears in my notes on the right screen, where I can arrange it and mark it up as I see fit. I can scroll back and forth on the left screen to find a photo that meets my current needs, while the page that I am authoring on the right screen always remains visible. The two screens are invaluable because I always have the reference material in the context of what I am working on, instead of feverishly flipping between them on a single screen.

Can I do this on a single large screen? Well, sure, I could monkey with the window placements and get everything arranged just so. But that takes a lot of effort and the temptation to expand windows to take over the full screen is hard
to resist if I have to expend effort to do so. A dual-screen device that understands the partition between the screens gives a much simpler experience where I don’t have to constantly manage the set-up of the windows.

Navigate without Losing the Big Picture

Here’s another example. I’m further along with authoring my blog post now, and I have a bunch of material floating around in my notes. I create a page that is a Table of Contents, with links to several themes that I’ve identified in my Flickr moleskine investigations. If I open a link, such as my Creative Collage page, it opens on the opposite page. I don’t even need a “back” command to return to where I was – I still have my navigational structure on the left, side-by-side with my content page on the right.

Table of Contents on two screens of Codex

Unlike a traditional dead-tree book, I have no physical restriction that forces me to view consecutive pages – The Codex lets me follow links or flip through the screens separately to view any two pages together.

Working Big and Small

We can draw some inspiration from traditional media as well. In magazines and books, sidebars are a distinct section of a page that augments the main text with auxiliary information. Well, the Codex has sidebars on steroids. I can take any chunk of my notes, make it into a sidebar, and then arrange a bunch of these on a page. Here, I’ve made a storyboard page consisting of six sidebars. When I tap on a sidebar in one of the storyboard cells, it expands to full size on the opposite screen. I can plot out the broad sweep of my story on one screen, while maintaining full access to the zoomed-in details on the other.

Example of Sidebars in a storyboard layout on Codex

Your Mother Was Right – Posture is Important!

Perhaps the coolest property of a dual-screen device is that the compact, mobile form-factor encourages shifting the device around. I can orient the screens with respect to one another, stand the device up, look at the screens in portrait or landscape, and so forth. In fact, the Codex supports about a dozen different configurations. We call these postures. (A colleague, Michael Miller, coined this term).

Add two cups of accelerometers, a dash of flex sensors, and bake with some simple software to fuse it all together. Out of the oven pops an intelligent dual-screen display system that configures itself depending on how you arrange it.

For example, if I want to work in landscape, I can flip the device into the laptop posture, as opposed to the book posture that I’ve been showing so far. Now I have one screen that’s angled for easy reading, plus another screen that’s horizontal for easy writing.

Codex in the "laptop" posture

Or I can use the laptop posture to hook up one of my screens to a projector. I put the public part of my presentation on the top screen, while the controls to drive the presentation and my private notes are confined to the bottom screen.

Running a presentation from the dual screens of the Codex

Now perhaps I meet a friend in a café and I want to show him what I’m up to. I can just lift up on the binding of the Codex to angle the screens so that one is facing me and one is facing my friend (below, left). We call this the battleship posture – as in “You sank my Battleship!” Each person has one screen with a private view. The Codex automatically configures the software for shared whiteboarding so the collaborators can mark up the screens, pass notes back and forth, and other such foolishness.

Codex "battleship" posture for collaboration

Next I drop the binding back flat. Now both of us can view the screens, but one screen is oriented towards each person.

Don’t Get Too Attached

Something that turned out to be surprisingly useful is that the Codex allows me to pull each screen right out of the binding. A firm pull pops it out, a firm press pops it back in. This is really handy for laying out the screens to suit my work, or to review a video while I jot down some notes. I can even hand the screen to another person to show them something – without actually giving them an electronic copy of the information.

Detachable screen on Codex

What? You Still Have a Desktop Computer?!?

The Codex is just another wireless device so of course I can connect it to my desktop computer. Now, any screen capture that I take from my desktop screen shows up in my Codex notes right where I left off. For web pages, the snapshot
comes across with a hyperlink back to the source, so I can easily revisit it later when I’m reviewing my notes on my Codex. This makes it the perfect companion no matter how I’m working.

A Parting Shot at Text Entry

The Codex is designed around doing most things with a pen, but I’m no idiot. When it comes to text entry, a keyboard is a good thing.

And that, in part, is why I’m puzzled that so many of the dual-display concept designs that I’ve seen take up a whole screen with a virtual keyboard. I’m sure people in focus groups ask for this. But I still think it’s a really bad idea. What is the point of having two screens if you are instantly going to cover one of them with a picture of a keyboard? Your hands immediately occlude it from sight anyway. At that point I’m basically using a single screen again, so I might as well just grab my laptop to get a decent keyboard.

Text entry on the Codex

The Codex offers a solution for text entry that I much prefer, by virtue of the OQO Model 02’s mechanical keyboard. I just slide the screen up and that reveals the keyboard when I need it. Works great. No dorky touchscreen keyboard. I still enjoy the full benefits of partitioning my work between the two screens. That’s the way I like it.

When Can I Buy One?

The Codex is a prototype– and a rather flaky, cobbled-together one at that. But it uses off-the-shelf devices, and there’s nothing magical about the software. So the crass answer is that you can have one now if you are willing to spend some dollars, build yourself a custom binder, and write a little bit of code. That’s how I started. My first prototype was a repurposed day-planner with Velcro holding the screens in there. Install a shared clipboard utility and you can start copying and pasting between screens. That will give you just enough of the experience that you will hunger for more.

What would you want out of a dual-screen device? What capabilities would make it most useful to you? How do you see dividing your own work between two screens?

In the meantime, the trajectory of ultra low-power “e-ink” displays bears watching. Check out the recent Plastic Logic device, for example. Right now e-ink is an abomination for anything interactive, but eventually some display technology will get where we want it to go. Low-cost, low-power screens are crucial to make dual display devices a practical consumer device, rather than a research lab curiosity.

The sheer number of concept devices that have popped up in the last 6 months suggests that dual-screen devices are poised to take off in the near future. My hope is that our research on the Codex can help in some small way to unearth the full promise of such devices. I, for one, am convinced that dual-display devices have a well-motivated role to play in a future ecosystem of mobile devices.

In my office, at least, that future is already here. It’s just not evenly distributed.


I would like to thank the following key contributors to this project:

  • Morgan Dixon
  • Raman Sarin
  • Francois Guimbretiere
  • Ravin Balakrishnan

The Codex dual-screen tablet computerHinckley, K., Dixon, M., Sarin, R., Guimbretiere, F., and Balakrishnan, R. 2009. Codex: a Dual-Screen Tablet Computer. In Proce. CHI 2009 Conf. on Human Factors in Computing Systems, Boston, MA, pp. 1933-1942. [PDF] [video .MOV] [OfficeLabs Thought Leadership Award]

9 responses to “Classic AlpineInker Post #1: The Microsoft Research Codex

  1. Thanks, Aaron, much appreciated.

    I still believe dual-screen devices have an important future. It’s probably not prime time for them quite yet though, the single-screen ipad-like devices have the limelight for now. But 10 or 20 years from now we’ll all be carrying around nice little leather portfolios that just fit in a pocket and let us read and write and post inane handwritten comments and doodles on facebook and twitter :-)

    At least, that’s the way it is in my science fiction stories :-)

  2. It’s a good thing that MS has patented the gestures from the “Manual Deskterity” research you worked on. At least we are moving in the right direction to such a device!

    I am currently using the HTC Flyer and I am mildly satisfied. Overall the Flyer seems like a toy and its not what I really want/need. It’s a shame HTC didn’t integrate their pen a little bit more with the interface.

    Thanks for responding!

    PS if you have any tablets laying around I’m more than willing to test them for you ;-)

    PPS If you made a future techy comic I would SOOOOO read it!

  3. I have filed somewhere around 100 patents in my career (to be honest, I have lost count in recent years) and while many of those, I’m sure, will turn out to be useless, there are probably some real gems in there too :-)

    I’ve played with the HTC Flyer a bit, but did not get excited enough that I felt like I needed to have one for heavier use. My ardent personal wish is that Amazon will come out with an e-reader / tablet that you can write on. I have dozens of tablet PC’s and an iPad– so I have my pick of devices– and I do find that I like the Kindle best for the reading experience.

    Hmmm, maybe I should self-publish a high-tech comic. I guess I’d have to call it “The Pen is Mightier Than the Touchscreen” or somesuch :-)

  4. Hi ken,
    I really enjoy reading your blog. I Completely agree that the pen is mightier than the touchscreen. I recently bought my first tablet PC, an Asus eee Slate EP121. I love it, I love writing on the screen and taking notes in OneNote. The first app that I installed on my eee slate was Inkseine and I love using it. What is the future of Inkseine? How is it different to Windows Journal? Thanks

  5. Hi Rami,
    Thanks and welcome into the fold :-)

    Glad to hear you are enjoying the EP121. There are a lot of things to like about it.

    For several years I used a TabletKiosk Sahara i440d slate running Vista as my main “laptop” and in fact for a while there I did all my talks on it, often using InkSeine. The i440d came with a nice folding stand which made that work really well, and it was great to be able to pause and mark up slides at any point or flip over to InkSeine to do demos or find something I needed.

    A lot of people like to make fun of Vista, but I still think it is the best OS for a tablet PC. Many things were changed in Windows 7, but at least as far as pen input on Tablet PC, not all were for the better in my opinion, at least. For example, the new TIP (Tablet Input Panel) has never grown on me because I hate how it immediately converts all the pen strokes to text; I found it much easier to fix mistakes when the pen strokes stayed there for a while, as they did in the Vista version of the TIP.

    As for InkSeine: that was a “concept car” type of program that we did for tablets to try and inject some new ideas and excitement into the Tablet PC landscape. In some ways it was similar to Journal, but the controls are really custom built for a pen and there is some functionality in there, like being able to circle handwritten phrases to jump-start a Personal Search (desktop search of all the files on your tablet) or the ability to easily grab snippits of content using the Tool Ring that make it unique.

    InkSeine is not being actively developed any more. The developer who was my partner on that project moved on to new things several years ago, actually, and developing robust and efficient code that’s suitable for public deployment is really not my forte (or in my job description, for that matter). It pains me to let it languish, but there’s really nothing I can do about it at this point.

    It’s still a pretty cool program and does a lot of cool things, even though some cracks of age are starting to appear (for example, the Web search feature no longer works, the feature that back-links web and Office document clippings back to the source may not work depending on which version of IE / Office you have, and so forth).

    I’d also really like to incorporate some of our Pen + Touch ideas in InkSeine, but that would be a big change.

    And actually there’s a much bigger and much longer story hiding in there, but that is a tale for another day…

  6. Pingback: Interesting New Tablets from Sony | The Past and Present Future

  7. Pingback: The Fractured State of Reading and Publishing | The Past and Present Future

  8. Pingback: Conference Notes: Workshop on the Impact of Pen & Touch Technology on Education (WIPTTE) | CID Spotlight

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s