This is my latest project, which I will present tomorrow (May 9th) at the CHI 2012 Conference on Human Factors in Computing Systems.
I’ll have a longer post up about this project after I return from the conference, but for now enjoy the video. I also link to the PDF of our short paper below which has a nice discussion of the motivation and design rationale for this work.
Above all else, I hope this work makes clear that there is still tons of room for innovation in how we interact with the e-readers and tablet computers of the future– as well as in terms of how we consume and manipulate content to produce new creative works.
Hinckley, K., Bi, X., Pahud, M., Buxton, B., Informal Information Gathering Techniques for Active Reading. 4pp Note. In Proc. CHI 2012 Conf. on Human Factors in Computing Systems, Austin, TX, May 5-10, 2012. [PDF]
Sun, M. Cao, X., Song, H., Izadi, S., Benko, H., Guimbretiere, F., Ren, X., and Hinckley, K. Enhancing Naturalness of Pen-and-Tablet Drawing through Context Sensing. In Proc. ACM International Conference on Interactive Tabletops and Surfaces (ITS ’11). ACM, New York, NY, USA, Kobe, Japan, November 13-16, 2011, pp. 212-221. [PDF] [video - WMV].
A while back Bill Buxton put up a brief quotation on billbuxton.com that I like a lot:
Just a thought: Slate computers and e-readers represent a new class of digital appliance– one targeted for casual use. With the growth of this market will emerge a new and long overdue approach to interaction– one that is in keeping with the casual intent and context of such usage and which will complement, rather than replace, interfaces that support more formal and structured activities.
He goes on to make a nice little plug for my InkSeine Tablet PC application, but that’s only a very small part of why I like his quotation.
To me Bill’s comment gets at the essence of the new approach to design and interaction demanded by this emerging class of casual-use devices.
And it’s one that’s already deeply rooted in human behavior, even back in those dark ages before our twilight dreams were set afire by the eerie blue-white glow of our iPad portals to never never land.
I like to call it informal organization. It’s a lot of why paper has remained so pervasive in the so-called “paperless” office of the modern information worker. (You could do a lot worse than to spend a weekend reading The Myth of the Paperless Officeby Sellen & Harper to understand this more deeply).
But now that e-reading and tablets are gaining more and more traction, we’re seeing at least some of those paper-based behaviors and expectations smash head-on into the limitations of current-generation readers and tablets.
I can send my documents to my Kindle to read them away from my desktop; I’ve never really liked doing much reading on my computer. And printing seems wasteful and moreover has a higher transaction cost than just sending the document to my reader– hitting print, selecting the right printer, remembering to pick the auto-staple function, walking down the hall to pick it up, coming back to my office and actually starting into reading before I get interrupted and forget why the hell I printed out the document in the first place…
But on the Kindle or iPad, I can get right into reading my document quicker, but then when that moment comes that I want to mark up some copy-edits, jot an annotation in the margin, highlight a passage, slap a post-it on it with a note about what to do next, thumb back and forth quickly between multiple documents, or god forbid casually toss the document into a pile with other like-minded papers– the workflow breaks down and I am into clumsy workarounds and tricks and little idiosyncratic habits I’ve developed to make the tools work for me as best they can.
This is where the frontier in e-reading experiences currently lies.
The e-ink reader and yes even our fancy-pants multi-touch iPads are going to look like antiquated glowing green-phosphor displays to our eyes a few short years from now.
Because these unmet needs are all about the interaction experience, figuring out what the appropriate digital analogs of these behaviors are, and figuring out what the right input modalities and interaction techniques and workflow are to allow users to express these notions as quickly and casually as they can conceive of them.
I talk about some of this in my recent paper Pen + Touch = New Tools. And a few techniques are shown in the video that goes along with that, although to be honest they are clumsy and have a long way to go to get to where I really think things need to be.
As another example, take a look at the Collections mechanism on the Kindle for organizing your e-books. My friend and former colleague Jeff Pierce wrote about the rather limited Kindle book organization mechanisms on his blog, which was actually what tipped me into this little rant. Just gathering together a half-dozen short stories, or documents that you’ve emailed to yourself, or your read vs. unread books and managing that on an ongoing basis is a huge pain in the ass.
But it doesn’t have to be that way. With the massive shift to electronic reading that the publishing industry is tipping towards, and as more and more “real” work versus casual recreational reading moves to such devices, there’s a huge incentive to get these reading devices and tablets right.
And making something that’s even half as good as the stacks of books and papers and post-its scattered around my desk wouldn’t be a bad start.
Hinckley, K., Wigdor, D., Input Technologies and Techniques. Chapter 9 in The Human-Computer Interaction Handbook – Fundamentals, Evolving Technologies and Emerging Applications, Third Edition, ed. by Jacko, J., Published by Taylor & Francis. To appear. [PDF of author's manuscript - not final]
This is an extensive revision of the 2007 and 2002 editions of my book chapter, and with some heavy weight-lifting from my new co-author Daniel Wigdor, it treats direct-touch input devices and techniques in much more depth. Lots of great new stuff. The book will be out in early 2012 or so from Taylor & Francis – keep an eye out for it!
Alright, here’s another blast from the not-so-distant past: our exploration of combined pen and touch input on the Microsoft Surface.
And this project was definitely a blast. A lot of fun and creative people got involved with the project and we just tried tons and tons of ideas, many that were stupid, many that were intriguing but wrong, and many cool ones that didn’t even make our demo reel. And as is clear from the demo reel, we definitely took a design-oriented approach in this work, meaning that we tried multiple possibilities without focusing too much on which was the “best” design in this work. Or, said another way, I would not advocate putting together a system that has all of the gestures that we explored in this work; but you can’t put together a map if you don’t explore the terrain, and this was most definitely a mapping expedition.
Since I did this original post, I’ve published a more definitive paper on the project called “Pen + Touch = New Tools” which appeared at the ACM UIST 2010 Symposium on User Interface Software and Technology. This is a paper I’m proud of; it really dissects this design space of pen + touch quite nicely. I’ll have to do another post about this work that gets into that next level of design nuances at some point.
I had a blast preparing the talk for this particular paper and to be honest it was probably one of the most entertaining academic talks that I’ve done in recent years. I have a very fun way of presenting this particular material, with help during the talk from a certain Mr. I.M.A. Bigbody:
Mr. Bigbody, a Corporate Denizen of Third Rate, Inc., is exactly the sort of arrogant prove-it-to-me, you’re-just-wasting-my-time sort of fellow that seems to inhabit every large organization.
Well, Mr. Bigbody surfaces from time to time throughout my talk to needle me about the shortcomings of the pen:
Why the pen? I can type faster than I can write.
Just tell me which is best, touch or pen.
Touch and pen are just new ways to control the mouse, so what’s the big deal?
And in the end, of course, because the good guys always win, Mr. Bigbody gets sacked and the world gets to see just how much potential there is in combined Pen + Touch input for the betterment of mankind.
One other comment about this work before I turn it over to the classic post. We originally did this work on the Microsoft Surface, because at the time this was the only hardware platform available to us where we could have full multi-touch input while also sensing a pen that we could distinguish as a unique type of contact. This is a critical point. If you can’t tell the pen from any other touch– as is currently a limitation of capacitive multi-touch digitizers such as those used on the iPad– it greatly limits the type of pen + touch interactions that a system can support.
These days, though, a number of slates and laptops with pen + touch input are available. The Asus EP121 Windows 7 slate is a noteworthy example; this particular slate contains a Wacom active digitizer for high-quality pen input, and it also includes a second digitizer with two-touch multi-touch input. The really cool thing about it from my perspective is that you can also use Wacom’s multi-touch API’s to support simultaneous pen + touch input on the device. This normally isn’t possible under Windows 7 because Windows turns off touch when the pen comes in range. But it is possible if you use Wacom’s multi-touch API and handle all the touch events yourself, so you can do some cool stuff if you’re willing to work at it.
Which gets us back to the Manual Deskterity demo on the Surface. To be honest, the whole theme in the video about the digital drafting table is a bit of a head fake. I was thinking slates the whole time I was working on the project, it just wasn’t possible to try the ideas in a slate form factor at the time. But that’s definitely where we intended to go with the research. And it’s where we still intend to go, using devices like the Asus EP121 to probe further ahead and see what other issues, techniques, or new possibilities arise.
Because I’m still totally convinced that combined pen and touch is the way of the future. It might not happen now, or two years from now, or even five years from now– but the device of my dreams, the sleek devices that populate my vision of what we’ll be carrying around as the 21st century passes out of the sun-drenched days of its youth– well, they all have a fantastic user experience that incorporates both pen and touch, and everyone just expects things to work that way.
Even Mr. Bigbody.
Manual Deskterity: An Exploration of Simultaneous Pen + Touch Direct Input
With certain obvious multi-touch devices garnering a lot of attention these days, it’s easy to forget that touch does not necessarily make an interface “magically delicious” as it were. To paraphrase my collaborator Bill Buxton, we have to remember that:
Next week at the annual CHI 2010 Conference on Human Factors in Computing Systems, I’ll be presenting some new research that investigates the little-explored area of simultaneous pen and touch interaction.
Now, what does this really mean? Building on the message we have articulated in black and white above, we observe the following:
The future of direct interaction on displays is not about Touch.
Likewise, it is not about the Pen.
Nor is it about direct interaction on displays with Pen OR Touch.
It is about Pen AND Touch, simultaneously, designed such that one complements the other.
That is, we see pen and touch as complementary, not competitive, modalities of interaction. By leveraging people’s natural use of pen and paper, in the real world, we can design innovative new user experiences that exploit the combination of pen and multi-touch input to support non-physical yet natural and compelling interactions.
Our research delves into the question of how one should use pen and touch in interface design. This really boils down to three questions: (1) What is the role of the pen? (2) What is the role of multi-touch? And (3) What is the role of simultaneous pen and touch? The perspective that we have arrived at in our research is the following: the pen writes, touch manipulates, and the
combination of pen + touch yields new tools:
I’ve now posted a video of the research on YouTube that shows a bunch of the techniques we explored. We have implemented these on the Microsoft Surface, using a special IR-emitting pen that we constructed. However, you can imagine this technology coming to laptops, tablets, and slates in the near future– the N-Trig hardware on the Dell-XT2, for example, already has this capability, although as a practical matter it is not currently possible to author applications that utlize simultaneous pen and touch; hence our exploration of the possibilities on the Microsoft Surface.
The name, of course, is a simple pun on “Manual Dexterity” – in the context of shuffling papers and content on a “digital desk” in our case. Hence “manual deskterity” would be the metric of efficacy in paper-shuffling and other such activities of manual organization and arrangement of documents in your workspace. This name also has the virtue that it shot a blank on <name your favorite search engine>. Plus I have a weakness for unpronounceable neologisms.
Special thanks to my colleagues (co-authors on the paper) who contributed to the work, particularly Koji Yatani, who contributed many of the novel ideas and techniques and did all of the heavy lifting in terms of the technical implementation:
Koji Yatani (Microsoft Research Intern, Ph.D. from the University of Toronto, and as of 2011 a full-time employee at Microsoft Research’s Beijing lab)
Song, H., Benko, H., Guimbretiere, F., Izadi, S., Cao, X., Hinckley, K., Grips and Gestures on a Multi-Touch Pen, In Proc. CHI 2011 Conf. on Human Factors in Computing Systems. [PDF] [video .WMV]
Hinckley, K., Yatani, K., Pahud, M., Coddington, N., Rodenhouse, J., Wilson, A., Benko, H., Buxton, B., Pen + Touch = New Tools. In Proc. UIST 2010 Symposium on User interface Software and Technology, New York, NY, pp. 27-36. [PDF] [video .WMV]
Hinckley, K., Pahud, M., Buxton, B., Direct Display Interaction via Simultaneous Pen + Multi-touch Input. In Society for Information Display (SID) Symposium Digest of Technical Papers, May 2010, Volume 41(1), Session 38, pp. 537-540. [PDF]
This paper had no accompanying video, but you can see the system in action in this YouTube video:
Hinckley, K., Yatani, K., Pahud, M., Coddington, N., Rodenhouse, J., Wilson, A., Benko, H., Buxton, B., Manual Deskterity: An Exploration of Simultaneous Pen + Touch Direct Input. In CHI 2010 Extended Abstracts: Proc. CHI 2010 Conf. on Human Factors in Computing Systems, Atlanta, GA, pp. 2793-2802. [PDF] [video .WMV]
Grossman, T., Baudisch, P., and Hinckley, K. 2009. Handle Flags: Efficient and Flexible Selections for Inking Applications. In Proc. Graphics Interface 2009, Kelowna, British Columbia, Canada, May 25 – 27, 2009. Canadian Information Processing Society, Toronto, Ont., Canada, pp. 167-174. [PDF] [video .AVI]
Hinckley, K., Bi, X., Pahud, M., Buxton, B., Informal Information Gathering Techniques for Active Reading. 4pp Note. In Proc. CHI 2012 Conf. on Human Factors in Computing Systems, Austin, TX, May 5-10, 2012. [PDF] 
