Category Archives: Large Displays

Paper: Wearables as Context for Guiard-abiding Bimanual Touch

This particular paper has a rather academic-sounding title, but at its heart it makes a very simple and interesting observation regarding touch that any user of touch-screen technology can perhaps appreciate.

The irony is this: when interaction designers talk about “natural” interaction, they often have touch input in mind. And so people tend to take that for granted. What could be simpler than placing a finger — or with the modern miracle of multi-touch, multiple fingers — on a display?

And indeed, an entire industry of devices and form-factors — everything from phones, tablets, drafting-tables, all the way up to large wall displays — has arisen from this assumption.

Yet, if we unpack “touch” as it’s currently realized on most touchscreens, we can see that it remains very much a poor man’s version of natural human touch.

For example, on a large electronic-whiteboard such as the 84″ Surface Hub, multiple people can work upon the display at the same time. And it feels natural to employ both hands — as one often does in a wide assortment of everyday manual activities, such as indicating a point on a whiteboard with your off-hand as you emphasize the same point with the marker (or electronic pen).

Yet much of this richness — obvious to anyone observing a colleague at a whiteboard — represents context that is completely lost with “touch” as manifest in the vast majority of existing touch-screen devices.

For example:

  • Who is touching the display?
  • Are they touching the display with one hand, or two?
  • And if two hands, which of the multiple touch-events generated come from the right hand, and which come from the left?

Well, when dealing with input to computers, the all-too-common answer from the interaction designer is a shrug, a mumbled “who the heck knows,” and a litany of assumptions built into the user interface to try and paper over the resulting ambiguities, especially when the two factors (which user, and which hand) compound one another.

The result is that such issues tend to get swept under the rug, and hardly anybody ever mentions them.

But the first step towards a solution is recognizing that we have a problem.

This paper explores the implications of one particular solution that we have prototyped, namely leveraging wearable devices on the user’s body as sensors that can augment the richness of touch events.

A fitness band worn on the non-preferred hand, for example, can sense the impulse resulting from making finger-contact with a display through its embedded motion sensors (accelerometers and gyros). If the fitness band and the display exchange information and id’s, the touch-event generated can then be associated with the left hand of a particular user. The inputs of multiple users instrumented in this manner can then be separated from one another, as well, and used as a lightweight form of authentication.

That then explains the “wearable” part of “Wearables as Context for Guiard-abiding Bimanual Touch,” the title of my most recent paper, but what the heck does “Guiard-abiding” mean?

Well, this is a reference to classic work by a research colleague, Yves Guiard, who is famous for a 1987 paper in which he made a number of key observations regarding how people use their hands — both of them — in everyday manual tasks.

Particularly, in a skilled manipulative task such as writing on a piece of paper, Yves pointed out (assuming a right-handed individual) three general principles:

  • Left hand precedence: The action of the left hand precedes the action of the right; the non-preferred hand first positions and orients the piece of paper, and only then does the pen (held in the preferred hand, of course) begin to write.
  • Differentiation in scale: The action of the left hand tends to occur at a larger temporal and spatial scale of motion; the positioning (and re-positioning) of the paper tends to be infrequent and relatively coarse compared to the high-frequency, precise motions of the pen in the preferred hand.
  • Right-to-Left Spatial Reference: The left hand sets a frame of reference for the action of the right; the left hand defines the position and orientation of the work-space into which the preferred hand inserts its contributions, in this example via the manipulation of a hand-held implement — the pen.

Well, as it turns out these three principles are very deep and general, and they can yield great insight into how to design interactions that fully take advantage of people’s everyday skills for two-handed (“bimanual”) manipulation — another aspect of “touch” that interaction designers have yet to fully leverage for natural interaction with computers.

This paper is a long way from a complete solution to the paucity of modern touch-screens but hopefully by pointing out the problem and illustrating some consequences of augmenting touch with additional context (whether provided through wearables or other means), this work can lead to more truly “natural” touch interaction — allowing for simultaneous interaction by multiple users, both of whom can make full and complementary use of their hard-won manual skill with both hands — in the near future.

Wearables (fitness band and ring) provide missing context (who touches, and with what hand) for direct-touch bimanual interactions.Andrew M. Webb, Michel Pahud, Ken Hinckley, and Bill Buxton. 2016. Wearables as Context for Guiard-abiding Bimanual Touch. In Proceedings of the 29th Annual ACM Symposium on User Interface Software and Technology (UIST ’16). ACM, New York, NY, USA, 287-300. Tokyo, Japan, Oct. 16-19, 2016.
[PDF] [Talk slides PDF] [Full video – MP4] [Watch 30 second preview on YouTube]

Field Notes from an Expedition into the Paleohistory of Personal Computing

After a time-travel excursion consisting of thirty years in the dusty hothouse of  fiberglass insulation that is my parent’s attic, I’ll be durned if my trusty old TI-99/4A computer didn’t turn up on my doorstep looking no worse for its exotic journey.

Something I certainly wish I could say about myself.

So I pried my fossil from the Jurassic age of personal computing out of the battered suitcase my Dad had shipped it in, and — with the help of just the right connector conjured through the magic of eBay — I was able to connect this ancient microprocessor to my thoroughly modern television, resulting in a wonderful non sequitur of old and new:

TI 994A on my large screen TV

Yep, that’s the iconic home screen from a computer that originally came with a 13″ color monitor — which seemed like an extravagant luxury at the time — but now projected onto the 53″ larger-than-life television in my secret basement redoubt of knotty pine.

This is the computer that got me started in programming, so I suppose I owe my putative status as a visionary (and occasional gadfly) of human-computer interaction to this 16-bit wonder. Its sixteen-color graphics and delightful symphonic sound generators were way ahead of its time.

Of course, when I sat down with my kids and turned it on, Exhibit A of What Daddy’s Old Computer Can Do had to be a reprise of the classic game Alpiner which requires you to spur your doughty 16-bit mountaineer to the top of increasingly treacherous mountains.

In my mind, even after the passage of three decades, I could hear Alpiner’s catchy soundtrack  — which takes excellent advantage of the 99’s sound generators — before I even plugged the cartridge in.

Here’s my seven-year-old daughter taking up the challenge:

Alpiner on the TI-99/4aAlpiner redux after the passage of three decades — and in the hands of a new generation. Unfortunately for our erstwhile mountaineer, he has dodged the rattlesnake only to be clobbered by a rockfall which (if you look closely) can be seen, captured in mid-plummet, exactly one character-row above his ill-fated digital noggin.

Next we moved on to some simple programs in the highly accessible TI-Basic that came with the computer, and (modifying one of the examples in the manual) we ginned up a JACKPOT!!! game.

And yes, the triple exclamation points do make it way, way better.

Here’s one of my 8-year-old twins showing off the first mega-jackpot ever struck, with a stunning payoff of 6,495 imaginary dollars, which my daughter informs me she will spend on rainbow ponies.

Powerball ain’t got nothin’ on that.


My daughter awaits verification from the pit boss while I capture photographic evidence of the first ever mega-jackpot payout for striking five consecutive multipliers with a sixth $ kicker redoubling the bonus.

I’m not quite sure what will come next for our paleontological expedition into this shale of exquisitely preserved microprocessors. My other twin daughter has informed me in no uncertain terms that we must add a unicorn to the jackpot symbols — a project for which extensive research is already underway, despite a chronic lack of funding — and which will presumably make even more dramatic payoffs possible in the near future.

And if I can get the TI’s “Program Recorder” working again — and if enough of the program DNA remains intact on my old cassette tapes — then in Jurassic-Park fashion I also hope to resuscitate some classics that a primeval version of myself coded up, including smash hits such as Skyhop, Rocket-Launch, and Karate Fest!

But with only one exclamation point to tout the excellence of the latter title,  I wouldn’t get your hopes up too much for the gameplay in that one (grin).

Book Chapter: Input/Output Devices and Interaction Techniques, Third Edition

Thumbnail for Computing Handbook (3rd Edition)Hinckley, K., Jacob, R., Ware, C. Wobbrock, J., and Wigdor, D., Input/Output Devices and Interaction Techniques. Appears as Chapter 21 in The Computing Handbook, Third Edition: Two-Volume Set, ed. by Tucker, A., Gonzalez, T., Topi, H., and Diaz-Herrera, J. Published by Chapman and Hall/CRC (Taylor & Francis), May 13, 2014.  [PDF – Author’s Draft – may contain discrepancies]

Project: Bimanual In-Place Commands

Here’s another interesting loose end, this one from 2012, which describes a user interface known as “In-Place Commands” that Michel Pahud, myself, and Bill Buxton developed for a range of direct-touch form factors, including everything from tablets and tabletops all the way up to electronic whiteboards a la the modern Microsoft Surface Hub devices of 2015.

Microsoft is currently running a Request for Proposals for Surface Hub research, by the way, so check it out if that sort of thing is at all up your alley. If your proposal is selected you’ll get a spiffy new Surface Hub and $25,000 to go along with it.

We’ve never written up a formal paper on our In-Place Commands work, in part because there is still much to do and we intend to pursue it further when the time is right. But in the meantime the following post and video documenting the work may be of interest to aficionados of efficient interaction on such devices. This also relates closely to the Finger Shadow and Accordion Menu explored in our Pen +Touch work, documented here and here, which collectively form a class of such techniques.

While we wouldn’t claim that any one of these represent the ultimate approach to command and control for direct input, in sum they illustrate many of the underlying issues, the rich set of capabilities we strive to support, and possible directions for future embellishments as well.

Thumbnail for In-Place CommandsKnies, R. In-Place: Interacting with Large Displays. Reporting on research by Pahud, M., Hinckley, K., and Buxton, B. TechNet Inside Microsoft Research Blog Post, Oct 4th, 2012. [Author’s cached copy of post as PDF] [Video MP4] [Watch on YouTube]

In-Place Commands Screen Shot

The user can call up commands in-place, directly where he is working, by touching both fingers down and fanning out the available tool palettes. Many of the functions thus revealed act as click-through tools, where the user may simultaneously select and apply the selected tool — as the user is about to do for the line-drawing tool in the image above.

Watch Bimanual In-Place Commands video on YouTube