Symposium Abstract: Issues in bimanual coordination: The props-based interface for neurosurgical visualization

I have a small backlog of updates and new posts to clear out, which I’ll be undertaking in the next few days.

The first of these is the following small abstract that actually dates from way back in 1996, shortly before I graduated with my Ph.D. in Computer Science from the University of Virginia.

It was a really fun symposium organized by the esteemed Yves Guiard, famous for his kinematic chain model of human bimanual action, that included myself and Bill Buxton, among others. For me this was a small but timely recognition that came early in my career and made it possible for me to take the stage alongside two of my biggest research heroes.

Thumbnail for Symposium on Human Bimanual SpecializationHinckley, K., 140.3: Issues in bimanual coordination: The props-based interface for neurosurgical visualization. Appeared in Symposium 140: Human bimanual specialization: New perspectives on basic research and application, convened by Yves Guiard, Montréal, Quebec, Canada, Aug. 17, 1996. Abstract published in  International Journal of Psychology, Volume 31, Issue 3-4, Special Issue: Abstracts of the XXVI INTERNATIONAL CONGRESS OF PSYCHOLOGY, 1996. [PDF – Symposium 140 Abstracts]


I will describe a three-dimensional human-computer interface for neurosurgical visualization based on the bimanual manipulation of real-world tools. The user’s nonpreferred hand holds a miniature head that can be “sliced open” or “pointed to” using a cross-sectioning plane or a stylus held in the preferred hand. The nonpreferred hand acts as a dynamic frame-of-reference relative to which the preferred hand articulates its motion. I will also discuss experiments that investigate the role of bimanual action in virtual manipulation and in the design of human-computer interfaces in general.

Contribute to MobileHCI 2015 and Help Advance the Frontiers of Mobility: Submissions Due Feb 6th, 2015.

Mobile HCI 2015 bannerSend us your work. If it makes us go “Wow!” we want it.

Along with Hans Gellersen of Lancaster University (UK), I’m proud to announce that I’m co-chairing the papers selection committee for the 2015 installment of the long-running MobileHCI conference (sponsored by the ACM and SIGCHI), to take place Aug 24th-Aug 27th, 2015, in wonderful and historic Copenhagen, Denmark.

MobileHCI is the premiere venue to publish and learn about state-of-the-art innovations and insights for all aspects of human-computer interaction as it pertains to mobility–whether in terms of the devices we use, the services we engage with, or the new patterns of human behavior emerging from the wilderness of the modern-day digital ecology.

Submissions due Feb 6th, 2015.

Call for Papers

MobileHCI seeks contributions in the form of innovations, insights, or analyses related to human experiences with mobility.

Our interpretation of mobility is inclusive and broadly construed. Likewise, our view of contribution encompasses technology, experience, methodology, and theory—or any mix thereof, and beyond. We seek richness and diversity in topic as well as approach, method, and viewpoint. If you can make a convincing case that you have something important to say about mobility, in all its many forms, we want to see your work.

In no particular order, this includes contributions in the form of:

Systems & infrastructures. The design, architecture, deployment, and evaluation of systems and infrastructures that support development of or interaction with mobile devices and services.

Devices & techniques: The design, construction, usage, and evaluation of devices and techniques that create valuable new capabilities for mobile human-computer interaction.

Applications & experiences. Descriptions of the design, empirical study of interactive applications, or analysis of usage trends that leverage mobile devices and systems.

Methodologies & tools. New methods and tools designed for or applied to studying or building mobile user interfaces, applications, and mobile users.

Theories & models. Critical analysis or organizing theory with clearly motivated relevance to the design or study of mobile human-computer interaction; taxonomies of design or devices; well-supported essays on emerging trends and practice in mobile human-computer interaction.

Visions & wildcards. Well-argued and well-supported visions of the future of mobile computing; non-traditional topics that bear on mobility; under-represented viewpoints and perspectives that convincingly bring something new to mobile research and practice. Surprise us with something new and compelling.

We seek contribution of ideas, as opposed to convention of form.

If you write a good paper—present clear, well-argued and well-cited ideas that are backed up with some form of compelling evidence (proof-of-concept implementations, system demonstrations, data analysis, user studies, or whatever methodology suits the contribution you are trying to make)—then we want to see your work, and if we agree it is good, we will accept it.

We are not particularly picky about page lengths or the structure of papers. Use the number of pages you need to convey a contribution, no more, no less.

Reviewers traditionally expect about 4pp for shorter contributions, and about 10pp for long-form contributions, but these are simply guideposts of what authors most commonly submit.

If you have a great 10 page paper with an intriguing set of ideas and the references spill over onto page 12, we are happy with that.

If you can convey a solid idea in 8 pages, that is fine too.

Or a four-pager with a clearly articulated nugget of contribution is always welcome.

Finally, keep the “Wow!” test in mind.

We are always happy to consider thought-provoking work that might not be perfect but clearly does inject new ideas into the discourse on mobile interaction, what it is now, what it could be in the future.

We would rather have 10 thought-provoking papers that break new ground in their own unique ways, than that one perfect paper that is dull and unassailable.

Send us your work. If it makes us go “Wow!” we want it. By the same token there is nothing wrong with solid work that advances the state of the art. We are excited to expand the many frontiers of mobility and we need your contributions to help us get there.

You can find full details in the online call for or papers at the MobileHCI 2015 website.

And be sure to spread the word to your peers and collaborators so that we can have a rich conference programme with a great diversity of neat projects and results to showcase the cutting edge of mobility.

Interacting with the Undead: A Crash Course on the “Inhuman Factors” of Computing

I did a far-ranging interview last week with Nora Young, the host of CBC Radio’s national technology and trend-watching show called Spark.

But the most critical and timely topic we ventured into was the burning question on everyone’s mind as All Hallows’ Eve rapidly approaches:

Can zombies use touchscreens?

This question treads (or shall we say, shambles) into the widely neglected area of Inhuman Factors, a branch of Human-Computer Interaction that studies technological affordances for the most disenfranchised and unembodied users of them all–the undead.

Fortunately for Nora, however, I am the world’s foremost authority on the topic.

And I was only too happy to speak to this glaring oversight in how we design today’s technologies, one that I have long campaigned to redress.

Needless to say, Zombie-Computer Interaction (ZCI) is an area rife with dire usability problems.

You can listen to the podcast and see how Nora sparked the discussion here.

But to clear up some common myths and misconceptions of ZCI, let me articulate seven critical design observations to keep in mind when designing technology for the undead:

  1.  Yes, zombies can use touchscreens–with appropriate design.
  2. Thus, like everything else in design, the correct answer is:
    “It Depends.”
  3. The corpse has to be fresh. Humans are essentially giant bags of water; touchscreens are sensitive to the capacitance induced by the moisture in our bodies. So long as the undead creature has recently departed the realm of the living, then, the capacitive touchscreens commonplace in today’s technology should respond appropriately.
  4. Results also may be acceptable if the zombie has fed on a sufficient quantity of brains in the last 24-36 hours.
  5. MOAR BRAINS! are better.
  6. Nonetheless, the water content of a motive corpse can be a significant barrier in day-to-day (or, to speak more precisely, night-to-night) interactions of the undead with tablets, smartphones, bank kiosks, and the like. In particular, touchscreens often completely fail to respond to mummies, ghasts, vampires, and the rarely-studied windigo of Algonquian legend–all due to the extreme desiccation of the corporeal form.
  7. Fortunately for these dried-up souls, the graveyard of devices-past is replete with resistive touchscreen technology such as the once-revered Palm Pilot handheld computer, as document in the frightening and deeply disturbing Buxton Collection of Input Devices and Technologies. These devices respond successfuly to the finger-taps of the desiccated undead because they sense contact pressure, not capacitance.

So let me recap the lessons:
Zombies can definitely use touchscreens; brains are good, MOAR BRAINS are better; and if you see a zombie sporting a Palm Pilot run like hell, because that sucker is damned hungry.

But naturally, the ground-breaking discussion on Zombie-Computer Interaction sparked by Nora’s provocation has triggered a flurry of follow-on questions from concerned citizens to my inbox:

What about ghosts? Can a ghost use a touchscreen?

A ghost is an unholy manifestation of non-corporeal form. Lacking an embodied form, a ghost therefore cannot use a touchscreen–their hand passes right through it. But ghosts can be sensed by light, such as laser rangefinders, or the depth-sensing technology of the Kinect camera for the XBox.

However, ghosts frequently can and do leave behind traces of ectoplasmic goo, which can cause touchscreens to respond in a strange and highly erratic manner.

If you have ever made a typo on a touchscreen keyboard, or triggered Angry Birds by accident when you could swear you were reaching for some other icon–chances are that “ghost contact” was triggered by a disembodied spirit trying to communicate with you from the beyond.

If this happens to you, I highly recommend that you immediately stop what you are doing and install every touchscreen Ouija board app you can find so that you can open a suitable communication channel with the realm of the dead.

What about Cthulu–H. P. Lovecraft’s terrifying cosmic deity that is part man, part loathsome alien form, and part giant squid? Can Cthulu use a touchscreen?

Studies are inconclusive. Scott’s great expedition to the Transantarctic mountains–where records of Cthulu are rumored to be hidden–vanished in the icy wastes, never to be heard from again. R. Carter et al. studied the literature extensively and promptly went insane.

Other researchers, including myself, have been understandably dissuaded from examining the issue further.

My opinion, unsupported by data, is that as a pan-dimensional being Cthulu can touch whatever the hell he wants–when the stars are right and the lost city of R’lyeh rises once again from the slimy eons-deep vaults of the black Pacific.

A lot of PEOPLE are WORRIED about Lawyers. Can lawyers use touchscreens as well?

Sadly, it is widely believed (and backed up by scientific studies) that most lawyers have no soul.

Therefore the majority of lawyers cannot use a touchscreen at all.

This is why summons and lawsuits always arrive in paper form from a beady-eyed courier.


Other noteworthy challenges to conventional INHUMAN FACTORS design wisdom

I’ve also fielded a variety of questions and strongly-held opinions from the far and dark corners of the Twittersphere.

Needless to say, these are clearly highly disturbed individuals, so I recommend that you interact with them at your own risk.

All right. I think I’ve put this topic to rest.

But keep the questions coming.

And be careful tonight.

Be sure to post in the comments below, or tweet me after midnight @ken_hinckley and I’ll do my best to give you a scientifically rigorous (if not rigor-mortis-ish) response.

Paper: LightRing: Always-Available 2D Input on Any Surface

In this modern world bristling with on-the-go-go-go mobile activity, the dream of an always-available pointing device has long been held as a sort of holy grail of ubiquitous computing.

Ubiquitous computing, as futurists use the term, refers to the once-farfetched vision where computing pervades everything, everywhere, in a sort of all-encompassing computational nirvana of socially-aware displays and sensors that can respond to our every whim and need.

From our shiny little phones.

To our dull beige desktop computers.

To the vast wall-spanning electronic whiteboards of a future largely yet to come.

How will we interact with all of these devices as we move about the daily routine of this rapidly approaching future? As we encounter computing in all its many forms, carried on our person as well as enmeshed in the digitally enhanced architecture of walls, desktops, and surfaces all around?

Enter LightRing, our early take on one possible future for ubiquitous interaction.

LightRing device on a supporting surface

By virtue of being a ring always worn on the finger, LightRing travels with us and is always present.

By virtue of some simple sensing and clever signal processing, LightRing can be supported in an extremely compact form-factor while providing a straightforward pointing modality for interacting with devices.

At present, we primarily consider LightRing as it would be configured to interact with a situated display, such as a desktop computer, or a presentation projected against a wall at some distance.

The user moves their index finger, angling left and right, or flexing up and down by bending at the knuckle. Simple stuff, I know.

But unlike a mouse, it’s not anchored to any particular computer.

It travels with you.

It’s a go-everywhere interaction modality.

Close-up of LightRing and hand angles inferred from sensors

Left: The degrees-of-freedom detected by the LightRing sensors. Right: Conceptual mapping of hand movement to the sensed degrees of freedom. LightRing then combines these to support 2D pointing at targets on a display, or other interactions.

LightRing can then sense these finger movements–using a one-dimensional gyroscope to capture the left-right movement, and an infrared sensor-emitter pair to capture the proximity of the flexing finger joint–to support a cursor-control mode that is similar to how you would hold and move a mouse on a desktop.

Except there’s no mouse at all.

And there needn’t even be a desktop, as you can see in the video embedded below.

LightRing just senses the movement of your finger.  You can make the pointing motions on a tabletop, sure, but you can just as easily do them on a wall. Or on your pocket. Or a handheld clipboard.

All the sensing is relative so LightRing always knows how to interpret your motions to control a 2D cursor on a display. Once the LightRing has been paired with a situated device, this lets you point at targets, even if the display itself is beyond your physical reach. You can sketch or handwrite characters with your finger–another scenario we have explored in depth on smartphones and even watches.

The trick to the LightRing is that it can automatically, and very naturally, calibrate itself to your finger’s range of motion if you just swirl your finger. From that circular motion LightRing can work backwards from the sensor values to how your finger is moving, assuming it is constrained to (roughly) a 2D plane. And that, combined with a button-press or finger touch on the ring itself, is enough to provide an effective input device.

The LightRing, as we have prototyped it now, is just one early step in the process. There’s a lot more we could do with this device, and many more practical problems that would need to be resolved to make it a useful adjunct to everyday devices–and to tap its full potential.

But my co-author Wolf Kienzle and I are working on it.

And hopefully, before too much longer now, we’ll have further updates on even more clever and fanciful stuff that we can do through this one tiny keyhole into this field of dreams, the verdant golden country of ubiquitous computing.


LightRing thumbnailKienzle, W., Hinckley, K., LightRing: Always-Available 2D Input on Any Surface. In the 27th ACM Symposium on User Interface Software and Technology (UIST 2014), Honolulu, Hawaii, Oct. 5-8, 2014, pp. 157-160. [PDF] [video.mp4 TBA] [Watch on YouTube]

Watch LightRing video on YouTube

Project: The Analog Keyboard: Text Input for Small Devices

With the big meaty man-thumbs that I sport, touchscreen typing–even on a full-size tablet computer–can be challenging for me.

Take it down to a phone, and I have to spend more time checking for typographical errors and embarrassing auto-miscorrections than I do actually typing in the text.

But typing on a watch?!?

I suppose you could cram an entire QWERTY layout, all those keys, into a tiny 1.6″ screen, but then typing would become an exercise in microsurgery, the augmentation of a high-power microscope an absolute necessity.

But if you instead re-envision ‘typing’ in a much more direct, analog fashion, then it’s entirely possible. And in a highly natural and intuitive manner to boot.

Enter the Analog Keyboard Project.

Analog Watch Keyboard on Moto 360 (round screen)

Wolf Kienzle, a frequent collaborator of mine, just put out an exciting new build of our touchscreen handwriting technology optimized for watches running the Android Wear Platform, including the round Moto 360 device that everyone seems so excited about.

Get all the deets–and the download–from Wolf’s project page, available here.

This builds on the touchscreen writing prototype we first presented at the MobileHCI 2013 conference, where the work earned an Honorable Mention Award, but optimized in a number of ways to fit on the tiny screen (and small memory footprint) of current watches.

All you have to do is scrawl the letters that you want to type–in a fully natural manner, not in some inscrutable secret computer graffiti-code like in those dark days of the late 1990’s–and the prototype is smart enough to transcribe your finger-writing to text.

It even works for numbers and common punctuation symbols like @ and #, indispensable tools for the propagation of internet memes and goofy cat videos these days.

Writing numbers and punctuation symbols on the Analog Keyboard

However, to fit the resource-constrained environment of the watch, the prototype currently only supports lowercase letters.

Because we all know that when it comes to the internet, UPPERCASE IS JUST FOR TROLLZ anyway.

Best of all, if you have an Android Wear device you can try it out for yourself. Just side-load the Analog Keyboard app onto your watch and once again you can write the analog way, the way real men did in the frontier days. Before everyone realized how cool digital watches were, and all we had to express our innermost desires was a jar of octopus ink and a sharpened bald eagle feather. Or something like that.

Y’know, the things that made America great.

Only now with more electrons.

You can rest easy, though, if these newfangled round watches like the Moto 360 are just a little bit too fashionable for you. As shown below, it works just fine on the more chunky square-faced designs such as the Samsung Gear Live as well.

Analog Keyboard on Samsung Gear Live watch

Check out the video embedded below, and if you have a supported Android Wear device, download the prototype and give it a try. I know Wolf would love to get your feedback on what it feels like to use the Analog Keyboard for texting on your watch.

Bring your timepiece into the 21st century.

You’ll be the envy of every digital watch nerd for miles around.

Besides: it’s clearly an idea whose time has come.

Thumbnail - Analog Keyboard ProjectKienzle, W., Hinckley, K., The Analog Keyboard Project. Handwriting keyboard download for Android Wear. Released October 2014. [Project Details and Download] [Watch demo on YouTube]


Watch Analog Keyboard video on YouTube

Nature Futures 2 Anthology, featuring “The Ostracons of Europa”

Leading speculative fiction publisher Tor Books has just come out with Nature Futures 2, an anthology of 100 provocative science-fictional visions of the future. (Available on Kindle and the other usual suspects.)

And that cover! Pretty darned spectacular:

Nature Futures 2 (front cover) (Credit: JACEY

Credit: JACEY

The editors, Colin Sullivan and Henry Gee, hand-picked their favorite stories for this anthology, all drawn from the last several  years of the award-winning Futures column from Nature.

The anthology thereby features many award winning authors, from Elizabeth Bear and Rachel Swirsky to Gregory Benford and Mike Resnick, to a personal favorite short-story writer of mine, the inimitable Ken Liu.

And I’m thrilled to say that my contribution, The Ostracons of Europa, made the cut as well!

The Ostracons of Europa (book cover)


The anthology is only available as an e-book, but just for fun, and by way of celebration, I put together a special print edition of The Ostracons of Europa available as a stand-alone story–a collector’s item of sorts.

It’s a very short story and makes for a very short book, but what the heck.

The paper-book format draws out the tension of the story–by judicious use of chapter breaks–plus it’s hard to beat the feeling of riffling through those creamy antique-white sheets of finely compacted pulp…

But just to be safe, and since in my scientific work I push at the crackling-with-electricity and fuming-with-sulfur frontiers of technology, the book also includes a coupon code so you can download a free electronic edition at your leisure :-)

Paper: Experimental Study of Stroke Shortcuts for a Touchscreen Keyboard with Gesture-Redundant Keys Removed

Text Entry on Touchscreen Keyboards: Less is More?

When we go from mechanical keyboards to touchscreens we inevitably lose something in the translation. Yet the proliferation of tablets has led to widespread use of graphical keyboards.

You can’t blame people for demanding more efficient text entry techniques. This is the 21st century, after all, and intuitively it seems like we should be able to do better.

While we can’t reproduce that distinctive smell of hot metal from mechanical keys clacking away at a typewriter ribbon, the presence of the touchscreen lets keyboard designers play lots of tricks in pursuit of faster typing performance. Since everything is just pixels on a display it’s easy to introduce non-standard key layouts. You can even slide your finger over the keys to shape-write entire words in a single swipe, as pioneered by Per Ola Kristensson and Shumin Zhai (their SHARK keyboard was the predecessor for Swype and related techniques).

While these type of tricks can yield substantial performance advantages, they also often demand a substantial investment in skill acquisition from the user before significant gains can be realized. In practice, this limits how many people will stick with a new technique long enough to realize such gains. The Dvorak keyboard offers a classic example of this: the balance of evidence suggests it’s slightly faster than QWERTY, but the high cost of switching to and learning the new layout just isn’t worth it.

In this work, we explored the performance impact of an alternative approach that builds on people’s existing touch-typing skills with the standard QWERTY layout.

And we do this in a manner that is so transparent, most people don’t even realize that anything is different at first glance.

Can you spot the difference?

Snap quiz time


What’s wrong with this keyboard?  Give it a quick once-over. It looks familiar, with the standard QWERTY layout, but do you notice anything unusual? Anything out of place?

Sure, the keys are arranged in a grid rather than the usual staggered key pattern, but that’s not the “key” difference (so to speak). That’s just an artifact of our quick ‘n’ dirty design of this research-prototype keyboard for touchscreen tablets.

Got it figured out?

All right. Pencils down.

Time to check your score. Give yourself:

  • One point if you noticed that there’s no space bar.
  • Two points if you noticed that there’s no Enter key, either.
  • Three points if the lack of a Backspace key gave you palpitations.
  • Four points and a feather in your cap if you caught the Shift key going AWOL as well.

Now, what if I also told you removing four essential keys from this keyboard–rather than harming performance–actually helps you type faster?


All we ask of people coming to our touchscreen keyboard is to learn one new trick. After all, we have to make up for the summary removal of Space, Backspace, Shift, and Enter somehow. We accomplish this by augmenting the graphical touchscreen keyboard with stroke shortcuts, i.e. short straight-line finger swipes, as follows:marking-menu-overlay-5

  • Swipe right, starting anywhere on the keyboard, to enter a Space.
  • Swipe left to Backspace.
  • Swipe upwards from any key to enter the corresponding shift-symbol. Swiping up on the a key, for example, enters an uppercase A; stroking up on the 1 key enters the ! symbol; and so on.
  • Swipe diagonally down and to the left for Enter.



In addition to possible time-motion efficiencies of the stroke shortcuts themselves, the introduction of these four gestures–and the elimination of the corresponding keys made redundant by the gestures–yields a graphical keyboard with number of interesting properties:

  • Allowing the user to input stroke gestures for Space, Backspace, and Enter anywhere on the keyboard eliminates fine targeting motions as well as any round-trips necessary for a finger to acquire the corresponding keys.
  • Instead of requiring two separate keystrokes—one to tap Shift and another to tap the key to be shifted—the Shift gesture combines these into a single action: the starting point selects a key, while the stroke direction selects the Shift function itself.
  • Removing these four keys frees an entire row on the keyboard.
  • Almost all of the numeric, punctuation, and special symbols typically relegated to the secondary and tertiary graphical keyboards can then be fit in a logical manner into the freed-up space.
  • Hence, the full set of characters can fit on one keyboard while holding the key size, number of keys, and footprint constant.
  • By having only a primary keyboard, this approach affords an economy of design that simplifies the interface, while offering further potential performance gains via the elimination of keyboard switching costs—and the extra key layouts to learn.
  • Although the strokes might reduce round-trip costs, we expect articulating the stroke gesture itself to take longer than a tap. Thus, we need to test these tradeoffs empirically.


Our studies demonstrated that overall the removal of four keys—rather than coming at a cost—offers a net benefit.

Specifically, our experiments showed that a stroke keyboard with the gesture-redundant keys removed yielded a 16% performance advantage for input phrases containing mixed-case alphanumeric text and special symbols, without sacrificing error rate. We observed these performance advantages from the first block of trials onward.

Even in the case of entirely lowercase text—that is, in a context where we would not expect to observe a performance benefit because only the Space gesture offers any potential advantage—we found that our new design still performed as well as a standard graphical keyboard. Moreover, people learned the design with remarkable ease: 90% wanted to keep using the method, and 80% believed they typed faster than on their current touchscreen tablet keyboard.

Notably, our studies also revealed that it is necessary to remove the keys to achieve these benefits from the gestural stroke shortcuts. If both the stroke shortcuts and the keys remain in place, user hesitancy about which method to use undermines any potential benefit. Users, of course, also learn to use the gestural shortcuts much more quickly when they offer the only means of achieving a function.

Thus, in this context, less is definitely more in achieving faster performance for touchscreen QWERTY keyboard typing.

The full results are available in the technical paper linked below. The paper contributes a careful study of stroke-augmented keyboards, filling an important gap in the literature as well as demonstrating the efficacy of a specific design; shows that removing the gesture-redundant keys is a critical design choice; and that stroke shortcuts can be effective in the context of multi-touch typing with both hands, even though previous studies with single-point stylus input had cast doubt on this approach.

Although our studies focus on the immediate end of the usability spectrum (as opposed to longitudinal studies over many input sessions), we believe the rapid returns demonstrated by our results illustrate the potential of this approach to improve touchscreen keyboard performance immediately, while also serving to complement other text-entry techniques such as shape-writing in the future.

Stroke-Keyboard-GI-2014-thumbArif, A. S., Pahud, M., Hinckley, K., and Buxton, B.,  Experimental Study of Stroke Shortcuts for a Touchscreen Keyboard with Gesture-Redundant Keys Removed In Proc. Graphics Interface 2014 (GI’14).  Canadian Information Processing Society, Toronto, Ont., CanadaMontreal, Quebec, Canada, May 7-9, 2014. Received the Michael A. J. Sweeney Award for Best Student Paper.  [PDF] [Talk Slides (.pptx)] [Video .MP4] [Video .WMV]

Watch A Touchscreen Keyboard with Gesture-Redundant Keys Removed video on YouTube