And for a proper scientific write-up and analysis, see our Science Robotics cover article, The uncanny valley of haptics. Note the main article is short (per the format favored by Science) but the supplementary materials go into considerable depth. The full data, materials, and Azure notebook for the analysis are available as well.
naturematerials also featured this work (“Zombie Materials“) but alas their article is paywalled.
Christopher C. Berger, Mar Gonzalez-Franco, Eyal Ofek and Ken Hinckley, The uncanny valley of haptics. Science Robotics, 2018. 3(17). https://doi.org/10.1126/scirobotics.aar7010
Scientific American Observations If (Virtual) Reality Feels Almost Right, It’s Exactly Wrong: How adding touch to VR can lead to an “uncanny valley” of sensations—and what we can do about it.
Mar Gonzalez-Franco, Christopher C Berger, and Ken Hinckley. April 19, 2018.
A general-interest overview of the fascinating sensory illusions revealed by our Science Robotics paper The uncanny valley of haptics.
Text Entry on Touchscreen Keyboards: Less is More?
When we go from mechanical keyboards to touchscreens we inevitably lose something in the translation. Yet the proliferation of tablets has led to widespread use of graphical keyboards.
You can’t blame people for demanding more efficient text entry techniques. This is the 21st century, after all, and intuitively it seems like we should be able to do better.
While we can’t reproduce that distinctive smell of hot metal from mechanical keys clacking away at a typewriter ribbon, the presence of the touchscreen lets keyboard designers play lots of tricks in pursuit of faster typing performance. Since everything is just pixels on a display it’s easy to introduce non-standard key layouts. You can even slide your finger over the keys to shape-write entire words in a single swipe, as pioneered by Per Ola Kristensson and Shumin Zhai (their SHARK keyboard was the predecessor for Swype and related techniques).
While these type of tricks can yield substantial performance advantages, they also often demand a substantial investment in skill acquisition from the user before significant gains can be realized. In practice, this limits how many people will stick with a new technique long enough to realize such gains. The Dvorak keyboard offers a classic example of this: the balance of evidence suggests it’s slightly faster than QWERTY, but the high cost of switching to and learning the new layout just isn’t worth it.
In this work, we explored the performance impact of an alternative approach that builds on people’s existing touch-typing skills with the standard QWERTY layout.
And we do this in a manner that is so transparent, most people don’t even realize that anything is different at first glance.
Can you spot the difference?
Snap quiz time
What’s wrong with this keyboard? Give it a quick once-over. It looks familiar, with the standard QWERTY layout, but do you notice anything unusual? Anything out of place?
Sure, the keys are arranged in a grid rather than the usual staggered key pattern, but that’s not the “key” difference (so to speak). That’s just an artifact of our quick ‘n’ dirty design of this research-prototype keyboard for touchscreen tablets.
Got it figured out?
All right. Pencils down.
Time to check your score. Give yourself:
One point if you noticed that there’s no space bar.
Two points if you noticed that there’s no Enter key, either.
Three points if the lack of a Backspace key gave you palpitations.
Four points and a feather in your cap if you caught the Shift key going AWOL as well.
Now, what if I also told you removing four essential keys from this keyboard–rather than harming performance–actually helps you type faster?
One Trick TO WOO THEM ALL
All we ask of people coming to our touchscreen keyboard is to learn one new trick. After all, we have to make up for the summary removal of Space, Backspace, Shift, and Enter somehow. We accomplish this by augmenting the graphical touchscreen keyboard with stroke shortcuts, i.e. short straight-line finger swipes, as follows:
Swipe right, starting anywhere on the keyboard, to enter a Space.
Swipe left to Backspace.
Swipe upwards from any key to enter the corresponding shift-symbol. Swiping up on the a key, for example, enters an uppercase A; stroking up on the 1key enters the ! symbol; and so on.
Swipe diagonally down and to the left for Enter.
DESIGN PROPERTIES OF A STROKE-AUGMENTED GRAPHICAL KEYBOARD
In addition to possible time-motion efficiencies of the stroke shortcuts themselves, the introduction of these four gestures–and the elimination of the corresponding keys made redundant by the gestures–yields a graphical keyboard with number of interesting properties:
Allowing the user to input stroke gestures for Space, Backspace, and Enter anywhere on the keyboard eliminates fine targeting motions as well as any round-trips necessary for a finger to acquire the corresponding keys.
Instead of requiring two separate keystrokes—one to tap Shift and another to tap the key to be shifted—the Shift gesture combines these into a single action: the starting point selects a key, while the stroke direction selects the Shift function itself.
Removing these four keys frees an entire row on the keyboard.
Almost all of the numeric, punctuation, and special symbols typically relegated to the secondary and tertiary graphical keyboards can then be fit in a logical manner into the freed-up space.
Hence, the full set of characters can fit on one keyboard while holding the key size, number of keys, and footprint constant.
By having only a primary keyboard, this approach affords an economy of design that simplifies the interface, while offering further potential performance gains via the elimination of keyboard switching costs—and the extra key layouts to learn.
Although the strokes might reduce round-trip costs, we expect articulating the stroke gesture itself to take longer than a tap. Thus, we need to test these tradeoffs empirically.
RESULTS AND PRELIMINARY CONCLUSIONS
Our studies demonstrated that overall the removal of four keys—rather than coming at a cost—offers a net benefit.
Specifically, our experiments showed that a stroke keyboard with the gesture-redundant keys removed yielded a 16% performance advantage for input phrases containing mixed-case alphanumeric text and special symbols, without sacrificing error rate. We observed these performance advantages from the first block of trials onward.
Even in the case of entirelylowercase text—that is, in a context where we would not expect to observe a performance benefit because only the Space gesture offers any potential advantage—we found that our new design still performed as well as a standard graphical keyboard. Moreover, people learned the design with remarkable ease: 90% wanted to keep using the method, and 80% believed they typed faster than on their current touchscreen tablet keyboard.
Notably, our studies also revealed that it is necessary to remove the keys to achieve these benefits from the gestural stroke shortcuts. If both the stroke shortcuts and the keys remain in place, user hesitancy about which method to use undermines any potential benefit. Users, of course, also learn to use the gestural shortcuts much more quickly when they offer the only means of achieving a function.
Thus, in this context, less is definitely more in achieving faster performance for touchscreen QWERTY keyboard typing.
The full results are available in the technical paper linked below. The paper contributes a careful study of stroke-augmented keyboards, filling an important gap in the literature as well as demonstrating the efficacy of a specific design; shows that removing the gesture-redundant keys is a critical design choice; and that stroke shortcuts can be effective in the context of multi-touch typing with both hands, even though previous studies with single-point stylus input had cast doubt on this approach.
Although our studies focus on the immediate end of the usability spectrum (as opposed to longitudinal studies over many input sessions), we believe the rapid returns demonstrated by our results illustrate the potential of this approach to improve touchscreen keyboard performance immediately, while also serving to complement other text-entry techniques such as shape-writing in the future.
Arif, A. S., Pahud, M., Hinckley, K., and Buxton, B., Experimental Study of Stroke Shortcuts for a Touchscreen Keyboard with Gesture-Redundant Keys Removed. In Proc. Graphics Interface 2014 (GI’14). Canadian Information Processing Society, Toronto, Ont., Canada. Montreal, Quebec, Canada, May 7-9, 2014. Received the Michael A. J. Sweeney Award for Best Student Paper. [PDF] [Talk Slides (.pptx)] [Video .MP4] [Video .WMV]
Arif, A., Pahud, M., Hinckley, K., Buxton, W., A Tap and Gesture Hybrid Method for Authenticating Smartphone Users (Poster). In ACM 15th International Conference on Human-Computer Interaction with Mobile Devices and Services(MobileHCI 2013), Munich, Germany, Aug. 27-30, 2013, pp. 486-491. [Paper PDF] [Poster Presentation PDF] [Video .WMV] [Video .MP4]
I have three papers coming out this week at MobileHCI 2013, the 15th International Conference on Human-Computer Interaction with Mobile Devices and Services, which convenes this week in Munich. It’s one of the great small conferences that focuses exclusively on mobile interaction, which of course is a long-standing interest of mine.
This post focuses on the first of those papers, and right behind it will be short posts on the other two projects that my co-authors are presenting this week.
I’ve explored many directions for viewing and moving through information on small screens, often motivated by novel hardware sensors as well as basic insights about human motor and cognitive capabilities. And I also have a long history in three-dimensional (spatial) interaction, virtual environments, and the like. But despite doing this stuff for decades, every once in a while I still get surprised by experimental results.
That’s just part of what keeps this whole research gig fun and interesting. If the all answers were simple and obvious, there would be no point in doing the studies.
In this particular paper, my co-authors and I took a closer look at a long-standing spatial, or through-the-lens, metaphor for interaction– akin to navigating documents (or other information spaces) by looking through your mobile as if it were a camera viewfinder– and subjected it to experimental scrutiny.
While this basic idea of using your mobile as a viewport onto a larger virtual space has been around for a long time, the idea hasn’t been subjected to careful scrutiny in the context of moving a mobile device’s small screen as a way to view virtually larger documents. And the potential advantages of the approach have not been fully articulated and realized either.
This style of navigation (panning and zooming control) on mobile devices has great promise because it allows you to offload the navigation task itself to your nonpreferred hand, leaving your preferred hand free to do other things like carry bags of grocieries — or perform additional tasks such as annotation, selection, and tapping commands — on top of the resulting views.
But, as our study also shows, it is an approach not without its challenges; sensing the spatial position of the device, and devising an appropriate input mapping, are both difficult challenges that will need more progress to fully take advantage of this way of moving through information on a mobile device. For the time being, at least, the traditional touch gestures of pinch-to-zoom and drag-to-pan still appear to offer the most efficient solution for general-purpose navigation tasks.
Pahud, M., Hinckley, K., Iqbal, S., Sellen, A., and Buxton, B., Toward Compound Navigation Tasks on Mobiles via Spatial Manipulation. In ACM 15th International Conference on Human-Computer Interaction with Mobile Devices and Services, (MobileHCI 2013), Munich, Germany, Aug. 27-30, 2013, pp. 113-122. [PDF] [video – MP4]
The March 2013 issue of the International Journal of Human-Computer Studies features a clever new technique for automatically (implicitly) bookmarking recently-visited locations in documents, which (as our paper reveals) eliminates 66% of all long-distance scrolling actions for users in active reading scenarios.
The technique, devised by Chun Yu (Tsinghua University Department of Computer Science and Technology, Beijing, China) in collaboration with Ravin Balakrishnan, myself, Tomer Moscovish, and Yuanchun Shi, requires only minimal modification of existing scrolling behavior in document readers — in fact, our prototype works by implementing a simple layer on top of the standard Adobe PDF Reader.
The technique would be particularly valuable for students or information workers whose activities necessitate deep engagements with texts such as technical documentation, non-fiction books on e-readers, or– of course, my favorite pastime– scientific papers.
Yu, C., Balakrishnan, R., Hinckley, K., Moscovich, T., Shi, Y., Implicit bookmarking: Improving support for revisitation in within-document reading tasks. International Journal of Human-Computer Studies, Vol. 71, Issue 3, March 2013, pp. 303-320. [Definitive Version] [Author’s draft PDF — may contain discrepancies]
Bragdon, A., Nelson-Brown, E., Li, Y., Hinckley, K., Experimental Analysis of Touch-Screen Gesture Designs in Mobile Environments, In Proc. CHI 2011 Conf. on Human Factors in Computing Systems. [PDF]
Guimbretiére, F., Dixon, M., and Hinckley, K., ExperiScope: An Analysis Tool for Interaction Data. In Proc. CHI 2007 Conference on Human Factors in Computing Systems, San Jose, CA, April 28 – May 03, 2007, pp. 1333-1342. [PDF] [video .MOV] Note: See also the experiment reported in the Springboard paper, which this study further analyzes.
Hinckley, K., Guimbretiere, F., Baudisch, P., Sarin, R., Agrawala, M., and Cutrell, E., The Springboard: Multiple Modes in One Spring-Loaded Control. In Proc. CHI 2006 Conf. on Human Factors in Computing Systems, Montréal, Québec, Canada, April 22 – 27, 2006, pp. 181-190. [PDF] [video .MOV] Note: See also the ExperiScope paper for some further analysis and discussion of this experiment’s results.
Christopher C. Berger, Mar Gonzalez-Franco, Eyal Ofek and Ken Hinckley, The uncanny valley of haptics. Science Robotics, 2018. 3(17).
The Past and Present Future 