I’m excited about this work — it really looks hard at what some of the next steps in sensing systems might be, particularly when one starts considering how users can most effectively interact with one another in the context of the rapidly proliferating Society of Devices we are currently witnessing.
I think our paper on the GroupTogether system, in particular, does a really nice job of exploring this with strong theoretical foundations drawn from the sociological literature.
F-formations are the various type of small groups that people form when engaged in a joint activity.
GroupTogether starts by considering the natural small-group behaviors adopted by people who come together to accomplish some joint activity. These small groups can take a variety of distinctive forms, and are known collectively in the sociological literature as f-formations. Think of those distinctive circles of people that form spontaneously at parties: typically they are limited to a maximum of about 5 people, the orientation of the partipants clearly defines an area inside the group that is distinct from the rest of the environment outside the group, and there are fairly well established social protocols for people entering and leaving the group.
A small group of two users as sensed via GroupTogether’s overhead Kinect depth-cameras.
GroupTogether also senses the subtle orientation cues of how users handle and posture their tablet computers. These cues are known as micro-mobility, a communicative strategy that people often employ with physical paper documents, such as when a sales representative orients a document towards to to direct your attention and indicate that it is your turn to sign, for example.
Our system, then, is the first to put small-group f-formations, sensed via overhead Kinect depth-camera tracking, in play simultaneously with the micro-mobility of slate computers, sensed via embedded accelerometers and gyros.
GroupTogether uses f-formations to give meaning to the micro-mobility of slate computers. It understands which users have come together in a small group, and which users have not. So you can just tilt your tablet towards a couple of friends standing near you to share content, whereas another person who may be nearby but facing the other way — and thus clearly outside of the social circle of the small group — would not be privy to the transaction. Thus, the techniques lower the barriers to sharing information in small-group settings.
Check out the video to see what these techniques look like in action, as well as to see how the system also considers groupings of people close to situated displays such as electronic whiteboards.
My co-author Nic Marquardt was the first author and delivered the talk. Saul Greenberg of the University of Calgary also contributed many great insights to the paper.
Marquardt, N., Hinckley, K., and Greenberg, S., Cross-Device Interaction via Micro-mobility and F-formations. In ACM UIST 2012 Symposium on User Interface Software and Technology (UIST ’12). ACM, New York, NY, USA, Cambridge, MA, Oct. 7-10, 2012, pp. (TBA). [PDF] [video - WMV]. Known as the GroupTogether system.
This is my latest project, which I will present tomorrow (May 9th) at the CHI 2012 Conference on Human Factors in Computing Systems.
I’ll have a longer post up about this project after I return from the conference, but for now enjoy the video. I also link to the PDF of our short paper below which has a nice discussion of the motivation and design rationale for this work.
Above all else, I hope this work makes clear that there is still tons of room for innovation in how we interact with the e-readers and tablet computers of the future– as well as in terms of how we consume and manipulate content to produce new creative works.
Hinckley, K., Bi, X., Pahud, M., Buxton, B., Informal Information Gathering Techniques for Active Reading. 4pp Note. In Proc. CHI 2012 Conf. on Human Factors in Computing Systems, Austin, TX, May 5-10, 2012. [PDF]
Bragdon, A., DeLine, R., Hinckley, K., and Morris, M. R., Code space: Touch + Air Gesture Hybrid Interactions for Supporting Developer Meetings. In Proc. ACM International Conference on Interactive Tabletops and Surfaces (ITS ’11). ACM, New York, NY, USA, Kobe, Japan, November 13-16, 2011, pp. 212-221. [PDF] [video - WMV]. As featured on Engadget and many other online forums.
Sun, M. Cao, X., Song, H., Izadi, S., Benko, H., Guimbretiere, F., Ren, X., and Hinckley, K. Enhancing Naturalness of Pen-and-Tablet Drawing through Context Sensing. In Proc. ACM International Conference on Interactive Tabletops and Surfaces (ITS ’11). ACM, New York, NY, USA, Kobe, Japan, November 13-16, 2011, pp. 212-221. [PDF] [video - WMV].
Lasting Impact Award, for Sensing Techniques for Mobile Interaction, UIST 2000. “Awarded for its scientific exploration of mobile interaction, investigating new interaction techniques for handheld mobile devices supported by hardware sensors, and laying the groundwork for new research and industrial applications.” Awarded to Ken Hinckley, Jeff Pierce, Mike Sinclair, and Eric Horvitz at the 24th ACM UIST October 2011 (Sponsored by the ACM, SIGCHI, and SIGGRAPH). October 18, 2011. Check out the original paper or watch the video appended below.
I’ve had a number of conversations with people recently about the new opportunities for mobile user interfaces afforded by the increasingly sophisticated sensors integrated with hand-held devices.
I’ve been doing research on sensors on and off for over twelve years now, and it’s a topic I keep coming back to every few years. The possibilities offered by these sensors have never been more promising. They increasingly will be integrated right on the microchip with all the other specialized computational units, so they are only going to become more widespread to the point that it will be practically impossible to buy a mobile gadget of any sort that doesn’t contain sensors. In practical terms there will be no incremental cost to include the sensors, and it’s just a matter of smart software to take advantage of them and enrich the user experience.
I continue to be excited about this line of work and think there’s a lot more that could be done to leverage these sensors. In particular, I believe the possibilities afforded by modern high-precision gyroscopes– and their combination with other sensors and input modalities– are not yet well-understood. And I believe the whole area of contextual sensing in general remains rich with untapped possibilities.
I posted about this on my old blog a while back, but I definitely wanted to make this post available here as well, so here it is. If you just want to cut to the chase, I’ve embedded the video demonstration at the bottom of the post.
The Hidden Dimension of Touch
What’s the gesture of one hand zooming?
This might seem like a silly question, but it’s not. The beloved multi-touch pinch gesture is ubiquitous, but it’s almost impossible to articulate with one hand. Need to zoom in on a map, or a web page? Are you using your phone while holding a bunch of shopping bags, or the hand of your toddler?
Well then, you’re a better man than I am if you can zoom in without dropping your darned phone on the pavement. You gotta hold it in one hand, and pinch with the other, and that ties up both hands. Oh, sure, you can double-tap the thing, but that doesn’t give you much control, and you’ll probably just tap on some link by mistake anyway.
So what do you do? What’s the gesture of one hand zooming?
Well, I found that if you want an answer to that, first you have to break out of the categorical mindset that seems to pervade so much of mainstream thinking, the invisible cubicle walls that we place around our ideas and our creativity without even realizing it. And Exhibit A in the technology world is the touch-is-best-for-everything stance that seems to be the Great Unwritten Rule of Natural User Interfaces these days.
Here’s a hint: The gesture of one hand zooming isn’t a touch-screen gesture.
Well, that’s not completely true either. It’s more than that.
Got any ideas?
- # -
Every so often in my research career I stumble across something that reminds me that this whole research gig is way easier than it seems.
And way harder.
Because I’ve repeatedly found that some of my best ideas were hiding in plain sight. Obvious things. Things I should have thought of five years ago, or ten.
The problem is they’re only obvious in retrospect.
Of course touch is all the rage; every smartphone these days has to have a touchscreen.
But people forget that every smartphone has motion sensors too– accelerometers and gyroscopes and such– that let the device respond to physical movement, such as when you hold your phone in landscape and the display follows suit.
I first prototyped that little automatic screen rotation interaction, by the way, over twelve years ago, so if you don’t like it, you can blame it on me. Come on, admit it, you’ve cussed more than once when you lay down in bed with your smartphone and the darned screen flipped to landscape. It’s ok, let loose your volley of curses. You won’t be judged here.
Because the first step to a solution is admitting you have a problem.
I started thinking hard about all of this- touch and motion sensing, zooming with one hand and automatic screen rotation gone wild– a while back and gradually realized that there’s an interesting new class of gestures for handhelds hiding in plain sight here. And it’s always been there. Any fool– like me, twelve years ago, for example– could have taken the inputs from a touchscreen and the signals from the sensors and started to build out a vocabulary of gestures based on that.
But well, um… nope. Never been explored in any kind of systematic way, as it turns out.
Call it the Hidden Dimension of Touch, if you like, an uncharted continent of gestures just laying there under the surface of your touchscreen, waiting to be discovered.
- # -
So now that we’re surveying this new landscape, let me show you the way to the first landmark, the Gesture of One Hand Zooming:
Hold your thumb on the screen, at the point you want to zoom.
Tip the device back and forth to zoom in or zoom out.
Lift your thumb to stop.
Yep, it’s that simple and that hard.
It’s a cross-modal gesture: that is, a gesture that combines both motion and touch. Touch: hold your thumb at a particular location on the screen. Motion sensing: your phone’s accelerometer senses the tilt of the device, and maps this to the rate of expansion for the zoom.
It’s not any faster or more intuitive than pinch-to-zoom.
But, gosh darn it, you can do it with one hand.
One-Handed Zooming by holding the screen and subtly tilting the device back and forth.
- # -
All right then, what about this problem of your smartphone gone wild in your bed? Ahem. The problem with the automatic screen rotation, that is.
Well, just hold your finger on the screen as you lay down. Or as you pivot the screen to a new viewing orientation.
Call it Pivot-to-Lock, another monument on this new touch-plus-motion landscape: just hold the screen while rotating the device.
Lock engaged. Just flip the screen to a new orientation to slip out of the lock. Simple, and fun to use.
- # -
Is that it? Is there more?
Sure, there’s a bunch more touch-and-motion gestures that we have experimented with. For example, here’s one more: you can collect bits of content that you encounter on your phone- say, crop out piece of a picture that you like- just by framing it with your fingers and then flipping the phone back in a quick motion. Here, holding two fingers still plus the flipping motion defines the cross-modal gesture, as demonstrated in our prototype for Windows Phone 7:
Check out the video below to see all of these in action, and some other ideas that we’ve tried out so far.
But there’s something else.
Another perspective. Something completely different from all the examples above.
There’s really two ways to look at interaction with motion sensors.
We can use them to support explicit new gestures– like giving your device a shake, for example– or the phone can use them in a more subtle way, by just sitting there in the background and seeing what the sensors have to say about how the device is being used. Did the user just pick up the phone? Is the user walking around with the phone? Is the phone sitting flat and motionless on a desk? Yep, you can infer all these things with high confidence.
And we can bring this perspective back to our thinking about combined touch and motion.
Imagine your touchscreen as the surface of a pond on a windless day. Perfectly flat. Smooth.
Motionless.
Now what happens when you set your finger to the surface of that pond?
Yep, ripples.
Touch the surface of the pond again, somewhere else. More ripples, expanding from a different spot.
Now take your finger and sweep it along the surface of the water. Another disturbance– a wake in the trail of your finger this time. That’s another pattern. A different pattern.
Touch and motion are inextricably linked. The sensors on these devices– particularly the new generation of low-cost gyroscopes that are making their way onto handhelds– are increasingly sensitive, even to rather subtle motions and vibrations.
When you touch the screen of your device, or place a finger anywhere on the case of your device for that matter, we have a good sense of how you’re touching it and about where you’re touching it and how you’re holding it.
And all of this can be used to optimize how your device reacts, how it interprets your gestures, how accurately it can respond to you. And maybe some more stuff that nobody even realizes is possible yet.
Frankly, I’m not even sure myself. We’ve probably only just scratched the surface of the possibilities here.
Yeah, there’s a hidden dimension of touch all right, and to be honest I still feel like we’re a long way from surveying all the landmarks of this new world.
Check out the paper for a full and nuanced discussion of this design space, as well as references to a whole bunch of exciting work that has been conducted by other researchers in recent years.
The paper was presented at the conference by my co-author Hyunyoung Song of the University of Maryland. Hyunyoung worked with me for her internship at Microsoft Research in the summer of 2010 and her contributions to this project were tremendous– very, very impressive work by a great young researcher.
Sony announced some new tablet designs today to stir the visions of our collective tablet dreams: a sleek airfoil slate design, and a dual-screen tablet that intrigues as well.
The Sony Tablet S airfoil slate design
First off I have to say that I love the industrial design of the Tablet S slate, an asymmetric foldback airfoil-like design straight out of a smarter future. I can’t speak to the build quality, since I haven’t held one in my hands (and the comments and video posted on Engadget seem to call this into question), but I love that Sony’s designers have stepped away from the me-too design mentality of pancake slate designs: flat, thin, and boring.
I’ve held other asymmetric design concept devices for slates in my hands, though, and they offer a number of distinct advantages (even if all of them aren’t fully realized in Sony’s current offering due to its thickness). The off-kilter weight distribution seems like a bad idea at first, but when you grasp one you quickly realize that this makes a slate much more comfortable and less fatiguing to hold with a single hand. All the weight rests in your strong hand, and by virtue of accelerometer-based automatic screen rotation, you can flip it over to your other hand any time you feel like you need a break (and of course this accommodates left-handers as well).
The wedge-shaped profile of the slate also means that it’s canted just a few degrees towards you when you set it down on a tabletop. This makes the screen easier to read, and easier to interact with as well. Whenever I use my iPad (a passe generation-one model that seems oh-so-2010 by now) on a table I resent that I have to lean way forward to look straight down on it, or go grab a book or, more likely at my kitchen table, a folded-up dish towel (hopefully one without too much little-kiddo goop all over it) to prop up the thing. And yeah, I know the case lets you prop it up, but it’s pretty flimsy and floppy.
The other thing that I like about the Tablet S design is they way it’s recessed on the ends (see photo above). Not only does this highlight the sleek curve of the design, and create an immediate emotional connection with the familiar shape of a glossy folded-over magazine, but it also tucks all the extra buttons and controls out of sight. But perhaps even more significant than the resulting aesthetics, this design also places the buttons out of the way of fumbling fingers so that you don’t hit them by accident when you hold or reposition the tablet.
Now if only we could design touch screens smart enough to recognize when I’ve brushed them by mistake.
The dual-screen Sony Tablet P
Any of you who’ve been following me for a while know that I have been a big advocate of dual-screen designs in the past, and have even conducted original research to explore the possibilities of such form-factors.
To be honest the industrial design on the Tablet P seems a little clumsy– it’s a little too thick, and the curved contour on the top screen doesn’t match the bottom and seems to make it a little harder to handle (in the video below, you can see that the device keeps sliding around on the table as the person interacts with it).
But Sony’s software demos for the device show a glimmering of understanding of how to leverage two interconnected screens to their best advantage. They have several demos that partition UI controls from content (video playback on top, play/pause/fast-forward controls on the bottom; video game on top, game controls on the bottom; text on top, touchscreen keyboard on bottom; and so forth). Perhaps the most interesting of the lot is the brief glimpse of an email client that we see with the text of the current message on one screen and the scrolling list of messages on the other screen.
There’s only one demo that uses the screens in portrait orientation, that of an e-book reader, which leverages the two-screened aesthetic perfectly, although the page flip animation in the current demo software leaves much to be desired (it’s an animation that takes time to play, and to my eye at least only serves to confuse, rather than guiding the eye gently through the transition to the new pages.)
The industrial design does have one nice property: the hinge design pivots the screens so that they are very close to one anther when the device is opened, and there is no raised screen bezel, so you can slide your fingers across the two screens without hitting a speed bump in the middle.
The Sony Tablet S and Tablet P, whether or not they are a success in the marketplace, are good examples of the proliferation of the design space of slates, handhelds, and booklet devices. There are some really exciting possibilities opening up here with continued advances in electronics and materials science– as well as the application of good old-fashioned design chops– and it makes me wonder what the devices held by my grandkids will look like.
And in my mind, at least, when I am visited by these visions of the near future, they aren’t just ho-hum pancaked layers of plastic, silicon, and glass any longer, but rather they take flight on the fancies of mad geometers and crazed topologists, digital displays contorted and multiplied into a gleaming sculpture of the human potential.
Hinckley, K., Wigdor, D., Input Technologies and Techniques. Chapter 9 in The Human-Computer Interaction Handbook – Fundamentals, Evolving Technologies and Emerging Applications, Third Edition, ed. by Jacko, J., Published by Taylor & Francis. To appear. [PDF of author's manuscript - not final]
This is an extensive revision of the 2007 and 2002 editions of my book chapter, and with some heavy weight-lifting from my new co-author Daniel Wigdor, it treats direct-touch input devices and techniques in much more depth. Lots of great new stuff. The book will be out in early 2012 or so from Taylor & Francis – keep an eye out for it!
Hinckley, K., and Song, H., Sensor Synaesthesia: Touch in Motion, and Motion in Touch, In Proc. CHI 2011 Conf. on Human Factors in Computing Systems. CHI 2011 Honorable Mention Award. [PDF] [video .WMV].
