GroupTogether video available on YouTube

I’m excited about this work — it really looks hard at what some of the next steps in sensing systems might be, particularly when one starts considering how users can most effectively interact with one another in the context of the rapidly proliferating Society of Devices we are currently witnessing.

I think our paper on the GroupTogether system, in particular, does a really nice job of exploring this with strong theoretical foundations drawn from the sociological literature.

F-formations are the various type of small groups that people form when engaged in a joint activity.

GroupTogether starts by considering the natural small-group behaviors adopted by people who come together to accomplish some joint activity.  These small groups can take a variety of distinctive forms, and are known collectively in the sociological literature as f-formations. Think of those distinctive circles of people that form spontaneously at parties: typically they are limited to a maximum of about 5 people, the orientation of the partipants clearly defines an area inside the group that is distinct from the rest of the environment outside the group, and there are fairly well established social protocols for people entering and leaving the group.

A small group of two users as sensed via GroupTogether’s overhead Kinect depth-cameras.

GroupTogether also senses the subtle orientation cues of how users handle and posture their tablet computers. These cues are known as micro-mobility, a communicative strategy that people often employ with physical paper documents, such as when a sales representative orients a document towards to to direct your attention and indicate that it is your turn to sign, for example.

Our system, then, is the first to put small-group f-formations, sensed via overhead Kinect depth-camera tracking, in play simultaneously with the micro-mobility of slate computers, sensed via embedded accelerometers and gyros.

GroupTogether uses f-formations to give meaning to the micro-mobility of slate computers. It understands which users have come together in a small group, and which users have not. So you can just tilt your tablet towards a couple of friends standing near you to share content, whereas another person who may be nearby but facing the other way — and thus clearly outside of the social circle of the small group — would not be privy to the transaction. Thus, the techniques lower the barriers to sharing information in small-group settings.

Check out the video to see what these techniques look like in action, as well as to see how the system also considers groupings of people close to situated displays such as electronic whiteboards.

The full text of our scientific paper on GroupTogether and the citation is also available.

My co-author Nic Marquardt was the first author and delivered the talk. Saul Greenberg of the University of Calgary also contributed many great insights to the paper.

Image credits: Nic Marquardt

See also my post with some further perspective on the GroupTogether project.

Watch the GroupTogether video on YouTube

I’ll have a longer post up about this project after I return from the conference, but for now enjoy the video. I also link to the PDF of our short paper below which has a nice discussion of the motivation and design rationale for this work.

Above all else, I hope this work makes clear that there is still tons of room for innovation in how we interact with the e-readers and tablet computers of the future– as well as in terms of how we consume and manipulate content to produce new creative works.

Hinckley, K., Bi, X., Pahud, M., Buxton, B., Informal Information Gathering Techniques for Active Reading. 4pp Note. In Proc. CHI 2012  Conf. on Human Factors in Computing Systems, Austin, TX, May 5-10, 2012. [PDF]

[Watch Informal Information Gathering Techniques for Active Reading on YouTube]

And yes, scientific research is very much a creative endeavor as well.

In this short essay I also recount a story that Gary Starkweather, the inventor of the laser printer, used to tell about how he evaluated start-up companies, and I promise you will laugh when you hear how he did it.

I used to frequent the hardware lab where Gary worked before he retired so I consider myself very lucky to have spent a lot of time talking shop with him and enjoying his jokes. And this one is one of my favorite of his stories that he would tell about working in Silicon Valley. Great fun.

How to Write Literary Masterpieces of Sheer Gilded Genius by Dumpster Diving

PS: Gary was also prominently featured in Malcolm Gladwell’s recent essay in the New Yorker about The Creation Myth. Seek it out – it’s a great read on the truth about innovation!

I have a new post in the works about the remarkable proliferation of devices in my briefcase which will be up soon… quite a fascinating little study in the digital ecology of e-readers and other gadgets that is turning out to be.

In the meantime, I’ve added a Fiction tab to the blog header– I’m a research scientist by day, a writer of offbeat speculative tales and other literary abominations by night. Occasionally I even write stories that other people actually like to read. I’ll post any new publications in my treasury of wayward fictions up there as they come through the pipeline.

I recently made a sale to Penumbra, a new professional-paying speculative fiction magazine from Musa Publishing, and the issue with my short story just went live a few days ago. It speculates on an unlikely bit of direct-input display technology that every guy secretly wishes he had. Check out my funny little tale and all the other great stories in the issue if you want an entertaining read.

Watch CodeSpace video on YouTube

Watch Enhancing Naturalness of Pen through Context Sensing video on YouTube

It was not that long ago really, that I would get strange glances from practical-minded people– those folks who would look at me with heads tilted downwards ever so slightly, eyebrows raised, and eyeballs askew– when I would mention how I was painting mobile devices with conductive epoxy and duct-taping accelerometers and infrared range-finders to them.

The dot-com bubble was still expanding, smartphones didn’t exist yet, and accelerometers were still far too expensive to reasonably consider on a device’s bill of materials. Many people still regarded the apex of handheld nirvana as the PalmPilot, although its luster was starting to fade.

And this Frankensteinian contraption of sensors, duct tape, and conductive epoxy was taking shape on my laboratory bench-top:

The Idea

I’d been dabbling in the area of sensor-enhanced mobile interaction for about a year, trying one idea here, another idea there, but the project had stubbornly refused to come together. For a long time I felt like it was basically a failure. But every so often myself and my colleagues who worked with me on the project– Jeff Pierce, Mike Sinclair, and Eric Horvitz– would come up with one new example, or another type of idea to try out, and slowly we populated a space of interesting new ways to use the sensors to make mobile devices smarter– or to be more honest about it, just a little bit less stupid– in how they responded to the physical environment, how the user was handling the device, or the orientation of the screen.

The latter led to the idea of using the accelerometer to automatically re-orient the display based on how the user was holding the device. The accelerometer gave us a constant signal of this-way-up, and at some point we realized it would make a great way to switch between portrait and landscape display formats without any need for buttons or menus, or indeed without even explicitly having to think about the interaction at all. The handheld, by being perceptive about it, could offload the decision from the user– hey, I need to look at this table in landscape– to the background of the interaction, so that the user could simply move the device to the desired orientation, and our sensors and our software would automatically optimize the display accordingly.

There were also some interesting subtleties to it. Just using the raw angle of the display, relative to gravity, was not that satisfactory. We built in some hysteresis so the display wouldn’t chatter back and forth between different orientations. We added special handling when you put the handheld down flat on a desk, or picked it back up, so that the screen wouldn’t accidentally flip to a different orientation because of this brief, incidental motion. We noticed that flipping the screen upside-down, which we initially thought wouldn’t be useful, was an effective way to quickly show the contents of the screen to someone seated across the table from you. And we also added some layers of logic in there so that other uses of the accelerometer could co-exist with automatic screen rotation.

Once we had this automatic screen rotation idea working well, I knew we had something. We worked furiously right up to the paper deadline, hammering out additional techniques, working out little kinks and details, figuring out how to convey the terrain we’d explored in the paper we were writing.

The reviewers all loved the paper, and it received a Best Paper Award at the conference. We had submitted it to the Association of Computing Machinery’s annual UIST Symposium– the UIST 2000 13th Annual Symposium on User Interface Software and Technology, held in San Diego, California– because we knew the UIST community was ideally suited to evaluate this research. The paper had a novel combination of sensors. It was a systems paper– that is, it did not just propose a one-off technique but rather a suite of techniques that all used the sensors in a variety of creative ways that complemented one another. And UIST is a rigorously peer-reviewed single-track conference. It’s not the largest conference in the field of Human-Computer Interaction by a long shot– for many years it averaged about two hundred attendees– but as my Ph.D. advisor Randy Pausch (now known for “The Last Lecture“) would often say, “UIST is only 200 people, but its the right 200 people.”

This is the video, recorded back in the year 2000, that accompanied the paper. I think it’s stood the test of time pretty well– or at least a lot better than the hair on top of my head .

Sensing Techniques for Mobile Interaction on YouTube

The Award

Fast forward ten years, and the vast majority of handhelds and slates being produced today include accelerometers and other micro-electromechanical wonders. The cost of these sensors has dropped to essentially nothing. Increasingly, they’re included as a co-processor right on the die with other modules of mobile microprocessors. The day will soon come where it will be all but impossible to purchase a device without sensors directly integrated into the microscopic Manhattan of its silicon gates.

And our mobile screens all automatically rotate, like it or not

So, it was with great pleasure last week that I attended the 2011 24th annual ACM UIST Symposium, and received a Lasting Impact Award, presented to me by Stanford professor Dr. Scott Klemmer, for the contributions of our UIST 2000 paper “Sensing Techniques for Mobile Interaction.”

The inscription on the award reads:

Awarded for its scientific exploration of mobile interaction, investigating new interaction techniques for handheld mobile devices supported by hardware sensors, and laying the groundwork for new research and industrial applications.

In the Meantime…

I remember demonstrating my prototype on-stage with Bill Gates at a media event here in Redmond, Washington in 2001. Gates spoke about the importance of keeping spending– both in the public and private sectors– on R & D and he used my demo as an example of some up-and-coming research, but what I most strongly recall is lingering in the green room backstage with him and some other folks. It wasn’t the first time that I’d met Gates, but it was the first occasion where I chit-chatted with him a bit in a casual, unstructured context. I don’t remember what we talked about but I do remember his foot twitching, always in motion, driving the pedal of a vast invisible loom, weaving a sweeping landscape surmounted by the towering summits of his electronic dreams.

I remember my palms sweating, nervous about the demo, hoping that the sensors I’d duct-taped to my transmogrified Cassiopeia E-105 Pocket PC wouldn’t break off or drain the battery or go crazy with some unforseen nuance of the stage lighting (yes, infrared proximity sensors most definitely have stage fright).

And then less than a week later came the 9/11 attacks. Suddenly spiffy little sensors for mobile devices didn’t seem so important any more. Many product groups, including Windows Mobile at the time, got excited about my demonstration but then the realities of a thousand other crushing demands and priorities rained down on the fragile bubble of technological wonderland I’d been able to cobble together with my prototype. The years stretched by and sensors still hadn’t become mainstream like I had expected them to be.

Then some laptops started shipping with accelerometers to automatically park the hard-disk when you dropped the laptop. I remember seeing digital cameras that would sense the orientation you snapped a picture in, so that you could view it properly when you downloaded it. And when the iPhone shipped in 2007, one of the coolest features on it was the embedded accelerometer, which enabled automatic screen rotation and tilt-based games.

A View to the Future

It took about five years longer than I expected, but we have finally reached an age where clever uses of sensors– both for obvious things like games, as well as for subtle and not-so-obvious things like counting footfalls while you are walking around with the device– abound.

Any my take on all this?

We ain’t seen nothin’ yet.

Since my initial paper on sensing techniques for mobile interaction, every couple of years another idea has struck me. How about answering your phone, or cuing a voice-recognition mode, just by holding your phone to your ear? How about bumping devices together as a way to connect them? What of dual-screen devices that can sense the posture of the screens, and thereby support a breadth of automatically sensed functions? What about new types of motion gestures that combine multi-touch interaction with the physical gestures, or vibratory signals, afforded by these sensors?

And I’m sure there’s many more. My children will never know a world where their devices are not sensitive to motion and proximity, to orientation and elevation and all the headings of the compass.

The problem is, the future is not so obvious until you’ve struck upon the right idea, until you’ve found the one gold nugget in acres and acres of tailings from the mine of your technological ambitions.

A final word of advice: if your aim is to find these nuggets– whether in research or in creative endeavors– what you need to do is dig as fast as you possibly can. Burrow deeper. Dig side-tunnels where no-one has gone before. Risk collapse and explosion and yes, worst of all, complete failure and ignominious rejection of your diligently crafted masterpieces.

Above all else, fail faster.

Because sometimes those “failed” projects turn out to be the most rewarding of all.

***

This project would not have been possible without standing on the shoulders of many giants. Of course, there are my colleagues on the project– Jeff Pierce, who worked with me as a Microsoft Research Graduate Fellowship recipient at the time, and did most of the heavy lifting on the software infrastructure and contributed many of the ideas and nuances of the resulting techniques. Mike Sinclair, who first got me thinking about accelerometers and spent many, many hours helping me cobble together the sensing hardware. And Eric Horvitz, who helped to shape the broad strokes of the project and who was always an energetic sounding board for ideas.

With the passing of time that an award like this entails, one also reflects on how life has changed, and the people who are no longer there. I think of my advisor Randy Pausch, who in many ways has made my entire career possible, and his epic struggle with pancreatic cancer. I think of my first wife, Kerrie Exely, who died in 1997, and of her father, Bill, who also was claimed by cancer a couple of years ago.

Then there are the many scientists whose work I built upon in our exploration of sensing systems. Beverly Harrison’s explorations of embodied interactions. Albrecht Schmidt’s work on context sensing for mobile phones. Jun Rekimoto’s exploration of tilting user interfaces. Bill Buxton’s insights into background sensing. And many others cited in the original paper.

Sensing Techniques for Mobile Interaction on YouTube

I’ve been doing research on sensors on and off for over twelve years now, and it’s a topic I keep coming back to every few years. The possibilities offered by these sensors have never been more promising. They increasingly will be integrated right on the microchip with all the other specialized computational units, so they are only going to become more widespread to the point that it will be practically impossible to buy a mobile gadget of any sort that doesn’t contain sensors. In practical terms there will be no incremental cost to include the sensors, and it’s just a matter of smart software to take advantage of them and enrich the user experience.

I continue to be excited about this line of work and think there’s a lot more that could be done to leverage these sensors. In particular, I believe the possibilities afforded by modern high-precision gyroscopes– and their combination with other sensors and input modalities– are not yet well-understood. And I believe the whole area of contextual sensing in general remains rich with untapped possibilities.

I posted about this on my old blog a while back, but I definitely wanted to make this post available here as well, so here it is. If you just want to cut to the chase, I’ve embedded the video demonstration at the bottom of the post.

The Hidden Dimension of Touch

What’s the gesture of one hand zooming?

This might seem like a silly question, but it’s not. The beloved multi-touch pinch gesture is ubiquitous, but it’s almost impossible to articulate with one hand. Need to zoom in on a map, or a web page? Are you using your phone while holding a bunch of shopping bags, or the hand of your toddler?

Well then, you’re a better man than I am if you can zoom in without dropping your darned phone on the pavement. You gotta hold it in one hand, and pinch with the other, and that ties up both hands.  Oh, sure, you can double-tap the thing, but that doesn’t give you much control, and you’ll probably just tap on some link by mistake anyway.

So what do you do? What’s the gesture of one hand zooming?

Well, I found that if you want an answer to that, first you have to break out of the categorical mindset that seems to pervade so much of mainstream thinking, the invisible cubicle walls that we place around our ideas and our creativity without even realizing it. And Exhibit A in the technology world is the touch-is-best-for-everything stance that seems to be the Great Unwritten Rule of Natural User Interfaces these days.

Here’s a hint: The gesture of one hand zooming isn’t a touch-screen gesture.

Well, that’s not completely true either. It’s more than that.

Got any ideas?

- # -

Every so often in my research career I stumble across something that reminds me that this whole research gig is way easier than it seems.

And way harder.

Because I’ve repeatedly found that some of my best ideas were hiding in plain sight. Obvious things. Things I should have thought of five years ago, or ten.

The problem is they’re only obvious in retrospect.

Of course touch is all the rage; every smartphone these days has to have a touchscreen.

But people forget that every smartphone has motion sensors too– accelerometers and gyroscopes and such– that let the device respond to physical movement, such as when you hold your phone in landscape and the display follows suit.

I first prototyped that little automatic screen rotation interaction, by the way, over twelve years ago, so if you don’t like it, you can blame it on me. Come on, admit it, you’ve cussed more than once when you lay down in bed with your smartphone and the darned screen flipped to landscape. It’s ok, let loose your volley of curses. You won’t be judged here.

Because the first step to a solution is admitting you have a problem.

I started thinking hard about all of this- touch and motion sensing, zooming with one hand and automatic screen rotation gone wild– a while back and gradually realized that there’s an interesting new class of gestures for handhelds hiding in plain sight here. And it’s always been there. Any fool– like me, twelve years ago, for example– could have taken the inputs from a touchscreen and the signals from the sensors and started to build out a vocabulary of gestures based on that.

But well, um… nope. Never been explored in any kind of systematic way, as it turns out.

Call it the Hidden Dimension of Touch, if you like, an uncharted continent of gestures just laying there under the surface of your touchscreen, waiting to be discovered.

- # -

So now that we’re surveying this new landscape, let me show you the way to the first landmark, the Gesture of One Hand Zooming:

• Hold your thumb on the screen, at the point you want to zoom.
• Tip the device back and forth to zoom in or zoom out.
• Lift your thumb to stop.

Yep, it’s that simple and that hard.

It’s a cross-modal gesture: that is, a gesture that combines both motion and touch. Touch: hold your thumb at a particular location on the screen. Motion sensing: your phone’s accelerometer senses the tilt of the device, and maps this to the rate of expansion for the zoom.

It’s not any faster or more intuitive than pinch-to-zoom.

But, gosh darn it, you can do it with one hand.

One-Handed Zooming by holding the screen and subtly tilting the device back and forth.

- # -

All right then, what about this problem of your smartphone gone wild in your bed? Ahem. The problem with the automatic screen rotation, that is.

Well, just hold your finger on the screen as you lay down. Or as you pivot the screen to a new viewing orientation.

Call it Pivot-to-Lock, another monument on this new touch-plus-motion landscape: just hold the screen while rotating the device.

Lock engaged. Just flip the screen to a new orientation to slip out of the lock. Simple, and fun to use.

- # -

Is that it? Is there more?

Sure, there’s a bunch more touch-and-motion gestures that we have experimented with. For example, here’s one more: you can collect bits of content that you encounter on your phone- say, crop out piece of a picture that you like- just by framing it with your fingers and then flipping the phone back in a quick motion. Here, holding two fingers still plus the flipping motion defines the cross-modal gesture, as demonstrated in our prototype for Windows Phone 7:

Check out the video below to see all of these in action, and some other ideas that we’ve tried out so far.

But there’s something else.

Another perspective. Something completely different from all the examples above.

There’s really two ways to look at interaction with motion sensors.

We can use them to support explicit new gestures– like giving your device a shake, for example– or the phone can use them in a more subtle way, by just sitting there in the background and seeing what the sensors have to say about how the device is being used.  Did the user just pick up the phone? Is the user walking around with the phone? Is the phone sitting flat and motionless on a desk? Yep, you can infer all these things with high confidence.

And we can bring this perspective back to our thinking about combined touch and motion.

Imagine your touchscreen as the surface of a pond on a windless day. Perfectly flat. Smooth.

Motionless.

Now what happens when you set your finger to the surface of that pond?

Yep, ripples.

Touch the surface of the pond again, somewhere else. More ripples, expanding from a different spot.

Now take your finger and sweep it along the surface of the water. Another disturbance– a wake in the trail of your finger this time. That’s another pattern. A different pattern.

Touch and motion are inextricably linked. The sensors on these devices– particularly the new generation of low-cost gyroscopes that are making their way onto handhelds– are increasingly sensitive, even to rather subtle motions and vibrations.

When you touch the screen of your device, or place a finger anywhere on the case of your device for that matter, we have a good sense of how you’re touching it and about where you’re touching it and how you’re holding it.

And all of this can be used to optimize how your device reacts, how it interprets your gestures, how accurately it can respond to you. And maybe some more stuff that nobody even realizes is possible yet.

Frankly, I’m not even sure myself. We’ve probably only just scratched the surface of the possibilities here.

Yeah, there’s a hidden dimension of touch all right, and to be honest I still feel like we’re a long way from surveying all the landmarks of this new world.

But I like what we see so far.

- VIDEO-

Here’s a video of our system in action:

YouTube Video of Touch and Motion Gestures for Mobiles.

- PUBLICATION DETAILS -

Our scientific paper on the work described in this post won an Honorable Mention Award for CHI 2011 Best Paper.  The paper appeared May 9th at the ACM CHI 2011 Conference on Human Factors in Computing Systems in Vancouver, British Columbia, Canada.

Check out the paper for a full and nuanced discussion of this design space, as well as references to a whole bunch of exciting work that has been conducted by other researchers in recent years.

Sensor Synaesthesia: Touch in Motion, and Motion in Touch, by Ken Hinckley and Hyunyoung Song. CHI 2011 Conf. on Human Factors in Computing Systems.

The paper was presented at the conference by my co-author Hyunyoung Song of the University of Maryland. Hyunyoung worked with me for her internship at Microsoft Research in the summer of 2010 and her contributions to this project were tremendous– very, very impressive work by a great young researcher.