I’m excited about this work — it really looks hard at what some of the next steps in sensing systems might be, particularly when one starts considering how users can most effectively interact with one another in the context of the rapidly proliferating Society of Devices we are currently witnessing.
I think our paper on the GroupTogether system, in particular, does a really nice job of exploring this with strong theoretical foundations drawn from the sociological literature.
F-formations are the various type of small groups that people form when engaged in a joint activity.
GroupTogether starts by considering the natural small-group behaviors adopted by people who come together to accomplish some joint activity. These small groups can take a variety of distinctive forms, and are known collectively in the sociological literature as f-formations. Think of those distinctive circles of people that form spontaneously at parties: typically they are limited to a maximum of about 5 people, the orientation of the partipants clearly defines an area inside the group that is distinct from the rest of the environment outside the group, and there are fairly well established social protocols for people entering and leaving the group.
A small group of two users as sensed via GroupTogether’s overhead Kinect depth-cameras.
GroupTogether also senses the subtle orientation cues of how users handle and posture their tablet computers. These cues are known as micro-mobility, a communicative strategy that people often employ with physical paper documents, such as when a sales representative orients a document towards to to direct your attention and indicate that it is your turn to sign, for example.
Our system, then, is the first to put small-group f-formations, sensed via overhead Kinect depth-camera tracking, in play simultaneously with the micro-mobility of slate computers, sensed via embedded accelerometers and gyros.
GroupTogether uses f-formations to give meaning to the micro-mobility of slate computers. It understands which users have come together in a small group, and which users have not. So you can just tilt your tablet towards a couple of friends standing near you to share content, whereas another person who may be nearby but facing the other way — and thus clearly outside of the social circle of the small group — would not be privy to the transaction. Thus, the techniques lower the barriers to sharing information in small-group settings.
Check out the video to see what these techniques look like in action, as well as to see how the system also considers groupings of people close to situated displays such as electronic whiteboards.
My co-author Nic Marquardt was the first author and delivered the talk. Saul Greenberg of the University of Calgary also contributed many great insights to the paper.
Marquardt, N., Hinckley, K., and Greenberg, S., Cross-Device Interaction via Micro-mobility and F-formations. In ACM UIST 2012 Symposium on User Interface Software and Technology (UIST ’12). ACM, New York, NY, USA, Cambridge, MA, Oct. 7-10, 2012, pp. (TBA). [PDF] [video - WMV]. Known as the GroupTogether system.
Sun, M. Cao, X., Song, H., Izadi, S., Benko, H., Guimbretiere, F., Ren, X., and Hinckley, K. Enhancing Naturalness of Pen-and-Tablet Drawing through Context Sensing. In Proc. ACM International Conference on Interactive Tabletops and Surfaces (ITS ’11). ACM, New York, NY, USA, Kobe, Japan, November 13-16, 2011, pp. 212-221. [PDF] [video - WMV].
Last week I received a significant award for some of my early work in mobile sensing.
It was not that long ago really, that I would get strange glances from practical-minded people– those folks who would look at me with heads tilted downwards ever so slightly, eyebrows raised, and eyeballs askew– when I would mention how I was painting mobile devices with conductive epoxy and duct-taping accelerometers and infrared range-finders to them.
The dot-com bubble was still expanding, smartphones didn’t exist yet, and accelerometers were still far too expensive to reasonably consider on a device’s bill of materials. Many people still regarded the apex of handheld nirvana as the PalmPilot, although its luster was starting to fade.
And this Frankensteinian contraption of sensors, duct tape, and conductive epoxy was taking shape on my laboratory bench-top:
The Idea
I’d been dabbling in the area of sensor-enhanced mobile interaction for about a year, trying one idea here, another idea there, but the project had stubbornly refused to come together. For a long time I felt like it was basically a failure. But every so often myself and my colleagues who worked with me on the project– Jeff Pierce, Mike Sinclair, and Eric Horvitz– would come up with one new example, or another type of idea to try out, and slowly we populated a space of interesting new ways to use the sensors to make mobile devices smarter– or to be more honest about it, just a little bit less stupid– in how they responded to the physical environment, how the user was handling the device, or the orientation of the screen.
The latter led to the idea of using the accelerometer to automatically re-orient the display based on how the user was holding the device. The accelerometer gave us a constant signal of this-way-up, and at some point we realized it would make a great way to switch between portrait and landscape display formats without any need for buttons or menus, or indeed without even explicitly having to think about the interaction at all. The handheld, by being perceptive about it, could offload the decision from the user– hey, I need to look at this table in landscape– to the background of the interaction, so that the user could simply move the device to the desired orientation, and our sensors and our software would automatically optimize the display accordingly.
There were also some interesting subtleties to it. Just using the raw angle of the display, relative to gravity, was not that satisfactory. We built in some hysteresis so the display wouldn’t chatter back and forth between different orientations. We added special handling when you put the handheld down flat on a desk, or picked it back up, so that the screen wouldn’t accidentally flip to a different orientation because of this brief, incidental motion. We noticed that flipping the screen upside-down, which we initially thought wouldn’t be useful, was an effective way to quickly show the contents of the screen to someone seated across the table from you. And we also added some layers of logic in there so that other uses of the accelerometer could co-exist with automatic screen rotation.
Once we had this automatic screen rotation idea working well, I knew we had something. We worked furiously right up to the paper deadline, hammering out additional techniques, working out little kinks and details, figuring out how to convey the terrain we’d explored in the paper we were writing.
The reviewers all loved the paper, and it received a Best Paper Award at the conference. We had submitted it to the Association of Computing Machinery’s annual UIST Symposium– the UIST 2000 13th Annual Symposium on User Interface Software and Technology, held in San Diego, California– because we knew the UIST community was ideally suited to evaluate this research. The paper had a novel combination of sensors. It was a systems paper– that is, it did not just propose a one-off technique but rather a suite of techniques that all used the sensors in a variety of creative ways that complemented one another. And UIST is a rigorously peer-reviewed single-track conference. It’s not the largest conference in the field of Human-Computer Interaction by a long shot– for many years it averaged about two hundred attendees– but as my Ph.D. advisor Randy Pausch (now known for “The Last Lecture“) would often say, “UIST is only 200 people, but its the right 200 people.”
This is the video, recorded back in the year 2000, that accompanied the paper. I think it’s stood the test of time pretty well– or at least a lot better than the hair on top of my head .
Fast forward ten years, and the vast majority of handhelds and slates being produced today include accelerometers and other micro-electromechanical wonders. The cost of these sensors has dropped to essentially nothing. Increasingly, they’re included as a co-processor right on the die with other modules of mobile microprocessors. The day will soon come where it will be all but impossible to purchase a device without sensors directly integrated into the microscopic Manhattan of its silicon gates.
And our mobile screens all automatically rotate, like it or not
So, it was with great pleasure last week that I attended the 2011 24th annual ACM UIST Symposium, and received a Lasting Impact Award, presented to me by Stanford professor Dr. Scott Klemmer, for the contributions of our UIST 2000 paper “Sensing Techniques for Mobile Interaction.”
The inscription on the award reads:
Awarded for its scientific exploration of mobile interaction, investigating new interaction techniques for handheld mobile devices supported by hardware sensors, and laying the groundwork for new research and industrial applications.
In the Meantime…
I remember demonstrating my prototype on-stage with Bill Gates at a media event here in Redmond, Washington in 2001. Gates spoke about the importance of keeping spending– both in the public and private sectors– on R & D and he used my demo as an example of some up-and-coming research, but what I most strongly recall is lingering in the green room backstage with him and some other folks. It wasn’t the first time that I’d met Gates, but it was the first occasion where I chit-chatted with him a bit in a casual, unstructured context. I don’t remember what we talked about but I do remember his foot twitching, always in motion, driving the pedal of a vast invisible loom, weaving a sweeping landscape surmounted by the towering summits of his electronic dreams.
I remember my palms sweating, nervous about the demo, hoping that the sensors I’d duct-taped to my transmogrified Cassiopeia E-105 Pocket PC wouldn’t break off or drain the battery or go crazy with some unforseen nuance of the stage lighting (yes, infrared proximity sensors most definitely have stage fright).
And then less than a week later came the 9/11 attacks. Suddenly spiffy little sensors for mobile devices didn’t seem so important any more. Many product groups, including Windows Mobile at the time, got excited about my demonstration but then the realities of a thousand other crushing demands and priorities rained down on the fragile bubble of technological wonderland I’d been able to cobble together with my prototype. The years stretched by and sensors still hadn’t become mainstream like I had expected them to be.
Then some laptops started shipping with accelerometers to automatically park the hard-disk when you dropped the laptop. I remember seeing digital cameras that would sense the orientation you snapped a picture in, so that you could view it properly when you downloaded it. And when the iPhone shipped in 2007, one of the coolest features on it was the embedded accelerometer, which enabled automatic screen rotation and tilt-based games.
A View to the Future
It took about five years longer than I expected, but we have finally reached an age where clever uses of sensors– both for obvious things like games, as well as for subtle and not-so-obvious things like counting footfalls while you are walking around with the device– abound.
And I’m sure there’s many more. My children will never know a world where their devices are not sensitive to motion and proximity, to orientation and elevation and all the headings of the compass.
The problem is, the future is not so obvious until you’ve struck upon the right idea, until you’ve found the one gold nugget in acres and acres of tailings from the mine of your technological ambitions.
A final word of advice: if your aim is to find these nuggets– whether in research or in creative endeavors– what you need to do is dig as fast as you possibly can. Burrow deeper. Dig side-tunnels where no-one has gone before. Risk collapse and explosion and yes, worst of all, complete failure and ignominious rejection of your diligently crafted masterpieces.
Above all else, fail faster.
Because sometimes those “failed” projects turn out to be the most rewarding of all.
***
This project would not have been possible without standing on the shoulders of many giants. Of course, there are my colleagues on the project– Jeff Pierce, who worked with me as a Microsoft Research Graduate Fellowship recipient at the time, and did most of the heavy lifting on the software infrastructure and contributed many of the ideas and nuances of the resulting techniques. Mike Sinclair, who first got me thinking about accelerometers and spent many, many hours helping me cobble together the sensing hardware. And Eric Horvitz, who helped to shape the broad strokes of the project and who was always an energetic sounding board for ideas.
With the passing of time that an award like this entails, one also reflects on how life has changed, and the people who are no longer there. I think of my advisor Randy Pausch, who in many ways has made my entire career possible, and his epic struggle with pancreatic cancer. I think of my first wife, Kerrie Exely, who died in 1997, and of her father, Bill, who also was claimed by cancer a couple of years ago.
Then there are the many scientists whose work I built upon in our exploration of sensing systems. Beverly Harrison’s explorations of embodied interactions. Albrecht Schmidt’s work on context sensing for mobile phones. Jun Rekimoto’s exploration of tilting user interfaces. Bill Buxton’s insights into background sensing. And many others cited in the original paper.
Lasting Impact Award, for Sensing Techniques for Mobile Interaction, UIST 2000. “Awarded for its scientific exploration of mobile interaction, investigating new interaction techniques for handheld mobile devices supported by hardware sensors, and laying the groundwork for new research and industrial applications.” Awarded to Ken Hinckley, Jeff Pierce, Mike Sinclair, and Eric Horvitz at the 24th ACM UIST October 2011 (Sponsored by the ACM, SIGCHI, and SIGGRAPH). October 18, 2011. Check out the original paper or watch the video appended below.
Sony announced some new tablet designs today to stir the visions of our collective tablet dreams: a sleek airfoil slate design, and a dual-screen tablet that intrigues as well.
The Sony Tablet S airfoil slate design
First off I have to say that I love the industrial design of the Tablet S slate, an asymmetric foldback airfoil-like design straight out of a smarter future. I can’t speak to the build quality, since I haven’t held one in my hands (and the comments and video posted on Engadget seem to call this into question), but I love that Sony’s designers have stepped away from the me-too design mentality of pancake slate designs: flat, thin, and boring.
I’ve held other asymmetric design concept devices for slates in my hands, though, and they offer a number of distinct advantages (even if all of them aren’t fully realized in Sony’s current offering due to its thickness). The off-kilter weight distribution seems like a bad idea at first, but when you grasp one you quickly realize that this makes a slate much more comfortable and less fatiguing to hold with a single hand. All the weight rests in your strong hand, and by virtue of accelerometer-based automatic screen rotation, you can flip it over to your other hand any time you feel like you need a break (and of course this accommodates left-handers as well).
The wedge-shaped profile of the slate also means that it’s canted just a few degrees towards you when you set it down on a tabletop. This makes the screen easier to read, and easier to interact with as well. Whenever I use my iPad (a passe generation-one model that seems oh-so-2010 by now) on a table I resent that I have to lean way forward to look straight down on it, or go grab a book or, more likely at my kitchen table, a folded-up dish towel (hopefully one without too much little-kiddo goop all over it) to prop up the thing. And yeah, I know the case lets you prop it up, but it’s pretty flimsy and floppy.
The other thing that I like about the Tablet S design is they way it’s recessed on the ends (see photo above). Not only does this highlight the sleek curve of the design, and create an immediate emotional connection with the familiar shape of a glossy folded-over magazine, but it also tucks all the extra buttons and controls out of sight. But perhaps even more significant than the resulting aesthetics, this design also places the buttons out of the way of fumbling fingers so that you don’t hit them by accident when you hold or reposition the tablet.
Now if only we could design touch screens smart enough to recognize when I’ve brushed them by mistake.
The dual-screen Sony Tablet P
Any of you who’ve been following me for a while know that I have been a big advocate of dual-screen designs in the past, and have even conducted original research to explore the possibilities of such form-factors.
To be honest the industrial design on the Tablet P seems a little clumsy– it’s a little too thick, and the curved contour on the top screen doesn’t match the bottom and seems to make it a little harder to handle (in the video below, you can see that the device keeps sliding around on the table as the person interacts with it).
But Sony’s software demos for the device show a glimmering of understanding of how to leverage two interconnected screens to their best advantage. They have several demos that partition UI controls from content (video playback on top, play/pause/fast-forward controls on the bottom; video game on top, game controls on the bottom; text on top, touchscreen keyboard on bottom; and so forth). Perhaps the most interesting of the lot is the brief glimpse of an email client that we see with the text of the current message on one screen and the scrolling list of messages on the other screen.
There’s only one demo that uses the screens in portrait orientation, that of an e-book reader, which leverages the two-screened aesthetic perfectly, although the page flip animation in the current demo software leaves much to be desired (it’s an animation that takes time to play, and to my eye at least only serves to confuse, rather than guiding the eye gently through the transition to the new pages.)
The industrial design does have one nice property: the hinge design pivots the screens so that they are very close to one anther when the device is opened, and there is no raised screen bezel, so you can slide your fingers across the two screens without hitting a speed bump in the middle.
The Sony Tablet S and Tablet P, whether or not they are a success in the marketplace, are good examples of the proliferation of the design space of slates, handhelds, and booklet devices. There are some really exciting possibilities opening up here with continued advances in electronics and materials science– as well as the application of good old-fashioned design chops– and it makes me wonder what the devices held by my grandkids will look like.
And in my mind, at least, when I am visited by these visions of the near future, they aren’t just ho-hum pancaked layers of plastic, silicon, and glass any longer, but rather they take flight on the fancies of mad geometers and crazed topologists, digital displays contorted and multiplied into a gleaming sculpture of the human potential.
My old blog, the AlpineInker, is currently read-only (I can’t even access the blog stats any more) and it will go away completely by the end of the year, but there’s still a number of posts there that are near and dear to my heart, and which I occasionally like to direct people to, so I figured why not get this blog on its feet by moving some of the real classics over here.
This particular post was probably one of the most fun blogs to write that I ever put together, and the project was a blast. Since I did this project, OQO went bankrupt and the Model 02 fell into the dustbin of history with it, the Courier came and went like a glint in the eye of the girl who got away, and e-readers and slates have exploded and are on the verge of toppling traditional publishing like a loose boulder teetering on the brink of a nine thousand foot cliff.
But I’m still here and plugging away on my sane and not-so-sane visions of the near future, looking for that next idea that’s oh-so-obvious in retrospect. Sometimes they come easy, sometimes they’re like pulling a loose molar from your jaw with no anesthetic. And I ain’t never been to no dental school, folks, although I did have the fine honor of working with some incredibly gifted and hard-working neurosurgeons back when I was in grad school.
So I’ve worked side-by side with brain surgeons and with Turing award winners and I figure all I need to do is come up with some zany project involving rocket scientists and then I should be about set to call it a wild ride of a career.
I hear NASA might be looking for the next great thing too now that the Space Shuttle is history, so maybe there is hope for me yet on that front.
As for the Codex. Since we’re on WordPress here, I can even enter the 21st century and embed a YouTube video right in my post without pulling yet another molar by hand. Below I’ve linked to a copy of my Codex Video that someone pirated and put up on YouTube on my behalf violating every law of copyright known to man, but heck, once something’s on the ‘net and everyone’s hyperlinks point to it, having them take it down is just counterproductive, so I just went with it. The video has almost 100,000 views so I guess someone must have checked it out. (This video, by the way, was one of the most difficult to shoot that I ever did because at the time we shot it the software kept crashing and the screens were dim and had a lot of glare and, and, and… so a belated apology that it looks a bit clunky in the video, but the thing was pretty damn cool to use and I wish I had the time and energy to resuscitate the software and do more with it).
Anyway, enough of my blathering. Here’s the Classic AlpineInker Post #1, The Microsoft Research Codex:
The Microsoft Research Codex: Are Dual Screens the Future of Mobile Devices?
Never buy one of anything. That’s advice you should stand by when you’re buying unusual gadgets. The advice was good when Randy Pausch offered it to me some 15 years ago, and it’s still good now.
Of course, with 18 month old twin girls at home, this has become second nature to me. Two boxes of diapers. Two gallons of milk. Two Elmo plush dolls.
Oh, and yes, of course. Two screens for my tablet computer.
Hinckley, K., and Song, H., Sensor Synaesthesia: Touch in Motion, and Motion in Touch, In Proc. CHI 2011 Conf. on Human Factors in Computing Systems. CHI 2011 Honorable Mention Award. [PDF] [video .WMV].
Song, H., Benko, H., Guimbretiere, F., Izadi, S., Cao, X., Hinckley, K., Grips and Gestures on a Multi-Touch Pen, In Proc. CHI 2011 Conf. on Human Factors in Computing Systems. [PDF] [video .WMV]
Hinckley, K., The MindEx Timepiece: It Takes Your Thinking and Keeps on Linking. (Non-archival; 5 minute talk only– no manuscript was published). One of five winning entries in the “Demos Optional” UIST 2030 Contest organized by Ken Perlin for UIST 2010, New York, NY, Oct. 5, 2010.
The MindEx is a concept for a cortical sensing timepiece offering multi-modal fusion of sensor inputs with a projected retinal display. Please note: this is purely a speculative fictional concept and not an actual project I am working on. I had the idea for it back in 2003.
.
