Category Archives: device form-factors

Paper: LightRing: Always-Available 2D Input on Any Surface

In this modern world bristling with on-the-go-go-go mobile activity, the dream of an always-available pointing device has long been held as a sort of holy grail of ubiquitous computing.

Ubiquitous computing, as futurists use the term, refers to the once-farfetched vision where computing pervades everything, everywhere, in a sort of all-encompassing computational nirvana of socially-aware displays and sensors that can respond to our every whim and need.

From our shiny little phones.

To our dull beige desktop computers.

To the vast wall-spanning electronic whiteboards of a future largely yet to come.

How will we interact with all of these devices as we move about the daily routine of this rapidly approaching future? As we encounter computing in all its many forms, carried on our person as well as enmeshed in the digitally enhanced architecture of walls, desktops, and surfaces all around?

Enter LightRing, our early take on one possible future for ubiquitous interaction.

LightRing device on a supporting surface

By virtue of being a ring always worn on the finger, LightRing travels with us and is always present.

By virtue of some simple sensing and clever signal processing, LightRing can be supported in an extremely compact form-factor while providing a straightforward pointing modality for interacting with devices.

At present, we primarily consider LightRing as it would be configured to interact with a situated display, such as a desktop computer, or a presentation projected against a wall at some distance.

The user moves their index finger, angling left and right, or flexing up and down by bending at the knuckle. Simple stuff, I know.

But unlike a mouse, it’s not anchored to any particular computer.

It travels with you.

It’s a go-everywhere interaction modality.

Close-up of LightRing and hand angles inferred from sensors

Left: The degrees-of-freedom detected by the LightRing sensors. Right: Conceptual mapping of hand movement to the sensed degrees of freedom. LightRing then combines these to support 2D pointing at targets on a display, or other interactions.

LightRing can then sense these finger movements–using a one-dimensional gyroscope to capture the left-right movement, and an infrared sensor-emitter pair to capture the proximity of the flexing finger joint–to support a cursor-control mode that is similar to how you would hold and move a mouse on a desktop.

Except there’s no mouse at all.

And there needn’t even be a desktop, as you can see in the video embedded below.

LightRing just senses the movement of your finger.  You can make the pointing motions on a tabletop, sure, but you can just as easily do them on a wall. Or on your pocket. Or a handheld clipboard.

All the sensing is relative so LightRing always knows how to interpret your motions to control a 2D cursor on a display. Once the LightRing has been paired with a situated device, this lets you point at targets, even if the display itself is beyond your physical reach. You can sketch or handwrite characters with your finger–another scenario we have explored in depth on smartphones and even watches.

The trick to the LightRing is that it can automatically, and very naturally, calibrate itself to your finger’s range of motion if you just swirl your finger. From that circular motion LightRing can work backwards from the sensor values to how your finger is moving, assuming it is constrained to (roughly) a 2D plane. And that, combined with a button-press or finger touch on the ring itself, is enough to provide an effective input device.

The LightRing, as we have prototyped it now, is just one early step in the process. There’s a lot more we could do with this device, and many more practical problems that would need to be resolved to make it a useful adjunct to everyday devices–and to tap its full potential.

But my co-author Wolf Kienzle and I are working on it.

And hopefully, before too much longer now, we’ll have further updates on even more clever and fanciful stuff that we can do through this one tiny keyhole into this field of dreams, the verdant golden country of ubiquitous computing.

_____________________________________________________

LightRing thumbnailKienzle, W., Hinckley, K., LightRing: Always-Available 2D Input on Any Surface. In the 27th ACM Symposium on User Interface Software and Technology (UIST 2014), Honolulu, Hawaii, Oct. 5-8, 2014, pp. 157-160. [PDF] [video.mp4 TBA] [Watch on YouTube]

Watch LightRing video on YouTube

Project: The Analog Keyboard: Text Input for Small Devices

With the big meaty man-thumbs that I sport, touchscreen typing–even on a full-size tablet computer–can be challenging for me.

Take it down to a phone, and I have to spend more time checking for typographical errors and embarrassing auto-miscorrections than I do actually typing in the text.

But typing on a watch?!?

I suppose you could cram an entire QWERTY layout, all those keys, into a tiny 1.6″ screen, but then typing would become an exercise in microsurgery, the augmentation of a high-power microscope an absolute necessity.

But if you instead re-envision ‘typing’ in a much more direct, analog fashion, then it’s entirely possible. And in a highly natural and intuitive manner to boot.

Enter the Analog Keyboard Project.

Analog Watch Keyboard on Moto 360 (round screen)

Wolf Kienzle, a frequent collaborator of mine, just put out an exciting new build of our touchscreen handwriting technology optimized for watches running the Android Wear Platform, including the round Moto 360 device that everyone seems so excited about.

Get all the deets–and the download–from Wolf’s project page, available here.

This builds on the touchscreen writing prototype we first presented at the MobileHCI 2013 conference, where the work earned an Honorable Mention Award, but optimized in a number of ways to fit on the tiny screen (and small memory footprint) of current watches.

All you have to do is scrawl the letters that you want to type–in a fully natural manner, not in some inscrutable secret computer graffiti-code like in those dark days of the late 1990’s–and the prototype is smart enough to transcribe your finger-writing to text.

It even works for numbers and common punctuation symbols like @ and #, indispensable tools for the propagation of internet memes and goofy cat videos these days.

Writing numbers and punctuation symbols on the Analog Keyboard

However, to fit the resource-constrained environment of the watch, the prototype currently only supports lowercase letters.

Because we all know that when it comes to the internet, UPPERCASE IS JUST FOR TROLLZ anyway.

Best of all, if you have an Android Wear device you can try it out for yourself. Just side-load the Analog Keyboard app onto your watch and once again you can write the analog way, the way real men did in the frontier days. Before everyone realized how cool digital watches were, and all we had to express our innermost desires was a jar of octopus ink and a sharpened bald eagle feather. Or something like that.

Y’know, the things that made America great.

Only now with more electrons.

You can rest easy, though, if these newfangled round watches like the Moto 360 are just a little bit too fashionable for you. As shown below, it works just fine on the more chunky square-faced designs such as the Samsung Gear Live as well.

Analog Keyboard on Samsung Gear Live watch

Check out the video embedded below, and if you have a supported Android Wear device, download the prototype and give it a try. I know Wolf would love to get your feedback on what it feels like to use the Analog Keyboard for texting on your watch.

Bring your timepiece into the 21st century.

You’ll be the envy of every digital watch nerd for miles around.

Besides: it’s clearly an idea whose time has come.

Watch Analog Keyboard video on YouTube

 

Paper: Informal Information Gathering Techniques for Active Reading

This is my latest project, which I will present tomorrow (May 9th) at the CHI 2012 Conference on Human Factors in Computing Systems.

I’ll have a longer post up about this project after I return from the conference, but for now enjoy the video. I also link to the PDF of our short paper below which has a nice discussion of the motivation and design rationale for this work.

Above all else, I hope this work makes clear that there is still tons of room for innovation in how we interact with the e-readers and tablet computers of the future– as well as in terms of how we consume and manipulate content to produce new creative works.

Informal Information Gathering Techniques for Active ReadingHinckley, K., Bi, X., Pahud, M., Buxton, B., Informal Information Gathering Techniques for Active Reading. 4pp Note. In Proc. CHI 2012  Conf. on Human Factors in Computing Systems, Austin, TX, May 5-10, 2012. [PDF]

[Watch Informal Information Gathering Techniques for Active Reading on YouTube]

The Fractured State of Reading and Publishing

The bad news: I dropped my Kindle this morning.

The good news: I caught it before it hit the floor.

The even worse news: In so doing, I slammed it against the corner of my desk, smashing the e-ink screen into a starburst of gray, black, and white-plaid shards:

The newly fractured landscape of my kindle screen.The man pictured in the screen saver offers his disapproval with a withering half-frown, a my-oh-my-what-have-thee done expression as he finds himself trapped forever in this doomed terrain of shattered e-ink.

So, I guess it’s back to paper for me until my new Kindle arrives.

For a long time I never thought I would have any use for a Kindle. After all, who wants to read on a computer? And what about marking up the text, dogearing pages, or having more than one book open on my desk at a time?

Well, those behaviors are mostly my self-fueled obsessions when authoring original works of nonfiction. For recreational reading, the mechanisms for highlighting passages and bookmarking pages on the Kindle are, while somewhat clumsy and indirect, still good enough to get the job done.

And then there’s the instant gratification aspect.

This weekend up I was up at my cabin, at 3000′ elevation and nestled deep in the alpine pinnacles of the Cascade Crest, and I decided that I wanted to read another one of the mystery anthologies edited by Ed Gorman and Martin H. Greenberg because I recently read By Hook or By Crook on the recommendation of Kristine Kathryn Rusch and it was fantastic.

So I just brought up the book in the Kindle store, paged through the related reads, and within sixty seconds of the impulse I was reading Between the Dark and the Daylight.

But now I have to read the bloody thing on my smartphone until my new Kindle arrives.

And while I wait, it occured to me that the fractured Kindle screen pictured above strikes a perfect image of the publishing industry and the entire state of reading these days. The old world has been shattered by feedback loops in technology and ongoing market forces that just keep reinforcing one another. Paper books ain’t going away soon, but I’ll probably live to see the day where they are uncommon for most titles. Bookstores will be relegated to specialty boutique status, like the camera and stationery stores populating the deserted shoals of strip-malls.

And you know what that smells like to me?

Opportunity.

The Courier was one example of how these shifts might spawn whole new experiences or categories of devices. The Amazon Tablet might well be another. But whatever the next hot gadget or gizmo is, rest assured, I feel like a technological wolf, scenting a long series of innovations-to-come in the shifting winds, and I’ll be looking to make a killing. :-) What of tablets with pen and multi-touch? What of Nicholas Chen’s Multi-Slate Reading System, a federation of cheap slates that you can scatter about your office like the glossy marketing brochures you get in the mail, tossed aside for the day where you may or may not read them? What of flexible, paper-like displays?

We’re still in the stone age here, folks, as far as e-readers are concerned. We’ll look back fondly on the Kindle and its ilk as the quaint auto-buggies that presaged a sleek, sophisticated, and nearly unrecognizable future.

That’s where I want to be, even if I have to cobble it together with clunky prototypes, Frankenstein monsters of acrylic and delrin etched out by the laser cutter of my dreams.

In the meantime, you could do a lot worse than to follow Kristine Kathryn Rusch and her husband, Dean Wesley Smith, as they talk about what this means for readers and writers and the publishing industry writ large.

Interesting New Tablets from Sony

Sony announced some new tablet designs today to stir the visions of our collective tablet dreams: a sleek airfoil slate design, and a dual-screen tablet that intrigues as well.

The Sony Tablet S airfoil slate design

First off I have to say that I love the industrial design of the Tablet S slate, an asymmetric foldback airfoil-like design straight out of a smarter future. I can’t speak to the build quality, since I haven’t held one in my hands (and the comments and video posted on Engadget seem to call this into question), but I love that Sony’s designers have stepped away from the me-too design mentality of pancake slate designs: flat, thin, and boring.

I’ve held other asymmetric design concept devices for slates in my hands, though, and they offer a number of distinct advantages (even if all of them aren’t fully realized in Sony’s current offering due to its thickness). The off-kilter weight distribution seems like a bad idea at first, but when you grasp one you quickly realize that this makes a slate much more comfortable and less fatiguing to hold with a single hand. All the weight rests in your strong hand, and by virtue of accelerometer-based automatic screen rotation, you can flip it over to your other hand any time you feel like you need a break (and of course this accommodates left-handers as well).

The wedge-shaped profile of the slate also means that it’s canted just a few degrees towards you when you set it down on a tabletop. This makes the screen easier to read, and easier to interact with as well. Whenever I use my iPad (a passe generation-one model that seems oh-so-2010 by now) on a table I resent that I have to lean way forward to look straight down on it, or go grab a book or, more likely at my kitchen table, a folded-up dish towel (hopefully one without too much little-kiddo goop all over it)  to prop up the thing. And yeah, I know the case lets you prop it up, but it’s pretty flimsy and floppy.

Image credit: Engadget

The other thing that I like about the Tablet S design is they way it’s recessed on the ends (see photo above). Not only does this highlight the sleek curve of the design, and create an immediate emotional connection with the familiar shape of a glossy folded-over magazine, but it also tucks all the extra buttons and controls out of sight. But perhaps even more significant than the resulting aesthetics, this design also places the buttons out of the way of fumbling fingers so that you don’t hit them by accident when you hold or reposition the tablet.

Now if only we could design touch screens smart enough to recognize when I’ve brushed them by mistake.

The dual-screen Sony Tablet P

Any of you who’ve been following me for a while know that I have been a big advocate of dual-screen designs in the past, and have even conducted original research to explore the possibilities of such form-factors.

To be honest the industrial design on the Tablet P seems a little clumsy– it’s a little too thick, and the curved contour on the top screen doesn’t match the bottom and seems to make it a little harder to handle (in the video below, you can see that the device keeps sliding around on the table as the person interacts with it).

But Sony’s software demos for the device show a glimmering of understanding of how to leverage two interconnected screens to their best advantage. They have several demos that partition UI controls from content (video playback on top, play/pause/fast-forward controls on the bottom; video game on top, game controls on the bottom; text on top, touchscreen keyboard on bottom; and so forth). Perhaps the most interesting of the lot is the brief glimpse of an email client that we see with the text of the current message on one screen and the scrolling list of messages on the other screen.

There’s only one demo that uses the screens in portrait orientation, that of an e-book reader, which leverages the two-screened aesthetic perfectly, although the page flip animation in the current demo software leaves much to be desired (it’s an animation that takes time to play, and to my eye at least only serves to confuse, rather than guiding the eye gently through the transition to the new pages.)

The industrial design does have one nice property: the hinge design pivots the screens so that they are very close to one anther when the device is opened, and there is no raised screen bezel, so you can slide your fingers across the two screens without hitting a speed bump in the middle.

Check out the Engadget coverage of the Tablet P for more details.

Conclusion and a Reflection on the Future

The Sony Tablet S and Tablet P, whether or not they are a success in the marketplace, are good examples of the proliferation of the design space of slates, handhelds, and booklet devices. There are some really exciting possibilities opening up here with continued advances in electronics and materials science– as well as the application of good old-fashioned design chops– and it makes me wonder what the devices held by my grandkids will look like.

And in my mind, at least, when I am visited by these visions of the near future, they aren’t just ho-hum pancaked layers of plastic, silicon, and glass any longer, but rather they take flight on the fancies of mad geometers and crazed topologists, digital displays contorted and multiplied into a gleaming sculpture of the human potential.

Informal Organization and the Emerging Class of Casual-Use Devices

A while back Bill Buxton put up a brief quotation on billbuxton.com that I like a lot:

Just a thought: Slate computers and e-readers represent a new class of digital appliance– one targeted for casual use. With the growth of this market will emerge  a new and long overdue approach to interaction– one that is in keeping with the casual intent and context of such usage and which will complement, rather than replace, interfaces that support more formal and structured activities. 

He goes on to make a nice little plug for my InkSeine Tablet PC application, but that’s only a very small part of why I like his quotation.

To me Bill’s comment gets at the essence of the new approach to design and interaction demanded by this emerging class of casual-use devices.

And it’s one that’s already deeply rooted in human behavior, even back in those dark ages before our twilight dreams were set afire by the eerie blue-white glow of our iPad portals to never never land.

I like to call it informal organization. It’s a lot of why paper has remained so pervasive in the so-called “paperless” office of the modern information worker. (You could do a lot worse than to spend a weekend reading The Myth of the Paperless Office by Sellen & Harper to understand this more deeply).

But now that e-reading and tablets are gaining more and more traction, we’re seeing at least some of those paper-based behaviors and expectations smash head-on into the limitations of current-generation readers and tablets.

I can send my documents to my Kindle to read them away from my desktop; I’ve never really liked doing much reading on my computer. And printing seems wasteful and moreover has a higher transaction cost than just sending the document to my reader– hitting print, selecting the right printer, remembering to pick the auto-staple function, walking down the hall to pick it up, coming back to my office and actually starting into reading before I get interrupted and forget why the hell I printed out the document in the first place…

But on the Kindle or iPad, I can get right into reading my document quicker, but then when that moment comes that I want to mark up some copy-edits, jot an annotation in the margin, highlight a passage, slap a post-it on it with a note about what to do next, thumb back and forth quickly between multiple documents, or god forbid casually toss the document into a pile with other like-minded papers– the workflow breaks down and I am into clumsy workarounds and tricks and little idiosyncratic habits I’ve developed to make the tools work for me as best they can.

This is where the frontier in e-reading experiences currently lies.

The e-ink reader and yes even our fancy-pants multi-touch iPads are going to look like antiquated glowing green-phosphor displays to our eyes a few short years from now.

Because these unmet needs are all about the interaction experience, figuring out what the appropriate digital analogs of these behaviors are, and figuring out what the right input modalities and interaction techniques and workflow are to allow users to express these notions as quickly and casually as they can conceive of them.

I talk about some of this in my recent paper Pen + Touch = New Tools. And a few techniques are shown in the video that goes along with that, although to be honest they are clumsy and have a long way to go to get to where I really think things need to be.

As another example, take a look at the Collections mechanism on the Kindle for organizing your e-books. My friend and former colleague Jeff Pierce wrote about the rather limited Kindle book organization mechanisms on his blog, which was actually what tipped me into this little rant. Just gathering together a half-dozen short stories, or documents that you’ve emailed to yourself, or your read vs. unread books and managing that on an ongoing basis is a huge pain in the ass.

But it doesn’t have to be that way. With the massive shift to electronic reading that the publishing industry is tipping towards, and as more and more “real” work versus casual recreational reading moves to such devices, there’s a huge incentive to get these reading devices and tablets right.

And making something that’s even half as good as the stacks of books and papers and post-its scattered around my desk wouldn’t be a bad start.

Book Chapter: Input Technologies and Techniques, 2012 Edition

Input Technologies and Techniques, 3rd EditionHinckley, K., Wigdor, D., Input Technologies and Techniques. Chapter 9 in The Human-Computer Interaction Handbook – Fundamentals, Evolving Technologies and Emerging Applications, Third Edition, ed. by Jacko, J., Published by Taylor & Francis. To appear. [PDF of author's manuscript - not final]

This is an extensive revision of the 2007 and 2002 editions of my book chapter, and with some heavy weight-lifting from my new co-author Daniel Wigdor, it treats direct-touch input devices and techniques in much more depth. Lots of great new stuff. The book will be out in early 2012 or so from Taylor & Francis – keep an eye out for it!