[Geeks are Sexy] technology news

Monday, May 29, 2006

Multi-Touch Screen

The interfaces between people and technology have always interested me. It's also a critical part of my work as a web designer. And when I watch movies, it's interesting to see what kinds of fictional "shinies" and other gimmicks people have come up with to portray the future of interface technologies. For the most part, the graphical wonders that have come out of Hollywood are dumb, useless, or strickly plot-driven in their design. But every now and then, I see a creation that is thoroughly thought-out, and beautifully rendered. And then, I turn around and find some aspects of these fictional technologies beginning to be realized.

Seeing the
Multi-Touch Interaction Research demo video made me very happy today. Watch the entire video. It's well worth it.

People are finally beginning to build technology that can truly take advantage of all ten of our fingers. Mainstream technology (in the form of the mouse pointer) is essentially a single "finger" on our screen. It can be a very slow and annoying experience to work with just a mouse. In fact, the "ancient" technology of the keyboard is still much faster, especially in the hands of a talented keyboard jockey. Stylus technology, such as
Wacom's excellent tablet lines, are a step up. In some cases, I'm faster with a digital pen than I am with a mouse. And check out this video of a master artist working magic on a Wacom Cintiq. It's amazing. But these devices and applications still represent a single point of contact.

The Multi-Touch project is one of multiple projects that are beginning to break out of this model. How soon until we start seeing implementations akin to those shown in such movies as
Minority Report and The Island? I don't know about you, but I can't wait.


  • This video shows how a user interacts with Google Earth and Warcraft 3 using multimodal tabletop interaction (gestures and voice). kinda cool too... http://video.google.com/videoplay?docid=6420668728353654549

    By Blogger YeeJen, at 10:13 PM  

  • These kinds of systems are really impressive in rigged demos. The only problem is, well, they're rigged demos. These systems never, and in fact cannot, work in the real world for very simple ergonomic reasons having to do with human anatomy.

    Number one, it used to be that monitors were very small and so a user's fat fingers would obscure the touchscreen. This is no longer a consideration but it's been replaced by an equally intractable problem.

    Essentially, what you can see at a glance with your eyes is far more than what you can touch at a moment's notice with your hands. There is a vast difference in scale between the two. A pointer or graph tablet performs some kind of magnification to bring everything within your reach. Touch screens just generate frustration since they aren't as responsive once you've positioned yourself to using them. Touch screens seem more responsive because it's faster to position yourself to move them, but once you've done that they're actually far less responsive by about an order of magnitude.

    Number two, the angle of view is all wrong. Neutral posture means that your hands are far away from your eyes because they move in a completely different plane from the one your eyes move in. They're parallel and completely separate.

    And why do we care about neutral posture? Because maintaining a non-neutral posture in the long term is extremely tiresome. This was known decades ago and is known as the gorilla shoulder effect. Because you look like a gorilla when using a touchscreen, and you'd have to be a gorilla to be able to use one all day long.

    This doesn't mean that touchscreens aren't going to come into use, they already are. What it does mean is that they will always be condemned to strictly niche applications. So hey, not a big deal.

    What we're dealing with here aren't problems that can be solved. They're fundamental tradeoffs that all happen to be more or less unacceptable for any long-term use of touchscreens. For instance, touchscreens bind eye movement to hand movement. As a result you are trading off between neck strain and shoulder / arm strain.

    The problem with armchair technologists is the same as with armchair quarterbacks. They don't appreciate that all of their proposed solutions have already been tried long ago and there are good reasons why they just don't work.

    Let's take the case of using brainwaves as an input system. Can you say stress headaches? I knew you could. Making weird facial expressions all day long gives people stress headaches. The problem is that users very quickly figure out that they can use facial expressions instead of brainwaves. And they got stress headaches for it.

    Virtual reality? Stress headaches, nausea and other problems.

    3D glasses? Stress headaches and eye strain.

    Touch screens? Sore shoulder & arm muscles, sore back, sore neck.

    But there's more. Oh how there's more!

    The idiot notion of "ten mouse pointers" is based on a fundamental misunderstanding of what the mouse is and what it is for. Many very naive people believe in mouse or hand gestures to replace mouse buttons and the keyboard. Well, you know what? If you make the gestures extensible then you have a full Sign Language on your hands. And why is sign language bad? Oh yeah, because voice recognition alone (without natural language processing) already works much, much better now.

    What does this all mean? Well, it means that mouse buttons aren't an essential part of the mouse. You could have a zero-button mouse and still have a fully workable input system, so long as it's augmented (as it must be) with either a keyboard or voice input.

    The point of the mouse is to provide continuous input, nothing else. A user sees a screen not as a grid of blocks but as a continuous plane. That's why you need the mouse, to provide a continuous variable. Now, since continuous variables are important (otherwise they could be discrete) the user's attention must be focused on them. And humans aren't capable of splitting their attention effectively. So you can only do one continuous variable at a time, which is almost always fine, and so you only need one mouse pointer in the general case.

    Finally, oh did I fail to mention that multiple pointers have been tried and don't work? People "think" they would be convenient but the truth is that multiple mouse pointers aren't convenient, they're just confusing. If you don't understand why, lookup currying.

    Mouse operations can always be curried in theory and almost always in practice. For example, in the demo, when the guy was enlarging objects with two fingers, he was actually performing several operations; first selecting the object to be operated on, then selecting the operation (enlargement), then selecting how much to enlarge by. The first and last are continuous input, strictly seperated by discrete input.

    And this completes the lesson of, well, why touchscreens are overhyped crap.

    By Anonymous Anonymous, at 4:47 AM  

  • Wow, dude, it seems like you've got some serious hatred for this sort of stuff... not that I'm knocking your opinion or anything, but what's your experience with these technologies? You sound pretty knowledgable about the problems attached to each one.

    By Blogger vansau, at 5:12 AM  

  • To test your hypotheses on why this doesn't work I did the following and have concluded that you are wrong in suggesting that these systems cannot work in the real world for ergonomic reasons.

    Using my twelve inch powerbook (which typically sits with the keyboard part flat and the screen at a about a 70 degree angle,) I tilted it backwards so that the screen was at about a 35 degree angle to the floor/table and moved the screen so it was right under my fingers (the keyboard - which for the purpose of this test I pretend doesn't actually exist itself i titled upwards at about the same angle so that it wouldn't be pushing the screen away from me.) The screen's location therefore was very similar to where the keyboard would be, on a traditional laptop, only on a 30 degree angle rather than flat. I then proceed to put my hands over the screen, presuming that the "multi-touch screen" could display a keyboard for typing purposes.

    First, reaching any point on the screen (menus, scroll bars, etc...) was way faster than with a mouse, as I didn't have to remove my hand from the screen area. Scross bars were accessible to my right baby finger so naturally, I almost forgot that in all these demos there's an even easier way to scroll which is to point-drag. Menus were also very convenient to reach. So that kind of blows away point number one.

    Point number two that you make is confusing, and I believe altogether wrong. Back in the days of typewriters, what you typed appeard essentially on the same plane, or only a few inches away from your fingers and that is precisely how I envisioned this multi touch typewriter working. Again, scrolling my text box in which I'm typing this reply to the top half of the properly positioned screen, I pretended to type underneath it, then periodically, touch the parts of the text I wanted to edit, and it felt a lot more convenient than using a mouse or trackpad which would require me to remove my hand from the screen or keyboard, find the device with my hand, wiggle the device to find the pointer with my eyes and then coordinate the movement of the two. Changing the plane of my eyes is actually not a very difficult nor an unnatural task. Pointing my eyes to near where my hands are I have to believe is a more natural location for them than on a totally different plane, which only the use of modern machines has forced us to adopt. I strongly discourage the use of a hammer, wrench, drill, screwdriver or any other less modern tool with your eyes on a different plane (or even pointed anywhere but on the specific action you're undertaking) than where your hands are at.

    Finally, to your third point`I moved into this position beside a mirror and realized that my posture was exactly the same as when my computer is in its natural position.

    So while I agree with you that screen angle and position are very important, I do not see these issues as even remotely insurmountable, but in fact, easily resolved ones.

    By Anonymous Anonymous, at 8:39 AM  

  • I think the multi-touch stuff is extremely cool, but I would never integrate the touch in with the screen.

    I'd basically just take what Apple has down with two-touch scrolling on their laptops, make the touchpad even bigger (like 10" diagonal or something), and then work from there.

    The real trick, of course, would be figuring out a meaningful way to process all of those signals. Is it better to create a language of elaborate multi-finger gestures? Or could you just absolutely map the 10" touchpad to a 15" monitor, and show some kind of translucent pointer for every finger down? Perhaps make the "click" action when a second finger taps within a given radius of an existing finger?

    Either way, it seems like it could be a pretty powerful HID. Especially if you could create text input by drawing PalmOS-y graffiti with your index finger.

    By Anonymous Anonymous, at 9:59 AM  

  • Actually, palm's graffiti is way way slower than using a keyboard to type. This is why they sell folding keyboards for palm pilots.

    As far as your hands being in a different, paralell plane... Right now, my hand moves in a different, not paralell plane whenever I use my mouse, so I think it would take a little getting used to (maybe 5 minutes) If people can swing baseball bats, tennis rackets, etc... in a totally different plane with amazing amounts of control, then I think learning how to control things in paralell planes would be much simpler.

    Of course, I'd still want to keep the normal keyboard off to the side for typing(I play piano, and I like to make keys move when I touch them. That's why you can't practice paino on a desk very well: the desk doesn't go anywhere when you press on it. I am eagerly looking forward to such technology becoming commonplace. Richard can stick with his regular mouse for all I care, but I, and many others I am sure, could use a change like this.

    By Anonymous Anonymous, at 1:18 AM  

  • Hi, I did a Computer Science Masters at York University and when I applied in 2001 they demonstrated a visual hand recognition system that didnt need a glove as in minority report. It could detect and box a hand at any angle, and could also measure depth. A later student added an interface very similar to the minority report GUI. Just goes to show reality is sometimes further ahead of hollywood than people realise.

    By Anonymous Anonymous, at 2:34 AM  

  • richard tell me that u study human anatomy, and i'll take back what i said here.
    if u dont study human anatomy, then stop tryin to bullshit ur way through here. just because u think that something doesnt work doesnt mean it doesnt work. that's why developement team exists, they are hired to overcome the obstacles u've listed. all the reasons u've suggested are pretty much ur opionion, unless u've proven it to be true or have other people's expirement to back u up. and ur saying that virtual reality, 3d glasses, and touch screens all have their negative aspect, but u kno what, even the most simple personal computer at home can cause all those problems. just sit there and use the computer for 8 hours and u mite find urself at ur family doctor's office.

    ps if your so close-minded, y r u here? just to flame the people tryin to develope/improve the touchscreen system? being open-minded is what science and technology is about, this place is obviously not for u

    By Anonymous Anonymous, at 3:05 AM  

  • i see valid points here(thank you kiltak) and i am going to RIT in jsut under one month. I saw a seniors project that was very promising, it used the motion of the eye to control the display of the scree(depending on the angle\ direction of the eye). I instantly improved, mentally of course, on the product and realized that we may not be far from wireless sunglasses with built in eye driven mouse capabilities. Although this may seem far fetched i can twist it into a palpable use here. Given that the standard human can handle on to two tasks maximum at a time. Given that the exceptional human can handle 3-4 task at a time. I assume that there will be no "10 pointer mouse system". This is not completely implausable, although definitley not compeltely functional either.


    now that i have told you what you know, now let me tell you what you dont know.

    i forsee there being many ways to acess the workings of a computer, for example the key differences of regular boot mode vs. safe mode; or the difference of settings say in your internet browser.

    i feel that there will eventually(soon) be available monitors that are both touchscreen, come with a built in 1-2mpg camera for webcasting and possible eye recognition, and for those who cherish nostalgia or those resistant to change a standard monitor.

    i feel that this will happen within the next 5-10 years and that it will aid in the mutability of the technology world immensely.

    as for the implausability of a fully touchscreen command system, there would ahve to be some sort of annoying\disruptive taskbar\dock menu, or an euqally annoying side button on the screen frame itself; to acess simple features such as a keyboard.

    as to the 10 touch aproach would it be able to handle multiple more touches, perhaps for multiple human interaction. or the event of falling objects "gracing" the screen. or would it ignore it if it were only a small infrequent touch? would it be able to deal with distinguishing time recognition delays? such as a single click vs a double click?

    or would they become obsolete and be replaced by a simple one touch method.

    I may be wrong on this matter, as i have only breifly worked with touchpad systems but i felt that my experience couldnt be completely of null value.

    a question i had, if you know, is how did the computer differentiate what commands it was being sent, and by which "pointer"? for example the select enlarge and drag options? al;l within seconds of each other.

    that is all

    i may be back for more

    i will at least check back for some comments and thoguths i hope to stir up


    By Blogger Andrew Azad Alem, at 3:36 AM  

Post a Comment

<< Home