What happened to.. "Put-That-There"

Back in 1980, Richard A. Bolt from MIT wrote Put-that-there:voice and gesture at the graphics interface. It was a pioneering multimodal application that combined speech and gesture recognition. I graduated on a pen and speech interface in 1998. At that time I think everyone in the field of multimodality paid tribute to Bolts work. Source: original paper (pdf) The MIT Media Lab, which Bolt helped to found, is a place where the memory of Put-That-There still resides. He doesn't appear to work there anymore, though his site is kept there. Bolt has written down much of his thoughts on HCI and multimodality here At the Interface... It is not very accessible web info, but if you take the time to read it you will be rewarded with plenty of insights. So, does anyone know what happened to.. "Put-That-There"? Does it still exist? Is the memory kept alive?

Tags: gesture, multimodal, HCI, history, MIT, literature

Gesture in the USA

America. What can I say of the gestures of the United States of America? Often cross-cultural web info compares culture X to America, like here, here or here on gestures. Or unless stated otherwise the context of given information is the USA like in the nonverbal dictionary. So, are we to believe that Texas is the same as Maine in this respect? Do Latinos gesture like Amish? Do Jews gesture like Italians? According to David Efron, Gesture, Race and Culture, they start to do so if you put them in a meltingpot long enough. The American Indians used sign languages, which have been documented quite a bit. Modern Plains Indians Sign? I found it rather surprising then that if you are going to study in America, you are advised on how to point and smile. I do agree that people exaggerate the importance of the OK sign. But lets wrap it up. America gestures and signs as vibrantly as elsewhere in the world. Immigration played its role, and probably still does. Good night, America.

Tags: gesture, USA, culture, emblems, cross-cultural

When in Court, Act Like the Romans

A great classical text is now online: Quintilian: Institutio Oratoria It is a massive work of twelve books used to educate boys to become orator. Part of becoming a good speaker is mastering the delivery of your message, through appropriate use of voice and gesture. This is the topic of section 3, Book XI. (lines 1-2, 14-15, 61-149 especially). Quintillian came from Spain but taught in Rome in the first century AD. His very influential work builds on that of Cicero, who discussed gesture in oratory as something to be cultivated, and before that Aristotle, who discouraged gesture and other theatrical means as mainly working on the emotions of the public, favoring pure reason instead. Quintillian separates what is appropriate for orators and for actors though. For orators he gives some very specific guidelines on gestures, how to make them, and when to make them. So, do you want to be seen as a captivating presenter, a convincing lawyer, or a trustworthy politician? You will have plenty to learn from Quintillian. Somehow, I don't think the instructors of our days such as Lenny Laskowski, these guys, this guy, or any of these for that matter have much better to say.

Tags: gesture, history, law, education, presenting, delivery, classical

Reward for Gesture Mix-Up Evidence

Of course they fascinate me. The classic cross-cultural gesture mix-up stories like OK versus Asshole and many, many others. I will pay you for a good video of a gesture that was misunderstood But it's always about someone doing this which could have been mistaken for that. Or someone being told his gesture or behaviour is rude or even obscene in the culture he is visiting. That in itself says it all. We see the gesture and recognize it as not intentionally rude or obscene. We guess the intentions of other people from the situation, the location, their signs of communication (be it speech, gesture or otherwise). But someone else could have taken offense! We are not stupid but we are afraid other people are. The only first hand evidence, like here, is given by people with a commercial interest, for example exploiting the stories in lectures or courses to business travelers. So that is why I have put out a reward. Just send me a video, or preferably a link in the comments. If it is good (not an enacted scene, nor with commercial connections) I will contact you and pay you. Update: I discussed this reward with some colleagues (GL, EO and A). They felt I shouldn't be the judge so they volunteered to be a panel. Update: The reward has gone up from 50 to 100 to 150 euro. ps. Here's the nice generator Update - An 'almost' example: this bit of cross-atlantic communication between Bush and Merkel comes fairly close. Yet I do not think Bush's gesture (giving a shoulder rub) was misinterpreted. It is a gesture of closeness, saying it is okay to let your guard down. This can be accepted by the receiver, who is then however in a bit of an underdog position. They both seem aware of this. Merkel, however seems surprised and her response is clearly one of not accepting. Instead of undergoing the shoulder rub and all its implications, she puts her hands up and smiles uneasily. To me this indicates she makes a compromise between shrugging him off and letting him save face.

Tags: cross-cultural, gesture, meaning, society, pragmatics

Twenty Thousand Signs Under the Sea

If you're something of a gesture fan you probably heard about diving signals aka underwater hand signals. The Active Divers Association (ADA) keeps pictures of the essentials. Other than that there's not a lot to check out on the web. (img source: ADA) Now here's the story. Some Dutch divers put up their own creative contributions to diving signals. Their playful sign inventions nicely highlight the way we go about gesturing in our daily lives. I recently invented a joke gesture with my friend A. It was a challenge best translated as 'do you need dick extension?'. It was done by running the index finger down the nose. We'll keep the story behind it secret, so we'll be able to use it as a side joke. Look Ray, a Fish! Are diving signals exceptional? Are those Dutch divers, or A. and me, doing something extraordinary? I doubt it. I believe any bunch of people put in the right circumstances will invent gestures, a gesture system, or if so required a sign language. Monkey see, monkey do. Man can say, man can sign.

Tags: gesture system, diving, creative, gesture, human

A Quick Brazilian Trick

Trust the Brazilians to come up with a great move, be it in football or gesturing. I already read and saw the Brazilian gesture for speed, but you really need to see the action. Can you do it? Hold your thumb and middle finger together, relax your finger muscles, but hold your wrist. Then shake your hand so your index finger snaps against the middle finger. It sounds a bit like snapping your fingers, which in the U.S. can mean quickly as well, but is mostly a rude way to draw attention from a waiter or a dog, or a nice way to accompany singing. I've been practising for a while now, but failed to achieve any result. My fingers remain silent. The best I got were barely audible pats. But it took me a while to learn to snap my fingers as well when I was a kid like this one. My colleague B. in the video reassured me that I'm not the only hopeless wannabee Brazilian. Perhaps I'd better try my luck with football?

Tags: gesture, meaning, Brazil, emblems

Signs of a Champion

The 2006 FIFA World Cup is dominating the hearts and minds of most western Europeans at the moment. One of these is Rob Oudkerk who wrote a column about the gestures Marco van Basten was making at Netherlands vs Serbia-Montenegro. The man with a plan? Oudkerk appears to be a gesture conaisseur. He can tell Marco van Basten is a visionary leader with a mission, as he shows the signs of a champion. He can also see the opposite from Balkenendes cheers (a political adversary). And he doesn't stop there. The entire inner life of San Marco, his emotions, thoughts, and feelings became clear as Oudkerk watched a video summary of the coach during the match. The confident, alert body language of a prime minister to be? Nothing escaped his keen eye for politics, or did it? As an experienced Dutch politician Oudkerk always enjoyed the gestural displays in parliamentary debates. They say so much more than words. Some of his own famous last words as a ambitious centre-left politician were a nasty comment about immigrants. It was accidently recorded. I guess sometimes words do say more than gestures.

Tags: gesture, body language, ToM, folk psychology, politics, sports, mysticism

Various Gestes Assorti

I came across a strangely informative site by a couple of students. They collected quite a bit of 'Nonverbal Communication' bits and pieces for their web project. For die-hards only, be warned. Perhaps the most popular form of nonverbal communication among students? They seem to promote Public Displays of Affection, or PDA. They put together a reasonable cross-cultural comparison of common greetings, courtesies and practices. There's a fairly indepth piece on Eye Contact and Movement. The bit on Sign Language is ridiculously incomplete. Next are chaotic collections about Facial Expression and putting up a Front. Last but not least there are essays on Posture and Body Positioning and The Handshake. All in all I would say it is not a bad job and a solid B is in order (the wonderfully inappropriate graphics sadly prevent an A). But can someone please tell them nobody calls it nonverbal communication anymore?

Tags: gesture, body language

Indecent Fan Proposals

According to Desmond Morris (Gestures, 1979) the old, widespread, mocking gesture known as 'thumbing your nose (at someone)' may be 'a crude pastiche' of a fan gesture. Though it's unlikely to be the original source the mild resemblance has probably played a role at the time when fans were very popular. The gesture is also known as Queen Anne's Fan, Spanish Fan, and Japanese Fan. Can't afford a fan to blow someone off? Just cock a snook at him! Here are some online manuals or dictionaries: Fan Language which seems to be mirrored here as 32 Victorian Fan Signs. And here is Spanish fan language where it appears still to be taught. I suspect a system of fan gestures was restricted to the court where it was used. These Spanish signs are different from the Victorian ones. More fan language is shown here at Kara's Flamenco Page. I love you - Tomorrow - Write me - I Can't - I'm Alone - Stop - Yes (Bleschunov Museum). Would you get the hint? Is contemporary etiquette so loose that such messages are simply said?

Tags: gesture, gesture system, fan language, culture, history

Gesture Control, Just a Gadget?

A while ago (April '05) BBC News already had a topic on project Audioclouds by researchers at the University of Glasgow. The aim is to control gadgets using movement and sound. Motion is sensed using accelerometers, not via cameras. The coverage by the BBC led to a discussion on engadget about the desirability of such gesture-controlled gadgets. Most of the obvious pros and cons are given, as well as more careful arguments. Since then the project moved on, featuring contributions to lots of HCI conferences mostly. How would you feel about dialing a number by tracing it in the air? (PC World) Gesture-controlled, or motion-controlled gadgets: DJammer, iPod, Nintendo Wii, Vodaphone's Sharp V603SH handset, Pantech's PH-S6500, LG Electronics' SV360, Samsung Electronics' SCH-S310, Antar,

Tags: gesture, phone, gadget, motion sensing, gesture recognition, HCI

G-Speak selling Minority Report Promise

Gesture recognition by G-Speak was reviewed the other day. Promises, promises, Minority Report. Their own website features a remarkable lack of info. The news movies are nice though. Raytheon is mentioned as well as the US military. I guess the enemies of the USA better watch out, or Uncle Sam may shake his fist at you! Wanna shake hands with the President? (source)

Tags: gesture recognition, military, business

All Blacks will Kill for Victory

In a rare display of utter sportsmanship, the New Zealand rugby team, aka the All Blacks, included a cutthroat gesture in their new Kapa o Pango haka. The old haka of the AB's, Ka Mate, was also meant to intimidate but was not as explicit in saying 'we will kill you'. It sparked a bit of interest, and then some, until Paul Lewis (NZHerald) sat down and treated it properly. A challenge yes, a tribute yes, but don't insult the people that can turn off the hot water when you hit the shower after the game. Your manhood may suffer otherwise. Which position would you like to play against these gentlemen? (PETER MEECHAM/The Press)

Tags: gesture, meaning, haka, All Blacks, New Zealand, sports, threat

The New Fidgeting

Fidgeting, nuisance or necessary? A good study by James A. Levine (science mag) of non-exercise activity thermogenesis (NEAT), or fidgeting, links it to staying thin. Thin is good, obese is bad, mind you. Unfortunately NEAT may well be innate, according to dr. Ravussin, meaning it's probably useless to encourage children to fidget (or else you'll get fat, Elsie!). But you could build classrooms that cater to the hunter-gatherer within, who obviously aches for a life in the wild, not for a nice fat hamburger. You may get these lovely kids silent but can you freeze their fidgeting? Should you want to? (news-record.com) Meanwhile, back home, I am trying to see how well people discriminate between gestures and fidgeting. The story goes that we attend to gestures because we can see they are intended to communicate and we ignore the unimportant fidgeting (Kendon Gesture 2004, and stuff from Goffman). Mind you, it's not a black and white thing. You can learn to attend fidgeting as well, see the Levine study for example. Sofar I have however found in my experiments that people are at least very well able to separate gestures and fidgeting, though they are sometimes temporarily 'fooled' by a fidget. (Submitted to Gesture last week) I surely hope that parents and teachers do not suddenly start paying too much attention to fidgeting. If Neat is innate, then perhaps so is our need to be able to move about without arousing everyone's curiosity. You may look at my fidgeting, as long as you don't see it.

Tags: fidgeting, intention, theory, science, child development, movement, perception

A new Mudra for a small lump on my leg

Learning of my interest in Mudras, colleague JM, who is also a yoga teacher, lent me his Mudra Werkboek by Gertrud Hirschi, a swiss yoga teacher (a 1999 Dutch translation of: Mudras – Yoga mit dem kleinen Finger, 1998). Hirschi published a new Mudras book in 2006, which I guess is an update. The most intruiging bit of the book is on page 199, which shows how you can create your own mudra:

• Study the book and the energies of your hand
• Show respect!
• Study the characteristics of each finger and choose the posture that fits [the selected bits?]
• Formulate the goal of the mudra in positive terms [i.e. I have a smooth leg]
• Speak the sentence loud and slow thrice with a breath each
• Picture the outcome
• Try feeling the result
• Dedicate your mudra to a divine power or deity of your choice
• Wish the outcome with all your heart, but remain patient and trusting

Does this sound a bit like creating your own magic spell? Well it wouldn't be the first time magic and yoga are considered twins. Anyway, I have yet to try it out. The energies of my hands still baffle me. Some serious contemplation is in order, methinks.

Tags: mudras, mysticism, yoga, creative, magic

The Good, the Bad and the Finger

I just rewatched Top Gun featuring young buck Tom Cruise as a feisty US fighterpilot. In the beginning he gives the finger to a mysterious bogey in a MIG. He is not allowed to pull the trigger so this is how he scares the attacker off: How old and widespread is the classical digitus impudicus? The event becomes part of the plot and the tension between girl Charlie and boy Maverick (scene transcript from IMDb): Charlie: Eh lieutenant, what were you doing there? Maverick: Communicating. Keeping up foreign relations. You know, giving him the bird! Goose: [Charlie looks puzzled, so Goose clarifies] You know, the finger [gestures appropriately] Charlie: Yes, I know the finger, Goose. Goose: I-I'm sorry, I hate it when it does that, I'm sorry. Excuse me.

Tags: the finger, insult, gesture, movie

Teaching an iPod New Tricks?

Instead of moving to the beat of the music, some Apple genius also patented an idea to have iPod change the beat to your pace of activities. And they filed a couple more patents for good measure. With motion sensing and an accelerometer it could even be respond to .. well, you name the movement. With a little imagination you could have it do the same as a Nintendo Wii remote. Why on earth you would want to is beyond me though. Will your iPod bring your slippers one day? Or just waggle if it likes the tune? It won't be long before my iPod is my intelligent trainer instead of my humble servant: Me: Playing tunes on the iPod while jogging iPod: Senses that I am at a faster pace than the beat and turns it up Me: Hearing the beat go faster I get excited and start a run iPod: Also quickens, then we run perfect synchrony for a while Me: Getting tired, slowing down iPod: Tries to motivate me by keeping the pace high for just a bit longer and shouting "half a minute, Jeroen!" Me: I go for it and then slow to a crawl iPod: Shifts the tune to a slower one and we are in harmony for a while Me: Heartbeat is getting good again iPod: Notices the heartbeat and starts secretly pumping up the pace again Me: Follow iPods lead and put in a extra mile iPod: Tells me "good job! now hit the shower"

Tags: iPod, motion sensing, HCI, movement

Will Wii Win?

The Wii is the new Nintendo gaming console that you could call a gesture interface. It's a motion sensitive remote control with his second hand sidekick Nunchuk. Gamers appear to like the idea very much, see for example the response to this demo-video of Wii gesture-drumming. Meanwhile, Bill Gates plays it down, blowing smoke up the Xbox chimney, which features real GestureTek. It's one up in the campaign to get gamers off of their butts. Which golden oldie is serving which hip youngster at E3? (Buffalo News)

Tags: gaming, gesture recognition, movement, Wii, GestureTek, motion sensing

My Baby Just Cares .. for Milk

Hotter than hot with US Parents today: teach your nine month old baby a few signs instead of just waiting a couple of months for his words to start. Why? Well, obviously to give him an advantage. How so? According to a Californian mix of commercial and academic experts, there are many positive effects. Does a parental investment in baby sign have the highest return? Or would you be better of with music and rhyme sessions? So what do babies sign? Guess what? By far the most used signs are More and Milk. A former colleague of mine from the UK mentioned had an anecdote about a signing baby: Whatever the problem was, be it hunger, thirst, a dirty diaper, etc, the baby would inevitable sign 'milk'. Our mother's breast, a cure for all ailments, eh? To be honest, I tried it with my first child but my limited investment had no visible return. My youngest was spared the exercise, apart from the regular attempts at using gestures such as waving goodbye. And as I gain parental experience an approach of reactive rather than pro-active stimulation seems to me to yield the most profitable ROI anyway. It is certainly less demanding and has many other positive effects on me.

Tags: gesture, baby sign, child development, intention