Friday, July 28, 2006

Holidays

Two weeks leave means two weeks without blogging. CU, Jeroen

Thursday, July 27, 2006

Help, my phone gestures

There are now several phones with motion sensors allowing you to interact with them using gestures (let's not worry about definitions for once). Imagine you are a software developer for such a phone. Obviously you would want to trace numbers in the air to dial them (?). What else would yor contribution be? Some examples for your inspiration:

Wednesday, July 26, 2006

Look! The Dots are Going Up!

CHICAGO (Reuters) - Scientists reported on Friday what they said was the first scientific evidence that people unconsciously gesture with their voices. What a world of assumptions lies hidden beneath this innocent looking pool of semiotic quicksand. What did Howard Nusbaum, Hadas Shintel, and Arika Okrent do? They let people look at dots and then say whether they went up or down. Separately, people read "It is going up" or "It is going down". If the dots went up, their pitch rose. It lowered when the dots went down. This also happened when the sentences were read. This is the 'unconscious gesturing' mentioned in the title. Look, the dots are going up (say in a rising pitch) First of all, the people did not 'gesture' with their voice, they modulated the fundamental frequency (F-0) of their speech signal to a higher frequency, or you could say they shortened its wavelength. If we assume the term gesture to form a pair with voice to encompass Man's communicative behaviour, surely intonation (and all other prosody) falls under voice. Why would it be useful to shove intonation under gesture? And in fact Nusbaum et al. talk about analog acoustic expression at first and only compare these to gestures. Do they mean gestures are analog? And if so, what are their digital counterparts? Sign language signs? I think this is what they have in mind. (More on this? Read Scott K. Liddell, Grammar, Gesture, and Meaning in American Sign Language. But in general I disagree with this presentation of gesture. I think even co-speech gesturing is a mix of 'analog' and 'digital' encoding of meaning. People simply have many strategies for semiosis which can be employed in just about any modality.) Second, talking about a pitch going 'up' or 'down' is a convention. There is nothing natural about it as such. On a piano it goes from left to right. Is praying to the Lord above simply a natural, analog manual expression? (Image by OYBB) Third, why would the intonation be any more or less 'conscious' than the words themselves?

Monday, July 24, 2006

Reward Raised to 100 Euro

Since there were no contenders since June 25, the reward for a gesture mix-up video is raised to 100 euro. Any takers?

Friday, July 21, 2006

Signing to Deaf Dogs

It appears sign language can be used with dogs as well as gorillas. Check this video (nb: ad leader). Two deaf Dalmatians, Hogan and Georgia, were taught some 65 signs by owner Connie Bombaci. Methinks this is but a classic use of some gestures to get a dog to do tricks. So what if the gestures happen to be ASL signs? What are the main differences between ASL and Dog Sign Language?

Thursday, July 20, 2006

Koko, the Talked Down Gorilla

In 1971 I was born. They called me Koko. Penny Patterson took me in. She teaches me stuff. She teaches me sign language. I am as much a person as you. Says Penny. I use 1000 signs. I hear you talk. I make small sentences. I never win a Pulitzer. But I am great blogger. My chat session was a waste. I didn't get any nipple. How many swear words would Koko use? (picture by A.L.I.) Barbet Shroeder and Nestor Almendros filmed the events back in 1977, and made a movie out of it. It was brought out on DVD recently with extra material (hence this post). Some reviewers focus on the strangeness of the relationship between Penny and Koko (the one scolds and talks down the other as an unfortunate child). Other reviewers stay with the main message about big apes. Funny thing, the movie is not mentioned on Penny's Koko's site. Dolphins 1 - Apes 1

Wednesday, July 19, 2006

GW 2007 Call for Papers

Upcoming event: GW 2007, or The 7th International Workshop on Gesture in Human-Computer Interaction and Simulation 2007 (23-24-25 May). Edificio ISCTE, Lisbon, Portugal. It's a biannual thing so take this opportunity or wait to 2009. The web site is empty but for the Call for Papers. For papers, the deadline is 19 January 2007.

The Dolphins Got the Point and Left

Pointing appears ever so natural to us. Everyone points (even though most of us feel it's rude to point at someone) and you don't need to learn it, right? As a young father of two I wonder: Could finger-pointing be innate? Well, an interesting point is raised by animal studies. Even though chimpanzees follow our gaze it seems difficult to point something out to them, contrary to dogs and wolves. Now, a study pointed out that dolphins understand pointing as well. If you have read So Long and Thanks for all the Fish it will not surprise you that dolphins understand us. The real question may well be: Why don't we understand dolphins? (with the possible exception of this guy.) Dolphins 1 - Chimps 0

Tuesday, July 18, 2006

Saint Peter Watches Over G8

In a world full of wonders, it is but a small step from a fish called Saint Peter, to the town of St Petersburg, where the G8 is now being held. And joining the online readers today with a nice analysis of the G8 summit body language is Dr Peter Bull of York University (for BBC News). Well it's one for the money, two for the show... Apparently we need hardly listen anymore to what they say. Dr. Phil Peter shows us how we can read the state of interpersonal affairs from a couple of pictures. And I have to say, it really fits well with what I already felt. It's uncanny, just like when I had my hand read last year. Do we not simply project that which we like and expect?

Monday, July 17, 2006

Body Tricks

Definitely not gestures by any definition I would use, but these body tricks are nice nevertheless. Does your brain ever itch too? I tend to categorize movement into gesture, fidgeting, practical actions, and moving around. Some things resist labeling however. Should more or less unvoluntary body language, like laughing or blushing, be considered as intended to communicate? If so, then a lot of animal body language should be included too. Is there another criterium at work? And where do these body tricks go? The way it is presented points the way. The body is treated as an instrument. If you handle it correctly you can achieve a set goal. Ergo, they are practical actions. Any contenders?

Sunday, July 16, 2006

John Dory, a Fish Born to Gesture?

There are many fish in the sea. And when divers come to admire them, it may not just the divers that gesture. There are some, like octopi, squids and cuttlefish that signal through colour changes. Or they blow themselves up in response to perceived threats, like terrorists pufferfish. But I once heard a very nice story About a gesturing fish named John Dory Who when faced with his fate Goes head on, stays straight Then flashes his evil eye to thee I've got my eye on you, friend. Thanks to EA for the links above and below and the following anecdote: When you get near a John Dory he will first face you directly, perhaps hoping you cannot see him because he is so thin. If you come closer still he turns and displays the extra eye on its flank, pretending he is a big fish. It's a clear signal to bystanders: "Bugger off". But beware: “Compared with the variety of human responses, however, that of a fish is stereotyped, not subject to much modification by “thought” or learning, and investigators must guard against anthropomorphic interpretations of fish behaviour.” (but then again, these image scoring fish appear none too backward)

Friday, July 14, 2006

Is An Octopus's Signal in his Shade?

Behold the Octopus. The James Bond of the sea. Ringo Starr once wrote: I'd like to be Under the sea In an Octopus's garden In the shade They pinned a romantic story on him about Octopi being aesthetic collectors. But an Octopus's garden is no more than the leftovers, the bones, spines and shells, outside a den. Of course, a true romantic could counter that an Octopus picks aesthetic food... The question I would ask is: does an octopus gesture? First, let us say that the term gesture originated in the distinction between voice and gesture, or oralité et gestualité as the French would say. The pair together refer to our total of communication means. It functions as a rough division of all the ways in which we create meaning for eachother. Since an octopus does not talk we could say that all his signaling behaviour falls under gesture. But that would be a fairly useless statement, unless we add that gesture is reserved to behaviour that is intended to communicate. Make my day, punk. At fUSION Anomaly, some gladly accept the linguistic status of the Octopus's signaling. They even feel it is superior to our own language, which as we all know is strictly limited to our tiny mouths. What do they do exactly? They change colour and texture (see explanation of chromatophores), they may use polarization of light. But why? Often the colour and texture changes are camouflage, but chey can also indicate arousal and/or threat. I find it all rather amazing. But then I saw what cuttlefish can do...

Thursday, July 13, 2006

Aymara Proof of Universal Truth: Humans have no Eyes in their Backs

A scientific finding (Nunez & Sweetser 2006, pdf) about Aymara is catching the news (more, Dutch): The Aymara people refer, in their words and their gestures, to the past as though it were before them, and to the future as though it were behind them. Did he see it before him with his own eyes? Is that where the gesture comes from? A nice analysis at physorg and interesting comments (more) suggests that the uniqueness of the Aymara way is exaggerated (there are many other occasions where the past is put in front and the future in the back). It is also stated that the finding is not new, as it has been described before in this book. A further complication is that the Aymara mark whether they were an eye-witness to an event or just heard about it. The writers of the original article appear to be aware of these things. Their main contribution seems to be that gestural and spoken data support each other in this case and that they should always be looked at together in similar studies. The authors do say that besides Aymara "so far all documented language appear to share a spatial metaphor mapping future events onto spatial locations in front of ego and past events onto locations behind ego". Well, I can not find it on the list of human universals and they provide no reference. Let's face it, it smells a bit like a strawman. But for the greater good of protecting the Aymara heritage and Earths cultural riches I am sure we can all put such petty criticisms behind us. I will end by saying: Aruskipasipxañanakasakipunirakispawa [?]

Tuesday, July 11, 2006

The Scoff, the Insult and the Headbutt

None of it's good. It's all bad and ugly. Zidane headbutted Materazzi (see video if they are not removed), after they exchanged words and gestures. How often have you wanted to do this to MM? It was a shamefull spectacle in the ending minutes of the world cup final. And now expert lip readers, you, me, and Everymen try to guess what happened exactly. Materazzi admits an insult, but regards Zidane's behaviour as offensive as well: ".. he turned to me, looked at me from top to bottom with utmost arrogance". Looking someone up and down is often seen as a form of sexual harrasment (here, here and here) if a man does it to a woman. I think it's more likely Zidane was looking down his nose at Materazzi, a display of contempt. As in 'the arrogant hot-blooded maestro looked down his nose at the barbaric uncultured wannabee'. Update: When Zidane said "you can have my shirt later if you want it that bad" Materazzi replied "I'd rather have your sister". That set Zidane off.

Monday, July 10, 2006

What Lies Beyond Poking

They may not want to call it a gesture interface, but it is definitely a cool demo. Mister Han presents: the Multi-Touch Interaction Research project. Trust me: Sit back and enjoy for 3 minutes Touch me, feel me Poke me, love me Apparently this was made possible by the FTIR (frustrated total internal reflection) sensing technique...

Sunday, July 09, 2006

Once Were Baboons?

Our proud human race... After the SLP more or less banned the topic from linguistic study in 1866, work published since the 1970's on gesture and (sign) language, action recognition (and mirror neurons) made it once again the scientific wild west: the (Gestural) Origins of Language. So, now in the news: Researchers Meguerditchian and Vauclair studied 60 captive baboons and especially a gesture of quick, repetitive rubbing or slapping on the ground, apparently to threaten or intimidate. More importantly, it appears to be lateralized (they use their right hand, thus their left hemisphere of the brain- here there be language centers). You talking to me? You talkin' to me?! A simplified sum of the arguments: Baboons gesture (mostly) right, chimps gesture right, children gesture right. In the brain language is left (connected to right hand). Gesture is part of language. We all come from the same ancestors. Humans must have kickstarted communication through gestures as well. The vocals, rude at first as a monkey's cries, accompanied the gestures at first but became sophisticated and dominant later. Solid as a rock. I think I should finally read that Jackendoff book on the foundations of language. No linguist can take a gesture guy serious without some of the essential baggage.

Saturday, July 08, 2006

Sporting Gestures

A noisy stadium packed with supporters, groups of players on a spacious field, coaches on the sideline, and a referee. Sports provide a perfect blend of ingredients for a banquet of gestures. Has modern dramatic action theatre discarded words? Or was it no different in classical times? News today: Darren Lehmann, a cricket player, was reprimanded for an obscene gesture to the crowd (not unlike this basketball player). But what did he do exactly? The web is silent, but perhaps you know?

Friday, July 07, 2006

Offensive Winker Christiano Ronaldo

It must be a coincidence, but here I am writing a third entry involving a world cup match with Portugal. First about Rooney's intentions regarding Carvalho (against UK), then about a gesture from Figo (against France). Now, on request, let us return to the Rooney incident and discuss Ronaldo's gestures after the incident. So, here is more exact footage: What did Ronaldo have to say to Rooney before the game? After winking to someone he shushes with his mouth. C. Ronaldo's behaviour during the world cup earned him a dungpile of UK abuse. How much of that is envy or looking for a scapegoat I will gladly leave to others. The wink is easily understood: it refers to more or less secret mutual understanding. The shushing or hush up mouth gesture confirms that. Given the immediate connection to the Rooney incident we can safely assume that Ronaldo & Co shared some ideas before the match which have bearing on it. Now we enter the realm of speculation. Did they set up Rooney, driving him into a case of intermittent explosive disorder? Or did they merely talk about Rooney's temperament and found it confirmed by his deliberate attack on Carvalho's manhood? What were their intentions? To play a malicious plot or to play fair but not foolish? Of one thing, and only one thing, we can be sure. They wanted to win. They won. Period.

A Gesture by Figo

Semi-final, France-Portugal, 16th minute.Figo is stretchered off after a collision and makes a gesture: He slams a hand with palm down on the opening of a half-opened fist. Meaning? Anyone? It looked like this one from South Africa I asked A., the Portuguese husband of my colleague EO. He said it meant 'damn'. Another Protuguese colleague thought it didn't have a specific meaning. This got me wondering: Can we trust it to be a Portuguese gesture? Figo played in other countries and together with people from even more different countries. He could've picked it up anywhere. What does it mean where you come from?

Thursday, July 06, 2006

Referees Rule by Signals

American football, regular football, basketball, volleyball, hockey, rugby or any other sport is full of gestures. Not only do they players and coaches sign to each other or their adversaries, but the referee relies almost entirely on gestures to communicate the calls he makes. Only in some cases does he talk to provide additional information. Which sport and what signal? Perhaps the sports unions of the world should start thinking about hiring deaf referees? They may have an easier time ignoring player abuse as well.

Wednesday, July 05, 2006

Fresh Flash for Deaf Access

So your New Zealand, Flemish, or other government acknowledged your sign language? It's a nice gesture but what does it mean? The hottest potato: Should important government information be accessible in sign language by law? The New Zealand bill says: A government department should, when exercising its functions and powers, be guided, so far as reasonably practicable, by the following principles: - NZSL should be used in the promotion to the public of government services and in the provision of information to the public: - government services and information should be made accessible to the Deaf community through the use of appropriate means (including the use of NZSL). I rather doubt many Kiwi government agencies went to pieces trying to get their info ASAP into NZSL with this bundle of escape clauses. Is there hope? Yes, there are projects and intitiatives. Sign synthesis is investigated (eSign, VisiCast, SignSynth, VSigns) as a means to automatically output information that would otherwise not be considered worth translating. In the Name of the Law: Tickle Me! But if you are serious about being heard by the Deaf use good sign language video clips on your web site. Check out the fresh flash from Avon and Somerset Police (BBC news). The classic use of feed-forward mechanisms (see what you will get) make it much better than for example the Quicktime movies of Flemish politician Helga Stevens, which aren't bad by themselves. Certainly nothing as clumsy as the Dutch NGC movies which seem to suffer down syndrome most of their online time.

Monday, July 03, 2006

Rooney, what are your intentions?

Do you ever marvel at your mysterious powers as a human being? Maybe we are not as good as prof X, but I think we all try to read minds a little bit to guess motives of co-humans. Are you reading this blog? You must be either gesture-crazy, family, or a lost soul. In 2004, Tom Troscianko wrote about how people can predict criminal behaviour from looking at someone's gestures and body position in security camera (CCTV) recordings (pdf). Lucky for the vandals we are still facing a huge challenge before we can program a computer to do the same. But it shows we use what we see of the movements to guess intentions (see Baldwin et.al.) What did you make of Rooney's dismissal? Did he commit a foul? Was it an accident? He probably did not intend to castrate Carvalho, but perhaps he wanted to unleash his anger? These questions are anything but academic. In fact, even when you have read all the academic stuff in the links above you will not be wiser. If you were the referee, you have only your own two eyes to trust and the intentions you can see. Could instant replay really help?

Saturday, July 01, 2006

Let's Give 'm a Hand

There is a good project on Hand Motion Modeling running at our lovely department of IDE, special branch ID Studiolab. Fed up with the ugly and unnatural hands of avatars Onno van Nierop fathered his own Hand. (source is better) What should you notice? First, look at the motion not at the skin. The pink triangular surfaces are the foundations of the skin, which can and should be smoothed and overlaid with actual skin texture. Second, the fingers (watch the pink) do not travel in a circular arc, but in a conical arc. The anatomy of the joint is such that the finger axes are not at straight angles with their turning axis, but rather at slightly bent angles. Third, extending is not the same as flexing, the turning axes are shifted slightly. This happens because different muscle systems are at work: extensors and flexors. It creates an ovoidal motion. Virtual Guido, the eSign Avatar (click for movie) It might be worth a try to give Guido a bit of a Hand in signing. One of the project members (IZ) once told me Guido has a weird thumb. It was modeled after one of the guys programming the Avatar who happened to possess a non-standard issue thumb.