Telling stories of gestures and sign language from around the world. A contemporary view and archive. Covers perception, semiotics, cross-cultural comparison, and technology.
Ninja Strike, a killer application for gesture recognition?
This is certainly an interesting development. Previously we have seen mobile phones using motion and acceleration sensors for gesture control (see here and here). There have also been applications where the camera was used to simply capture optical flow: something in front of the camera is moving/turning in direction A therefore the phone is moving/turning in A + 180 degrees (here). In this case the gesture recognition appears to go a step further and at least the hand appears to be extracted from the image. Or does it simply assume all movement is the hand? And then perhaps the position of the motion is categorized into left-middle-right? Maybe the velocity is calculated but I don't think so.
Update: I do like the setup of how people can hold their phone with the camera in one hand, throw with the other and check their virtual throw on the display. The virtual throwing hand on the display is more or less in the same position as your physical hand, which I think is nice.
EyeSight is a techno start-up of 2004 from the Kingdom of Heaven (Tel Aviv) aspiring to use nothing but Air and a Camera to achieve a divine interaction between true techno-believers and their mobile phones. They prophetize that their technology will 'offer users, including those who are less technologically-adept, a natural and intuitive way to input data, play games and use their mobile phone for new applications'. Heaven on Earth. Mind you, nothing is carved in stone these days. Besides, human nature and intuition are all too often deified these days anyway. Human nature is what usually gets us into trouble (not in the least in the Middle East).
Anyway, one of their angels called Amnon came to me in the night bearing the following message:
Hello Jeroen,
First Allow me to introduce myself. I'm Amnon Shenfeld, RND projects manager for eyeSight Mobile Technologies.
I’ve been following (and enjoying) your BLOG reports for a while, and I thought that the following news from my company, eyeSight Mobile Technologies, may make for an interesting post.
eyeSight has just launched “Ninja Strike”, an innovative mobile game featuring a unique touch free user interface technology we call eyePlay™. Allow me to provide some background information about eyeSight, eyePlay and Ninja Strike: I’m sure you are aware of the popularity and attention innovative user interfaces are getting since the introduction Apple’s IPhone and Nintendo’s Wii… My company’s vision is to bring this technology into the mobile market, and our first products are focused on changing the way mobile gamers play. Our new game, “Ninja Strike”, does exactly this.
You play a ninja warrior with Ninja Stars as your primary weapon. Your stars are thrown by making a throwing motion in front of the phone’s camera. Much like training in real life, during the game you will learn how to throw your weapon correctly, and improve your aim. Your enemies, the evil Kurai ninjas, will also gain strength as the game advances…
Looking forward to hear from you, I hope to see a new post in your blog soon, you've been quiet for a while… J
Amnon Shenfeld
Amnon, will you heed my calls? Have you answers to my burning questions above?
eyeSightgame demo
In the UK, the i-Limb has won an innovation prize. It is a prosthesis with fingers that can move independently (I would think that certain motor programs require a very close 'functioning together' of the fingers, but perhaps they incorporated such things). It gets its input from myoelectric signals from the arm's muscles.
i-Limb by Touch Bionics (source BBC)
Onno got a grant from the European Union that will allow him to start his own research group. He will investigate what happens 'on the other hand' in signed languages.
I have worked as an interaction designer and usability specialist for many years. At the moment, the fashionable thing to be designing for is 'user experience'. Several colleagues here at TU Delft are trying to define user experience to help designers and researchers. Arnold Vermeeren organized a workshop on the definition of user experience at the last SIGCHI conference, and he gave me this link to some reading material: MAUSE. Towards the MAturation of Information Technology USability Evaluation. Paul Hekkert and Rick Schifferstein, also colleagues wrote a book called Product Experience which contains many interesting chapters but is a bit of an investment at $170.
Arnold is also working on a nice graphic that contains the elements of User Experience. He is not the only one though, see this collection of images that try to capture user experience.
Emotion is the core
Utility/Usability, Personal Social Meaning, and Aesthetics are the first contributors
Finally, there is a timeline involved starting with anticipation and ending with reflection
But other people have other opinions. Here is a nice graph too. Here is a nice paper on it.
The link between user experience and gesture is that through gesture control one might expect an increased aesthetic appraisal. It might appeal to your senses because you like to move. Or because you like to move in ways that you are familiar with.
Typically, people can see whether a movement is intended to communicate (a.k.a. a gesture) or whether the movement’s producer has some other intention, be it practical or just fidgeting. There are however plenty of examples where the movement is ambiguous: it could be a gesture but it could also be a meaningless incidental movement. Barack Obama produced such a movement during a speech. Watch and judge for yourself.
Did Obama just flip off Clinton or was he merely scratching his cheek?
Again, like in many other cases where the nature of a movement was debated, there is a potential insult to be considered. It is almost as if people are more sensitive to potentially insulting gestures then to other gestures. Some people, like Lehmann or Mr Wood even use this sensitivity to their advantage. They camouflage their insulting gesture and thus create ambiguity on purpose. Those who have a reason to feel offended are insulted by the ‘gesture’. Other people only see a cactus or someone scratching his head.
I would predict that if people must judge if a movement is intended to communicate they will do so more often when that would mean it is an insult than when that would mean it is some other gesture. (Question: Can you think of an experiment to test this prediction?)
BTW, there is a very interesting related paper on this topic from a psychiatric perspective:
Bucci, Sandra, Mike Startup, Paula Wynn, Amanda Baker, & Terry J. Lewin. (2008). Referential delusions of communication and interpretations of gestures. Psychiatry Research, 158(1), 27-34. (Scopus)
Gestures are an important aspect of non-verbal communication, but people with schizophrenia have poor comprehension of them. However, the tests of gesture comprehension that have been used present only scenes in which interpersonal meaning is communicated, though there is evidence that people with psychotic disorders tend to perceive communications where none were intended. Such mistakes about non-verbal behaviour are the hallmark of a subtype of delusions of reference identified as delusions of communication. Thus we hypothesised that patients with delusions of communication would tend to misinterpret incidental movements as gestures and, since delusions are often derogatory to the self, they would also tend to misinterpret gestures as insulting. Patients with acute psychotic symptoms (n = 64) were recruited according to a 2 × 2 design (presence vs. absence of delusions of communication by presence vs. absence of auditory hallucinations). They, and 57 healthy controls, were presented with 20 brief video clips in which an actor either made a well-known gesture or an incidental movement. After each clip, they selected one of four interpretations: a correct interpretation if a gesture had been presented; the interpretation of a different gesture; an insulting interpretation; no gesture intended (correct for incidental movements). The patients made significantly more errors of all kinds than the controls, perceived significantly more of the incidental movements as gestures, and selected significantly more insulting interpretations of the clips. These differences between patients and controls were almost wholly due to patients with delusions of communication. These results suggest that the difficulties that people with delusions of communication experience in understanding gestures can be explained, at least in part, by the misattribution of self-generated internal events to external sources.
Perhaps we all suffer from delusions of communication to some degree when we are in a situation where we expect to be insulted (rightly or wrongly). I know I always check for fingers when I feel I did something impolite in traffic. Don’t you?
The Wanker gesture is the second most often occuring gesture in the news, with giving the finger a mile in the lead. In third place is the 'fuck-you' forearm jerk. What do these have in common? They are insults, and unexpected insults are apparently very newsworthy.
(source: Wikipedia)
The latest wanker case comes from the Austin, Texas, US...
From Keyetv.com: Travis County Court at Law #6 Judge Jan Breland put Adam Reposa into jail after he made what is described in court documents as "…a simulated masturbatory gesture with his hand while making eye contact with the Court…"
Do children learn language from rich (enough) input or do they invent it more or less on their own, driven by some innate program? That is a question that has kept great scientists busy, particularly Noam Chomsky.
And so with modern gesture research (post Chomsky) and modern sign language research (post Stokoe/Tervoort) the question became important which role gesture and emerging sign language skills plays in the development of language and cognition in hearing children and deaf children, see the work of Susan Goldin-Meadow and co-workers in particular.
A famous case is the discussion surrounding the documented invention of Nicaraguan Sign Language by successive generations of deaf children (by Judy Kegl and others).
But it appears that not only children can create language. A local newspaper here reported that the oldest man in the Netherlands (age 106) lost hearing and speech and invented a 'sign language' with his daughter in law to communicate.
Adrianus van der Vaart and daughter-in-law Corry created a sign language (source: AD)
Did 'Opa Arie' take a dip in the fountain of youth?
Is there no such thing as a critical age of acquiring/inventing a language?
Or did the newspaper exaggerate?
Given the nature of newspapers it is likely that the AD exaggerates. Besides, any sort of gesture system is quickly called a 'sign language' in the Netherlands, and little distinction is made by the general public between 'genuine Sign Language of the Netherlands (NGT)' and other 'gebarentaal'.
Further research is needed urgently however, before it is too late. The potential 'Wilnis Sign Language' (Wilnis is an isolated village in the Netherlands with a remarkable population of elderly people with bad hearing) should be documented by the likes of Judy Kegl? Can anybody send in a linguist?
Mark van Bommel is a Dutch football player who plays for Bayern Munich. He is also a bit of a drama queen, who already supplied us with a decent little gesture scandal before. After already being fined 6200 euro for making a 'fuck you' gesture (forearem jerk) to the crowds at Real Madrid's Bernabeu stadium, he repeated the gesture (now dubbed 'doing a Van Bommel') in the final minutes against Hamburg, this time insulting the referee. This time he got an extra two match ban and a 15000 euro fine (compare other fines) What sort of fine would be enough to stop him?
Van Bommel's little theatrical performance includes some sarcastic clapping. There is an interesting piece by Steve Tomkins on the correct and incorrect application of sarcasm. He also has some cases of football players clapping sarcastically. In a lovely cool sarcastic style Tomkins sets out to explain why sarcasm is not necessarily rude. If done correctly sarcasm can deliver venomous bites. The key lies in not overdoing what you say or how you gesture, and in the inclusion of a pinch of humour. Obviously, if your victim really did or said something stupid it will make sarcasm succeed easily (your victim can't get angry because he is too busy being ashamed).
In this case, Van Bommel overdoes the sarcasm. First, the referee is always right, so he is free to get angry, and indeed he responds by showing the red card. In such a case one must leave room for innocent interpretations to hide behind. If one simply keeps a straight face and claps twice it can always be explained as saying 'good call, ref!'. The fans in the stadium will understand the sarcasm but there will not be enough evidence to punish you. I think that in most cases even the slightest hint of sarcasm will be enough for other people to pick up the message. We humans are so sensitive to insults. Often the mere context will make a straightforward interpretation of an innocent phrase that is intended as sarcasm very unlikely.
Him: "did you watch the game yesterday?" (your team lost 2-0 to his, the bastard)
You: "yes, great game for the neutral spectators" (it was a teeth-grinding muddy fight)
Him: "we gave your lot a good whipping, hey?" (obnoxious little fella, everyone could tell the ref wrongly sent a man off after ten minutes)
You: "yes, a victory well deserved. They are really on fire these last few weeks aren't they?" (they only had one win against a second division club and three losses)
Every knowledgeable bystander (or at least those you care about) will get your point from the context. But in comparison to your obnoxious colleague you appear to be gracious about it all. He may or may not spot your sarcasm but he has no good option to respond. He can either take your words literally and be a fool, or he can acknowledge your sarcasm and respond to the unspoken allegations (that it was an ugly match, an undeserved victory and a team that sucks anyway). The latter choice will have him on the defensive ("I thought it was a proper red card for hands") in which case you can turn up the sarcasm ("of course, he should have stopped the ball with his genitals").
ps. Thanks to Elif for the link to Tomkin's piece
America, as usual, appears to have unique powers of division. In this case America is divided into Deaf America and deaf America, into oralists and manualists (?), into those who respect Alexander Graham Bell and those who reject and denounce his legacy. Bell, whose wife and mother were deaf, devoted much of his life to improve the teaching of speech to deaf and hard of hearing children. As such, he was hardly a champion of sign language, see the following excerpt from Through Deaf Eyes:
In 1884, Bell published a paper “Upon the Formation of a Deaf Variety of the Human Race,” in which he warned of a “great calamity” facing the nation: deaf people were forming clubs, socializing with one another and, consequently, marrying other deaf people. The creation of a “deaf race” that yearly would grow larger and more insular was underway. Bell noted that “a special language adapted for the use of such a race” already was in existence, “a language as different from English as French or German or Russian.” Some eugenicists called for legislation outlawing intermarriage by deaf people, but Bell rejected such a ban as impractical. Instead he proposed the following steps: “(1) Determine the causes that promote intermarriages among the deaf and dumb; and (2) remove them. The causes he sought to remove were sign language, deaf teachers, and residential schools. His solution was the creation of special day schools taught by hearing teachers who would enforce a ban on sign language.
It is hard to imagine a more ruthless approach to the 'problem' of a more or less isolated American Deaf subculture. Let me just state my position clearly: I would have been against it. No. Lucky for me I am Dutch and not American and therefore I do not feel called upon to state my position. I feel free to rise above the divide. Oralism existed here too, but I dare say it was less extreme, and the opposition was therefore less extreme too. Needless to say, the advocates of (also) using sign language are on top at the moment and oralism in its extreme form has all but vanished. Within this context, the following video was recently put online. It is a good illustration of the strong emotions felt by the victims of oralism.
Source: Daveynin: "Scene parody plot: Hilter is angry; his final defeat against sign language has now pushed him over the edge."
i-Limb by Touch Bionics (source BBC)
Arnold is also working on a nice graphic that contains the elements of User Experience. He is not the only one though, see this 
(source: 
Adrianus van der Vaart and daughter-in-law Corry created a sign language (
