Telling stories of gestures and sign language from around the world. A contemporary view and archive. Covers perception, semiotics, cross-cultural comparison, and technology.
Abstract: In this study we compare human and machine acceptability judgments for extreme variations in sign productions. We gathered acceptability judgments of 26 signers and scores of three different Automatic Gesture Recognition (AGR) algorithms that could potentially be used for automatic acceptability judgments, in which case the correlation between human ratings and AGR scores may serve as an ‘acceptability performance’ measure. We found high human-human correlations, high AGR-AGR correlations, but low human-AGR correlations. Furthermore, in a comparison between acceptability and classification performance of the different AGR methods, classification performance was found to be an unreliable predictor of acceptability performance.
Snapshots of the three signs used in the experiment.Examples of three manipulations of the sign SAW. We tested about 68 sign manipulations in total. These were run through the automatic recognition algorithms we had been working on and they were rated by human signers. The paper is about how humans and machines can be compared.
One of the more interesting lectures at the FG2008 conference was a keynote speech delivered by Nadia Magnenat-Thalmann, director of the MIRALab in Geneva. She talked about Communicating with a Virtual Human or a Robot that has Emotions, Memory and Personality. She went far beyond the simplistic notion of expressing 'the six basic emotions' and talked about how mood, personality and relationships may affect our facial expressions.
The talk by Magnenat-Thalmann focused on facial expression. (source)
By coincidence I got an invitation to write a paper for another conference, organized by Anton Nijholt and Nadia Magnenat-Thalmann (and others), called the Conference on Computer Animation and Social Agents (CASA 2009). It is organized by people from the University of Twente but held in Amsterdam. Call for papers: deadline February 2009.
Nadia also mentioned a researcher at Utrecht University called Arjan Egges. He got his PhD at the MIRALab and is now working on "the integration of motion capture animation with navigation and object manipulation".
The British are doing it again. Leading the world to a better place. This time it concern books, or stories might be a better word, in sign language, BSL to be exact. I wonder why they keep calling it books? Although my hearing kids sometimes listen to 'Spoken Books' on CDs, hmm. Thanks to Gavin Howard for the link.
MyBSLbooks: Welcome to myBSLbooks.com - the World's first free online library of signed books. We are delighted to share with you a range of popular children's books, available for the first time in British Sign Language. This site offers D/deaf children, their families and schools wider access to their favourite stories in the preferred language of the Deaf Community.
Well, the site only contains about eight DVD's so far. And it's hardly a library since it doesn't cover any books published by anyone else, and I don't know if lending instead of buying is an option. The site is copyrighted by Lexicon/Signstream, so I guess they somehow own it.
Come to think of it, the Dutch site Vi-Taal - De Gebarenwinkel has had a similar offering out there for years, and also offers a lot of other sign language goodies. And the Nederlands Gebarencentrum has a few DVDs as well. But well done all the same, you wonderful Britons.
Arend Harteveld died at the age of 50 years on Sunday 7 September 2008. Much too soon and entirely unexpected he was struck down by an accident in the blood circulation. Arend was a good man and a well respected colleague at Delft University of Technology. My thoughts go out to his family, especially to his mother who lived in with him and whom Arend was taking care of.
Arend Harteveld (source)
Arend contributed to much of the research based on which I hope to write my thesis, and these last years would not have been the same without him. He bore quite a burden in providing, more or less on his own, support to many courses and many labs, a burden he used to share with three colleagues in support who all left as a result of reorganisations. Meanwhile, his main interest was to work on research projects himself, and I found his contributions, both in creating software for experiments or for data analysis and in discussing the design of the experiments, to be very valuable. Arend always quickly grasped the ideas behind experiments and had a knack of pointing out flaws in the experimental design.
Arend also maintained a website with information that shows some of his technical prowess. The website is maintained now by one his radio amateur friends. Arend tells of radio and measuring equipment, chirps, about which he also gave lectures occasionally. From personal experience I know that if a subject gripped him he wouldn't rest until he understood it fully, which happened during our collaboration for example with capturing response times on a laptop. He tried out several clocks of the PC and its processor and experimentally tested delays and variance in delays. As a radio amateur, a passion he picked up in his teens, he was known as PA1ARE. And now, as his brother in law said during the departure ceremony: "PA1ARE is voorgoed uit de lucht".
Ruud de Wild, the barely disguised saviour of our ears?
Ruud de Wild is a DJ (and uomo universalis) who has prime time shows on the Dutch Radio. He switched to Q-Music in 2007 and this is one of their recent ads. Some people are complaining now but it is still going. There is a billboard poster too:
In the news: Brabants Dagblad: Op posters en in tv-spotjes is De Wild afgebeeld op een manier die doet denken aan een Christusfiguur. Op zijn T-shirt staat een hart omringd met stralen, net zoals Christus soms wordt weergegeven.
It took me a few moments to dig up this image of Christ that looks very similar:
Jesus and his wife (?) Mary posing as Ruud de Wild (source)
What is on the heart exactly? I cannot read it. Does anyone know?
Update: The heart on Ruud's shirt reads 'Q is good for you' while the heart of Christ the King is adorned with the crown or a ring of thorns.
A demo describing the MobileASL research project
The group of MobileASL researchers at the University of Washington features in a local news bulletin. They have been working for a few years now on efficient transmitting of ASL video over a channel with limited bandwidth. The idea is to enable mobile videophony, which has been the holy grail of mobile applications for quite some time already.
Personally, I am not convinced that specific technology for the transmission of sign language video will really have an impact. Here are a few reasons. Bandwidth will increase anyway with costs going down. Processing capacity in phones will increase. Videophony is an application that is desirable for many, not just signers. In other words, there is already a drive towards videophony that will meet the requirements for signing. Furthermore, I am not sure which requirements are specifically posed by sign language. People talk and gesture too, and I imagine they would want that to come across in the videophony as well. Finally, signers can and do adjust their signing to for example webcams. Does the technology address a real problem?
"The team tried different ways to get comprehensible sign language on low-resolution video. They discovered that the most important part of the image to transmit in high resolution is around the face. This is not surprising, since eye-tracking studies have already shown that people spend the most time looking at a person's face while they are signing."
Would this not be true for any conversation between people?
On the positive side: perhaps this initiative for signers will pay off for everyone. It wouldn't be the first time that designs for people with specific challenges actually addressed problems everyone had to some degree.
Hello my dear readers. Perhaps you missed it, but this website was down for almost two weeks. There were some technical difficulties and in the end I more or less started anew, with a less than perfect backup of the content. So, it is possible that pages are missing or that links malfunction. If you would be so kind to report these things it would be highly appreciated. At least the website is upgraded with new blogging software and a new look.
Enjoy, Jeroen
I noticed a flurry of gesture patents that mentioned a 'portable mutlifunction device'. That's patentspeak for iPhone. The patents were all from APPLE Inc. Well done Apple. That's how you manage a patent portfolio. Philips and IBM used to be the masters in this line of completely covering an area with a barrage of patents. It will give Apple something to negotiate with in future business deals with other vendors.
Here they all are as far as I could tell:
Who will be able to argue with this patent portfolio? Who will be able to claim that the things Apple has patented were already invented elsewhere? Who will be able to maintain that gestures are not technical inventions but natural human communicative actions? Who will pay the lawyers to fight these fights?
Here it all is in a fashion that is easier to digest than sifting through 22 patents.
I think Apple has won this fight before it could even get started.
Although the story of David Healy's flute gesture is getting a little moldy it has generated enough discourse to deserve another mentioning here. The interesting thing about this flute gesture is how it is part of the history of the Northern Ireland sectarian conflicts. Sensitive catholic Irish republicans will get inflamed over the gesture while others have no idea what the problem is.
David Healy making the flute gesture. (source)These flute bands on Orangist marches are what the gesture refers to.Get a glimpse of the triumphalist nature of these marches
By coincidence I am currently reading 'The Irish War' by Tony Geraghty. He sketches a long and messy conflict which has gone on for more than 300 years. It is clear that these marches are of an inflammatory nature, and therefore a gesture that refers to them is also inflammatory. It is not just a merry band of flute-playing men. They celebrate Orangist protestant dominance in Northern Ireland at the expense of the catholic part of the population.
The conflict carried over to a Scottish football match called 'the Old Firm' between the Rangers (protestant) and Celtic (catholic), see this nice historical overview by the BCC. Many Irish people moved to Scotland and brought the conflict with them. Paul Gasoigne made the mistake of making this gesture while he played for the Rangers and paid a heavy fine of 20.000 pounds.
Paul Gascoigne made the same flute gesture during the old firm (Picture: BBC News)
David Healy was not playing for the Rangers, in fact I don't think he ever did, but he is known as a Rangers fan. He is from Northern Ireland and he plays in their national side. However, in this game Healy was playing for Fulham (an English club) in a friendly match against Celtic, which sets the context for the gesture. Healy was 'provoked' by the Celtic fans who knew his sympathies and chanted 'where were you on The Twelfth' (a reference to an important march on the twelfth of July). In response, he seems to have made this gesture somewhat jokingly. The strange thing is that he seems to be escaping the sort of fine Gascoigne got. Why is that? Was Gazza perceived as doing it to inflame Celtic supporters whereas Healy was just fooling around? I think many people will take it more seriously than that. As always happens with sportsmen making inappropriate gestures, Healy is now apologizing and his club is investigating. It wouldn't surprise me if a fine came soon.
The Orange OrderWhat Irish Political Pundits have to say about itA similar incident at Belfast Zoo, involving a panda.CNN reports about the Orange Order marching seasonUpdate: I think an important difference between Healy and Gascoigne is that the latter played for the Rangers who were at that time trying to defuse a tense situation. Gascoigne's gesture was hurting that effort.
Control a Beamed Powerpoint Presentation with Gestures
These students appear to have created a gesture based application that we also considered about four years ago. I know IBM and Philips were interested in this sort of application. So, well done guys! And excellent presentation too. I think they managed to make the best of it, given a difficult application.
Why is a presentation system a difficult application? Well if someone is presenting he will usually gesture during talking. These gestures are directed at the audience and not at the presentation software. So, the first task of such a system is to discriminate between those gestures: what is for me and what is for the audience. Furthermore, a presenter may also be fidgeting during his talk which shouldn't be interpreted as a gesture. Unfortunately, it is unclear whether these students treated these issues.
The things they did do seem to be designed well enough. I think I like the calibration they designed: It creates a connection between the user's physical environment and the camera he must address. It grounds the interaction. The subsequent examples of the functionality they have built in is less impressive. The forward-back commands are okay, but the drawing and highlighting are not very valuable in my opinion. People in the audience can see that you are pointing at something so there is perhaps little need to do more. But maybe these are first steps which need a bit more maturity in their interaction design to become useful.
On the whole, excellent work.
Nijntje DVD in Nederlandse Gebarentaal (NGT)
There is a new DVD for kids created and sold bij Nederlands Gebarencentrum with translations or enactments of ten Nijntje stories in NGT.
It is the first of it's kind in NGT, but similar to this ASL video of Click-clack Moo. In both cases the images of the children's books animate the story. But in the Nijntje example the 'storyteller' dressed up like the Nijntje character. Funny though that may be it seems a bit beside the point. The voice-over in spoken Nijntje DVD's does not represent the voice of Nijntje but is a storyteller. But here I go again with the sour comments. Stop it! It's fun and good.
ps. It reminded me of Woof woof way, an ASL DVD where the storyteller dressed up as Paws, the dawg.
Ninja Strike, a killer application for gesture recognition?
This is certainly an interesting development. Previously we have seen mobile phones using motion and acceleration sensors for gesture control (see here and here). There have also been applications where the camera was used to simply capture optical flow: something in front of the camera is moving/turning in direction A therefore the phone is moving/turning in A + 180 degrees (here). In this case the gesture recognition appears to go a step further and at least the hand appears to be extracted from the image. Or does it simply assume all movement is the hand? And then perhaps the position of the motion is categorized into left-middle-right? Maybe the velocity is calculated but I don't think so.
Update: I do like the setup of how people can hold their phone with the camera in one hand, throw with the other and check their virtual throw on the display. The virtual throwing hand on the display is more or less in the same position as your physical hand, which I think is nice.
EyeSight is a techno start-up of 2004 from the Kingdom of Heaven (Tel Aviv) aspiring to use nothing but Air and a Camera to achieve a divine interaction between true techno-believers and their mobile phones. They prophetize that their technology will 'offer users, including those who are less technologically-adept, a natural and intuitive way to input data, play games and use their mobile phone for new applications'. Heaven on Earth. Mind you, nothing is carved in stone these days. Besides, human nature and intuition are all too often deified these days anyway. Human nature is what usually gets us into trouble (not in the least in the Middle East).
Anyway, one of their angels called Amnon came to me in the night bearing the following message:
Hello Jeroen,
First Allow me to introduce myself. I'm Amnon Shenfeld, RND projects manager for eyeSight Mobile Technologies.
I’ve been following (and enjoying) your BLOG reports for a while, and I thought that the following news from my company, eyeSight Mobile Technologies, may make for an interesting post.
eyeSight has just launched “Ninja Strike”, an innovative mobile game featuring a unique touch free user interface technology we call eyePlay™. Allow me to provide some background information about eyeSight, eyePlay and Ninja Strike: I’m sure you are aware of the popularity and attention innovative user interfaces are getting since the introduction Apple’s IPhone and Nintendo’s Wii… My company’s vision is to bring this technology into the mobile market, and our first products are focused on changing the way mobile gamers play. Our new game, “Ninja Strike”, does exactly this.
You play a ninja warrior with Ninja Stars as your primary weapon. Your stars are thrown by making a throwing motion in front of the phone’s camera. Much like training in real life, during the game you will learn how to throw your weapon correctly, and improve your aim. Your enemies, the evil Kurai ninjas, will also gain strength as the game advances…
Looking forward to hear from you, I hope to see a new post in your blog soon, you've been quiet for a while… J
Amnon Shenfeld
Amnon, will you heed my calls? Have you answers to my burning questions above?
eyeSightgame demo
In the UK, the i-Limb has won an innovation prize. It is a prosthesis with fingers that can move independently (I would think that certain motor programs require a very close 'functioning together' of the fingers, but perhaps they incorporated such things). It gets its input from myoelectric signals from the arm's muscles.
i-Limb by Touch Bionics (source BBC)
Onno got a grant from the European Union that will allow him to start his own research group. He will investigate what happens 'on the other hand' in signed languages.
I have worked as an interaction designer and usability specialist for many years. At the moment, the fashionable thing to be designing for is 'user experience'. Several colleagues here at TU Delft are trying to define user experience to help designers and researchers. Arnold Vermeeren organized a workshop on the definition of user experience at the last SIGCHI conference, and he gave me this link to some reading material: MAUSE. Towards the MAturation of Information Technology USability Evaluation. Paul Hekkert and Rick Schifferstein, also colleagues wrote a book called Product Experience which contains many interesting chapters but is a bit of an investment at $170.
Arnold is also working on a nice graphic that contains the elements of User Experience. He is not the only one though, see this collection of images that try to capture user experience.
Emotion is the core
Utility/Usability, Personal Social Meaning, and Aesthetics are the first contributors
Finally, there is a timeline involved starting with anticipation and ending with reflection
But other people have other opinions. Here is a nice graph too. Here is a nice paper on it.
The link between user experience and gesture is that through gesture control one might expect an increased aesthetic appraisal. It might appeal to your senses because you like to move. Or because you like to move in ways that you are familiar with.
Typically, people can see whether a movement is intended to communicate (a.k.a. a gesture) or whether the movement’s producer has some other intention, be it practical or just fidgeting. There are however plenty of examples where the movement is ambiguous: it could be a gesture but it could also be a meaningless incidental movement. Barack Obama produced such a movement during a speech. Watch and judge for yourself.
Did Obama just flip off Clinton or was he merely scratching his cheek?
Again, like in many other cases where the nature of a movement was debated, there is a potential insult to be considered. It is almost as if people are more sensitive to potentially insulting gestures then to other gestures. Some people, like Lehmann or Mr Wood even use this sensitivity to their advantage. They camouflage their insulting gesture and thus create ambiguity on purpose. Those who have a reason to feel offended are insulted by the ‘gesture’. Other people only see a cactus or someone scratching his head.
I would predict that if people must judge if a movement is intended to communicate they will do so more often when that would mean it is an insult than when that would mean it is some other gesture. (Question: Can you think of an experiment to test this prediction?)
BTW, there is a very interesting related paper on this topic from a psychiatric perspective:
Bucci, Sandra, Mike Startup, Paula Wynn, Amanda Baker, & Terry J. Lewin. (2008). Referential delusions of communication and interpretations of gestures. Psychiatry Research, 158(1), 27-34. (Scopus)
Gestures are an important aspect of non-verbal communication, but people with schizophrenia have poor comprehension of them. However, the tests of gesture comprehension that have been used present only scenes in which interpersonal meaning is communicated, though there is evidence that people with psychotic disorders tend to perceive communications where none were intended. Such mistakes about non-verbal behaviour are the hallmark of a subtype of delusions of reference identified as delusions of communication. Thus we hypothesised that patients with delusions of communication would tend to misinterpret incidental movements as gestures and, since delusions are often derogatory to the self, they would also tend to misinterpret gestures as insulting. Patients with acute psychotic symptoms (n = 64) were recruited according to a 2 × 2 design (presence vs. absence of delusions of communication by presence vs. absence of auditory hallucinations). They, and 57 healthy controls, were presented with 20 brief video clips in which an actor either made a well-known gesture or an incidental movement. After each clip, they selected one of four interpretations: a correct interpretation if a gesture had been presented; the interpretation of a different gesture; an insulting interpretation; no gesture intended (correct for incidental movements). The patients made significantly more errors of all kinds than the controls, perceived significantly more of the incidental movements as gestures, and selected significantly more insulting interpretations of the clips. These differences between patients and controls were almost wholly due to patients with delusions of communication. These results suggest that the difficulties that people with delusions of communication experience in understanding gestures can be explained, at least in part, by the misattribution of self-generated internal events to external sources.
Perhaps we all suffer from delusions of communication to some degree when we are in a situation where we expect to be insulted (rightly or wrongly). I know I always check for fingers when I feel I did something impolite in traffic. Don’t you?
The Wanker gesture is the second most often occuring gesture in the news, with giving the finger a mile in the lead. In third place is the 'fuck-you' forearm jerk. What do these have in common? They are insults, and unexpected insults are apparently very newsworthy.
(source: Wikipedia)
The latest wanker case comes from the Austin, Texas, US...
From Keyetv.com: Travis County Court at Law #6 Judge Jan Breland put Adam Reposa into jail after he made what is described in court documents as "…a simulated masturbatory gesture with his hand while making eye contact with the Court…"
Do children learn language from rich (enough) input or do they invent it more or less on their own, driven by some innate program? That is a question that has kept great scientists busy, particularly Noam Chomsky.
And so with modern gesture research (post Chomsky) and modern sign language research (post Stokoe/Tervoort) the question became important which role gesture and emerging sign language skills plays in the development of language and cognition in hearing children and deaf children, see the work of Susan Goldin-Meadow and co-workers in particular.
A famous case is the discussion surrounding the documented invention of Nicaraguan Sign Language by successive generations of deaf children (by Judy Kegl and others).
But it appears that not only children can create language. A local newspaper here reported that the oldest man in the Netherlands (age 106) lost hearing and speech and invented a 'sign language' with his daughter in law to communicate.
Adrianus van der Vaart and daughter-in-law Corry created a sign language (source: AD)
Did 'Opa Arie' take a dip in the fountain of youth?
Is there no such thing as a critical age of acquiring/inventing a language?
Or did the newspaper exaggerate?
Given the nature of newspapers it is likely that the AD exaggerates. Besides, any sort of gesture system is quickly called a 'sign language' in the Netherlands, and little distinction is made by the general public between 'genuine Sign Language of the Netherlands (NGT)' and other 'gebarentaal'.
Further research is needed urgently however, before it is too late. The potential 'Wilnis Sign Language' (Wilnis is an isolated village in the Netherlands with a remarkable population of elderly people with bad hearing) should be documented by the likes of Judy Kegl? Can anybody send in a linguist?
Mark van Bommel is a Dutch football player who plays for Bayern Munich. He is also a bit of a drama queen, who already supplied us with a decent little gesture scandal before. After already being fined 6200 euro for making a 'fuck you' gesture (forearem jerk) to the crowds at Real Madrid's Bernabeu stadium, he repeated the gesture (now dubbed 'doing a Van Bommel') in the final minutes against Hamburg, this time insulting the referee. This time he got an extra two match ban and a 15000 euro fine (compare other fines) What sort of fine would be enough to stop him?
Van Bommel's little theatrical performance includes some sarcastic clapping. There is an interesting piece by Steve Tomkins on the correct and incorrect application of sarcasm. He also has some cases of football players clapping sarcastically. In a lovely cool sarcastic style Tomkins sets out to explain why sarcasm is not necessarily rude. If done correctly sarcasm can deliver venomous bites. The key lies in not overdoing what you say or how you gesture, and in the inclusion of a pinch of humour. Obviously, if your victim really did or said something stupid it will make sarcasm succeed easily (your victim can't get angry because he is too busy being ashamed).
In this case, Van Bommel overdoes the sarcasm. First, the referee is always right, so he is free to get angry, and indeed he responds by showing the red card. In such a case one must leave room for innocent interpretations to hide behind. If one simply keeps a straight face and claps twice it can always be explained as saying 'good call, ref!'. The fans in the stadium will understand the sarcasm but there will not be enough evidence to punish you. I think that in most cases even the slightest hint of sarcasm will be enough for other people to pick up the message. We humans are so sensitive to insults. Often the mere context will make a straightforward interpretation of an innocent phrase that is intended as sarcasm very unlikely.
Him: "did you watch the game yesterday?" (your team lost 2-0 to his, the bastard)
You: "yes, great game for the neutral spectators" (it was a teeth-grinding muddy fight)
Him: "we gave your lot a good whipping, hey?" (obnoxious little fella, everyone could tell the ref wrongly sent a man off after ten minutes)
You: "yes, a victory well deserved. They are really on fire these last few weeks aren't they?" (they only had one win against a second division club and three losses)
Every knowledgeable bystander (or at least those you care about) will get your point from the context. But in comparison to your obnoxious colleague you appear to be gracious about it all. He may or may not spot your sarcasm but he has no good option to respond. He can either take your words literally and be a fool, or he can acknowledge your sarcasm and respond to the unspoken allegations (that it was an ugly match, an undeserved victory and a team that sucks anyway). The latter choice will have him on the defensive ("I thought it was a proper red card for hands") in which case you can turn up the sarcasm ("of course, he should have stopped the ball with his genitals").
ps. Thanks to Elif for the link to Tomkin's piece
America, as usual, appears to have unique powers of division. In this case America is divided into Deaf America and deaf America, into oralists and manualists (?), into those who respect Alexander Graham Bell and those who reject and denounce his legacy. Bell, whose wife and mother were deaf, devoted much of his life to improve the teaching of speech to deaf and hard of hearing children. As such, he was hardly a champion of sign language, see the following excerpt from Through Deaf Eyes:
In 1884, Bell published a paper “Upon the Formation of a Deaf Variety of the Human Race,” in which he warned of a “great calamity” facing the nation: deaf people were forming clubs, socializing with one another and, consequently, marrying other deaf people. The creation of a “deaf race” that yearly would grow larger and more insular was underway. Bell noted that “a special language adapted for the use of such a race” already was in existence, “a language as different from English as French or German or Russian.” Some eugenicists called for legislation outlawing intermarriage by deaf people, but Bell rejected such a ban as impractical. Instead he proposed the following steps: “(1) Determine the causes that promote intermarriages among the deaf and dumb; and (2) remove them. The causes he sought to remove were sign language, deaf teachers, and residential schools. His solution was the creation of special day schools taught by hearing teachers who would enforce a ban on sign language.
It is hard to imagine a more ruthless approach to the 'problem' of a more or less isolated American Deaf subculture. Let me just state my position clearly: I would have been against it. No. Lucky for me I am Dutch and not American and therefore I do not feel called upon to state my position. I feel free to rise above the divide. Oralism existed here too, but I dare say it was less extreme, and the opposition was therefore less extreme too. Needless to say, the advocates of (also) using sign language are on top at the moment and oralism in its extreme form has all but vanished. Within this context, the following video was recently put online. It is a good illustration of the strong emotions felt by the victims of oralism.
Source: Daveynin: "Scene parody plot: Hilter is angry; his final defeat against sign language has now pushed him over the edge."
I was wrong. Wrong about the necessity of hands for gesturing (actually, for waving, to be exact).
(source: ShoutWire)
Apparently, any old hook will do instead of hands. Or should we say that this case is a special case of projection? Perhaps it is the placement of the hook on Pirate Arm's End that enables the illusions? We have no problem seeing gestures because the hook is clearly a substitute hand?
Would the magic disappear if the hook was alone, without an arm attached? Perhaps it will and perhaps it won't. One thing is for sure though, my hopeless over-analyzing of anything gestural is killing this otherwise decent joke.
The 2008 8th IEEE International Conference on Automatic Face and Gesture Recognition is upcoming. It will be held in De Rode Hoed in Amsterdam, September 17-19, 2008.
FG2008 logo
22 March is the deadline for the submission of regular papers. There will also be a workshop on the 'Psychology of Face and Gesture Recognition'. That should be interesting. And Paul Ekman is one of the keynote speakers.
Detailed instructions for paper submission are provided. It should be six pages (unless you want to pay them $100 extra :-) ). I am told that getting accepted is not easy, but I will try nevertheless. You have to subscribe to an online submission system, it appears.
I hope to be contributing with two papers. The first I will write together with JL and covers our experiment with acceptability judgments (man vs. machine). The second will be written by GH and me and will be more of an overview of how we try to link perception to automatic recognition.
With great delight I read Herman Roodenburg's 2004 book called 'Eloquence of the Body. Perspectives on gesture in the Dutch Republic'. I already posted about it earlier with respect to Jelgerhuis and a lovely video about 'Welstand'. Roodenburg is a scholar based at the Meertens Instituut, who has written on various topics where history and sociology meet.
Cover of the book (source)
Some of his works have been translated into many languages. I previously read 'A Cultural History of Gesture from Antiquity to the Present Day, red. J.N. Bremmer en H.W. Roodenburg (Cambridge, 1991)', of which I found a 1993 Dutch translation in my local bookshop. But his documentations of the histories of honour, humor and corsets also captured my imagination.
The current work, Eloquence of the Body, is a time machine. It took me, a young Dutch man, back to the days of Constantijn Huygens (1596-1687), an important man in the history of the Dutch Republic. And Roodenburg makes it happen because he seems to have developed an ear for the stories that books tell about their previous owners. He scrutinizes the library lists of the Huygenses and compares them to their correspondence and is deeply engrossed in his studies of the old texts on civility (Castiglione, Erasmus, etc.). And from all this scholarly groundwork rises a clear picture of the role of gesture and bodily memory in the minds of the people at that time.
The chapters:
- Castiglione's Paradox
- Civility and the Dutch Republic
- Incarnating Civility
- Painting and Civility
- Acting and Civility
- Preaching and Civility
There are also many notes and a large bibliography, which unlock the old writings on gesture for any novel student such as myself. It is a guide to thoughts of the past. To stimulate the imagination many images illustrate the points made by the author. For me, it was also a free lesson on art history and in how painting and acting (and preaching) are related to gesture.
Sadly, what is lacking is a bridge from the past to the present. I would have loved to read whether the 'welstand' for example is still practised today in the Netherlands. Or how children are taught manners today. Or perhaps some remarks about the 'Dutch Identity', a vague term that has been hijacked by nationalistic politicians. I would hope that Roodenburg can make more sense of what is typically Dutch (from a historical perspective) than them.
Orange, the phone company, is exploring and developing a nice new niche market for gesture recognition: The Interactive Shop Window.
Could it be one of the killer apps for gesture recognition? I think it is possible. The advantages are quite clear. As a unique selling point gesture recognition offers the possibility to interact through existing windows, simply because a camera can see through them. I do not think any other interaction technique can do that. Voice recognition doesn't travel through windows. Touchscreens need to physically available, as any other haptic device (mouse, keyboard, etc.).
Review by Gizmodo
Made by The AlternativeUpdate Feb 3: Reviewed by the Dutch Marketing Facts
I just noticed that there is a new case of people trademarking a gesture. In this case it is tennisplayer Lleyton Hewitt . However, he did not 'invent' the gesture (if such a thing could be said about a gesture in the first place), another tennisplayer called Niclas Kroon was apparently the first to trademark this "Vicht" salute, and he shared the rights with Mats Wilander.
The gesture and accompanying "C'mon" are claimed to be known as "doing a Lleyton". (source & pic: Getty)
I you find it hard to believe, you may share it with the people on this forum. I merely find it interesting to witness (there are so many other 'ridiculous' things that can and will be trademarked). It is certainly not the first time it happens. Besides Kroon, there was once a wrestler called Diamond Dallas Page who fought a legal battle with rapper Jay-Z over a trademark violation of his 'Diamond Cutter Gesture'. One seems to hear about gesture trademarks only when there is a dispute (as it is often with patents as well).
If you want to study the matter, read Henry Abromson's good piece on gesture trademarking and the legal case of Page vs. Jay-Z, from which the following pictures were taken:
V.
Oh my god.
I just read the abstract of the first journal paper I have ever published (not counting my writings for the University's weekly, the Delta). It is horrible writing! After a long and tedious process of writing, revising, co-editing, and revising some more it is unreadable!
The present and past tense are alternated seemingly at random. Bizarre sentences arose from disputes and compromises. Strange ambiguities must leave even the most forgiving reader puzzled and irritated. I promise here and now to do a better job next time.
Well, leaving the abstract aside, the paper contains the results of my first experiments, which are not entirely uninteresting I think, regarding the perception of signs and fidgeting. It is called 'When and how well do people see the onset of gestures'.
To explain what it is about let me give you a little task. Next time when you are away from your computer and among the people, look for two or three people grouped together, talking or otherwise. Preferably they are at enough distance so as not to be overheard (or take offense at your snooping). Now, watch their movements, their hands. And as soon as you see a gesture you knock on something.
If you do this you will probably experience three things. First, it is an easy task. Second, you will probably ignore the fidgeting. Third, if you ask a friend to join your little game of gesture-spotting (as I have done on many occasions), he will probably knock in response to the same movements that you do.
Now, the ability to spot gestures, and ignore fidgeting, may seem trivial to you, but imagine you were a robot. Do you think computer vision experts would have been able to program you with this ability? At the moment, they don't even have a clue how to solve this puzzle. You would be forced to analyze any movement as 'possibly a gesture I might have to respond to'. And perhaps you would wish you had something to filter out the 50% 'just fidgeting' (estimate by intuition) to save you time that you need to keep from bumping into lantern posts and such.
Here is a wonderful video that sums up and illustrates the main ideas about gestures that are available in the work of Johannes Jelgerhuis, a famous artist of this time as a painter, illustrator and actor. At the same time the video follows the story of a recent book by Herman Roodenburg, Eloquence of the Body, which is however mentioned nowhere. But the illustrations are all in that book and the ideas are the same as well, although I am only halfway in the book so far. I scanned the book written by Jelgerhuis between 1827 and 1830, of which an original copy is available in the library of Utrecht. It is a wonderful series of ´lessons´ about gesture, posture and mimicry with a large collection of plates, witch Jelgerhuis etched himself. There is also a 1970 reprint avaliable with Dutch and English summaries and all the plates. Furthermore there is a full translation, with annotations, available in Golding's book on classicistic acting.
The ideas of the eighteenth century about the 'welstand' in Holland, or more generally about appropriate bodily behavior for the upper classes are inspiring. They contain an appealing myth: acquire a certain second nature of standing, walking, dancing, etc. and you will be lifted in society. People will value your behavior and may consider you their peers. Yet at the same time, paradoxically, such a second nature was only attainable for those of the right first nature. If you didn't have the right breeding then you might as well forget it.
Ever since I read the books I have been trying to walk more upright, keep my feet at right angles, and behave and talk as elegant as I can. Alas, I slip into my burly uncivilized manners as often as not. I blame my parents for not putting me into a straightjacket as a boy and sending me off to dancing and fencing lessons (I forgive them for my genes, they were all they had to give).
But Jelgerhuis' book is not about life. It is about acting. He wrote about the art of gesture on stage (see this earlier entry for links to Johann Jakob Engel, but also to the memory of Dene Barnett). But then, life and acting are not exactly two mutually exclusive categories.
Literature
JELGERHUIS RZ, JOHANNES Theoretische lessen over de gesticulatie en mimiek, gegeven aan de kweekelingen van het fonds ter opleiding en onderrigting van tooneel-kunstenaars aan den stads schouwburg te Amsterdam. Amsterdam, P. Meijer Warnars, [1827-1830].
Roodenburg, H.W. (2004) The Eloquence of the Body. Studies on Gesture in the Dutch Republic. Waanders, Zwolle, 208 p. 1e dr. (Studies in Netherlandish Art and Cultural History, nr. 6)
Golding, Alfred Siemon; Jelgerhuis, J. (1984). Classicistic Acting: Two Centuries of a Performance Tradition at the Amsterdam Schouwburg to Which Is Appended an Annotated Translation of the Lessons on the Principles of Gesticulation and Mimic Expression of Johannes Jelgerhuis, Rz. University Press of America
Here is a guy called roschler demonstrating how to (learn to) control an i-Sobot, the world's smallest humanoid robot, with Wii-gestures instead of using a complicated remote control. Because the robot's routines are mostly gestures you can create commands for them through imitation or iconicity. Probably, actions will often have to be simplified, since I do not expect people will want to make an actual somersault to tell the robot to do that. Also, for certain actions or scripts I imagine that the gestures will become arbitrary and not really intuitive, but probably still easier to use than the alternative RC.
Robot control, another good niche for gesture recognition?
i-SOBOT by ThinkGeek: World's Smallest Fully Articulated Humanoid RobotBuy an i-SOBOT at Amazon for about $250
Watch a fight: i-SOBOT vs. Godzilla
More about Robodance, a.k.a. Robosapiens Dance Machine, which can also be used to control WowWee Robots.
The ZCam, from 3DV Systems, is featuring in several recent movie demos on the YouTube. It is a camera with onboard technology (it emits infrared pulses and catches refections) that not only provides RGB values for a matrix of pixels, but also a Z-score, i.e. the depth of that pixel. It is reported to be quite accurate. It is also reported to become cheaper, and therefore more widely available to, for example, game developers.
The ZCam, small but deep?
So far, there are demos shown of a squash game, a boxing game, and a flight simulator. But the website from the company has a more extensive video gallery.
Here is a nice review on Gizmodo, where they always keep an eye out for interesting gadgets like this. Crowd response has been positive so far but it remains to be seen whether this technology will be adopted in the market. It is at the moment a bit hard to say how well the technology works, what sort of drawbacks there are, etc.
But if it does work well than I think it has less disadvantages than stereo-cameras. I have some experience with a stereo-camera (two synchronized cameras with a software algorithm to integrate their output and obtain depth). This is expensive, costs processing power, costs time to configure, is unreliable (it should not be moved or even touched after configuration), and it relies on skin color segmentation to track faces and hands.
But what about the ZCam, what are users required to do or not to do, that is the question.
See here a nice video on the YouTube featuring a lot of gestures (it is a group of Sicilians at the dinner table). Thanks to Kensy Cooperrider and his embodied talk weblog for digging up some video gems.
A Sicilian Argument
One of the things I like in this video is that you can see some emblems in action, such as the mano bursa (grip hand). It is made near the end of the video by the man in the blue shirt.
Kendon (1995, 2004) wrote extensively on how gestures like the ones you see here can be used with a high degree of conventionality. The ring hand alone is used in a variety of conventional ways. It is not hard to imagine that indeed this group of people knows how to make their gestures count.
Snapshots of the three signs used in the experiment.
Examples of three manipulations of the sign SAW. We tested about 68 sign manipulations in total. These were run through the automatic recognition algorithms we had been working on and they were rated by human signers. The paper is about how humans and machines can be compared.
The talk by Magnenat-Thalmann focused on facial expression. (
Arend Harteveld (
Jesus and his 
David Healy making the flute gesture. (
These flute bands on Orangist marches are what the gesture refers to.
Get a glimpse of the triumphalist nature of these marches
By coincidence I am currently reading 'The Irish War' by Tony Geraghty. He sketches a long and messy conflict which has gone on for more than 300 years. It is clear that these marches are of an inflammatory nature, and therefore a gesture that refers to them is also inflammatory. It is not just a merry band of flute-playing men. They celebrate Orangist protestant dominance in Northern Ireland at the expense of the catholic part of the population.
The conflict carried over to a Scottish football match called 'the Old Firm' between the Rangers (protestant) and Celtic (catholic), see this 
Paul Gascoigne made the same flute gesture during the old firm (Picture: 
i-Limb by Touch Bionics (source BBC)
Arnold is also working on a nice graphic that contains the elements of User Experience. He is not the only one though, see this 
(source: 
Adrianus van der Vaart and daughter-in-law Corry created a sign language (
(
FG2008 logo
22 March is the 
Cover of the book (
(
The gesture and accompanying "C'mon" are claimed to be known as "doing a Lleyton". (source & 
V. 
The ZCam, small but deep?
So far, there are demos shown of a squash game, a boxing game, and a flight simulator. But the website from the company has a more extensive 
