Name calling in SLN

There is an interesting initiative to include name calling or swear signs in the SLN lessons for parents of Deaf kids was organised by a group of parents of Deaf kids. According to their leaflet (in Dutch) they will be treating the following signs in the first lesson: klootzak (dysphemism for scrotum), idioot (idiot), and sukkel (idiot/loser). See posts on Doof.nl and FOK (with many comments). It made me wonder how much of the vulgar Dutch emblems are part of the SLN lexicon. In fact, I am quite curious in general about the relation between signs and a host culture's emblems (which according to Ekman & Friesen (1969) and Morris et al. (1979) are a recognizable set of symbolic gestures with a conventionalized meaning in a culture that can be used to substitute words or parts of speech. Kendon (1984, 1992) calls them 'quotable gestures'). I will check the set of 20 key gestures from Morris et al. (1979) against an SLN dictionary (Standaard Lexicon NGT, deel 1+2, NGC, 2002) to see which of those 20 emblems are also SLN signs. It should be noted that according to Morris et al. only 12 of the gestures are commonly known in the Netherlands, which I checked with 'Het Gebarenboekje' by Andrea & De Boer (1993) to see if they concurred, which by and large they did. One gesture is at least a well known SLN sign: the Fig. Do you know what it means in the Netherlands and in SLN? (Source: Morris et al. 1979)

Medal of Honor Heroes 2 - Gesture Trailer

There is a new release in the game series Medal of Honor (see also Medal of Honor Vanguard), which again bets heavily on the Wii gesture control.

Trailer: Experience WWII with an all new Wii-mote control scheme!

GameTrailers commenter: For once a Wii control scheme that actually looks intuitive and intricate as opposed to gimmicky. Ill be picking this up tomorrow

Lehmann's Perfect Finger Camouflage

Here is one of the nicest 'Fingers in camouflage' I have ever seen on video.

Lehmann middle finger
Uploaded by goalbumbum

Offside Bundesliga: When confronted Lehmann said: “It’s nonsense. I’ve never done anything like that”. It supposedly happened after one of Lehmann’s risky trademark excursions outside of the box, during the Germany - Cyprus match on Saturday (4-0). The crowd in Hannover started Robert Enke chants. Enke happens to be Hannover’s goalkeeper and Jogi Löw’s current third choice behind Lehmann and Hildebrand.

Lehmann, whose workout was also a bit ambiguous for some, has shown himself a true master of the camouflaged gesture. Playing into people's increased sensitivity he manages to insult those he wants to insult, while he can claim innocence in public. Bravo, a perfect grasp of the perception of insults. Much better than Joe Nedney, who got a $7.500 fine (which is average, see these other fines for similar offences) for his lousy camouflage attempt: Clearly the man is scratching his head (source) The only examples I have of an even more subtle camouflage is when a man painted an abstract work of art, a huge cactus, on the side of his house facing his (complaining) neighbors. They were highly offended, but he got away with it (and eventually removed it).

The Acceptability of Sign Manipulations

My latest research revolved around the question 'when is a sign production still acceptable', or rather, given that context and application of different norms have a big influence on acceptability, 'which (types of) variations are more acceptable?' We ran an experiment which I think has yielded interesting results. Tuesday Jan 22 I will be giving a presentation at the MPI Nijmegen about the experiment and the results as part of the Nijmegen Gesture Centre Lecture Series 2007. This will be in English. At our own workshop 'Een mooi gebaar 2007' I also gave preliminary results in a short 10 minute presentation in Dutch.

Gestures are Sterile, Touching is not

At the renowned Fraunhofer institute they may have built a killer gesture app: Gesture control that lets surgeons control a 3D-display of a head (for example) during surgery while remaining sterile (touching buttons would break sterility, I guess). Rotate the 3D image by gesturing (source)

Press Release: Non-contact image control As if by magic, the three-dimensional CAT scan image rotates before the physician’s eyes – merely by pointing a finger. This form of non-contact control is ideal in an operating room, where it can deliver useful information without compromising the sterile work environment. The physician leans back in a chair and studies the three-dimensional image floating before his eyes. After a little reflection, he raises a finger and points at a virtual button, likewise floating in the air. At the physician’s command, the CAT scan image rotates from right to left or up and down – precisely following the movement of his finger. In this way, he can easily detect any irregularities in the tissue structure. With another gesture, he can click on to the next image. Later, in the operating room, the surgeon can continue to refer to the scanner images. Using gesture control to rotate the images, he can look at the scan of the patient’s organs from the same perspective as he sees them on the operating table. There is no risk of contaminating his sterile gloves, because there is no mouse or keyboard involved. But how does the system know which way the finger is pointing? “There are two cameras installed above the display that projects the three-dimensional image,” explains Wolfgang Schlaak, who heads the department at the Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institut HHI in Berlin that developed the display. “Since each camera sees the pointing finger from a different angle, image processing software can then identify its exact position in space.” The cameras record one hundred frames per minute. A third camera, integrated in the frame of the display, scans the user’s face and eyes at the same frequency. The associated software immediately identifies the inclination of the person’s head and the direction in which the eyes are focused, and generates the appropriate pair of stereoscopic images, one for the left eye and one for the right. If the person moves their head a couple of inches to the side, the system instantly adapts the images. “In this way, the user always sees a high-quality three-dimensional image on the display, even while moving about. This is essential in an operating theater, and allows the physician to act naturally when carrying out routine tasks,” says Schlaak. “The unique feature of this system is that it combines a 3-D display screen with a non-contact user interface.” The three-dimensional display costs significantly less than conventional 3-D screens of comparable quality. Schlaak is convinced that “this makes our gesture-controlled 3-D display an affordable option even for smaller medical practices.” The research team will be presenting its prototype at the MEDICA trade fair from November 14 to 17, 2007, in Düsseldorf (Hall 16, Stand D55). Schlaak hopes to be able to commercialize the system within a year or so.

Things like this are probably the best bet for the near future of gesture recognition. Niche applications that exploit some specific benefit of using gestures instead of (or besides) other, more mundane interface technology. The biggest hit in gesture land is without a doubt the Nintendo Wii, which exploits another unique selling point of gestures: a higher (or more representative) physical involvement leading to a better 'experience' of a game. It specifically targets gamers who are interested in fun and exercise in a social context. I doubt that hardcore gamers, intent on getting to higher levels of killing sprees, will be very keen on the Wii. And so it will probably remain for the near future. Like with speech recognition, gesture recognition will have to find some nice niches to live in and multiply. Maybe one day, the general conditions will change (ubiquitous camera viewpoints? intention-aware machines?) and gesture can become the dominant form of HCI, driving buttons to niche applications. I wouldn't bet on it right now, though.

Larry Craig's Cottaging Gestures

Here is a nice cartoon, a caricature of a supposed gesture system used by Senator Larry Craig to not solicit gay sex in a restroom. The practice is known as cottaging. A cottaging gesture? (source) Frankly, I don't give a damn about whether Craig is gay or not, or whether he elicited sex with other gay men or not (fine by me), though I would rather he did so in a place where he could be sure the invited party was of the same mind (and not possibly some unwitting hetero). But the cartoon is nice, a perfect example of a man-invented little gesture system. The case bears a remarkable resemblance to that of George Michael, who was also invited by an undercover cop who apparently even showed his dick first (source). I think there is a good line of defence in the following reasoning: If you are guilty of 'soliciting sex' by using these gestures then it is firmly acknowledged that there is a code of behaviour that governs the interaction. Therefore the behaviour of the undercover cop in the toilet booth also has to have been in accordance to this code. Perhaps he wasn't making explicit hand gestures, but he had positioned himself in the right spot, assumed the right attitude, and perhaps kept silent where he was supposed to (while a normal response might have been "it's taken"). In my definition all these actions are also gestures (gesture-complex). In other words, both men were soliciting sex by their conduct, and the policeman was the initiator. Is that allowed under US law? A related 'gesture' system is the bandana code. Although I am not intimately aware of where and when this code applies.

In Memoriam: Washoe

Today I received news that Washoe died at age 42. Washoe was a chimpansee that was taught sign language in order to study the extent to which she would be able to gain language skills. An impression of Washoe talking with a researcher (source) Here is more information on attempts to talk with chimps. May she find peace in death. In her life she caused (through no fault of her own) more conflict between humans than most humans. Update: Here is Carl Schroeder's tribute (ASL vlog)

Future President's Gesture?

Here is a post that's good for a laugh (if you're not in love with US politicians) on Geenstijl, a popular Dutch news blog. Unfortunately for the majority of readers, the dodgy jokes are in Dutch. Which is why I'll translate them to English. Who knows, maybe the jokes will end up back in the US? Hi, I'm Hill You know, the wife of Bill Now, I'm looking for something bigger Yeah right, about this thick? I've only got this It's gotta be about this length Take me, take me! Shit! F*ck. Never mind. For those of you who can not believe I am lowering myself to such a cheap shot: The story exemplifies the limits of our ability to see just about anything in a gesture. Even though speech may be required to interpret gestures, the liberties taken here are clearly too much.

A Case of Co-Speech Gestures

A wonderfull new video on YouTube of two guys (programmers, it says) talking and 'co-speech-gesturing' (is that a verb?). "Real programmers use sign language" (by ekabanov) I think it is safe to assume that it is for real. Their whole behaviour looks too natural and wacky to be scripted. I also think this is a great case study to spend some time on while discussing some of the ideas of David McNeill. Because what we have here is what his theories and ideas are concerned with. There is (of course) no sign language nor did I spot any other 'emblematic gesture' (those vulgar things you get fined or jailed for or the goofy ones that seem to be must-haves for ad campaigns). I also do not see any pantomime. No, this is the stuff they like in Chicago: Co-speech gestures. An episode full of deictics, beats, iconic and metaphoric gestures, right?

From the McNeill lab: A misconception has arisen about the nature of the gesture categories described in Hand and Mind, to wit, that they are mutually exclusive bins into which gestures should be dumped. In fact, pretty much any gesture is going to involve more than one category. Take a classic upward path gesture of the sort that many speakers produce when they describe the event of the cat climbing up the pipe in our cartoon stimulus. This gesture involves an iconic path-for-path mapping, but is also deictic, in that the gesture is made with respect to an origo --that is, it is situated within a deictic field. Even "simple" beats are often made in a particular location which the speaker has given further structure (e.g. by setting up an entity there and repeatedly referring to it in that spatial location). Metaphoric gestures are de facto iconic gestures, given that metaphor entails iconicity. The notion of a type, therefore, should be considered as a continuum --with a given gesture having more or less iconicity, metaphoricity, etc.

Wrong! Apparently the main problems of McNeill's typology of gestures, that has sent many an engineer on a wild goose hunt for iconic gestures, are now even recognized at the source (McNeill, 1992). It is not mutually exclusive but rather an index of the functioning of a gesture ('as a beat' - 'through spatial reference (deictic)' - 'referring thorugh iconicity to something concrete' - 'referring via iconicity first to something concrete and second through metaphor to something abstract'). Good. I never liked 'beats' for example. I don't think I ever saw one. But to say that it was a misconception... I vaguely recall an annotation procedure called the 'beat filter' that begs to differ. Anyway, at least this clears up the discussions regarding 'metaphoric gestures' considerably [they are de facto also iconic, the metaphor functions on another level]. And it also clears the way for an annotation of this video. Any volunteers? Well, you would have to get a decent file of the movie instead of the YouTube flash stuff anyway, so let's forget about it. McNeill wrote a new book recently (2005) which is mostly about growth points. But before you read the summary by McNeill you might want to check Kendon's brilliant poem called 'The Growth Point', which he delivered at McNeill's festen. I find it neatly captures my feelings towards growth points (and more that is beyond my grasp). I am at once awed, baffled, and stupefied when I read about growth points and catchments. And so it goes. Again I tried to get it. Again I failed to learn anything from reading about growth points. One thing only. If David McNeill (or Susan Duncan) is right, then annotating gestures in episodes like this will be eternal hell :-) And without the speech it will not work. Thank God. I can go to bed with a clear conscience. Books: McNeill, D. (Fall 2005) Gesture and Thought. Chicago: University of Chicago Press. McNeill, D. (2000) (Ed.). Language and Gesture. Cambridge: Cambridge University Press. McNeill, D. (1992). Hand and Mind. Chicago: University of Chicago Press.

Gesture Definitions

I am going to try and coin some definitions regarding gestures.

A gesture is any act, except speech, by which we intend to communicate something beyond the act itself

This definition includes all normal gestures, that have meaning (the message that was communicated intentionally) because of cultural conventions (including languages) or iconicity, and all such acts (like giving flowers) that serve, mostly through context, as bearers of an additional meaning (like an apology). Speech is viewed as basically the same type of behavior (it fits the definition), but speech is given an exceptional status because of its importance and because it has certain characteristics that set it apart from other gestures. Excluded are all acts which either do not communicate anything (for example because the actor is not aware of an observer) or which only communicate themselves to an observer (”[look at me,] I am fishing/reading/sleeping/walking”).

I further wish to emphase the difference between normal, straightforward gestures and those acts that serve some other purpose in the first instance and only serve as gestures (or rather ‘gestures of something‘) in the second instance like the example of giving flowers to apologize. If I wish to distinguish between these different gestures I will add the term ’simple’ to the first category and ‘complex’ to the second category.

A gesture-simple is a gesture where the (sole) purpose of the act is to communicate

A gesture-complex is first some action but communicates an additional message in the second instance

I tried to find better words to express what I mean, but it’s the best I could come up with so far (it has been brewing for about a year, see one of my first posts).

Note that any gesture-simple may also be a gesture-complex (even a speech act can be a gesture-complex). I tried to explain this with this example of Pee Wee Reese standing by Jackie Robinson in the face of racist fan-abuse. The shoulder embrace was both a gesture-simple and a gesture-complex.

The gesture that touched a nation (source)

There are important reasons for making these distinctions (but I forgot them ). Well, at least it will allow me (and maybe you) to better analyze observations of gesture. And perhaps it is necessary to be precise if you want to make statements about gestures. For example, I think that people can typically see that a movement is a gesture-simple from just its appearance but this is not true for a gesture-complex. I think that people easily miss that an action was intended as a gesture-complex or, vice versa, see/read too much in what was just some action, for example in this cartoon from Garfield:

Jon mistakes Garfield’s intentions (source)

Let us see how far these definitions can take us. Or does anyone have better suggestions?

And a final wild speculation: Women are not able to see a man’s action as just that action but are always convinced it is some gesture-complex (if we do not bring flowers we do not care, if we do bring flowers we have something to apologize for [but we just thought they would be nice on the table]). Men conversely tend to miss most of the complex gestures made by women (when they wear something nice and new to show their appreciation of some event for example). Any takers?

In memoriam: Marcel Marceau, silent yet eloquent

Hero of the ‘once nearly lost art of pantomime’, Marcel Marceau (images), has recently died at the age of 84. Pop-out
A nice tribute (proceed to many others at YouTube at your leisure) Mime Marceau was born as Marcel Mangel in Strasbourg, France. He died on September 22 this year. From what I gather of the many tributes and obituaries he was an exceptional mime artist. He is often said to have revitalized the art (even singlehandedly ). That he enjoyed world wide recognition is illustrated by an older movie from a trip to Japan, where he is warmly received. Or possibly the Japanese are simply very fond of pantomime? It is strange how this art, which is so cherished and admired by some, is unappreciated by others. The wikipedia says: “Of [Marceau’s] summation of the ages of man in the famous Youth, Maturity, Old Age and Death, one critic said: “He accomplishes in less than two minutes what most novelists cannot do in volumes.”. But if you read research on gesture or sign language, pantomime is often what the writers set themselves off against. It is often used to make distinctions between that which is studied (linguistic or at least semiotic systems) and that which is not studied (mere pantomime). There are also those who seek to distinguish different gestural mechanism in signed discourse (where signers may alternate pantomimic and lexical/grammatical strategies to convey meaning) who talk about ‘gesture versus sign’. To me that is a strange distinction, for even ‘frozen’ (in the sense of Cuxac and Sallandre) lexical signs are gestures in my definition. What would be more appropriate is to label it ‘pantomime versus sign language’, if you wish to indicate a difference in level of conventionality. I think it could be very productive to study the mechanisms by which mime artists create meaning or express themselves. In countless gesture studies the iconic nature of gesture has been treated (from Tyler (1870) to Mueller (1998)). Each time the same mechanism emerge: meaning is created through imitation of acts, through embodiment, and through molding and sketching in the air. The same appears to be true of pantomime. At least those strategies are certainly used. But there may be more. Poetry pushes the boundaries of what can be done with spoken or written language. Perhaps mime as an art form pushes the boundaries of what can be done with gesture? I believe sign language poetry and pantomime are brothers in arms. Not only is beauty thus created, but people are shown new ways to express themselves more eloquently in gesture. Obituaries (among many others) in The Times, and on BBC News. Update October 2: There is a hilarious Dutch parody of a meeting between Marceau and Ivo Niehe, from Koefnoen (video).

Buckingham Palace Plonker

There is a funny little gesture story in the news these days. It is about a guard who is making little gestures (and doing a little dance) while he is supposed to be standing very still. The peak of the stroke of a wanker gesture? (source)

On the YouTube: Buckingham Palace Plonker. "Shocking behavior by one of the Queen's Guards in front of Buckingham Palace. Exclusive footage never seen before in front of Buckingham Palace." (little dance - checking time)

Elsewhere the Telegraph reports: "The video clip shows him turning his head - apparently to catch the attention of a colleague - before shaking his right fist up and down. Perhaps realizing that he is being watched, he quickly morphs the gesture into a more typical if slightly camp wave, before resuming his sentry duty." That is a nice and detailed analysis of the gesture that the Foot Guard is making. A wanker gesture that is camouflaged by morphing it into a wave. I concur. And whoever made the analysis, please keep up the good work. Update 1 hour later: It could also be a combination of 'wanker' and 'hurry up', possibly sending a message like 'hey wanker, hurry up". Maybe his colleague was slow on his routine? (see the comments in the Sun)

Evolution according to Tomas Persson & Co

[email] Hi Jeroen, Happened upon your blog. Thought you might enjoy this paper on a proposed iconic-gestural origin of language. Or perhaps another of the publications [from SEDSU]. All the best, Tomas Persson

Frontpage illustration from the paper. Well, I checked it out and for all those interested in evolution it might be nice to do the same. The paper's full title is 'Bodily mimesis as “the missing link” in human cognitive evolution', by Jordan Zlatev, Tomas Persson and Peter Gärdenfors. First impression: Strange how people tend to think that the topic of their study (in Lund's case it is a workpackage on 'imitation and mimesis') is the one decisive factor in human evolution. And I never have a shred of evidence to prove them wrong. But it will be interesting to read their case in more detail. {I, for one, believe that it is our ability to blog that sets us apart from other animals. And, of course, I mean blogging in a broad sense. For what is blogging if it is not the continual provision of unelicited non-information on how we feel about things and about what we know. Humans have always 'blogged', even before the internet and before the alphabet. We filled the world with our own thoughts and listened to ourselves, not to anyone else. This constant egoistic reflection created an evolutionary pressure whereby only individuals who could sustain this confrontation with the inner blogger, were still confident enough to reproduce. Since then, most of the strains of humanity who had any shame or humility left have died out in (relative) silence. What is left is what we are now: wanderers of the web, captains of comments, and slaves to our next posting}

[SEDSU's main hypothesis:] There remains, despite centuries of debate, no consensus about what makes human beings intellectually and culturally different from other species, and even less so concerning the underlying sources of these differences. The main hypothesis of the project Stages in the Evolution and Development of Sign Use (SEDSU) is that it is not language per se, but an advanced ability to engage in sign use that constitutes the characteristic feature of human beings; in particular the ability to differentiate between the sign itself, be it gesture, picture, word or abstract symbol, and what it represents, i.e. the “semiotic function” (Piaget 1945).

Substantial work has of course been done on gesture (or sign language) with primates (see this entire issue of Gesture). In some cases chimpansees or gorillas were taught to use gestures or pictures as signs (with a semiotic function). How does that fit into SEDSU's picture? By intuition, I would sooner propose that it is our ability to create 'systems of systems' of signs that sets us apart. Or maybe our ability to create and remember such large quantities and varieties of signs. I think even most animals and perhaps (what the hell) plants can be argued to 'gesture'. Do they differentiate between a signal and that which it represents? I think they do. Any animal that warns his group against predators is sending out a signal. The group members see the signal, not the predator, right? Or perhaps they can only communicate about what is actually present and not refer to things in other times and places? Enough speculation. It is time to read. I expect your reactions to the paper within this week... ps. Did you wonder about the semiotic function of the {curly brackets} as used above? Then you must be human. The answer: I signaled a humorous intermezzo.

Air Guitar Toy by Mannak

Ronald Mannak, a former colleague, is now developing toys at his own 1uptoys. His toys at hand are the SilverLit V-Beat AirDrums, AirGuitar and BoomBox. This week our university's 'newspaper' has an interview with him: Luchtgitaar met Geluid. And here he is in a video demonstrating his AirGuitar: Still a long way to go before he can try for world champion airguitar, I think. But the product is interesting to consider. At first I thought it looked quite nice and cool. But then I wondered: why would anyone want to actually have an AirGuitar? Isn't the point of playing air guitar that you don't have to have the damn thing? If I am going to buy something to play guitar I might as well, or even better, buy a real (toy) guitar, right? Is this going to be cheaper than a real guitar? I would guess that the additional electronics will not be cheaper than the bits of extra wood, metal or plastic needed for a physical guitar. But then again, microelectronics can be cheap if they are sold in large quantities. So, is this going to provide a better experience? I think that by definition that is impossible. The point of playing air guitar is to imitate the actual playing, to go thorugh the motions and almost 'feel like' you are really playing. In other words, it can never be better than the real thing, or can it? Maybe it can. Maybe it can help people who can not play guitar 'feel more like' they are playing guitar. Maybe the AirGuitar can take care of the difficult stuff like putting your fingers in the right position on the strings and remembering the chords and licks, and leave the exciting stuff to you, like strumming wildly, creating vibrato or smashing it. That would be neat, Ronald if you read this, can you make it so it can be smashed?

Art of Gesture on Stage

Here is nice article on the art of gesture in theatre: Music students help revive the art of Baroque gesture. Reviving an ancient art: students from the University’s Faculty of Music worked with theatre director Helga Hill to present a fully-staged and gestured season of Eccles’ The Judgment of Paris: Above, Paul Bentley as Paris and Janelle Hopman as Venus. [Photo: Mark Wilson] (source) Johann Jakob Engel (DE) wrote in a very interesting way about gestures, especially in Ideen zu einer Mimik. From the perspective of actors on stage, he analyzed how gestures function. I read only the paper by Sara Fortuna (2003) in Gesture: Gestural expression, perception and language. A discussion of the ideas of Johan Jakob Engel. It is intriguing reading material. A bit difficult to summarize in a few sentences here, so I will not try. An open mind, keen on philosophical musings is a good companion while chewing on Engel's thoughts. If we go further back in time, the work of Quintillian (and Cicero) is related. They wrote for orators, which were actors as much as they were politicians and lawyers. Wittgenstein is also referenced a lot.

Microsoft Surface

Microsoft is making a big deal out of their Surface. Basically, it is a regular computer with some fancy software that works together with a new type of table sized touchscreen. It enables people to work with the ten fingers of two hands or with artefacts (multi-touch) and it is sensitive to pressure. This idea was most eloquently presented by Jeff Han earlier, maybe Microsoft bought the idea? Anyway, here it is, one of the most expensive tables you will ever desire: Microsoft Surface parody (source) See also Ianus' Cabinet and Palette, the Studiolab Surface.

Deafblind Haptic Sign Language

A Dutch local newspaper, Leidsch Dagblad, has written a good report of the annual holiday gathering organized by the national foundation for the Deafblind (De Nederlandse Stichting voor Doofblinden). About 70 deafblind people (and their interpreters) apparently had a good time there. Impressions of the gathering (source) I previously wrote about their four hands sign language (vierhandengebarentaal): 'signs that made a life worth living'. I think this is a language that is not very well known or understood. So, wouldn't it be great to set up some research on this haptic sign language. There are plenty of people who are interested in sign language because it provides insight in the human language capacity. They compare how people listen and talk (and gesture) to how they watch and sign (and gesture). General human language processing must be separated from modality dependant processing stuff (though it is actually more like oral/auditory+visual/gestural vs. visual/gestural). Very interesting nevertheless. Lots of brain research with fMRI scanners... Just imagine what we could learn by studying deafblind people while letting them 'talk' or 'listen' in haptic sign language. They should probably go two-by-two? Or else, what would be the stimulus material to which to must respond? Prepared haptic sign language material? Hmm, maybe some observations should be the first step, or recordings using video or perhaps datagloves? Anyway, I would love to see more of it. Investigate how deafblind people manage to defy the odds and together create a language of their own. They are apparently already telling jokes. When shall we see/feel the first haptic sign language poem? And how can it be captured, transcribed or annotated? What sort of grammar does it have? Does iconicity play a role in sign formation and language use? Is iconicity achieved using similar strategies as in gesture and sign language? An ambitious man could write a research proposal for a nice post-doc position about it. Sometimes you don't have to go to small villages in Africa or the Middle East to find interesting languages. Sometimes you just need to hold out your hand. A professor in Utrecht who does haptic research: Astrid Kappers Professors in Nijmegen who study language: Levinson - Hagoort

In Love with SiSi

A wonderful bit of news has been hitting the headlines:

BBC News: Technique links words to signing: Technology that translates spoken or written words into British Sign Language (BSL) has been developed by researchers at IBM. The system, called SiSi (Say It Sign It) was created by a group of students in the UK. SiSi will enable deaf people to have simultaneous sign language interpretations of meetings and presentations. It uses speech recognition to animate a digital character or avatar. IBM says its technology will allow for interpretation in situations where a human interpreter is not available. It could also be used to provide automatic signing for television, radio and telephone calls.

Read the full story at IBM: IBM Research Demonstrates Innovative 'Speech to Sign Language' Translation System Demo or scripted scenario? Serendipity. Just this week a man called Thomas Stone inquired whether he could get access to the signing avatars of the eSign project. I passed him on to Inge Zwitserlood. She first passed him on to the eSign coordinator at Hamburg University, which was a dead end. Finally, he was pointed to the University of East Anglia, to John Glauert. And who is the man behind the sign synthesis in SiSi? From the press release from IBM:

John Glauert, Professor of Computing Sciences, UEA, said: "SiSi is an exciting application of UEA's avatar signing technology that promises to give deaf people access to sign language services in many new circumstances." This project is an example of IBM's collaboration with non-commercial organisations on worthy social and business projects. The signing avatars and the award-winning technology for animating sign language from a special gesture notation were developed by the University of East Anglia and the database of signs was developed by RNID (Royal National Institute for Deaf People).

Well done professor Glauert, thank you for keeping the dream alive. Now for some criticism: the technology is not very advanced yet. It is not at a level where I think it is wise to make promises about useful applications. The signing is not very natural and I think much still needs to be done to achieve of basic level of acceptability for users. But it is good to see that the RNID is on board, although they choose their words of praise carefully. It is amazing how a nice technology story gets so much media attention so quickly. Essentially these students have just linked a speech recognition module to a sign synthesis module. The inherent problems with machine translation (between any two languages) is not even discussed. And speech recognition only works under very limited conditions and produces limited results.

IBM says: "This type of solution has the potential in the future to enable a person giving a presentation in business or education to have a digital character projected behind them signing what they are saying. This would complement the existing provision, allowing for situations where a sign language interpreter is not available in person".

First, speech recognition is incredibly poor in a live event like a business presentation (just think of interruptions, sentences being rephrased, all the gesturing that is linked to the speech, etc.) and second, the idea that it will be (almost) as good as an interpreter is ludicrous for at least the next 50 years. The suggestion alone will probably be enough to put off some Deaf people. They might (rightly?) see it as a way for hearing people to try to avoid the costs of good interpreters. I think the media just fell in love at first sight with the signing avatar and the promises it makes. I also love SiSi, but as I would like to say to her and to all the avatars I've loved before: My love is not unconditional. If you hear what I say, will you show me a sign?

Doodling, Gesture, and Language Origins, the Movie

Here is a very entertaining video (nice music) that tells the tale of gesture and the origins of language in a nutshell. Much has been written about how the language capability may have evolved in humans with gesture as a stepping stone or how Man's first language may have been a signed language. Recent brain research findings (gesture+speech, mirror neurons, lateralization, sign language aphasia) have added more indirect 'evidence' for these theories. It is still hard to really prove anything about pre-historic events though... One thing that struck me is how the author talks about how people might be aided in their thinking when the gesture, or doodle and fidget. A reference to fidgeting! Hooray! Should I point out that I think gesture and fidgeting are quite different? No, I will just let it be.

Lead Guitar Body Language

Here is the Air Guitar World Champion 2007, Ochi "Dainoji" Yosuke (Japan) performing at Air Guitar World Championships 2007, Oulu, Finland: What a nice gesture performance: the pantomime, the gestures, the emotional expressions, the mimicry of the actual guitar play, and of course the dramatic gestures of a lead guitar player on stage. It makes me realize that a language may be found around in every hidden corner of human activity. In this case Dainoji shows a hilarious command of the body language of lead guitars. It also makes me wonder what exactly would remain of 'musical gestures', when all of a musicians 'body language' were hidden to the audience? I guess something would remain, and that would then be the real musical gesture.

Tags: guitar, body language, play, music

Baby Sign Mini Dictionary Flash

Here is a wonderful flash animation from Babystrology, featuring a signing baby: Lord knows, I am not the world's biggest fan of baby signing, but this is positively funny. I hope the creators keep treating baby sign with the same sense of humor. It is far too important a subject to ever talk seriously about. *Another nice example of using flash to present sign language online: Avon and Somerset Police. *Another nice example of animated kids signing: XV Congreso Mundial de la WFD.

Tags: baby sign, child development, dictionary, flash

Plains Indian Sign Language, Browning 1930

A wonderful collection of videos with Plains Indian Sign Language has been put on YouTube by Tommy Foley. A short 'teaser' with subtitles The videos were recorded in 1930, Browning Montana, when sign talkers from 14 different Plains nations gathered as participants in a conference organized by General Hugh L. Scott for the purpose of demonstrating their use of sign language. The first four videos (see this playlist) contain material from the participants at the conference themselves: Indians telling stories. Another six videos are a video version of a dictionary of the language (see this playlist).

Following the 1930 Plains Indian Sign Language Conference, General Scott intended to produce a cinematic dictionary of over thirteen hundred signs. Due to the Great Depression it would have been too difficult to get a second appropriation bill passed through congress to finish the cinematic dictionary. He did manage to get over three hundred signs filmed. (Note from Tommy Foley)

An important documenter of the Plains Indian Sign Language was Col. Garrick Mallery. He wrote 'Sign Language Among North American Indians Compared With That Among Other People And Deaf-Mutes' a report for the Smithsonian Insitute which was published in 1881, which is avaliable for free download as an e-book via Project Gutenberg.

Wiki Interactive Gestures

Dan Saffer, from Adaptive Path, and a member and contributor to the Interaction Design Association (IxDA), calls upon interaction designers to share their knowledge on gestural interaction in this new Interactive Gestures wiki. Read his 'Call to Arms' for more info and some nice links to web resources. Today's screen capture of the Interactive Gestures wiki I think it is a a good idea, which is why I am repeating the news here. Share and enjoy.

Signed Story Click, Clack, Moo

This wonderful multimedia video is a signed children's book. Ah, the revolution that is YouTube.

Gesturing Vervets and Sexual Harassment

Thanks to Alexis Heloir, a fellow PhD working on gestures, for sending me this story: Wild Vervet Monkeys Wreak Havoc in Kenya (or check the BBC which is the source). The most interesting part of this nice story (which tells of a group of vervets monkeys stealing food from a village and threatening specifically the women) is the following quote:

"The monkeys grab their breasts, and gesture at us while pointing at their private parts. We are afraid that they will sexually harass us," said Mrs Njeri.

Well, that is an interesting statement by Mrs Njeri. In the picture below you can see a vervet monkey next to a couple of children (from the Colobus Trust website, which also has more info on pest behavior by vervet monkeys) Vervet Monkeys (source) It must be said, these monkeys are not very big and the idea of 'sexual harassment' seems to me at first glance to be a tale of imagination gone wild. What are they going to do? Pinch a ladies bottom? Squeeze a boob? Certainly that is as far as they can go? Or is it? Perhaps I am thinking in the wrong direction. Perhaps sexual harassment is more like psychological warfare? Indeed, wikipedia states on sexual harassment that it can include many types of behavior and has a variety of purposes, most of which appear to be psychological rather than directly involving sexual intercourse. Dominance and humiliation can be important parts of it. From wikipedia we also learn that Vervets seem to "possess what has been called the "rudiments of language". Vervet Monkey alarm calls vary greatly depending on the different types of threats to the community. There are distinct calls to warn of invading leopards, snakes, and eagles." Now, there is an excellent web page on The Phallic Threat: Giant Penises and Similar Threat Devices. From it, I gather that the idea of a phallic threat is not unheard of, but instead common in both men and monkeys. Specifically on Primata, it is stated that Vervets have the folllowing use of the penile display.

penile display: This is when an adult male vervet monkey will present his erect penis and scrotum so that a neighboring group will see them (Estes, 1991). This display is used to demarcate territory (Estes, 1991). red-white-and-blue display: This display is used to communicate dominance by one male over another within a group (Estes, 1991). The male walks back and forth with his penis and scrotum in full view for the receiver to see; the sender will encircle the receiver (Estes, 1991). Occasionally the sender will stand on his hind legs and present his penis and scrotum to the receiver (Estes, 1991).

Moreover, the pigmentation of the Vervet Monkey's scrotum is a vivid blue that pales when the animal falls in social rank. In other words, Vervets may perhaps refer to their dominance over someone else by referring to the color of their genitals. So now we may have (1) an ability to communicate a variety of message, (2) a phallic threat with (3) a reference to dominance. Suddenly it is not so difficult to imagine that it is real. Or at least as real as sexual harassment gets. If the monkeys mean to express their dominance, mark their territory or humiliate the women and the women feel dominated or humiliated then that is a successful (if you will pardon the expression) case of sexual harassment. Unfortunately we cannot be sure of anything from such a distance. The whole story could just be exaggerated. It could even be an excuse for the villagers to start physically harassing the monkeys. Elsewhere: Atheism Central on this story A YouTube playlist on monkeys and their penal displays

US Police vs. The Finger

Giving the finger to the police, it remains an interesting case. Earlier I wrote that the Dutch police had booked and fined a man for it, which was overturned in court by a judge. Now, a similar thing has happened in Mobile, Alabama (USA). Philly.com brought the news: Mobile Appeals $3K Payment Over Gesture.

MOBILE, Ala. - The city of Mobile is appealing a judge's decision to award $3,000 to a motorist who was arrested for making an obscene hand gesture to a police officer. City attorney Ashton Hill said Wednesday the city is seeking to have Addison DeBoi's civil suit heard in circuit court. On July 31, District Court Judge Michael McMaken ruled in favor of DeBoi in his wrongful arrest suit and ordered the police department to pay $3,000. DeBoi, 56, was arrested by Officer Bristol Hines on Sept. 2, 2005, on a charge of disorderly conduct after he made a hand gesture while the two men were in their vehicles. He was acquitted last year and sued the city for $10,000 in damages, citing time lost from work, the threat of losing his engineering job , which requires a government security clearance , and the embarrassment of being put in jail. In awarding him $3,000, the judge said police officers must have "thicker skin" than the general public.

The rulings by the judges in the Netherlands and Alabama are along the same lines: the police should be less sensitive about being insulted. Unfortunately, being sensitive to insults is probably required if you do not want your authority challenged. Sensitivity lies at the heart of the perception of insults, which is a very subtle process. If the police are instructed to 'not be too sensitive' it probably greatly hinders their functioning. After all, we all rely on their judgment in all sorts of situations regarding aggression, violence, or misconduct, which all require a policeman to rely on his personal perception of other people's behavior. Why can't we rely on them to judge whether someone was 'insulting'? On the other hand, I was once taught how to referee a football game by none other than Mario van der Ende (a well known referee in Holland). He said that verbal abuse was always flying around on the pitch, most of it directed at him. He recommended to pretend not to hear it the first time (tempers can fly). A second time he would rebuff the perpetrator with a sneer or mocking insult of his own. Only if that wouldn't take care of it, a booking (yellow card) would be given. I think it made him one of the most respected referees in the competition. Maybe respect is gained as much from restraint in handing out punishment as it is from punishment itself. ps. I just finished this post when I saw that the UK police gave a man a 80 pound fine for giving the finger to a speed camera (and/or the cops operating it). I hope Simon Thompson (a respected citizen and school headmaster) will fight the decision, and let it go to court. I wonder whether the UK judge will follow the above examples from Alabama and the Netherlands. Elsewhere: The legal history of the finger (also showing many examples of verdicts and appeals where fines for fingers are overturned and arrests judged to be wrongful) - Jalopnik

Virginia is for Gangsters and Lovers

Some agency spent about $400.000 on a ad campaign featuring among others, this picture: My first impression was that it shows a very happy and very busy young lady. It's not often one sees someone simultaneously stamping grapes for wine with one foot, perform a balancing act and still have time to make a perfect heart gesture: bravo! What a nightmare photo shoot that must have been. Or a good bit of photoshopping. But the news (see Fox) is:

"The Virginia is for Lovers "Live Passionately" campaign will remove images of models making the hand gesture, one of several signs associated with the Gangster Disciples, Virginia Tourism Corp. officials said Friday. The gesture shows thumbs and index fingers formed into a heart."

I looked up the Gangster Disciples, and I checked Gang Signs, and more Gang Signs, and the most complete Gangster Hand Sign Index, where I found the following sign indeed shown as one of the Black Gangster Disciples: We have a match. Now the question remains: is it a wise decision to remove the image from the ad campaign? It is hard to judge from a distance. However, I do not think it was necessary, because it is nearly impossible that actual misunderstandings would arise from the use of the picture. Given the clarity of the image and the text on it, I cannot imagine anyone believing that this lady is flashing a gangster sign. Any gesture that is used as a conventional semiotic sign, or symbol, is what people choose it to be. If the producers of this campaign and the public that is adressed with it both agree that in this context it conveys something like 'love/heart/passion', then so be it. Other people in other contexts or other countries are free to attach their own meaning to the gesture. But, even though I can see no principled reason to scratch the image, it is understandable to do so anyway. Personally, I do not find it a very good picture, nor do I think that introducing a special gesture for a fairly complex message is going to be a succes (do they actually think people are going to adopt it in some way?). And why grant a bunch of gangsters a good laugh if you can avoid it (although they are probably already laughing their heads off from all the free publicity). In that sense, it is damage control to remove the picture. The best thing might have been to completely ignore the complaints and turn a blind eye to the gangster's sign. In that way, the gesture might have even been seen as ridiculing the gangster sign. Anyone flashing it could be mocked as a supporter of the campaign 'Virginia is for Lovers' instead of being a hardcore gang member. I admit, it probably wouldn't work that way, but it would have been better than granting the gangster disciples this PR victory. Other post on MS13 and gang gestures.

Gesture Watch from Georgia Tech

Here is another aspiring wannabee HCI star at the gesture firmament: the gesture watch.

Activate! The Gesture Watch has five infrared sensors, four of which sense any hand motion that occurs above the watch. If the user is wearing the watch on his left hand, he can move his right hand over the watch in an up or down, left or right, or circular motion. Different combinations of these movements communicate an action to the watch. (source)

Why do such applications receive so much credit in the various tech news sites and magazines? The only thing happening is that a couple of engineers have put together a neat device that can do a trick. It's not commercially available, there are no real users yet, there is no positive market feedback. There is only a vague promise of solving a vague problem. Discovery Channel: It won’t be long now before all electronic devices go "nano," and shrink to the size of frosted mini wheat square. You won’t know whether to turn it on or eat it. But the real question is: How do you press those teeny buttons? I know that writing an opening line can be hard, but this one has fallen straight from the sky on the willing imagination of Tracy Staedter (the reporter in question). Did she not notice the big display on the iPhone? People may not want tiny devices at all, because they need displays. And yes, they may also require decent buttons from their devices. In other words, the premises of the promises are promiscuous (sorry, couldn't resist); reporters are trading in their objective reflection for a nice soundbite.

Gesture Remote Control for TV

Two guys in Australia have given humanity the ultimate killer application for gesture recognition: Just the thing we needed, really. I am going to throw my remote away as soon as I can get this little gem of technology: something that solves the giant problems we are having with TV remote controls (and replaces them with a whole new set of problems). I think about twenty problem scenarios popped up simultaneously in my head fighting for priority. But I am just too lazy to type them all in. Instead I will just shrug this one off and save myself the calories.

Gesture Wellformedness

I am running an experiment on the acceptability of variation in sign language. One of the things that touch upon this matter is sign wellformedness, which supposes a certain sign language phonology with rules that tell whether a sign is wellformed or not. I am not done thinking that one over but it did get me thinking: is there such a thing a gesture wellformedness. According to these guys, there definitely is a way to do a gesture and a way that you don't do it. Listen to the comments for details :-)

Can I Learn How to Sign?

Here is a nice master's thesis by Chantal Mülders called 'Can I Learn How to Sign: Exploring aptitude for spoken language and visual stimuli in connection with sign language'. (Master of Arts, Radboud University Nijmegen, 2007). Usually these theses aren't published but since there is so little published on Sign Language of the Netherlands (SLN) I offered her to publish it here. Download the full pdf (138 pages, 3.8 Mb) here

Summary: This thesis explores the relationship between linguistic and visual aptitude and sign language learning to see what abilities are necessary for sign language acquisition and whether these differ from spoken language acquisition. 29 Students enrolled in a Sign Language minor took four tests before the start of their practical sign language course: a Sound Discrimination test, a Sign Language test, a Shape Discrimination test and a Visual Spatial Discrimination test. The Sign Language test was constructed for this thesis and focused on phonological and phonetic alterations of handshape. The other tests originated from aptitude test batteries. After seven weeks of sign language instruction, the students took a proficiency test that tested their receptive and productive sign language skills. This proficiency test was constructed by the sign language teacher and was the student’s final exam for the course. 18 Students remained who had taken all five tests. Correlations between the four tests and the proficiency test show that the Sign Language test has a decent, but insignificant correlation with Reception. The Sound Discrimination test did not show a relationship and the Shape Discrimination test had a steady, but low and insignificant relationship. The Visual Spatial Discrimination test correlated negatively with Reception. This was the only significant correlation between the four tests and the proficiency test. It is likely that the subject group was not varied enough and all subjects performed above a certain critical level. I conclude that sign language learning appears to require different abilities from spoken language acquisition, but the current subject group is too small for a definite answer.

Chantal's spot on the web (in Dutch). Some believe even dogs can learn sign language ... (I disagree, they are learning hand signals, or gestures, but nothing beyond that)

Small Penis Gesture

In the news today: the Australian Roads and Traffic Agency (RTA) has launched a campaign to demotivate drivers from speeding: The video is good too. The RTA hopes this small gesture will have a bigger impact on young men than images of bloody car crash victims. I think it probably will.

A Band called Obscene Gesture

I just found out that there is a trashcore (?) band called Obscene Gesture. Don't they look all tough and mean? (source) And then that menacing gesture that is just 'disgusting to the senses', and/or 'repulsive by reason of crass disregard of moral or ethical principles'. These men-boys will have to watch out for the US laws on obscenity. A final note: should their music be considered one giant musical gesture?

Pallete, the Studiolab Surface

It seems a student in my own backyard has been working on a multi-touch surface computer. His name is Arjen Klinkenberg and he is graduating on this design and the tests he did on it. Participants in a usability test playing 'Airhockey' (source) Here is the invitation he sent around for his presentation, in case you happen to be in the neighbourhood of Delft University of Technology :-)

Dear All, Next week on wednesday I'm graduating on the design of a rich media tool for supporting conceptual design sessions. The result of this project is a gesture-based multi touch interactive table-top called Palette. With Palette multiple people can collect and generate design relevant content during conceptual design sessions by having direct control over the touchable media. I would like to invite you to my presentation which is on wednesday the 27th of june starting at 13:45h in room 4B-65/66 in the faculty [map]. Kind regards, Arjen Klinkenberg