The Acceptability of Sign Manipulations

My latest research revolved around the question 'when is a sign production still acceptable', or rather, given that context and application of different norms have a big influence on acceptability, 'which (types of) variations are more acceptable?' We ran an experiment which I think has yielded interesting results. Tuesday Jan 22 I will be giving a presentation at the MPI Nijmegen about the experiment and the results as part of the Nijmegen Gesture Centre Lecture Series 2007. This will be in English. At our own workshop 'Een mooi gebaar 2007' I also gave preliminary results in a short 10 minute presentation in Dutch.

Gestures are Sterile, Touching is not

At the renowned Fraunhofer institute they may have built a killer gesture app: Gesture control that lets surgeons control a 3D-display of a head (for example) during surgery while remaining sterile (touching buttons would break sterility, I guess). Rotate the 3D image by gesturing (source)

Press Release: Non-contact image control As if by magic, the three-dimensional CAT scan image rotates before the physician’s eyes – merely by pointing a finger. This form of non-contact control is ideal in an operating room, where it can deliver useful information without compromising the sterile work environment. The physician leans back in a chair and studies the three-dimensional image floating before his eyes. After a little reflection, he raises a finger and points at a virtual button, likewise floating in the air. At the physician’s command, the CAT scan image rotates from right to left or up and down – precisely following the movement of his finger. In this way, he can easily detect any irregularities in the tissue structure. With another gesture, he can click on to the next image. Later, in the operating room, the surgeon can continue to refer to the scanner images. Using gesture control to rotate the images, he can look at the scan of the patient’s organs from the same perspective as he sees them on the operating table. There is no risk of contaminating his sterile gloves, because there is no mouse or keyboard involved. But how does the system know which way the finger is pointing? “There are two cameras installed above the display that projects the three-dimensional image,” explains Wolfgang Schlaak, who heads the department at the Fraunhofer Institute for Telecommunications, Heinrich-Hertz-Institut HHI in Berlin that developed the display. “Since each camera sees the pointing finger from a different angle, image processing software can then identify its exact position in space.” The cameras record one hundred frames per minute. A third camera, integrated in the frame of the display, scans the user’s face and eyes at the same frequency. The associated software immediately identifies the inclination of the person’s head and the direction in which the eyes are focused, and generates the appropriate pair of stereoscopic images, one for the left eye and one for the right. If the person moves their head a couple of inches to the side, the system instantly adapts the images. “In this way, the user always sees a high-quality three-dimensional image on the display, even while moving about. This is essential in an operating theater, and allows the physician to act naturally when carrying out routine tasks,” says Schlaak. “The unique feature of this system is that it combines a 3-D display screen with a non-contact user interface.” The three-dimensional display costs significantly less than conventional 3-D screens of comparable quality. Schlaak is convinced that “this makes our gesture-controlled 3-D display an affordable option even for smaller medical practices.” The research team will be presenting its prototype at the MEDICA trade fair from November 14 to 17, 2007, in Düsseldorf (Hall 16, Stand D55). Schlaak hopes to be able to commercialize the system within a year or so.

Things like this are probably the best bet for the near future of gesture recognition. Niche applications that exploit some specific benefit of using gestures instead of (or besides) other, more mundane interface technology. The biggest hit in gesture land is without a doubt the Nintendo Wii, which exploits another unique selling point of gestures: a higher (or more representative) physical involvement leading to a better 'experience' of a game. It specifically targets gamers who are interested in fun and exercise in a social context. I doubt that hardcore gamers, intent on getting to higher levels of killing sprees, will be very keen on the Wii. And so it will probably remain for the near future. Like with speech recognition, gesture recognition will have to find some nice niches to live in and multiply. Maybe one day, the general conditions will change (ubiquitous camera viewpoints? intention-aware machines?) and gesture can become the dominant form of HCI, driving buttons to niche applications. I wouldn't bet on it right now, though.

Larry Craig's Cottaging Gestures

Here is a nice cartoon, a caricature of a supposed gesture system used by Senator Larry Craig to not solicit gay sex in a restroom. The practice is known as cottaging. A cottaging gesture? (source) Frankly, I don't give a damn about whether Craig is gay or not, or whether he elicited sex with other gay men or not (fine by me), though I would rather he did so in a place where he could be sure the invited party was of the same mind (and not possibly some unwitting hetero). But the cartoon is nice, a perfect example of a man-invented little gesture system. The case bears a remarkable resemblance to that of George Michael, who was also invited by an undercover cop who apparently even showed his dick first (source). I think there is a good line of defence in the following reasoning: If you are guilty of 'soliciting sex' by using these gestures then it is firmly acknowledged that there is a code of behaviour that governs the interaction. Therefore the behaviour of the undercover cop in the toilet booth also has to have been in accordance to this code. Perhaps he wasn't making explicit hand gestures, but he had positioned himself in the right spot, assumed the right attitude, and perhaps kept silent where he was supposed to (while a normal response might have been "it's taken"). In my definition all these actions are also gestures (gesture-complex). In other words, both men were soliciting sex by their conduct, and the policeman was the initiator. Is that allowed under US law? A related 'gesture' system is the bandana code. Although I am not intimately aware of where and when this code applies.

In Memoriam: Washoe

Today I received news that Washoe died at age 42. Washoe was a chimpansee that was taught sign language in order to study the extent to which she would be able to gain language skills. An impression of Washoe talking with a researcher (source) Here is more information on attempts to talk with chimps. May she find peace in death. In her life she caused (through no fault of her own) more conflict between humans than most humans. Update: Here is Carl Schroeder's tribute (ASL vlog)

Future President's Gesture?

Here is a post that's good for a laugh (if you're not in love with US politicians) on Geenstijl, a popular Dutch news blog. Unfortunately for the majority of readers, the dodgy jokes are in Dutch. Which is why I'll translate them to English. Who knows, maybe the jokes will end up back in the US? Hi, I'm Hill You know, the wife of Bill Now, I'm looking for something bigger Yeah right, about this thick? I've only got this It's gotta be about this length Take me, take me! Shit! F*ck. Never mind. For those of you who can not believe I am lowering myself to such a cheap shot: The story exemplifies the limits of our ability to see just about anything in a gesture. Even though speech may be required to interpret gestures, the liberties taken here are clearly too much.

A Case of Co-Speech Gestures

A wonderfull new video on YouTube of two guys (programmers, it says) talking and 'co-speech-gesturing' (is that a verb?). "Real programmers use sign language" (by ekabanov) I think it is safe to assume that it is for real. Their whole behaviour looks too natural and wacky to be scripted. I also think this is a great case study to spend some time on while discussing some of the ideas of David McNeill. Because what we have here is what his theories and ideas are concerned with. There is (of course) no sign language nor did I spot any other 'emblematic gesture' (those vulgar things you get fined or jailed for or the goofy ones that seem to be must-haves for ad campaigns). I also do not see any pantomime. No, this is the stuff they like in Chicago: Co-speech gestures. An episode full of deictics, beats, iconic and metaphoric gestures, right?

From the McNeill lab: A misconception has arisen about the nature of the gesture categories described in Hand and Mind, to wit, that they are mutually exclusive bins into which gestures should be dumped. In fact, pretty much any gesture is going to involve more than one category. Take a classic upward path gesture of the sort that many speakers produce when they describe the event of the cat climbing up the pipe in our cartoon stimulus. This gesture involves an iconic path-for-path mapping, but is also deictic, in that the gesture is made with respect to an origo --that is, it is situated within a deictic field. Even "simple" beats are often made in a particular location which the speaker has given further structure (e.g. by setting up an entity there and repeatedly referring to it in that spatial location). Metaphoric gestures are de facto iconic gestures, given that metaphor entails iconicity. The notion of a type, therefore, should be considered as a continuum --with a given gesture having more or less iconicity, metaphoricity, etc.

Wrong! Apparently the main problems of McNeill's typology of gestures, that has sent many an engineer on a wild goose hunt for iconic gestures, are now even recognized at the source (McNeill, 1992). It is not mutually exclusive but rather an index of the functioning of a gesture ('as a beat' - 'through spatial reference (deictic)' - 'referring thorugh iconicity to something concrete' - 'referring via iconicity first to something concrete and second through metaphor to something abstract'). Good. I never liked 'beats' for example. I don't think I ever saw one. But to say that it was a misconception... I vaguely recall an annotation procedure called the 'beat filter' that begs to differ. Anyway, at least this clears up the discussions regarding 'metaphoric gestures' considerably [they are de facto also iconic, the metaphor functions on another level]. And it also clears the way for an annotation of this video. Any volunteers? Well, you would have to get a decent file of the movie instead of the YouTube flash stuff anyway, so let's forget about it. McNeill wrote a new book recently (2005) which is mostly about growth points. But before you read the summary by McNeill you might want to check Kendon's brilliant poem called 'The Growth Point', which he delivered at McNeill's festen. I find it neatly captures my feelings towards growth points (and more that is beyond my grasp). I am at once awed, baffled, and stupefied when I read about growth points and catchments. And so it goes. Again I tried to get it. Again I failed to learn anything from reading about growth points. One thing only. If David McNeill (or Susan Duncan) is right, then annotating gestures in episodes like this will be eternal hell :-) And without the speech it will not work. Thank God. I can go to bed with a clear conscience. Books: McNeill, D. (Fall 2005) Gesture and Thought. Chicago: University of Chicago Press. McNeill, D. (2000) (Ed.). Language and Gesture. Cambridge: Cambridge University Press. McNeill, D. (1992). Hand and Mind. Chicago: University of Chicago Press.

Gesture Definitions

I am going to try and coin some definitions regarding gestures.

A gesture is any act, except speech, by which we intend to communicate something beyond the act itself

This definition includes all normal gestures, that have meaning (the message that was communicated intentionally) because of cultural conventions (including languages) or iconicity, and all such acts (like giving flowers) that serve, mostly through context, as bearers of an additional meaning (like an apology). Speech is viewed as basically the same type of behavior (it fits the definition), but speech is given an exceptional status because of its importance and because it has certain characteristics that set it apart from other gestures. Excluded are all acts which either do not communicate anything (for example because the actor is not aware of an observer) or which only communicate themselves to an observer (”[look at me,] I am fishing/reading/sleeping/walking”).

I further wish to emphase the difference between normal, straightforward gestures and those acts that serve some other purpose in the first instance and only serve as gestures (or rather ‘gestures of something‘) in the second instance like the example of giving flowers to apologize. If I wish to distinguish between these different gestures I will add the term ’simple’ to the first category and ‘complex’ to the second category.

A gesture-simple is a gesture where the (sole) purpose of the act is to communicate

A gesture-complex is first some action but communicates an additional message in the second instance

I tried to find better words to express what I mean, but it’s the best I could come up with so far (it has been brewing for about a year, see one of my first posts).

Note that any gesture-simple may also be a gesture-complex (even a speech act can be a gesture-complex). I tried to explain this with this example of Pee Wee Reese standing by Jackie Robinson in the face of racist fan-abuse. The shoulder embrace was both a gesture-simple and a gesture-complex.

The gesture that touched a nation (source)

There are important reasons for making these distinctions (but I forgot them ). Well, at least it will allow me (and maybe you) to better analyze observations of gesture. And perhaps it is necessary to be precise if you want to make statements about gestures. For example, I think that people can typically see that a movement is a gesture-simple from just its appearance but this is not true for a gesture-complex. I think that people easily miss that an action was intended as a gesture-complex or, vice versa, see/read too much in what was just some action, for example in this cartoon from Garfield:

Jon mistakes Garfield’s intentions (source)

Let us see how far these definitions can take us. Or does anyone have better suggestions?

And a final wild speculation: Women are not able to see a man’s action as just that action but are always convinced it is some gesture-complex (if we do not bring flowers we do not care, if we do bring flowers we have something to apologize for [but we just thought they would be nice on the table]). Men conversely tend to miss most of the complex gestures made by women (when they wear something nice and new to show their appreciation of some event for example). Any takers?

In memoriam: Marcel Marceau, silent yet eloquent

Hero of the ‘once nearly lost art of pantomime’, Marcel Marceau (images), has recently died at the age of 84. Pop-out
A nice tribute (proceed to many others at YouTube at your leisure) Mime Marceau was born as Marcel Mangel in Strasbourg, France. He died on September 22 this year. From what I gather of the many tributes and obituaries he was an exceptional mime artist. He is often said to have revitalized the art (even singlehandedly ). That he enjoyed world wide recognition is illustrated by an older movie from a trip to Japan, where he is warmly received. Or possibly the Japanese are simply very fond of pantomime? It is strange how this art, which is so cherished and admired by some, is unappreciated by others. The wikipedia says: “Of [Marceau’s] summation of the ages of man in the famous Youth, Maturity, Old Age and Death, one critic said: “He accomplishes in less than two minutes what most novelists cannot do in volumes.”. But if you read research on gesture or sign language, pantomime is often what the writers set themselves off against. It is often used to make distinctions between that which is studied (linguistic or at least semiotic systems) and that which is not studied (mere pantomime). There are also those who seek to distinguish different gestural mechanism in signed discourse (where signers may alternate pantomimic and lexical/grammatical strategies to convey meaning) who talk about ‘gesture versus sign’. To me that is a strange distinction, for even ‘frozen’ (in the sense of Cuxac and Sallandre) lexical signs are gestures in my definition. What would be more appropriate is to label it ‘pantomime versus sign language’, if you wish to indicate a difference in level of conventionality. I think it could be very productive to study the mechanisms by which mime artists create meaning or express themselves. In countless gesture studies the iconic nature of gesture has been treated (from Tyler (1870) to Mueller (1998)). Each time the same mechanism emerge: meaning is created through imitation of acts, through embodiment, and through molding and sketching in the air. The same appears to be true of pantomime. At least those strategies are certainly used. But there may be more. Poetry pushes the boundaries of what can be done with spoken or written language. Perhaps mime as an art form pushes the boundaries of what can be done with gesture? I believe sign language poetry and pantomime are brothers in arms. Not only is beauty thus created, but people are shown new ways to express themselves more eloquently in gesture. Obituaries (among many others) in The Times, and on BBC News. Update October 2: There is a hilarious Dutch parody of a meeting between Marceau and Ivo Niehe, from Koefnoen (video).

Buckingham Palace Plonker

There is a funny little gesture story in the news these days. It is about a guard who is making little gestures (and doing a little dance) while he is supposed to be standing very still. The peak of the stroke of a wanker gesture? (source)

On the YouTube: Buckingham Palace Plonker. "Shocking behavior by one of the Queen's Guards in front of Buckingham Palace. Exclusive footage never seen before in front of Buckingham Palace." (little dance - checking time)

Elsewhere the Telegraph reports: "The video clip shows him turning his head - apparently to catch the attention of a colleague - before shaking his right fist up and down. Perhaps realizing that he is being watched, he quickly morphs the gesture into a more typical if slightly camp wave, before resuming his sentry duty." That is a nice and detailed analysis of the gesture that the Foot Guard is making. A wanker gesture that is camouflaged by morphing it into a wave. I concur. And whoever made the analysis, please keep up the good work. Update 1 hour later: It could also be a combination of 'wanker' and 'hurry up', possibly sending a message like 'hey wanker, hurry up". Maybe his colleague was slow on his routine? (see the comments in the Sun)

Evolution according to Tomas Persson & Co

[email] Hi Jeroen, Happened upon your blog. Thought you might enjoy this paper on a proposed iconic-gestural origin of language. Or perhaps another of the publications [from SEDSU]. All the best, Tomas Persson

Frontpage illustration from the paper. Well, I checked it out and for all those interested in evolution it might be nice to do the same. The paper's full title is 'Bodily mimesis as “the missing link” in human cognitive evolution', by Jordan Zlatev, Tomas Persson and Peter Gärdenfors. First impression: Strange how people tend to think that the topic of their study (in Lund's case it is a workpackage on 'imitation and mimesis') is the one decisive factor in human evolution. And I never have a shred of evidence to prove them wrong. But it will be interesting to read their case in more detail. {I, for one, believe that it is our ability to blog that sets us apart from other animals. And, of course, I mean blogging in a broad sense. For what is blogging if it is not the continual provision of unelicited non-information on how we feel about things and about what we know. Humans have always 'blogged', even before the internet and before the alphabet. We filled the world with our own thoughts and listened to ourselves, not to anyone else. This constant egoistic reflection created an evolutionary pressure whereby only individuals who could sustain this confrontation with the inner blogger, were still confident enough to reproduce. Since then, most of the strains of humanity who had any shame or humility left have died out in (relative) silence. What is left is what we are now: wanderers of the web, captains of comments, and slaves to our next posting}

[SEDSU's main hypothesis:] There remains, despite centuries of debate, no consensus about what makes human beings intellectually and culturally different from other species, and even less so concerning the underlying sources of these differences. The main hypothesis of the project Stages in the Evolution and Development of Sign Use (SEDSU) is that it is not language per se, but an advanced ability to engage in sign use that constitutes the characteristic feature of human beings; in particular the ability to differentiate between the sign itself, be it gesture, picture, word or abstract symbol, and what it represents, i.e. the “semiotic function” (Piaget 1945).

Substantial work has of course been done on gesture (or sign language) with primates (see this entire issue of Gesture). In some cases chimpansees or gorillas were taught to use gestures or pictures as signs (with a semiotic function). How does that fit into SEDSU's picture? By intuition, I would sooner propose that it is our ability to create 'systems of systems' of signs that sets us apart. Or maybe our ability to create and remember such large quantities and varieties of signs. I think even most animals and perhaps (what the hell) plants can be argued to 'gesture'. Do they differentiate between a signal and that which it represents? I think they do. Any animal that warns his group against predators is sending out a signal. The group members see the signal, not the predator, right? Or perhaps they can only communicate about what is actually present and not refer to things in other times and places? Enough speculation. It is time to read. I expect your reactions to the paper within this week... ps. Did you wonder about the semiotic function of the {curly brackets} as used above? Then you must be human. The answer: I signaled a humorous intermezzo.