Me at the FG2008

I would almost forget, but I also presented some work at the FG2008 conference: Acceptability Ratings by Humans and Automatic Gesture Recognition for Variations in Sign Productions.

Abstract: In this study we compare human and machine acceptability judgments for extreme variations in sign productions. We gathered acceptability judgments of 26 signers and scores of three different Automatic Gesture Recognition (AGR) algorithms that could potentially be used for automatic acceptability judgments, in which case the correlation between human ratings and AGR scores may serve as an ‘acceptability performance’ measure. We found high human-human correlations, high AGR-AGR correlations, but low human-AGR correlations. Furthermore, in a comparison between acceptability and classification performance of the different AGR methods, classification performance was found to be an unreliable predictor of acceptability performance.

Snapshots of the three signs used in the experiment. Examples of three manipulations of the sign SAW. We tested about 68 sign manipulations in total. These were run through the automatic recognition algorithms we had been working on and they were rated by human signers. The paper is about how humans and machines can be compared.

Nadia Magnenat-Thalmann at the FG2008

One of the more interesting lectures at the FG2008 conference was a keynote speech delivered by Nadia Magnenat-Thalmann, director of the MIRALab in Geneva. She talked about Communicating with a Virtual Human or a Robot that has Emotions, Memory and Personality. She went far beyond the simplistic notion of expressing 'the six basic emotions' and talked about how mood, personality and relationships may affect our facial expressions. The talk by Magnenat-Thalmann focused on facial expression. (source) By coincidence I got an invitation to write a paper for another conference, organized by Anton Nijholt and Nadia Magnenat-Thalmann (and others), called the Conference on Computer Animation and Social Agents (CASA 2009). It is organized by people from the University of Twente but held in Amsterdam. Call for papers: deadline February 2009. Nadia also mentioned a researcher at Utrecht University called Arjan Egges. He got his PhD at the MIRALab and is now working on "the integration of motion capture animation with navigation and object manipulation".

My BSL Books

The British are doing it again. Leading the world to a better place. This time it concern books, or stories might be a better word, in sign language, BSL to be exact. I wonder why they keep calling it books? Although my hearing kids sometimes listen to 'Spoken Books' on CDs, hmm. Thanks to Gavin Howard for the link.

MyBSLbooks: Welcome to myBSLbooks.com - the World's first free online library of signed books. We are delighted to share with you a range of popular children's books, available for the first time in British Sign Language. This site offers D/deaf children, their families and schools wider access to their favourite stories in the preferred language of the Deaf Community.

Well, the site only contains about eight DVD's so far. And it's hardly a library since it doesn't cover any books published by anyone else, and I don't know if lending instead of buying is an option. The site is copyrighted by Lexicon/Signstream, so I guess they somehow own it. Come to think of it, the Dutch site Vi-Taal - De Gebarenwinkel has had a similar offering out there for years, and also offers a lot of other sign language goodies. And the Nederlands Gebarencentrum has a few DVDs as well. But well done all the same, you wonderful Britons.

In Memoriam: Arend Harteveld

Arend Harteveld died at the age of 50 years on Sunday 7 September 2008. Much too soon and entirely unexpected he was struck down by an accident in the blood circulation. Arend was a good man and a well respected colleague at Delft University of Technology. My thoughts go out to his family, especially to his mother who lived in with him and whom Arend was taking care of. Arend Harteveld (source) Arend contributed to much of the research based on which I hope to write my thesis, and these last years would not have been the same without him. He bore quite a burden in providing, more or less on his own, support to many courses and many labs, a burden he used to share with three colleagues in support who all left as a result of reorganisations. Meanwhile, his main interest was to work on research projects himself, and I found his contributions, both in creating software for experiments or for data analysis and in discussing the design of the experiments, to be very valuable. Arend always quickly grasped the ideas behind experiments and had a knack of pointing out flaws in the experimental design. Arend also maintained a website with information that shows some of his technical prowess. The website is maintained now by one his radio amateur friends. Arend tells of radio and measuring equipment, chirps, about which he also gave lectures occasionally. From personal experience I know that if a subject gripped him he wouldn't rest until he understood it fully, which happened during our collaboration for example with capturing response times on a laptop. He tried out several clocks of the PC and its processor and experimentally tested delays and variance in delays. As a radio amateur, a passion he picked up in his teens, he was known as PA1ARE. And now, as his brother in law said during the departure ceremony: "PA1ARE is voorgoed uit de lucht".

Ruud de Wild as Christ on the Radio

Ruud de Wild, the barely disguised saviour of our ears? Ruud de Wild is a DJ (and uomo universalis) who has prime time shows on the Dutch Radio. He switched to Q-Music in 2007 and this is one of their recent ads. Some people are complaining now but it is still going. There is a billboard poster too:

In the news: Brabants Dagblad: Op posters en in tv-spotjes is De Wild afgebeeld op een manier die doet denken aan een Christusfiguur. Op zijn T-shirt staat een hart omringd met stralen, net zoals Christus soms wordt weergegeven.

It took me a few moments to dig up this image of Christ that looks very similar: Jesus and his wife (?) Mary posing as Ruud de Wild (source) What is on the heart exactly? I cannot read it. Does anyone know? Update: The heart on Ruud's shirt reads 'Q is good for you' while the heart of Christ the King is adorned with the crown or a ring of thorns.

Gestures in language development

Gesture 8:2 came out recently. It is a special issue on 'Gestures in language development'. Amanda Brown, a friend who stayed at the MPI doing PhD research, published a paper on Gesture viewpoint in Japanese and English: Cross-linguistic interactions between two languages in one speaker. Marianne Gullberg, Kees de Bot and Virginia Volterra wrote an introductory chapter 'Gestures and some key issues in the study of language development'. Kees de Bot (LinkedIn) is a professor in Groningen working on (second) language acquisition.

MobileASL progress

A demo describing the MobileASL research project The group of MobileASL researchers at the University of Washington features in a local news bulletin. They have been working for a few years now on efficient transmitting of ASL video over a channel with limited bandwidth. The idea is to enable mobile videophony, which has been the holy grail of mobile applications for quite some time already. Personally, I am not convinced that specific technology for the transmission of sign language video will really have an impact. Here are a few reasons. Bandwidth will increase anyway with costs going down. Processing capacity in phones will increase. Videophony is an application that is desirable for many, not just signers. In other words, there is already a drive towards videophony that will meet the requirements for signing. Furthermore, I am not sure which requirements are specifically posed by sign language. People talk and gesture too, and I imagine they would want that to come across in the videophony as well. Finally, signers can and do adjust their signing to for example webcams. Does the technology address a real problem?

"The team tried different ways to get comprehensible sign language on low-resolution video. They discovered that the most important part of the image to transmit in high resolution is around the face. This is not surprising, since eye-tracking studies have already shown that people spend the most time looking at a person's face while they are signing."

Would this not be true for any conversation between people? On the positive side: perhaps this initiative for signers will pay off for everyone. It wouldn't be the first time that designs for people with specific challenges actually addressed problems everyone had to some degree.

Website Renovations

Hello my dear readers. Perhaps you missed it, but this website was down for almost two weeks. There were some technical difficulties and in the end I more or less started anew, with a less than perfect backup of the content. So, it is possible that pages are missing or that links malfunction. If you would be so kind to report these things it would be highly appreciated. At least the website is upgraded with new blogging software and a new look. Enjoy, Jeroen

Apple Gesture Patents around iPhone

I noticed a flurry of gesture patents that mentioned a 'portable mutlifunction device'. That's patentspeak for iPhone. The patents were all from APPLE Inc. Well done Apple. That's how you manage a patent portfolio. Philips and IBM used to be the masters in this line of completely covering an area with a barrage of patents. It will give Apple something to negotiate with in future business deals with other vendors. Here they all are as far as I could tell:

1. PORTABLE MULTIFUNCTION DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR INTERPRETING A FINGER GESTURE ON A TOUCH SCREEN DISPLAY (WO 2008/086302)
2. PORTABLE ELECTRONIC DEVICE SUPPORTING APPLICATION SWITCHING (WO 2008/086298)
3. SYSTEM, METHOD, AND GRAPHICAL USER INTERFACE FOR INPUTTING DATE AND TIME INFORMATION ON A PORTABLE MULTIFUNCTION DEVICE (WO 2008/086073)
4. APPLICATION PROGRAMMING INTERFACES FOR GESTURE OPERATIONS (WO 2008/085848)
5. MULTI-TOUCH GESTURE DICTIONARY (WO 2008/085784)
6. GESTURE LEARNING (WO 2008/085783)
7. PORTABLE MULTIFUNCTION DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR INTERPRETING A FINGER SWIPE GESTURE (WO 2008/085770)
8. PORTABLE ELECTRONIC DEVICE, METHOD AND GRAPHICAL USER INTERFACE FOR DISPLAYING INLINE MULTIMEDIA CONTENT (WO 2008/085747)
9. PORTABLE MULTIFUNCTION DEVICE,METHOD, AND GRAPHICAL USER INTERFACE FOR TRANSLATING DISPLAYED CONTENT (WO 2008/085744)
10. OVERRIDE OF AUTOMATIC PORTRAIT-LANDSCAPE ROTATION FOR A PORTABLE MULTIFUNCTION DEVICE WITH ACCELEROMETERS (WO 2008/085741)
11. METHOD, SYSTEM, AND GRAPHICAL USER INTERFACE FOR VIEWING MULTIPLE APPLICATION WINDOWS (WO 2008/085739)
12. METHOD, SYSTEM, AND GRAPHICAL USER INTERFACE FOR PROVIDING WORD RECOMMENDATIONS (WO 2008/085737)
13. Somewhat earlier this year: DELETION GESTURES ON A PORTABLE MULTIFUNCTION DEVICE (WO 2008/030975)
14. SOFT KEYBOARD DISPLAY FOR A PORTABLE MULTIFUNCTION DEVICE (WO 2008/030974)
15. PORTABLE ELECTRONIC DEVICE PERFORMING SIMILAR OPERATIONS FOR DIFFERENT GESTURES (WO 2008/030972)
16. EMAIL CLIENT FOR A PORTABLE MULTIFUNCTION DEVICE (WO 2008/030970)
17. PORTABLE ELECTRONIC DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR DISPLAYING STRUCTURED ELECTRONIC DOCUMENTS (WO 2008/030879)
18. PORTABLE MULTIFUNCTION DEVICE, METHOD, AND GRAPHICAL USER INTERFACE FOR CONFIGURING AND DISPLAYING WIDGETS (WO 2008/030875)
19. PORTABLE ELECTRONIC DEVICE FOR PHOTO MANAGEMENT (WO 2008/030779)
20. PORTABLE ELECTRONIC DEVICE FOR INSTANT MESSAGING (WO 2008/030776)
21. 2007: UNLOCKING A DEVICE BY PERFORMING GESTURES ON AN UNLOCK IMAGE (WO 2007/076210)
22. 2006: GESTURES FOR TOUCH SENSITIVE INPUT DEVICES (WO 2006/020305)

Who will be able to argue with this patent portfolio? Who will be able to claim that the things Apple has patented were already invented elsewhere? Who will be able to maintain that gestures are not technical inventions but natural human communicative actions? Who will pay the lawyers to fight these fights? Here it all is in a fashion that is easier to digest than sifting through 22 patents. I think Apple has won this fight before it could even get started.

Healy's Flute is an Orangist Salute

Although the story of David Healy's flute gesture is getting a little moldy it has generated enough discourse to deserve another mentioning here. The interesting thing about this flute gesture is how it is part of the history of the Northern Ireland sectarian conflicts. Sensitive catholic Irish republicans will get inflamed over the gesture while others have no idea what the problem is. David Healy making the flute gesture. (source) These flute bands on Orangist marches are what the gesture refers to. Get a glimpse of the triumphalist nature of these marches By coincidence I am currently reading 'The Irish War' by Tony Geraghty. He sketches a long and messy conflict which has gone on for more than 300 years. It is clear that these marches are of an inflammatory nature, and therefore a gesture that refers to them is also inflammatory. It is not just a merry band of flute-playing men. They celebrate Orangist protestant dominance in Northern Ireland at the expense of the catholic part of the population. The conflict carried over to a Scottish football match called 'the Old Firm' between the Rangers (protestant) and Celtic (catholic), see this nice historical overview by the BCC. Many Irish people moved to Scotland and brought the conflict with them. Paul Gasoigne made the mistake of making this gesture while he played for the Rangers and paid a heavy fine of 20.000 pounds. Paul Gascoigne made the same flute gesture during the old firm (Picture: BBC News) David Healy was not playing for the Rangers, in fact I don't think he ever did, but he is known as a Rangers fan. He is from Northern Ireland and he plays in their national side. However, in this game Healy was playing for Fulham (an English club) in a friendly match against Celtic, which sets the context for the gesture. Healy was 'provoked' by the Celtic fans who knew his sympathies and chanted 'where were you on The Twelfth' (a reference to an important march on the twelfth of July). In response, he seems to have made this gesture somewhat jokingly. The strange thing is that he seems to be escaping the sort of fine Gascoigne got. Why is that? Was Gazza perceived as doing it to inflame Celtic supporters whereas Healy was just fooling around? I think many people will take it more seriously than that. As always happens with sportsmen making inappropriate gestures, Healy is now apologizing and his club is investigating. It wouldn't surprise me if a fine came soon. The Orange Order What Irish Political Pundits have to say about it A similar incident at Belfast Zoo, involving a panda. CNN reports about the Orange Order marching season Update: I think an important difference between Healy and Gascoigne is that the latter played for the Rangers who were at that time trying to defuse a tense situation. Gascoigne's gesture was hurting that effort.