Signing avatars for railway travel announcements in the Netherlands – Initial results

Britt van Gemert, Richard Cokart, Lyke Esselink, Maartje de Meulder, Nienke Sijm and Floris Roelofsen

Various researchers have stressed the importance of collaboration between deaf sign language specialists and (deaf/hearing) AI specialists in the development of sign language technologies (e.g. Bragg et al. 2019). We present initial results of a project in which we develop a sign language avatar for communicating railway travel announcements in Dutch Sign language (NGT), in collaboration with the Dutch national railway company (NS). The goal of our presentation will be twofold: (1) to share our experiences with two collaboration methods: co-design and focus groups, (2) to share some specific insights/results obtained in the project so far.

Our team consists of three deaf researchers with a background in Applied Sign Linguistics (Cokart, de Meulder) and Deaf Studies (de Meulder, Sijm) and three hearing researchers with a background in AI and Linguistics (Esselink, van Gemert, Roelofsen). Esselink and van Gemert have elementary proficiency in NGT, Roelofsen intermediate.

Starting point
First, we obtained a list of railway announcement templates from NS (e.g. ‘The intercity to destination X departs at time Y from platform Z’). The Dutch Sign Language Centre (Nederlandse Gebarencentrum) provided NGT translations of these templates (for randomly picked X’s, Y’s, and Z’s). Based on these translations, we created a first basic system, with an avatar mimicking the video translations as closely as possible, and performing several variations as well (with different X’s, Y’s, and Z’s). We used the JASigning avatar engine (e.g. Ebling and Glauert 2016), which makes it possible to efficiently create many variants of a given template, without expensive equipment (e.g. motion capture).

After this initial stage, the entire team worked together in four co-design sessions to improve the system (manual movements, facial expressions, mouthing, grammar, transitions between signs) as well as non-linguistic aspects of the animations (e.g., camera angle, speed). The first session resulted in a general strategy for an iterative co-design process, consisting of three additional sessions. In between these, the developers integrated all suggestions, sometimes in multiple versions, so that these could be considered and further refined during the next session.

Focus group
After the co-design phase, we organised a focus group with six deaf participants representing different regions and age groups. The discussion concentrated on eight topics, including timing, mouthing, indexing and choice of vocabulary. In each case, three variants were presented for comparison. Interpreters NGT – Dutch were present at all meetings, though hearing team members communicated in NGT as much as possible.

The combined expertise from various disciplines in the co-design process and the input from a diverse group of potential end users in the focus group led to major adjustments of the avatar. To exemplify, one specific finding was that the distinction between formal and informal signing styles proved to be highly relevant for the comprehension and perception of the avatar. Clear signing is not sufficient; the avatar’s signing style and choice of vocabulary also need to fit the particular context of use. Other specific findings will be discussed in-depth during our presentation.