SMILE II: Scalable Multimodal Sign Language Technology for Sign Language Learning and Assessment (Phase II)

SMILE-II aims to research and build advanced technology for sign language learning. More precisely, the project builds on the groundwork of the SNSF Sinergia project SMILE I, which dealt with assessment of the manual activity of Swiss German Sign Language (Deutschschweizerische Gebärdensprache, DSGS) in isolated signs produced by L1 users and L2 learners. SMILE II extends this technology to continuous sign language assessment including both manual and non-manual components of signs so that a DSGS learner’s sentence-level production can be assessed in an automatic manner.
The proposed goal is faced with several challenges: (a) lack of DSGS resources; (b) continuous sign language can contain both manual information (information on the hands/arms) and non-manual information (information on the torso, head, and face), and almost all work on recognition to date has focused on manual information only. Furthermore, continuous sign language recognition and continuous sign language production are still open research problems; (c) lack of standardized instruments for DSGS assessment; and (d) lack of automatic methods to assess continuous sign productions at both the manual and non-manual level.
To achieve the goal of the project, SMILE-II brings together researchers from different fields, namely, sign language linguistics, sign language assessment, sign language technology, and spoken language technology, across four different institutions:

  • Idiap Research Institute, Martigny
  • University of Applied Sciences of Special Needs Education (Interkantonale Hochschule für Heilpädagogik Zürich, HfH)
  • University of Zurich
  • University of Surrey (UK)
SMILE I demo
Zoom (PNG, 963 KB)
Demonstrator SMILE I project (screenshot, video courtesy of University of Surrey)

 

Funding

The SMILE II project is funded by the Swiss National Science Foundation (SNSF) Sinergia instrument under grant agreement number CRSII5_193686.

Project duration

January 2021 to December 2024

Project head

Dr. Sarah Ebling

Researchers

Alessia Battisti

More information