Master thesis: language modeling with pose estimation data for sign language processing

In a nutshell

In this thesis you are going to develop a language model for sign languages, based on human pose estimation.

Details

Language models are an important technology for text-based NLP systems. For sign languages, which are usually not represented as text, language models do not exist. The goal of this work is to develop one.

Instead of working with video data, sign language data will be represented as human pose estimation data. Widely used systems for pose estimation are OpenPose (https://github.com/CMU-Perceptual-Computing-Lab/openpose) and Mediapipe Holistic (https://google.github.io/mediapipe/solutions/holistic.html).

You will conduct a thorough review of how current languages models are trained and evaluated, to see which components and principles can be applied to non-textual representations of language.

Then your task is to develop a training procedure and evaluation method that make sense for modeling sign language data. Your approach can also be inspired by recent advances in sign language processing.

Your profile

  • You are currently studying Computational Linguistics or Computer Science.
  • You have an interest in, and willingness to learn more about, sign languages and sign language processing.

Supervisor