Quality Estimation for Text Simplification / Simplicity Scoring

Supervisor:  Annette Rios

This PP is embedded in the Inclusive Information and Communication Technologies" (IICT)  . The goal is to develop a tool to score the simplicity of a given text automatically, using both rule-based and machine learning approaches.

Automatic evaluation metrics commonly used in text simplification include BLEU and ROUGE, but also more specialized metrics such as SARI [Xu 2016] or Flesch-Kincaid Grade Levels (FKGL) [Flesch 1979, Kincaid 1975] that are supposed to measure simplicity. However, there is some doubt that these metrics can measure simplicity reliably [Tanprasert and Kauchak 2021]. For this reason, we want to implement a linguistically informed simplicity metric. This first version will be for German only, but extensions to the other languages of the project are possible (English, Italian, French).

 

You will receive a set of documents that describe how to write in Easy German, and you will convert these guidelines (where possible) into "rules" that can be implemented. Some of these rules are suitable to be implemented e.g. with spacy, others might work better as a learned score e.g. with LLMs.
Additionally, you will  receive a set of human annotations (scores) that can be used to fine-tune the weights of the individual components to create a final simplicity score.

• good python skills, familiarity with spacy + huggingface is a plus
• good knowledge of the German language
• familiarity with Unix/shell scripts is a plus

• information on how to write in Easy German from theForschungsstelle Leichte Sprache Universität Hildesheim
• guidelines by the Netzwerk Leichte Sprache

[Flesch, 1979] Flesch, R. (1979). How to write plain English : A Book for Lawyers and Consumers.

[Kincaid et al., 1975] Kincaid, J. P., Fishburne, R. P., Rogers, R. L., and Chissom, B. S. (1975). Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Millington. Institute for Simulation and Training, University of Central Florida.

[Xu et al., 2016] Xu, W., Napoles, C., Pavlick, E., Chen, Q., and Callison-Burch, C. (2016). Optimizing Statistical Machine Translation for Text Simplification. Transactions of the Association for Computational Linguistics, 4:401–415.

[Tanprasert and Kauchak, 2021] Tanprasert, T. and Kauchak, D. (2021). Flesch-Kincaid is not a Text Simplification Evaluation Metric. In Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pages 1–14, Online. Association for Computational Linguistics.