Pilot studies

The project officially kicks off in 2023. However, I have co-authored several studies that were pilots for the research that will be conducted within this project:

Kimmelman, V., A. Imashev, M. Mukushev & A. Sandygulova. (2020). Eyebrow position in grammatical and emotional expressions in Kazakh-Russian Sign Language: A quantitative study. PLOS ONE 15(6). https://doi.org/10.1371/journal.pone.0233731 (open access)

• In this study, we applied a Computer Vision tool (OpenPose) to quantitatively analyze eyebrow position as affected by three sentence types and three different emotions in utterances produced by ten signers. See a video below demonstrating application of the tool and sentence types.

Kuznetsova, A., A. Imashev, M. Mukushev, A. Sandygulova & V. Kimmelman (2021). Using Computer Vision to Analyze Non-manual Marking of Questions in KRSL. In D. Shterionov (ed.) Proceedings of the 1st International Workshop on Automatic Translation for Signed and Spoken Languages (AT4SSL), (pp. 49-59). Association for Machine Translation in the Americas. https://aclanthology.org/2021.mtsummit-at4ssl.6/ (open access)

• In this study, we re-analyzed parts of the data from the previous study using a different CV-tool (OpenFace) and applying Machine Learning to improve the measurements of eyebrow movements.

Kuznetsova, A., Imashev, A., Mukushev, M., Sandygulova, A., & Kimmelman, V. (2022). Functional Data Analysis of Non-manual Marking of Questions in Kazakh-Russian Sign Language. In E. Efthimiou, S.-E. Fotinea, T. Hanke, J. A. Hochgesang, J. Kristoffersen, J. Mesch, & M. Schulder (Eds.), Proceedings of the LREC2022 10th Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources (pp. 124-131). European Language Resources Association (ELRA). https://www.sign-lang.uni-hamburg.de/lrec/pub/22024.pdf

• In this study, we improved the statistical analysis of the data from the previous study in order to account for the dynamic nature of eyebrow and head movements. The figure below shows application of FDA analysis to head movement and inner and outer eyebrow movement across different sentence types with and without landmark registration (time alignment to sign boundaries).

Photo:

Anna Kuznetsova

Chizhikova, A., & Kimmelman, V. (2022). Phonetics of Negative Headshake in Russian Sign Language: A Small-Scale Corpus Study. In E. Efthimiou, S.-E. Fotinea, T. Hanke, J. A. Hochgesang, J. Kristoffersen, J. Mesch, & M. Schulder (Eds.), Proceedings of the LREC2022 10th Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources (pp. 29-36). European Language Resources Association (ELRA). https://www. sign-lang.uni-hamburg.de/lrec/pub/22011.pdf

• In this study, we analyzed phonetic properties of headshake expressing negation in Russian Sign Language using OpenFace. Example of a headshake measured as head rotation in OpenFace:

Photo:

Anastasiia Chizhikova