"Transformers and Pretrained Language Models"

This is a trial lecture linked with the recruitment into a tenure track position in Machine learning.

Abstract: "In this talk I will introduce the currently most common architecture for language modeling, called transformer. The transformer architecture has revolutionized the field of Natural Language Processing (NLP). Transformers have two key mechanisms: self-attention and positional encoding, which helps temporal representation and permits focus on the relationship between words even over long distances. I will show how these types of models can be applied to the task of language modeling. I will also discuss the notion of pretraining, which is the process of learning how to represent words or sentences in very large amounts of texts. We can pretrain a language model, and refer to the resulting model as a pretrained language model."