Denis Emelin
Researcher in Artificial Intelligence and Natural Language Processing
University of Edinburgh, ILCC
Biography
Hello world! I recently finalized my PhD studies at the University of Edinburgh, where I was advised by Dr. Rico Sennrich and Dr. Ivan Titov. My research aims to explore and improve the extent of language understanding in neural machine translation (NMT) and (multi-lingual) language modeling. Additionally, I’m interested in developing methods that enable large language models (LLMs) to reason about and generate natural language in a manner that is aligned with principles of fairness and safety.
Work completed in the course of my PhD candidacy has demonstrated that NMT models rely on shallow heuristics when inferring the right sense of ambiguous words, and improved the ability of transformer models to represent lexical and contextual information. More recently, we showcased that state-of-the-art translation and multi-lingual language models perform poorly on tasks that incorporate commonsense reasoning. Related phenomena, such as co-reference resolution, discourse processing, and translation of figurative language are also among my varied research interests.
In the past, I completed several research internships, including one with the MOSAIC group at the Allen Institute of Artificial Intelligence, where I investigated commonsense reasoning abilities of state-of-the-art models of language. My most recent internship experiences centered around injecting factual knowledge into task-oriented dialogue systems at Amazon and the development of training objectives for LLMs that are informed by insights from language processing in the human brain at the University of Zurich.
In my spare time, I enjoy bouldering, acrobatics, music, learning new things, and caring for my miniature orchard.
Interests
- Natural language generation / generative AI
- Large language models
- Machine translation and multilingual technologies
- Grounded natural language understanding
- Commonsense, social, and moral reasoning
- Interpretability, fairness, and alignment
Education
-
PhD Informatics, 2024
University of Edinburgh, United Kingdom
-
MSc Language Science & Technology, 2018
Saarland University, Germany
-
BA German Studies (Linguistics Focus)
University of Tübingen, Germany
News
| Feb 2024 | I have completed my PhD! Viva has been passed, minor corrections to the thesis have been accepted. Awaiting graduation ceremony! |
| Feb 2023 | I started an internship at / research visit with the University of Zurich, advised by Dr. Rico Sennrich! Working on improvements to large language modeling. |
| Dec 2022 | ‘Injecting Domain Knowledge in Language Models for Task-Oriented Dialogue Systems’ was published in the Proceedings of EMNLP 2022! Much thanks to my amazing co-authors! |
| Jul 2022 | ‘Reducing Disambiguation Biases in NMT by Leveraging Explicit Word Sense Information’ was published in the Proceedings of NAACL 2022! Happy to have contributed to such an interesting project! |
| Jun 2022 | ‘Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models’. Thrilled to be involved in such an important and massive collaboration! |
| Nov 2021 | Presented ‘Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences’ and Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution at [EMNPLP 2021] |
| Nov 2021 | Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution. Many thanks to my wonderful collaborators and advisors! |
| Jun 2021 | I started an internship with Amazon, advised by Saab Mansour! Working on task-oriented dialogue systems and knowledge injection into neural networks. |
| Dec 2020 | Published ‘Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences’. Many thanks to my wonderful collaborators! |
| Nov 2020 | Presented ‘Detecting Word Sense Disambiguation Biases in Machine Translation for Model-Agnostic Adversarial Attacks’ at EMNPLP 2020 |
| Sep 2020 | ‘Detecting Word Sense Disambiguation Biases in Machine Translation for Model-Agnostic Adversarial Attacks’ has been accepted as a long paper to EMNLP 2020! |
| Jun 2020 | I started an internship with the MOSAIC group at AI2, advised by Ronan Le Bras and Yejin Choi! Working on goal-directed, commonsense reasoning. |
| Sep 2019 | I attended WeCNLP2019 in Menlo Park, USA! Impressions: High industry representation, excellent speakers, engaging panel discussion. Definitely worth attending. |
| Jul 2019 | I attended ACL 2019 and presented ‘Widening the Representation Bottleneck in Neural Machine Translation with Lexical Shortcuts’ at the Fourth Conference on Machine Translation (WMT19)! |
| Jul 2019 | I started an internship at the Information Sciences Institute (ISI), advised by Dr. Jonathan May! Working on translation of figurative language and commonsense resoning for NMT. |
| Sep 2018 | I attended the 13th Machine Translation Marathon, hosted at Charles University, Czech Republic! Participated in lectures and workshops on recent developments in MT and implemented a hierarchical character-to-word decoder as part of the week-long hackathon. |
| Jul 2018 | I attended the Microsoft Research AI Summer School 2018, hosted at Microsoft Research Cambridge, UK. Attended lectures and workshops at MSR Cambridge as one of 100 invited PhD students. |
| Mar 2018 | I started my PhD at the University of Edinburgh, ILCC! |
Recent Publications
Work Experience
Research Intern
University of Zurich
Explored strategies for aligning the predictive behaviour of pre-trained large language models with anticipatory mechanisms evidenced in the human brain for the prediction of N next words during language generation.
Accomplishments:
- Implemented a custom training objective for large language models (GPT model family), based on insights from human language processing.
Research Intern
Amazon / Amazon Web Services (AWS)
Ivestigated and implemented strategies for injecting factual knowledge into task-oriented dialogue systems to improve the consistency of generated dialogue responses with task-specific knowledge bases.
Accomplishments:
- Developed an adapter-based method for injecting factual knowledge into the parameters of neural langauge models for dialogue generation
- Assessed accuracy of knowledge retrieved at inference time through thorough experimental evaluation
- Performed extensive ablation studies and model analysis across different dialogue domains
Research Intern
Allen Institute for Artificial Intelligence (AI2)
Explored social commonsense reasoning capabilities of contemporary natural language understanding and generation models.
Accomplishments:
- Crowd-sourced a high-quality dataset
- Developed effective, task-specific neural decoding algorithms
- Performed extensive experimentation and model analysis
Research Intern
University of Southern California, Information Sciences Institute (ISI)
Developed methods for improved translation of figurative language.
Accomplishments:
- Constructed a novel, wide-coverage idiom explicitation corpus from web data
- Evaluated the ability of NMT systems to translate non-literal, non-compositional expressions
- Implemented initial strategies for improved translation of figurative language
Teaching Assistant, Tutor, Marker
University of Edinburgh
Assisted in preparing and teaching undergraduate courses on machine translation and natural language understanding.
Accomplishments:
- Created and evaluated coursework submitted by several hundred students
- Taught tutorials accompanying the primary lecture, helping students to develop a better understanding of theory and practical considerations