publications

2026

1. Linguistics 26

   They saw it, onu, 它, coming: An information theoretic study of cross-linguistic variation in personal pronouns,

   Luigi Talamo, Julius Steuer, and Annemarie Verkerk

   Linguistics, 2026

   Abs

   Personal pronouns share with other nominal constructions, such as common nouns and demonstratives, the basic function of selecting and identifying a referent. However, they are special in that they depend on the context and are inherently rooted in interlocutor roles (speaker, addressee, other). As proxies for information that is already known to the hearer, we could expect their usage to be similar across different languages, as well as little variation between pronouns in the same language. But as is known from previous studies, many languages of the world do not require overt independent personal pronouns in subject position; and different pronouns have very different roles in spoken and written text, depending most importantly on person and case. In this study, we aim to capture the predictability of personal pronouns using the information theoretic measure of surprisal, which characterizes the information value of a word given its preceding context. Our data come from mini-CIEP+, a parallel corpus of literary texts, from which we sample 17 languages from 8 language families. We compute the surprisal of personal pronouns in these languages at different context sizes, using mGPT language models. Then, linear mixed effects models are fitted with surprisal-based response variables and a range of independent variables, including the frequency of the pronoun and various morpho-syntactic parameters (syntactic role, number, person, and others). These parameters are extracted from grammars and from mini-CIEP+, which is automatically annotated in the Universal Dependencies framework using pre-trained models. We find universal effects of frequency and near-universal effects of position on surprisal, but other variable estimates differ widely between languages both in terms of which variables are relevant and their polarity. We conclude by stating that this type of quantitative study could shed further light on the usage of different types of nominal referents across languages, for which corpus-based typology is ideally positioned.

2024

1. CRAC 24

   WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case

   Vagrant Gautam, Julius Steuer, Eileen Bingert, and 3 more authors

   In Proceedings of the Seventh Workshop on Computational Models of Reference, Anaphora and Coreference, Nov 2024

   Abs DOI PDF

   While measuring bias and robustness in coreference resolution are important goals, such measurements are only as good as the tools we use to measure them. Winogender Schemas (Rudinger et al., 2018) are an influential dataset proposed to evaluate gender bias in coreference resolution, but a closer look reveals issues with the data that compromise its use for reliable evaluation, including treating different pronominal forms as equivalent, violations of template constraints, and typographical errors. We identify these issues and fix them, contributing a new dataset: WinoPron. Using WinoPron, we evaluate two state-of-the-art supervised coreference resolution systems, SpanBERT, and five sizes of FLAN-T5, and demonstrate that accusative pronouns are harder to resolve for all models. We also propose a new method to evaluate pronominal bias in coreference resolution that goes beyond the binary. With this method, we also show that bias characteristics vary not just across pronoun sets (e.g., \textithe vs. \textitshe), but also across surface forms of those sets (e.g., \textithim vs. \textithis).

2. BUCC 24

   Modeling Diachronic Change in English Scientific Writing over 300+ Years with Transformer-based Language Model Surprisal

   Julius Steuer, Marie-Pauline Krielke, Stefan Fischer, and 3 more authors

   In Proceedings of the 17th Workshop on Building and Using Comparable Corpora (BUCC) @ LREC-COLING 2024, May 2024

   Abs PDF

   This study presents an analysis of diachronic linguistic changes in English scientific writing, utilizing surprisal from transformer-based language models. Unlike traditional n-gram models, transformer-based models are potentially better at capturing nuanced linguistic changes such as long-range dependencies by considering variable context sizes. However, to create diachronically comparable language models there are several challenges with historical data, notably an exponential increase in no. of texts, tokens per text and vocabulary size over time. We address these by using a shared vocabulary and employing a robust training strategy that includes initial uniform sampling from the corpus and continuing pre-training on specific temporal segments. Our empirical analysis highlights the predictive power of surprisal from transformer-based models, particularly in analyzing complex linguistic structures like relative clauses. The models’ broader contextual awareness and the inclusion of dependency length annotations contribute to a more intricate understanding of communicative efficiency. While our focus is on scientific English, our approach can be applied to other low-resource scenarios.

2023

1. SIGTYP 23

   Information-Theoretic Characterization of Vowel Harmony: A Cross-Linguistic Study on Word Lists

   Julius Steuer, Johann-Mattis List, Badr M. Abdullah, and 1 more author

   In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP, May 2023

   Abs DOI PDF

   We present a cross-linguistic study of vowel harmony that aims to quantifies this phenomenon using data-driven computational modeling. Concretely, we define an information-theoretic measure of harmonicity based on the predictability of vowels in a natural language lexicon, which we estimate using phoneme-level language models (PLMs). Prior quantitative studies have heavily relied on inflected word-forms in the analysis on vowel harmony. On the contrary, we train our models using cross-linguistically comparable lemma forms with little or no inflection, which enables us to cover more under-studied languages. Training data for our PLMs consists of word lists offering a maximum of 1000 entries per language. Despite the fact that the data we employ are substantially smaller than previously used corpora, our experiments demonstrate the neural PLMs capture vowel harmony patterns in a set of languages that exhibit this phenomenon. Our work also demonstrates that word lists are a valuable resource for typological research, and offers new possibilities for future studies on low-resource, under-studied languages.

2. BabyLM 23

   Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures

   Julius Steuer, Marius Mosbach, and Dietrich Klakow

   In Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, Dec 2023

   Abs DOI PDF

   Research on the cognitive plausibility of language models (LMs) has so far mostly concentrated on modelling psycholinguistic response variables such as reading times, gaze durations and N400/P600 EEG signals, while mostly leaving out the dimension of what Mahowald et al. (2023) described as formal and functional linguistic competence, and developmental plausibility. We address this gap by training a series of GPT-like language models of different sizes on the strict version of the BabyLM pretraining corpus, evaluating on the challenge tasks (BLiMP, GLUE, MSGS) and an additional reading time prediction task. We find a positive correlation between LM size and performance on all three challenge tasks, with different preferences for model width and depth in each of the tasks. In contrast, a negative correlation was found between LM size and reading time fit of linear mixed-effects models using LM surprisal as a predictor, with the second-smallest LM achieving the largest log-likelihood reduction over a baseline model without surprisal. This suggests that modelling processing effort and linguistic competence may require an approach different from training GPT-like LMs on a developmentally plausible corpus.