Speaker: Brian Dillon (UMass Amherst)
Title: Evaluating expectation-based comprehension with the Syntactic Ambiguity Processing benchmark
Abstract: Prediction has been proposed as an overarching principle that explains human information processing in language and beyond. In this talk, I will present work that asks to what degree processing difficulty in syntactically complex sentences – one of the major concerns of psycholinguistics – can be explained by predictability. To begin to address this question, I will present the Syntactic Ambiguity Processing (SAP) Benchmark, two large datasets of reading time data (one using self-paced reading, and one eye-tracking). The SAP benchmark comprises a diverse set of complex English sentences that have been influential in psycholinguistic research, with the aim of facilitating precise tests of computationally explicitly models of processing difficulty: The size of the SAP benchmark datasets makes it possible to measure processing difficulty associated with individual syntactic constructions, and even individual sentences, precisely enough to rigorously test the quantitative predictions of computational models of comprehension.
Across the SAP benchmark datasets, we find that the predictions of language models with two different neural architectures sharply diverge from our reading data. In self-paced reading, they dramatically underpredicting processing difficulty, failing to predict relative difficulty among different syntactic ambiguous constructions, and only partially explaining item-wise variability. However, our eye-tracking data reveals a more nuanced picture, with surprisal selectively predicting early indices of processing difficulty in garden path sentences. Overall, the SAP benchmark data affirm a selective role for prediction in syntactic processing, and help to delineate its role in a comprehensive model of language comprehension.