How literacy shapes cognition

Reading and writing are very 'new' skills, when you consider the timescale of human evolution. In order to learn to read and write, we have to make use of pre-existing brain structures and networks, including those for visual perception and speech perception and production. In acquiring literacy, however, these networks undergo changes that can affect their suitability for their original purpose.

In a project with the Cultural Brain group at the Max Planck Institute, we collected data from more than 150 socioeconomically matched literate and illiterate participants from Chennai (Tamil Nadu, India) we investigate how acquiring literacy affects visual perception (including object and face recognition and perception of mirror invariance) and speech perception and production (including awareness of phonemes/syllables and perceptual learning of phoneme distinctions).

What distributional statistics can tell us about semantics

Word embedding models based on simple co-occurrence (models such as word2vec, GLoVe, and fastText) explain much of the variance in human ratings on various semantic dimensions (e.g., concreteness, offensiveness, etc.). While the state of the art in NLP models improves seemingly every other week or so, I am primarily interested in exploring why this simple co-occurrence principle can generate such an effective model of human cognition.

In a line of research organized loosely around this idea, I've collaborated (with Bill Thompson) to generate word embeddings from a large corpus of speech transcriptions, rather than the usual Wikipedia/Common Crawl based embeddings. The paper describing this work is under review at Behavior Research Methods.

I've also explored (with Markus Ostarek and Guillermo Montero-Melis) the learnability of visual properties from spoken language; in a letter to the editor of PNAS we posit that blind individuals cannot learn animal-color associations from spoken language, since typical colors are rarely named explicitly.

Putting embodied cognition to the test

In a series of papers and experiments with Markus Ostarek and Guillermo Montero-Melis, we probe the validity of embodied theories of language processing.

Does tapping your feet make it harder to remember leg-related words? A widely cited paper in embodied language processing claims that motor activity interferes with memory for effector-specific verbs. This claim is based on an a small sample (n = 15) and subject-aggregate ANOVA, a statistical method that has been superseded by more sensitive methods. Phase I of our Registered Replication Report has been accepted-in-principle for publication Cortex, proposing to replicate the original study with an adequate sample size and Bayesian multilevel models. In addition to the improved statistical power and methodology, we have designed a touch-sensitive MIDI-interface to monitor and analyze the motor activity (hand and foot tapping).

In another embodiment-related project, we investigate whether the brain networks controlling eye movements are involved in the processing of spatial language. For this study, we implemented a cross-decoding searchlight approach: We divide the brain into many thousands of small overlapping spheres we test if an SVM classifier trained on fMRI activity associated with eye movements in one condition can be used to predict whether participants heard up-associated or down-associated words in another condition.

Architectural constraints on language processing

In this line of inquiry, central to my dissertation work, I examine how concurrency in language production and comprehension is constrained by the architecture of the language processing system.

One chapter from this dissertation was published in Language, Cognition and Neuroscience and proposes that performance differences between speech shadowing and simultaneous interpreting are caused by a bottleneck at the lexical level. To support this claim, I present a computational model with an architecture based on a meta-analysis of speech processing studies, fitted to behavioral data from Dutch-English bilinguals shadowing and interpreting narratives across a range of carefully controlled speech rates.

In a subsequent chapter, I analyze lexical and contextual factors affecting the speech latency of close to 100,000 words produced by 40 participants across two experiments in speech shadowing and simultaneous interpreting. This chapter is currently in preparation for submission to a peer-reviewed journal.

Papers I'm currently working on

Destructive competition or fine-tuning of object recognition mechanisms? The influence of learning to read on the recognition of faces: under review at Psychological Science
Novel corpus of word embeddings trained on pseudo-conversational speech transcriptions: in press at Behavior Research Methods
Registered replication report of a paper claiming motor activity interferes with effector-specific verbal working memory: Phase I accepted at Cortex
Cross-decoding fMRI activation patterns from eye movements and up/down-associated words: in preparation
Contextual facilitation effects in speech shadowing and simultaneous interpreting: in preparation