Klinton Bicknell

Surprisal theory has provided a unifying framework for understanding many phenomena in sentence processing (Hale, 2001; Levy, 2008a), positing that a word's conditional probability given all prior context fully determines processing difficulty. Problematically for this claim, one local statistic, word frequency, has also been shown to affect processing, even when conditional probability given context is held constant. Here, we ask whether other local statistics have a role in processing, or whether word frequency is a special case. We present the first clear evidence that more complex local statistics, word bigram and trigram probability, also affect processing independently of surprisal. These findings suggest a significant and independent role of local statistics in processing. Further, it motivates research into new generalizations of surprisal that can also explain why local statistical information should have an outsized effect.

Students using self-directed learning platforms, such as Duolingo, cannot be adequately assessed relying solely on responses to standard learning exercises due to a lack of control over learners' choices in how to utilize the platform: for example, how learners choose to sequence their studying and how much they choose to revisit old material. To provide accurate and well-controlled measurement of learner achievement, Duolingo developed two methods for injecting test items into the platform, which combined with Educational Data Mining techniques yield insights important for product development and curriculum design. We briefly discuss the unique characteristics and advantages of these two systems - Checkpoint Quiz and Review Exercises. We then present a case study investigating how different study approaches on Duolingo relate to learning outcomes as measured by these assessments. We demonstrate some of the unique benefits of these systems and show how educational data mining approaches are central to making use of this assessment data.

The emerging discipline of Learning Engineering is focused on putting into place tools and processes that use the science of learning as a basis for improving educational outcomes. An important part of Learning Engineering focuses on improving the effectiveness of educational software. In many software domains, A/B testing has become a prominent technique to achieve the software's goals. Many large companies (Amazon, Google, Facebook, etc.) run thousands of AB tests and present at the Annual Conference on Digital Experimentation (CODE), but that venue is too broad to address AB testing issues specific to EdTech platforms. We see a need to address issues with running large-scale A/B tests within the educational context, where the use of A/B testing lags other industries. This workshop will explore ways in which A/B testing in educational contexts differs from other domains and proposals to overcome current challenges so that this approach can become a more useful tool in the learning engineer's toolbox.

Reading is a highly complex learned skill in which humans move their eyes three to four times every second in response to visual and cognitive processing. The consensus view is that the details of these rapid eye-movement decisions---which part of a word to target with a saccade---are determined solely by low-level oculomotor heuristics. But maximally efficient saccade targeting would be sensitive to ongoing word identification, sending the eyes farther into a word the farther its identification has already progressed. Here, using a covert text-shifting paradigm, we showed just such a statistical relationship between saccade targeting in reading and trial-to-trial variability in cognitive processing. This result suggests that, rather than relying purely on heuristics, the human brain has learned to optimize eye movements in reading even at the fine-grained level of character-position targeting, reflecting efficiency-based sensitivity to ongoing cognitive processing.

Readers intentionally do not fixate some words, thought to be those they have already identified. In a rational model of reading, these word skipping decisions should be complex functions of the particular word, linguistic context, and visual information available. In contrast, heuristic models of reading only predict additive effects of word and context features. Here we test these predictions by implementing a rational model with Bayesian inference and predicting human skipping with the entropy of this model's posterior distribution. Results showed a significant effect of the entropy in predicting skipping above a strong baseline model including word and context features. This pattern held for entropy measures from rational models with a frequency prior but not from models with a 5‐gram prior. These results suggest complex interactions between visual input and linguistic knowledge as predicted by the rational model of reading, and a dominant role of frequency in making skipping decisions.

We present the task of Simultaneous Translation and Paraphrasing for Language Education (STAPLE). Given a prompt in one language, the goal is to generate a diverse set of correct translations that language learners are likely to produce. This is motivated by the need to create and maintain large, high-quality sets of acceptable translations for exercises in a language-learning application, and synthesizes work spanning machine translation, MT evaluation, automatic paraphrasing, and language education technology. We developed a novel corpus with unique properties for five languages (Hungarian, Japanese, Korean, Portuguese, and Vietnamese), and report on the results of a shared task challenge which attracted 20 teams to solve the task. In our meta-analysis, we focus on three aspects of the resulting systems: external training corpus selection, model architecture and training decisions, and decoding and filtering strategies. We find that strong systems start with a large amount of generic training data, and then finetune with in-domain data, sampled according to our provided learner response frequencies.

During reading, readers intentionally do not fixate a word when highly confident in its identity. In a rational model of reading, word skipping decisions should be complex functions of the particular word, linguistic context, and visual information available. In contrast, simple heuristic of reading only predicts additive effects of word and context features. Here we test these predictions by implementing a rational model with Bayesian inference, and predicting human skipping with the entropy of this model's posterior distribution. Results showed a significant effect of the entropy in predicting skipping above a strong baseline model including word and context features. This pattern held for entropy measures from rational models with a frequency prior but not from ones with a 5-gram prior. These results suggest complex interactions between visual input and linguistic knowledge as predicted by the rational model of reading, and a dominant role of frequency in making skipping decisions.

To ascertain the importance of phonetic information in the form of phonological distinctive features for the purpose of segment-level phonotactic acquisition, we compare the performance of two recurrent neural network models of phonotactic learning: one that has access to distinctive features at the start of the learning process, and one that does not. Though the predictions of both models are significantly correlated with human judgments of non-words, the feature-naive model significantly outperforms the feature-aware one in terms of probability assigned to a held-out test set of English words, suggesting that distinctive features are not obligatory for learning phonotactic patterns at the segment level.

Within human sentence processing, it is known that there are large effects of a word's probability in context on how long it takes to read it. This relationship has been quantified using information-theoretic surprisal, or the amount of new information conveyed by a word. Here, we compare surprisals derived from a collection of language models derived from n-grams, neural networks, and a combination of both. We show that the models' psychological predictive power improves as a tight linear function of language model linguistic quality. We also show that the size of the effect of surprisal is estimated consistently across all types of language models. These findings point toward surprising robustness of surprisal estimates and suggest that surprisal estimated by low-quality language models are not biased.

Many of the most significant impairments faced by individuals with autism spectrum disorder (ASD) relate to pragmatic (i.e. social) language. There is also evidence that pragmatic language differences may map to ASD-related genes. Therefore, quantifying the social-linguistic features of ASD has the potential to both improve clinical treatment and help identify gene-behavior relationships in ASD. Here, we apply vector semantics to transcripts of semi-structured interactions with children with both idiopathic and syndromic ASD. We find that children with ASD are less semantically similar to a gold standard derived from typically developing participants, and are more semantically variable. We show that this semantic similarity measure is affected by transcript word length, but that these group differences persist after removing length differences via subsampling. These findings suggest that linguistic signatures of ASD pervade child speech broadly, and can be automatically detected even in less structured interactions.

Individuals use semantic expectancy -- applying conceptual and linguistic knowledge to speech input -- to improve the accuracy and speed of language comprehension. This study tested how adults use semantic expectancy in quiet and in the presence of speech-shaped broadband noise at -7 dB and -12 dB signal-to-noise ratio (SNR). Twenty-four adults (22.1 $pm$ 3.6 years, mean $pm$ SD) were tested on a four-alternative-forced-choice task whereby they listened to sentences and were instructed to select an image matching the sentence-final word. The semantic expectancy of the sentences was unrelated to (neutral), congruent with, or conflicting with the acoustic target. Congruent expectancy improved accuracy and conflicting expectancy decreased accuracy relative to neutral, consistent with a theory where expectancy shifts beliefs toward likely words and away from unlikely words. Additionally, there were no significant interactions of expectancy and noise level when analyzed in log-odds, supporting the predictions of ideal observer models of speech perception.

The standard model is that word identification in reading is a holistic process, most efficient when words are centered in the visual field. In contrast, rational models of reading predict word identification to be a constructive process, where readers efficiently gather visual information about a word, and may combine visual information about different parts of the word obtained across multiple fixations to identify it. We tease apart these accounts by arguing that rational models should predict that the most efficient place in a word to make a second fixation (refixation) depends on the visual information the reader has already obtained. We present evidence supporting this prediction from an eye movement corpus. Computational model simulations confirm that a rational model predicts this finding, but a model implementing holistic identification does not. These results suggest that refixations can be well understood as rationally gathering visual information, and that word identification works constructively.

Christiansen & Chater (C&C) propose that language comprehenders must immediately compress perceptual data by ``chunking'' them into higher-level categories. Effective language understanding, however, requires maintaining perceptual information long enough to integrate it with downstream cues. Indeed, recent results suggest comprehenders do this. Although cognitive systems are undoubtedly limited, frameworks that do not take into account the tasks that these systems evolved to solve risk missing important insights.

In the past, most research on eye movements during reading involved a limited number of subjects reading sentences with specific experimental manipulations on target words. Such experiments usually only analyzed eye-movements measures on and around the target word. Recently, some researchers have started collecting larger data sets involving large and diverse groups of subjects reading large numbers of sentences, enabling them to consider a larger number of influences and study larger and more representative subject groups. In such corpus studies, most of the words in a sentence are analyzed. The complexity of the design of corpus studies and the many potentially uncontrolled influences in such studies pose new issues concerning the analysis methods and interpretability of the data. In particular, several corpus studies of reading have found an effect of successor word (n + 1) frequency on current word (n) fixation times, while studies employing experimental manipulations tend not to. The general interpretation of corpus studies suggests that readers obtain parafoveal lexical information from the upcoming word before they have finished identifying the current word, while the experimental manipulations shed doubt on this claim. In the present study, we combined a corpus analysis approach with an experimental manipulation (i.e., a parafoveal modification of the moving mask technique, Rayner & Bertera, 1979), so that, either (a) word n + 1, (b) word n + 2, (c) both words, or (d) neither word was masked. We found that denying preview for either or both parafoveal words increased average fixation times. Furthermore, we found successor effects similar to those reported in the corpus studies. Importantly, these successor effects were found even when the parafoveal word was masked, suggesting that apparent successor frequency effects may be due to causes that are unrelated to lexical parafoveal preprocessing. We discuss the implications of this finding both for parallel and serial accounts of word identification and for the interpretability of large correlational studies of word identification in reading in general.

Recent Electroencephalography/Magnetoencephalography (EEG/MEG) studies suggest that when contextual information is highly predictive of some property of a linguistic signal, expectations generated from context can be translated into surprisingly low-level estimates of the physical form-based properties likely to occur in subsequent portions of the unfolding signal. Whether form-based expectations are generated and assessed during natural reading, however, remains unclear. We monitored eye movements while participants read phonologically typical and atypical nouns in noun-predictive contexts (Experiment 1), demonstrating that when a noun is strongly expected, fixation durations on first-pass eye movement measures, including first fixation duration, gaze duration, and go-past times, are shorter for nouns with category typical form-based features. In Experiments 2 and 3, typical and atypical nouns were placed in sentential contexts normed to create expectations of variable strength for a noun. Context and typicality interacted significantly at gaze duration. These results suggest that during reading, form-based expectations that are translated from higher-level category-based expectancies can facilitate the processing of a word in context, and that their effect on lexical processing is graded based on the strength of category expectancy.

We present a method to jointly learn features and weights directly from distributional data in a log-linear framework. Specifically, we propose a non-parametric Bayesian model for learning phonological markedness constraints directly from the distribution of input-output mappings in an Optimality Theory (OT) setting. The model uses an Indian Buffet Process prior to learn the feature values used in the log-linear method, and is the first algorithm for learning phonological constraints without presupposing constraint structure. The model learns a system of constraints that explains observed data as well as the phonologically-grounded constraints of a standard analysis, with a violation structure corresponding to the standard constraints. These results suggest an alternative data-driven source for constraints instead of a fully innate constraint set.

While much previous work on reading in languages with alphabetic scripts has suggested that reading is word-based, reading in Chinese has been argued to be less reliant on words. This is primarily because in the Chinese writing system words are not spatially segmented, and characters are themselves complex visual objects. Here, we present a systematic characterization of the effects of a wide range of word and character properties on eye movements in Chinese reading, using a set of mixed-effects regression models. The results reveal a rich pattern of effects of the properties of the current, previous, and next words on a range of reading measures, which is strikingly similar to the pattern of effects of word properties reported in spaced alphabetic languages. This finding provides evidence that reading shares a word-based core and may be fundamentally similar across languages with highly dissimilar scripts. We show that these findings are robust to the inclusion of character properties in the regression models, and are equally reliable when dependent measures are defined in terms of characters rather than words, providing strong evidence that word properties have effects in Chinese reading above and beyond characters. This systematic characterization of the effects of word and character properties in Chinese advances our knowledge of the processes underlying reading and informs the future development of models of reading. More generally, however, this work suggests that differences in script may not alter the fundamental nature of reading.

It is well-known that word frequency and predictability affect processing time. These effects change magnitude across tasks, but studies testing this use tasks with different response types (e.g., lexical decision, naming, and fixation time during reading; Schilling, Rayner & Chumbley, 1998), preventing direct comparison. Recently, Kaakinen and Hyona (2010) overcame this problem, comparing fixation times in reading for comprehension and proofreading, showing that the frequency effect was larger in proofreading than in reading. This result could be explained by readers exhibiting substantial cognitive flexibility, and qualitatively changing how they process words in the proofreading task in a way that magnifies effects of word frequency. Alternatively, readers may not change word processing so dramatically, and instead may perform more careful identification generally, increasing the magnitude of many word processing effects (e.g., both frequency and predictability). We tested these possibilities with two experiments: subjects read for comprehension and then proofread for spelling errors (letter transpositions) that produce nonwords (e.g., trcak for track as in Kaakinen & Hyona) or that produce real but unintended words (e.g., trial for trail) to compare how the task changes these effects. Replicating Kaakinen and Hyona, frequency effects increased during proofreading. However, predictability effects only increased when integration with the sentence context was necessary to detect errors (i.e., when spelling errors produced words that were inappropriate in the sentence; trial for trail). The results suggest that readers adopt sophisticated word processing strategies to accommodate task demands.

It is generally assumed that the character position targeted within a particular word is not under direct cognitive control, but is rather determined by oculomotor processes sensitive only to word length and distance. An alternative view is that readers target more distant characters in words when they have parafoveally processed these words more. These possibilities are difficult to distinguish because the actual landing site within a word has large effects on subsequent word processing measures. In two experiments, we decoupled the targeted location from the actual landing site by shifting the text 3 characters during the saccade into a target word. Results show that subsequent word processing time given a particular landing site was lower/higher when the eyes would have landed further forward/backward in the word. This effect remains significant in some cases when controlling for saccade launch site. These data provide evidence against the oculomotor theory and support a cognitive account of saccade targeting.

Computational work in the past decade has produced several models accounting for phonetic category learning from distributional and lexical cues. However, there have been no computational proposals for how people might use another powerful learning mechanism: generalization from learned to analogous distinctions (e.g., from /b/--/p/ to /g/--/k/). Here, we present a new simple model of generalization in phonetic category learning, formalized in a hierarchical Bayesian framework. The model captures our proposal that linguistic knowledge includes the possibility that category types in a language (such as voiced and voiceless) can be shared across sound classes (such as labial and velar), thus naturally leading to generalization. We present two sets of simulations that reproduce key features of human performance in behavioral experiments, and we discuss the model's implications and directions for future research.

Whether readers always identify words in the order they are printed is subject to considerable debate. In the present study, we used the gaze-contingent boundary paradigm (Rayner, 1975) to manipulate the preview for a two-word target region (e.g. white walls in My neighbor painted the white walls black). Readers received an identical (white walls), transposed (walls white), or unrelated preview (vodka clubs). We found that there was a clear cost of having a transposed preview compared to an identical preview, indicating that readers cannot or do not identify words out of order. However, on some measures, the transposed preview condition did lead to faster processing than the unrelated preview condition, suggesting that readers may be able to obtain some useful information from a transposed preview. Implications of the results for models of eye movement control in reading are discussed.

Some of the most robust effects of linguistic variables on eye movements in reading are those of word length. Their leading explanation states that they are caused by visual acuity limitations on word recognition. However, Bicknell (2011) presented data showing that a model of eye movement control in reading that includes visual acuity limitations and models the process of word identification from visual input (Bicknell & Levy, 2010) does not produce humanlike word length effects, providing evidence against the visual acuity account. Here, we argue that uncertainty about word length in early word identification can drive word length effects. We present an extension of Bicknell and Levy's model that incorporates word length uncertainty, and show that it produces more humanlike word length effects.

This paper presents results from the first rational model of eye movement control in reading to make predictions for the full range of the eye movement record. The model identifies the text through Bayesian inference and makes eye movement decisions to maximize the efficiency of text identification, going beyond leading approaches which select model parameters to maximize the fit to human data. Two simulations with the model demonstrate that it can produce effects of word predictability and frequency on eye movements in reading similar to those produced by humans, providing evidence that many properties of human reading behavior may be understood as following from the nature of efficient text identification.

Decades of empirical work have shown that a range of eye movement phenomena in reading are sensitive to the details of the process of word identification. Despite this, major models of eye movement control in reading do not explicitly model word identification from visual input. This paper presents an argument for developing models of eye movements that do include detailed models of word identification. Specifically, we argue that insights into eye movement behaviour can be gained by understanding which phenomena naturally arise from an account in which the eyes move for efficient word identification, and that one important use of such models is to test which eye movement phenomena can be understood this way. As an extended case study, we present evidence from an extension of a previous model of eye movement control in reading that does explicitly model word identification from visual input, Mr. Chips (Legge, Klitz, & Tjan, 1997), to test two proposals for the effect of using linguistic context on reading efficiency.

Moving one's eyes while reading is one of the most complex everyday tasks humans face. To perform efficiently, readers must make decisions about when and where to move their eyes every 200--300ms. Over the past decades, it has been demonstrated that these fine-grained decisions are influenced by a range of linguistic properties of the text, and measuring eye movements during reading has become one of the primary methods of studying online sentence comprehension. However, it is still largely unclear why linguistic variables affect the eye movement record in the ways they do. The present work begins to answer this question by presenting a rational framework for understanding eye movement control in reading, in which probabilistic language knowledge plays a crucial role. Specifically, the task of reading is taken to be one of sentence identification: readers move their eyes to efficiently obtain visual input, which they combine with probabilistic language knowledge through Bayesian inference to yield posterior beliefs about sentence form and structure. Simulations with implemented models within this framework demonstrate that it can provide a principled account of many aspects of reading behavior, including the influence of a number of linguistic variables. In addition, the framework suggests a novel explanation for one of the least understood aspects of eye movements in reading -- regressive eye movements -- and we present evidence from an eye tracking corpus to support this proposal.

While the major models of eye movement control in reading propose very different mechanisms for the generation of saccades to previous words, there has been relatively little empirical data to distinguish these hypotheses. Here we provide a systematic statistical analysis of the factors that elicit these saccades in a corpus of eye movements. We show that the results are contrary to the predictions of a number of accounts, and provide new evidence to discriminate among the models.

This research tests whether comprehenders use their knowledge of typical events in real time to process verbal arguments. In self-paced reading and event-related brain potential (ERP) experiments, we used materials in which the likelihood of a specific patient noun (brakes or spelling) depended on the combination of an agent and verb (mechanic checked vs. journalist checked). Reading times were shorter at the word directly following the patient for the congruent than the incongruent items. Differential N400s were found earlier, immediately at the patient. Norming studies ruled out any account of these results based on direct relations between the agent and patient. Thus, comprehenders dynamically combine information about real-world events based on intrasentential agents and verbs, and this combination then rapidly influences online sentence interpretation.

A number of results in the study of realtime sentence comprehension have been explained by computational models as resulting from the rational use of probabilistic linguistic information. Many times, these hypotheses have been tested in reading by linking predictions about relative word difficulty to word-aggregated eye tracking measures such as go-past time. In this paper, we extend these results by asking to what extent reading is well-modeled as rational behavior at a finer level of analysis, predicting not aggregate measures, but the duration and location of each fixation. We present a new rational model of eye movement control in reading, the central assumption of which is that eye movement decisions are made to obtain noisy visual information as the reader performs Bayesian inference on the identities of the words in the sentence. As a case study, we present two simulations demonstrating that the model gives a rational explanation for between-word regressions.

While there exist a range of sophisticated models of eye movements in reading, it remains an open question to what extent human eye movement behavior during reading is adaptive given the demands of the task. In this paper, we help to answer this question by presenting a model of reading that corrects two problems with a rational model of the task, Mr. Chips (Legge, Klitz, & Tjan, 1997). We show that the resulting model is closer to human performance across two measures, supporting the idea that many components of eye movement behavior in reading can be well understood as a rational response to the demands of the task.

Human sentence processing involves integrating probabilistic knowledge from a variety of sources in order to incrementally determine the hierarchical structure for the serial input stream. While a large number of sentence processing effects have been explained in terms of comprehenders' rational use of probabilistic information, effects of local coherences have not. We present here a new model of local coherences, viewing them as resulting from a belief-update process, and show that the relevant probabilities in our model are calculable from a probabilistic Earley parser. Finally, we demonstrate empirically that an implemented version of the model makes the correct predictions for the materials from the original experiment demonstrating local coherence effects.

This paper presents the results of a sentence completion study and a self-paced reading experiment involving a relative clause attachment ambiguity. Our results provide evidence that comprehenders resolve local structural ambiguity during incremental sentence processing by combining expectations about upcoming discourse continuations with their knowledge of typical relationships between event participants. We interpret these results as supporting an account in which comprehenders dynamically integrate cues from an in-principle unbounded range of sources during language comprehension.

In prevailing approaches to human sentence comprehension, the outcome of the word recognition process is assumed to be a categorical representation with no residual uncertainty. Yet perception is inevitably uncertain, and a system making optimal use of available information might retain this uncertainty and interactively recruit grammatical analysis and subsequent perceptual input to help resolve it. To test for the possibility of such an interaction, we tracked readers' eye movements as they read sentences constructed to vary in (i) whether an early word had near neighbors of a different grammatical category, and (ii) how strongly another word further downstream cohered grammatically with these potential near neighbors. Eye movements indicated that readers maintain uncertain beliefs about previously read word identities, revise these beliefs on the basis of relative grammatical consistency with subsequent input, and use these changing beliefs to guide saccadic behavior in ways consistent with principles of rational probabilistic inference.

This paper provides support for the hypothesis that comprehenders form online expectations for upcoming verbal arguments using their knowledge of typical events. We test this hypothesis in a self-paced reading experiment and an experiment measuring event-related brain potentials. In both experiments, we use materials in which the likelihood of the verbal patient depends on event knowledge about the particular combination of agent and verb earlier in the sentence. By manipulating the agent for a given verb, we show that comprehenders experience more processing difficulty in sentences where the patient is less likely. Norming studies and a priming experiment provide evidence that this result is unlikely to have arisen from direct linguistic associations between patient and agent, suggesting that comprehenders use their event knowledge to form expectations.

This report presents the early results of the Frame+Schema project, a project to represent image and force-dynamic schemas in FrameNet. These structures are eventually intended to be used in leveraging the wealth of frame semantic knowledge available in the FrameNet database for situation-specific reasoning. We detail our image and forcedynamic schemas in the Embodied Construction Grammar formalism and give correspondence rules to represent these schemas in FrameNet. We also analyze much of the current force-dynamic structure in the FrameNet database and suggest changes to make it more compatible with image and force-dynamic schema-based inference. We note many of the issues raised by our representation as well as the unsolved problems and also make proposals for future work in the Frame+Schema project.