Computational Psycholinguistics, Fall 2020
1 Class information
| Lecture Times | Mondays & Wednesdays 9:30–11:00am |
| Recitation Time | Fridays 10–11am |
| Lecture & Recitation Location | On Zoom! |
| Class website | https://canvas-mit-edu.ezproxy.canberra.edu.au/courses/5574 |
| Syllabus | http://www.mit.edu.ezproxy.canberra.edu.au/~rplevy/teaching/2020fall/9.19 |
2 Instructor information
| Instructor | Roger Levy (rplevy@mit.edu) |
| Instructor's office | On Zoom! |
| Instructor's office hours | MW 11am–12pm |
| Teaching Assistants | Annika Heuser (aheuser@mit.edu); Carina Kauf (ckauf@mit.edu) |
| TA Office | On Zoom! |
| TA Office Hours | Tuesdays 4pm (Annika); Thursdays 3pm (Carina) |
3 Class Description
Over the last two and a half decades, computational linguistics has been revolutionized as a result of three closely related developments: increases in computing power, the advent of large linguistic datasets, and paradigm shifts toward probabilistic modeling and deep learning. At the same time, similar theoretical developments in cognitive science have led to a view of major aspects of human cognition as instances of rational statistical inference. These developments have set the stage for renewed interest in computational approaches to how humans use language. Correspondingly, this course covers some of the most exciting developments in computational psycholinguistics over the past decade. The course spans human language comprehension, production, and acquisition, and covers key phenomena spanning phonetics, phonology, morphology, syntax, semantics, and pragmatics. Students will learn technical tools including probabilistic models, formal grammars, neural networks, and decision theory, and how theory, computational modeling, and data can be combined to advance our fundamental understanding of human language acquisition and use.
4 Class organization
The fall 2020 edition of 9.19/9.190 will operate as a "flipped" virtual classroom. It will feature pre-recorded lecture videos by the instructor available in advance of the class meeting time; part of your out-of-class work will involve watching the video for the upcoming class and preparing questions about it. In-class activities will include in-class exercises, reviewing answering questions about lecture and exercise content, and open discussion.
Also new in fall 2020 will be a recitation session led by TAs, which will review relevant material on programming, linguistics, probability, and machine learning.
5 Intended Audience
Undergraduate or graduate students in Brain & Cognitive Sciences, Linguistics, Electrical Engineering & Computer Science, and any of a number of related disciplines. The undergraduate section is 9.19, the graduate section is 9.190. Postdocs and faculty are also welcome to participate!
The course prerequisites are:
- One semester of Python programming (fulfillable by 6.00/6.0001+6.0002, for example), plus
- Either:
- one semester of probability/statistics/machine learning (fulfilled by, for example, 6.041B or 9.40), or
- one semester of introductory linguistics (fulfilled by 24.900 or equivalent).
If you think you have the requisite background but have not taken the specific courses just mentioned, please talk to the instructor to work out whether you should take this course or do other prerequisites first.
We will be doing Python programming in this course, and also using programs that must be run from the Unix/Linux/OS X command line. If you have less than one semester of Python programming experience and/or would like to strengthen your Python programming background, I have listed some resources recommended by others here.
6 Readings & Textbooks
Readings will frequently be drawn from the following textbooks:
Daniel Jurafsky and James H. Martin. Speech and Language Processing. Draft chapters for the third edition can be found here. (I refer to this book as "SLP3" in the syllabus.)
This textbook is the single most comprehensive and up-to-date introduction available to the field of computational linguistics. Note: we will also be using some chapters from the second edition (2008); these will be referred to as J&M2008 on the syllabus, and will be available on Stellar.
Jacob Eisenstein. 2019. Introduction to Natural Language Processing. MIT Press. Pre-publication PDF available here.
This is an excellent recent textbook on natural language processing, with contemporary deep learning thoroughly integrated. We will be using the pre-publication version freely available, under the Creative Commons CC BY-NC-ND license, here.
Bird, Steven, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O'Reilly Media. (I refer to this book as "NLTK" in the syllabus.)
This is the book for the Natural Language Toolkit (or NLTK), which we will be using extensively to do programming We will also be doing some of our programming in the Python programming language, and will make quite a bit of use of Python. You can buy this book, or you can freely access it on the Web at http://www.nltk.org/book.
Christopher D. Manning and Hinrich Schütze. (1999). Foundations of statistical natural language processing. Cambridge: MIT press. Book chapter PDFs can be obtained through the MIT library website. (I refer to this book as "M&S" in the syllabus.)
This is an older but still very useful book on natural language processing (NLP).
We'll also occasionally draw upon other sources for readings, including original research papers in computational linguistics, psycholinguistics, and other areas of the cognitive science of language.
7 Syllabus (subject to modification!)
| Week | Day | Topic | Readings | Related readings | Problem sets |
|---|---|---|---|---|---|
| Week 1 | Wed 2 Sep | Course intro; intro to probability theory; speech perception | M&S 2.1 | . Goldsmith, 2007; Clayards et al., 2008 | Pset 1 out |
| Fri 4 Sep | Recitation 1: Probability theory | ||||
| Week 2 | Mon 7 Sep | Labor Day, no class | |||
| Wed 9 Sep | Speech perception and introductory rational analysis | Selection from Anderson 1990; PMSL 5.2.4 | |||
| Fri 11 Sep | Recitation 2: Python programming | ||||
| Week 3 | Mon 14 Sep | Word sequences, language models and $N$-grams | . SLP3 3 | Pset 2 out | |
| Wed 16 Sep | Psycholinguistic methods; prediction in human language understanding; surprisal theory | . Kutas et al. 2011; TBD | . McMurray et al., 2008; Smith & Levy, 2013; Rayner, 1998; Dahan et al., 2001 | ||
| Fri 18 Sep | Recitation 3: Regular expressions in practice | Pset 1 due | |||
| Week 4 | Mon 21 Sep | Regular expressions and finite-state machines | J&M2008 2.1-2.4, 3; Eisenstein, 2018, Section 9.1 | ||
| Wed 23 Sep | Is human language finite state? | SLP3 11; J&M2008 16 (under Readings on Stellar); | . Chomsky, 1956 | Pset 3 out | |
| Fri 25 Sep | Recitation 4: Linguistics fundamentals | ||||
| Week 5 | Mon 28 Sep | Context-free grammars; syntactic analysis. | SLP3 12; Eisenstein, 2018, Section 9.2; NLTK 8.1-8.5; Levy & Andrew, 2006 | . Gazdar, 1981 (esp. Section 2); Müller, 2018 (esp. Section 5.4); Joshi et al., 1991 (on formalisms beyond context-free) | Pset 2 due |
| Wed 30 Sep | Context-free grammars and syntactic analysis continued. | ||||
| Fri 2 Aug | Recitation 5: Writing grammar fragments | ||||
| Week 6 | Mon 5 Oct | Probabilistic context-free grammars (PCFGs), incremental parsing, human syntactic processing | SLP3 10 & 13; NLTK 8.6; Levy, 2013 | . Jurafsky, 1996; Hale, 2001; Levy, 2008 | Pset 4 out |
| Wed 7 Oct | Theory, models, and data for human language comprehension | ||||
| Fri 9 Oct | Recitation 6: Designing experiments & interpreting experimental data | Pset 3 due | |||
| Week 7 | Mon 12 Oct | Indigenous Peoples' Day, no class (our class is Tuesday) | |||
| Tue 13 Oct | Bayes Nets; the perceptual magnet | Russell & Norvig, 2010, chapter 14 (on Stellar); Feldman & Griffiths, 2006; Levy in progress, Directed Graphical Models appendix; | |||
| Wed 14 Oct | Multi-factor models: logistic regression; word order preferences in language. Hierarchical models; binomial construction. | SLP3 5; Graphical models intro ; Morgan & Levy, 2015 | . Bayes Nets lecture notes; Kraljic et al., 2008 | Pset 5 out | |
| Fri 16 Oct | Recitation 7: Midterm review | Pset 4 due | |||
| Week 8 | Mon 19 Oct | Midterm exam (take-home, 24 hours) | |||
| Wed 21 Oct | Logistic regression review; basic multi-layer neural networks | SLP3 7.1-7.4; Eisenstein, 2018, 3.1-3.3 | . Mikolov et al., 2013, Pennington et al., 2014, Gutierrez et al., 2016 | ||
| Fri 23 Oct | Recitation 8: feed-forward neural networks | ||||
| Week 9 | Mon 26 Oct | Word embeddings | SLP3 6; Eisenstein, 2018, Ch. 14; Young et al., 2018, Section IV; | . Goldberg, 2015; Collobert et al., 2011; Levy, Goldberg, & Dagan, 2015; | Pset 6 out |
| Wed 28 Oct | Neural networks for natural language | SLP3 7.5, 9.1-9.4; Eisenstein 2018, 6.3 | |||
| Fri 30 Oct | Recitation 8: language modeling with deep learning | Pset 5 due | |||
| Week 10 | Mon 2 Nov | What do neural networks learn about language structure – and what don't they learn? | . Linzen & Baroni, 2020 | . Linzen et al., 2016; Gulordava et al., 2018, Linzen 2019 | |
| Wed 4 Nov | Bringing together grammar and deep learning; controlled syntactic evaluation of neural language models | . Dyer et al., 2016; Futrell et al., 2019 | . Choe & Charniak, 2016; Kuncoro et al., 2017; Dyer et al., 2015 | Pset 7 out | |
| Fri 6 Nov | Recitation 9: comparing models with human data | ||||
| Week 11 | Mon 9 Nov | Transformers; filler-gap dependencies | . Vaswani et al., 2017; Sasha Rush's "The Annotated Transformer"; Wilcox et al., 2018 | . Radford et al., 2018; Radford et al., 2019 | Pset 6 due |
| Wed 11 Nov | Veterans Day, no class | ||||
| Fri 13 Nov | Recitation 10: how attention works | ||||
| Week 12 | Mon 16 Nov | Noisy-channel language comprehension | . Levy et al., 2009 | ||
| Wed 18 Nov | Noisy-channel language comprehension II | . Gibson et al., 2013; Futrell & Levy, 2017 | |||
| Fri 20 Nov | Recitation 11: hierarchical Bayesian models | Pset 7 due | |||
| Week 13 | Mon 23 Nov | Thanksgiving holiday week, no class | |||
| Wed 25 Nov | Thanksgiving holiday week, no class | Pset 8 out | |||
| Fri 27 Nov | Thanksgiving holiday week, no recitation | ||||
| Week 14 | Mon 30 Nov | Unsupervised learning and native language acquisition I | . Feldman et al., 2013 | ||
| Wed 2 Dec | Unsupervised learning and native language acquisition II | . Saffran et al., 1996; Goldwater, Griffiths, & Johnson, 2009 | 9.190 Class projects due Friday, December 4 | ||
| Fri 4 Dec | Recitation 12: topic TBD | Pset 8 due (no penalty for turning in up to 1 week late) | |||
| Week 15 | Mon 7 Dec | Computational semantics & pragmatics | . Frank & Goodman, 2016 | ||
| Wed 9 Dec | End-of-semester review | Use this class to review readings from the rest of the semester! | |||
| Final Exam | Tue 15 Dec | Official time 1:30-4:30pm; you may arrange an alternative time convenient to you |
8 Requirements & grading
You'll be graded on:
| Work | Grade percentage (9.19) | Grade percentage (9.190) |
| A number of homework assignments throughout the semester | 60% | 48% |
| A midterm exam | 15% | 12% |
| A final exam | 25% | 20% |
| If you are enrolled in 9.190, a class project | -- | 20% |
Active participation in the class is also encouraged and taken into account in borderline grade cases!
8.1 Pset late policy
Psets can be turned in up to 7 days late; 10% of your score will be deducted for each 24 hours of lateness (rounded up). For example, if a homework assignment is worth 80 points, you turn it in 3 days late, and earn a 70 before lateness is taken into account, your score will be (1-0.3)*70=49.
8.2 Personal or medical circumstances impacting psets, exams, or projects
If personal or medical circumstances such as illness impact your work on a pset or project, or your ability to take an exam on the scheduled date with adequate preparation, please work with Student Support Services (S3) to verify these circumstances and be in touch with the instructor. We are happy to work with you in whatever way is most appropriate to your individual circumstances to help ensure that you are able to achieve your best performance in class while maintaining your health, happiness, and well-being.
8.3 Mapping of class score to letter grade
We use new homework and exam materials every year, and it can be hard to perfectly predict the difficulty of assignments and exams. Therefore I determine standards for end-of-semester letter grades in light of student performance throughout the class (though I do not grade on a curve). However, I guarantee minimum grades on the basis of the following thresholds:
| Threshold | Guaranteed minimum grade |
| >=90% | A- |
| >=80% | B- |
| >=70% | C- |
| >=60% | D |
So, for example, an overall score of 90.0001% of points guarantees you an A-, but you could well wind up with a higher grade depending on the ultimate grade thresholds determined at the end of the semester.