Deep-learning and symbolic models of human language, Spring 2020

This once-weekly advanced reading seminar will be devoted to recent and ongoing work on computational models of human language. Deep learning, symbolic models, and a range of approaches bringing the two together will be covered. There will be special focus on research that sheds light on language acquisition and processing in human minds and brains. This seminar offers a regular forum for cross-disciplinary interaction among cognitive scientists, linguists, and computer scientists interested in understanding and improving contemporary computational models of language. Participating in this seminar will offer you opportunities to gain expertise in state-of-the-art methods in computational linguistics, develop and implement your own novel ideas for models, and forge relationships and collaborations with peers and colleagues.

We'll meet once a week; seminar participants will take turns leading open discussions of recent papers, including both technical details and how the work fits into the larger current research landscape.

Undergraduate or graduate students in Brain & Cognitive Sciences, Linguistics, Electrical Engineering & Computer Science, and any of a number of related disciplines. Postdocs and faculty are also welcome to participate. Participants should have prior experience with contemporary computational models of language, through an advanced class such as 9.19 (Computational Psycholinguistics), 6.864 (Natural Language Processing), or their own research. If you are interested in participating but unsure about preparation, or if you cannot make it to the first meeting, please contact the instructor.

To be determined based on the interests of seminar participants. We will make sure to cover core techniques widely used in the field (recurrent neural networks, Transformers, probabilistic symbolic grammars,…) and we will also cover newer and in-development approaches.

Readings are accessible on Stellar.

The following texts are all useful background resources for this seminar:

1. Yoav Goldberg. 2017. Neural network methods for natural language processing. Synthesis Lectures on Human Language Technologies, 10(1), 1-309. You can get a PDF version of this textbook through MIT Libraries.

   You'll find this text most accessible if you have some background in machine learning but there is a brisk introductory section that will get you up to speed regardless.

2. Daniel Jurafsky and James H. Martin. Speech and Language Processing. Third edition (draft). Draft chapters can be found here.

   This textbook is the single most comprehensive and up-to-date introduction available to the field of computational linguistics.

3. Christopher D. Manning and Hinrich Schütze. (1999). Foundations of statistical natural language processing. Cambridge: MIT press. Book chapter PDFs can be obtained through the MIT library website.

   This is an older but still very useful book on natural language processing (NLP).

4. Jacob Eisenstein. 2019. Introduction to Natural Language Processing. MIT Press.

   This is a brand-new textbook; I haven't used it in teaching before but it looks really quite good. You can get a very usable draft here and I will be pointing to chapters in that draft.

5. Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. MIT Press. Deep Learning.

   This book is not NLP-specific but may be useful to consult.

You need to do the readings in advance, show up to class, lead discussions when it is your turn, and do a final project for the class. There will be ample opportunities to collaborate with fellow seminar participants and to get guidance and advice from the instructor.

There is a mailing list for this seminar: http://mailman.mit.edu.ezproxy.canberra.edu.au/mailman/listinfo/9.s916-spring2020. This mailing list will be used both for organizational purposes and for communication about seminar content. If you plan to participate in the seminar, please subscribe!