Course Description:

In this course, you will create your own natural language training corpus for machine learning. This course guides you through the process of adding metadata to your training corpus to help ML algorithms work more efficiently. Using detailed examples at every step, you’ll learn how the MATTER Annotation Development Process helps you Model, Annotate, Train, Test, Evaluate, and Revise your training corpus. You also get a complete walkthrough of a real-world annotation project.

  • Define a clear annotation goal before collecting your dataset (corpus)
  • Learn tools for analyzing the linguistic content of your corpus
  • Build a model and specification for your annotation project
  • Examine the different annotation formats, from basic XML to the Linguistic Annotation Framework
  • Create a gold standard corpus that can be used to train and test ML algorithms
  • Select the ML algorithms that will process your annotated data
  • Evaluate the test results and revise your annotation task.
Tuesday and Friday, 9:30 - 10:50
Volen 106
Natural Language Annotation for Machine Learning
James Pustejovsky and Amber Stubbs
O'Reilly Publishers


Marie Meteer
Feldberg 127
Office Hours:
Tuesday 11 - 12 and Friday 12 - 1
and by appointment

Teaching Assistant

Zachary Yocum


Topics and assignments for each class are posted on the schedule page ( Please check this reguarly, as it may change throughout the year.
Details on the assignments are posted on the assignments page. Again, please check this reguarly, I'll update it as the assignments get closer.


Grades are based on the steps of the project as incidated below. Descriptions of each step is on the Assignments page..

5% - annotation goal, group contracts
5% - task description and corpus selection
10% - annotation schema
10% - task specification
10% - annotation
15% - adjudication, precision and recall calculations
15% - running Machine Learning algorithms
20% - write up and final presentation
10% - peer evaluation