Class GoogleDocs Folder

Assignment for April 24

1. Review the papers below

  • Figure out how the sections map to the general outline I gave in class (and below). Note that it won't be exact.
  • Read the abstract, into and conclusion for each and any other part that captures your interest. You don't have to read the whole paper unless it particularly interests you.
  • Be prepared to talk about the mapping.

Outline

  • abstract
  • 1. Intro
  • 2. Background/ Methodology / Related work / Bigger picture
  • 3. Description of the work itself (often multiple sections)
  • 4. Results (with numbers)
  • 5. Conclusion
  • References

Papers

Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory
Lynn Carlson, Daniel Marcu,Mary Ellen Okurowsk, 2004 ACL Workshop on Discourse Annotation

Image Annotation with ISO-Space: Distinguishing Content from Structure
James Pustejovsky and Zachary Yochum

Annotating Discourse Connectives And Their Arguments
Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, Bonnie Webber

Anaphoric Annotation in the ARRAU Corpus
Massimo Poesio, Ron Artstein

Discourse Annotation and Semantic Annotation in the GNOME Corpus
Massimo Poesio

Sentiment Analysis of Twitter Data
Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, Rebecca Passonneau

2. Find another ACL-like annotation paper that is related to your project

  • Understand how its sections map into the broad outline
  • Be prepared to give the "elevator pitch" of what it's about and why you thought it was interesting
  • NOTE: No two people in a group can have the same paper (though they can be from the same author/group

3. Read the paper following paper

Cheap and Fast — But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks

Be prepared to discuss how the methods described might work (or not) on your project.

MAE

  • MAE : https://code.google.com/p/mae-annotation/
  • MAI : https://code.google.com/p/mai-adjudication/
  • Keigh's mae and mai
    • https://www.github.com/keighrim/mae-annotation
    • https://www.github.com/keighrim/mai-adjudication

From Zach on Kappa:

1. An implementation of Fleiss’ kappa in Python I found on Google Code: https://code.google.com/p/hydrat/source/browse/src/hydrat/common/fleiss.py

IAA Starter code from Zach

  • https://github.com/zyocum/IAA


2. My implementation of Cohen’s kappa in Python: https://github.com/zyocum/cohens_kappa

Additionally, I have a couple spreadsheets with the worked out kappa calculation examples from NLAML up on Google Docs. I can put these up in ‘view only’ mode on the class Google Drive as well.

Fleiss: https://docs.google.com/a/brandeis.edu/spreadsheet/ccc?key=0AhnYYMZRL22ZdG9odlprM3E5ZFE0QVo2RGctTkFVbXc&usp=sharing
Cohen: https://docs.google.com/a/brandeis.edu/spreadsheet/ccc?key=0AhnYYMZRL22ZdF81YXJBVExtb29LdUhDVnhHUzE0SXc&usp=sharing