Computational Linguistics
Cosci 114
Spring 1997


Assignment #1: Tokenization


Due: February 11, 1997, (by class time)


Write a tokenizer for the text below, with the following properties:


NOTE
Handle the abbreviations and contractions in two different subroutines, and be sure to comment your code extensively!!.


Here is the text

At least two people were killed and several others injured Thursday morning when a wall at a Houston mall collapsed as employees and shoppers prepared for the beginning of the business day. Dr. Smith, an M.D. and P.M.D. from Health Works Inc., located in the S.E. part of the city, reported that he wouldn't expect anyone still alive under the rubble.

Rescuers began combing through the rubble at the Northline Mall, attempting to reach anyone who could be trapped underneath the debris. (1.4MB/25 sec. QuickTime movie)

"Reports are that some people were walking in that particular area and we just don't know how many," Houston Fire Chief Eddie Corral said. "We'll just have to see as we pull the rocks and things out," Mr. Corral said.

The wall collapsed at about 9 a.m. (10 a.m. EST) in a store that was undergoing renovations.But it was not immediately clear what caused the collapse, authorities said. The section of the wall that collapsed was about 18 feet high and 150 feet long.

Rick Flanigan, a spokesman for the Houston Fire Department, said an employee had just greeted shoppers walking into the mall when a loud noise echoed throughout the building. "Apparently that's when the collapse took place," he said.

Earlier officials had said three were dead. The identities of those killed have not been released. Flanigan said at least one of the injured is in critical condition.

This story can be found at the following URL: http://cnn.com/US/9701/30/wall.fall.texas/index.html

To access our general website, go to www.cnn.com, and click on "Full Story" under title "Wall collapses in Houston mall; at least 2 dead".