HARD 2004 Annotation Timeline

Last updated: November 2, 2004

Data	Description	Size	Due Date	Delivered	Notes
HARD 2004 Evaluation Corpus	English only newswire 5 sources, January-December 2003		mid April, 2004	May 21, 2004	LDC2004E30 Contact ldc@ldc.upenn.edu
Training topics	20 new or revised topic descriptions, with metadata	20 topic description text files	May 20, 2004	June 14, 2004	LDC2004E32 Contact ldc@ldc.upenn.edu
Evaluation topics	50 new topic descriptions with metadata	50 topic description files	June 25, 2004	June 25, 2004	LDC2004E34 Contact ldc@ldc.upenn.edu
Clarification forms	LDC returns HTML clarification forms, received July 16,2004, to research sites.	up to 3 forms per topic, per site	July 29, 2004	July 23, 2004	Emailed directly to sites
Baseline runs	Sites deliver baseline runs (no metadata) to NIST	n/a	July 9, 2004	n/a	received
Final runs	Sites deliver revised/final runs (metadata) to NIST	n/a	August 5, 2004	n/a	received
Evaluation relevance judgements	LDC assesses pooled lists of documents for each topic	85 pooled documents per site per topic	October 11, 2004	delivered to NIST	LDC2004E42 Contact ldc@ldc.upenn.edu
TREC meeting	Principal Investigator's meeting, Gaithersburg, MD	participating sites	November 16-19, 2004	n/a