(169) previous ~ index ~ next
To: doddington@nist.gov
From: Ralf Brown <Ralf_Brown@v.gp.cs.cmu.edu>
Subject: Re: The TDT3 second dry run and the October TDT workshop
Date: Wed, 01 Sep 1999 21:48:35 -0400
>Rich Schwartz wrote:
>> this data, 'but will work soon'. So we have not yet bothered to even
>> look at it. Perhaps this been the case for other sites as well.
>
>I think that you are exaggerating, Richard. The current "perfect"
>version of the TDT3 evaluation plan (version 2.7) was announced on
>Wednesday August 11th. Following this announcement, the first advisory
>that I am aware of regarding corpus problems was disseminated on Friday
>August 27th -- over two weeks later.
Some of us have been out of town and/or working on non-TDT tasks.
Only today did I get around to installing new data to test for
compatibility with the new directory structure (no problems other than
the expected need to update the script that creates our control file
listing the source and boundary files).
The only issue I have with eval 2.7 isn't with the actual evaluation
plan, but with Jon Fiscus' announcement that index files will not list
the partial source files just after the Nt_max'th YES-labeled story
for an event. In my opinion, that makes *all* tracking software using
those index files non-compliant with the spec:
Page 2, section 4.2
"The tracking task is then to correctly classify ALL SUBSEQUENT STORIES
as to whether or not they discuss the target topic."
[emphasis mine]
Page 4, section 5.2 (right-hand column)
[Training data is all stories up to last YES-labeled training story.]
"The test set will comprise the remainder of the corpus that follows."
(169) previous ~ index ~ next
Last updated Thu Sep 2 18:19:20 1999