(242) previous ~ index ~ next
To: allan@cs.umass.edu
From: doddingt@nist.gov
Subject: Re: Evaluation plan
Date: Wed, 07 Jun 2000 5:54:33 EDT
On Tue, 06 Jun 2000 18:35:35 -0400
James Allan wrote:
> Regarding Wessel's comments....
>
> Note that if you use the existing TDT-3 topics for your training, you
> are learning how to put them into clusters. Anything you do to
> improve their clustering might also improve your clustering of stories
> in the as-yet-untagged 60 new topics in the same corpus. Of course,
> people will try not to fall into a trap like that, but it's possible.
Huh? People will try not to fall into a trap like what? I don't know
how you would avoid it. And if there were some possible way, I seriously
doubt that people would try to avoid it. It seems very clear to me that
having 60 (old) topics identified in the test corpus will definitely bias
results in a favorable way, more so for topic detection and first story
detection than for topic tracking or link detection. Therefore the formal
evaluation can only be considered as being suggestive -- it will not give
reliable estimates of absolute performance. This is not to say that I
disagree with using TDT3 for experimentation and development. I do agree
that it would be more valuable to use it for R&D than to try to exclude it
(which, by the way, is not really possible in any case, since we have
already tested on it).
(242) previous ~ index ~ next
Last updated Mon Jun 12 13:26:39 2000