(341) previous ~ index ~ next
To: TDT distribution <tdt-distrib@ldc.upenn.edu>
From: George Doddington <doddington@nist.gov>
Subject: First Story Detection -- debate over definition and evaluation
Date: Mon, 19 Nov 2001 08:39:46 -0800
There appears to be considerable debate over how first story detection
should be defined and how it should be evaluated. Rather than respond
directly to the various points that have been raised, let me offer the
following perspective:
What I have suggested is that a topic affiliation be included in the
output for the FSD task. This is in agreement with the TDT framework
that we have created. Inclusion of topic assignment makes it possible
to perform ANY evaluation within the scope of our current conceptual
framework. This includes:
1) the current evaluation,
2) the evaluation that Charles Wayne suggested at our recent
TDT workshop, and
3) the evaluation based on temporal weighting that I suggested
when FSD was originally proposed a few years ago.
Alternately, if y'all don't wish to include a topic affiliation in FSD
output, the underlying TDT concepts will need to be changed in order to
accommodate those who wish to soften the current harsh all-or-nothing
nature of FSD evaluation. This may require:
1) defining a "novelty" detection task, assuming that we can
adequately define "novelty", or
2) accommodating two different kinds of false alarms -- temporally
close versus temporally distant from the true first story.
--
George Doddington in Orinda, CA: doddington@nist.gov or 925/250-8346
-------------------------------------------------------------
To unsubscribe from tdt-distrib, email majordomo@ldc.upenn.edu
with "unsubscribe tdt-distrib" in the body of the message.
(341) previous ~ index ~ next
Last updated Mon Nov 19 13:57:05 2001