(148) previous ~ index ~ next
To: TDT distribution <tdt-distrib@ldc.upenn.edu>
From: George Doddington <doddington@nist.gov>
Subject: The TDT3 Evaluation Plan -- summary of changes in version 2.4
Date: Fri, 23 Jul 1999 13:00:37 -0400
A new version of the TDT3 evaluation plan has been installed. You may
access it via NIST's TDT3 URL http://www.nist.gov/speech/tdt3/tdt3.htm
This version incorporates several changes that were suggested at the
last workshop. The most significant changes are:
* Miss and false alarm error rates are calculated using topic-weighting
only. Pooled (unweighted, or story-weighted) error rate calculations
are being abandoned because topic-weighting yields better (i.e., lower
variance) estimates of error rates.
* A normalized form of Cdet is introduced as the primary representation
of performance. This provides a cost measure that varies between 0
(for perfect performance) and 1 for a "just say no" decision strategy.
This normalized form is therefore more intuitively meaningful and more
stable with changes in the cost parameters.
* The relative cost of a false alarm was reduced (from 1 to 0.1) for topic
tracking, topic detection, first-story detection and link detection.
This better represents the expected range of application needs. This
also makes it easier for a system to do better than the "just say no"
decision strategy.
* The first-story detection task is limited to English source language.
* Performance evaluation will be conditioned on source language.
--
George Doddington at NIST: doddington@nist.gov or 301/975-3261
(148) previous ~ index ~ next
Last updated Thu Aug 19 16:14:47 1999