(305) previous ~ index ~ next
To: TDT Distrib <tdt-distrib@ldc.upenn.edu>
From: Jonathan Fiscus <jonathan.fiscus@nist.gov>
Subject: TDT Adjudicated Results Available
Date: Thu, 02 Nov 2000 16:07:43 -0500
Folks,
The adjudicated results of the TDT2000 evaluation are available from the
URLs
ftp://jaguar.ncsl.nist.gov/tdt/tdt2000/TDT2000_official_results_20001102/index.htm
ftp://jaguar.ncsl.nist.gov/tdt/tdt2000/TDT2000_official_results_20001102.tgz
The scores in this release are unchanged for the story segmentation
task. On average, the costs have slightly decreased for the topic
detection, first story detection, and link detection tasks as a result
of the adjudication.
On average, the topic tracking costs have increase by ~5%. I've
attributed the increase to two changes in scoring which I believe
conter-balanced the slight reduction in as a result of adjudication.
First, the fixes to the scoring scripts in the "unofficial" releases had
the effect of throwing out of the test all stories with "NO" judgments.
Since those stories are, in some sense, more difficult, the "unofficial"
scores were artificially deflated.
Second, Jon Yamron and George Doddington noted that for some systems,
the topic weighted actual decision costs were lower than the minimum DET
cost calculated from a DET curve. This was a result of improper
handling of topics that had no "on-topic" stories. I've modified the
evaluation scripts to handle this case. More details will follow if any
one's interested..
Cheers,
Jon
(305) previous ~ index ~ next
Last updated Mon Nov 13 15:12:46 2000