(306) previous ~ index ~ next
To: TDT Distrib <tdt-distrib@ldc.upenn.edu>
From: Jonathan Fiscus <jonathan.fiscus@nist.gov>
Subject: Re: TDT Adjudicated Results Available
Date: Fri, 03 Nov 2000 14:15:51 -0500
Folks,
An updated version of the TDT evaluation software can be found at the
URL 'ftp://jaguar.ncsl.nist.gov/tdt/tdt2000/TDT3eval_v2.1.tgz'. This
release includes the fixes needed to score this year's evaluation.
Aside from the corrections to handle "NO" annotations, the most noteable
change in the release is how topic weighted tracking costs are
computed. Using the new scripts, locally scored tests will show a new
topic weighted score only if the scored test includes topics that do not
have on-topic stories associated with them.
Jon
Jon_Yamron@dragonsys.com wrote:
>
> I would like new copies of the evaluation scripts with the correction, if
> possible.
>
> - Jon
>
> Jonathan Fiscus <jonathan.fiscus@nist.gov> on 11/02/2000 04:07:43 PM
>
> To: TDT Distrib <tdt-distrib@ldc.upenn.edu>
> cc:
> Subject: TDT Adjudicated Results Available
>
> Folks,
>
> The adjudicated results of the TDT2000 evaluation are available from the
> URLs
>
> ftp://jaguar.ncsl.nist.gov/tdt/tdt2000/TDT2000_official_results_20001102/index.htm
>
> ftp://jaguar.ncsl.nist.gov/tdt/tdt2000/TDT2000_official_results_20001102.tgz
>
> The scores in this release are unchanged for the story segmentation
> task. On average, the costs have slightly decreased for the topic
> detection, first story detection, and link detection tasks as a result
> of the adjudication.
>
> On average, the topic tracking costs have increase by ~5%. I've
> attributed the increase to two changes in scoring which I believe
> conter-balanced the slight reduction in as a result of adjudication.
>
> First, the fixes to the scoring scripts in the "unofficial" releases had
> the effect of throwing out of the test all stories with "NO" judgments.
> Since those stories are, in some sense, more difficult, the "unofficial"
> scores were artificially deflated.
>
> Second, Jon Yamron and George Doddington noted that for some systems,
> the topic weighted actual decision costs were lower than the minimum DET
> cost calculated from a DET curve. This was a result of improper
> handling of topics that had no "on-topic" stories. I've modified the
> evaluation scripts to handle this case. More details will follow if any
> one's interested..
>
> Cheers,
> Jon
(306) previous ~ index ~ next
Last updated Mon Nov 13 15:12:46 2000