(213) previous ~ index ~ next
To: TDT Distrib <tdt-distrib@ldc.upenn.edu>
From: Jonathan Fiscus <jonathan.fiscus@nist.gov>
Subject: TDT3eval 1.7 release, new index files
Date: Wed, 27 Oct 1999 16:00:31 -0400
Folks,
It has been a productive several weeks since the workshop. I have a new
version of the evaluation software ready, a new version of the index
files built, and additional links on our TDT3 web site. Resource
locations and modification summaries follow:
EVALUATION SOFTWARE:
--------------------
The new release of the evaluation software is at the URL:
ftp://jaguar.ncsl.nist.gov/tdt3/TDT3eval_v1.7.tgz
Modifications in this release include:
1. The TDT3BuildIndex.pl script generates shorter index file names.
For example,
'trk_SRC=nwt+bnasr_TRAIN:SL=eng,CL=eng_TEST:SL=mul,CL=eng_TOPIC=20001.ndx'
becomes
'trk_SR=nwt+bnasr_TR=eng,eng_TE=mul,eng_TP=20001.ndx'
This change necessitated re-building the index files so that people
would be familiar with the new naming conventions for the evaluation.
2. Modified the tracking software to do conditional evaluations, (i.e.
performance broken down by source data, Newswire text vs. Audio Sources,
and source language, English vs. Mandarin). The process is controlled
by a source file subset definition file, the formats and usage of which
are documented in the release. Subset-conditioned DET curves are also
supported.
3. The tracking evaluation script will read compressed system output
files and decompress them automatically. This is controlled by .Z or
.gz extensions on the system output files, and through the command line
option -Z.
4. To support systems built for last years' tracking evaluation, the
tracking evaluation script ignores the system output generated for
partial source files if the particular source file begins after the last
training story.
5. New proceedures have been implemented for selection the topic
training stories for tracking. Training stories are selected at random
within the training period.
6. The Story Link Detection task has been renamed to "Link Detection".
7. The proceedures to generate the Link Detection index files has been
corrected. The random selection process first selects the candidate
stories at random, then finds the appriopriate source file for the test
conditions. The previous version used different documents for each
condition, so comparing performance across source conditions were
suspect. Hence, this was another reason to release the index files.
EVALUATION INDEX FILES:
-----------------------
The new versions of the index files are at the URLs
ftp://jaguar.ncsl.nist.gov/tdt3/example_indexes/indexes_tdt2_0198-0698_DragonRec_v1.tar.Z
ftp://jaguar.ncsl.nist.gov/tdt3/example_indexes/indexes_tdt2_0198-0698_BBNRec_v1.tar.Z
These index files are provided for two reasons, to familiarized people
with the new filenames, and to correct the subtle problems discussed
above.
Jon
--
Jonathan Fiscus
National Inst. of Stds. and Tech.
100 Bureau Dr. Stop 8940
Gaithersburg, MD 20899-8940
Phone: (301) 975-3182
Email: jonathan.fiscus@nist.gov
(213) previous ~ index ~ next
Last updated Thu Jan 13 09:25:32 2000