(269) previous ~ index ~ next
To: tdt-distrib <tdt-distrib@ldc.upenn.edu>
From: Jonathan Fiscus <jonathan.fiscus@nist.gov>
Subject: Default evaluation conditions
Date: Mon, 14 Dec 1998 11:11:40 -0500
Folks,
I spoke to Charles late last week and he was concerned that no one would
submit results for the other than the default TDT evaluations, (e.g.
segmentation on ASR transcripts and Tracking and Detection on the News
wire text plus ASR transcripts.)
I'd like to encourage participants to run their systems on more than the
default evaluation conditions. I know that time is a precious commodity
during these dim hours before the submission deadline so I'd like to
propose an alternative.
In our many other evaluations, there are two submission deadlines, a
deadline for primary evaluation conditions and a deadline for
contrastive evaluation conditions.
TDT has only a single deadline, December 21. In the interest of
obtaining greater breadth of coverage in both source file conditions and
task parameter variations, (i.e., deferral periods) I propose a second,
contrastive evaluation deadline, January 6th, 12:00 noon EST.
The default evaluation conditions will still be due December 21st, (and
if you are planning other non-default evaluations conditions, we will
accept them at that time also).
The contrastive evaluation deadline will include two types of system
submissions:
1) results generated for non-default evaluation conditions by a system
previously submitted on the December 21 deadline.
2) results generated by a system that explicitly breaks any of the
evaluation rules. The intent here is to test the arguments that some
evaluation rules hurt performance unnecessarily.
I know that these additional non-default evaluations represent a burden
to participants, but in the interest of good science, please consider
these additional, and important, non-default evaluation conditions.
Regards,
Jon
--
Jonathan Fiscus Snailmail: Nat'l Inst. of Stds. and Tech.
NIST 100 Bureau Dr. Stop 8940
Phone: (301) 975-3182 Gaithersburg, MD 20899-8940
Email: jfiscus@nist.gov
(269) previous ~ index ~ next
Last updated Wed Feb 3 10:44:18 1999