(023) previous ~ index ~ next
To: tdt-distrib@unagi.cis.upenn.edu
From: john.garofolo@nist.gov
Subject: TDT-2 Dry Dry Run and Beta Scoring Tools
Date: Fri, 10 Apr 1998 13:49:40 -0400
Folks,
We (at NIST) are currently developing the scoring software for the
TDT-2 tasks. We intend to conduct a dry run to debug the scoring
process and tools prior to the May 7-8 meeting to grind through the
process of TDT scoring. As such, we plan to release a Beta version of
the scoring tools and related files, indices, etc. for the January
text data on April 30 so that you can test-drive the new scoring
software using your system output before the meeting. We would ask
that if you have time before the meeting, that you send us your system
output so that we can also score them on our end and really put the
scoring software through its paces. We also ask that you come
prepared to talk about implementation and evaluation issues using the
dry-run test data and Beta scoring tools at the May 7-8 meeting here
at NIST.
Given the tight schedule, it is likely that we will only have scoring
tools for the text data in time for the meeting. We plan to complete
a Beta release of the scoring tools for the ASR data in late May.
However, sites are strongly encouraged to work with the January '98
ASR data prior to the meeting and be prepared to discuss
implementation and scoring issues for ASR as well as text. We would
like to run a second dry-run on the Jan. '98 ASR data in late May to
debug the ASR TDT scoring tools. Discussion pertaining to that
dry-run will have to be via email since another meeting so soon is
impractical. If you perform runs on the ASR prior to the May 7-8
meeting, we would appreciate you submitting those as well so that we
have sample data to work with on the ASR TDT scoring tools.
Note that the NIST TDT scoring tools will be written in PERL. So, it
will help speed the process if you make sure that you have PERL
installed on your systems prior to April 30. PERL is freely available
via the Internet for most hardware/software platforms.
Current plans call for a dry-run system evaluation using the TDT-2
development test data in August-September with another meeting to
follow. The official TDT-2 evaluation using the evaluation test data
will then be in late '98.
Stay tuned this month for information regarding the NIST TDT
scoring software and submission protocols.
Regards,
The NIST Gang.
(023) previous ~ index ~ next
Last updated Wed Sep 9 09:40:47 1998