(142) previous ~ index ~ next
To: "firstname.lastname@example.org" <email@example.com>
From: Jonathan Fiscus <firstname.lastname@example.org>
Subject: TDT2 Dev-Test submission instructions
Date: Fri, 04 Sep 1998 11:38:44 -0400
In preparation for the workshop, we would like to specify, in detail,
what runs that participants should submit to NIST and the resources
that should be used. Since this is a debugging run for the
evaluation, we are going to limit the number submissions per site to
one submission per default condition of each of the three TDT2 tasks
(i.e. segmentation, tracking and detection).
Listed below are the data resources to be used and submission
instructions for sending system output to NIST.
Please note: the results for this test are for debugging purposes and
are not to be published.
Tests should be run on the tdt_deliv_980708 release from the
LDC. Please do not augment the corpora in any way.
NIST will use the latest release of the TDT2 evaluation
software, TDT2eval_v0.3. You can pick up the software at the URL:
TDT2 dev-test index files:
The index files for the dev-test are available from the URL
Default Evaluation conditions:
Segmentation: Automatic transcription
10,000 word deferral period
Index File: "indexes_devtest/seg_asr.ndx"
Tracking: Newswire + ASR transcriptions
Nt = 4 training stories
Story boundaries given
Detection: Newswire + ASR transcripts
Deferral period: 10 source files
Story boundaries given
Index file: indexes_devtest/det_nwt+asr.ndx
The submission process consists of 3 steps:
1) directory structure creation,
2) system output preparation and documentation,
3) transmission to NIST.
Step 1: Directory Structure Creation
Create a directory identifying your site ('SITE'). This directory
will serve as the root directory for all your submissions:
E.g. bbn, cmu, dragon, ibm, nyu, etc.
Organize your system outputs under this directory. When the
submissions are scored and subsequently published, this directory name
will be used to identify your site/organization.
For each test condition run, create a sub-directory, 'SYSID', under your
'SITE' directory identifying the condition's system name or key
attributes and the evaluation parameter settings. The 'SYSID' name is
to consist of 2 parts:
1) a free form system identification string 'DESC' chosen by
2) the evaluation parameters defined in the evaluation spec.
Example 'SYSID' names are NIST_seg_DEF=10K, NIST_trk_boundary_Nt=4,
See the "Filename and Directory Structure BNF Grammar" below for all
possible filenames and evaluation parameters. Place all files for a
particular tdt condition run under the corresponding 'SYSID'
Step 2: System output preparation and documentation.
Each condition run is to include a system description file and one or
more system output files. These files are to be placed under the
'SYSID' directory as above. Format these files as follows:
**** System descriptions are not required for the Dev-Test ****
For each submitted condition run, create a system description
file with the name 'sys-desc.txt' using the following format.
TDT2 TASK DESIGNATION
1) PRIMARY TEST SYSTEM DESCRIPTION:
3) DIFFERENCES FOR EACH CONTRASTIVE TEST:
4) NEW CONDITIONS FOR THIS EVALUATION:
TDT2 System Output files:
The system output files must be formatted according to the TDT2
evaluation plan. A system output file should be generated for each
index file. For each of the segmentation and detection tasks, there
will be one system output file per condition run. For the tracking
task, there will be a separate system output file for each topic in a
Each system output file will be named according to the basename of the
index file from which it was generated. The system output files will
have a .sys extension to identify them. For example, the index file
'trk_nwt+asr.ndx' would generate a system output file named
Step 3: Test Results Submission Protocol
Format your results as specified above before submitting them to NIST
for scoring. The following instructions assume that you are using the
operating system. If you do not have access to UNIX utilities or ftp,
please contact NIST to make alternate arrangements.
First change directory to the parent directory of your the <SITE>
directory. Next, type the following command:
tar -cvf - ./<SITE> | compress > <SITE>-<SUBM_ID>.tar.Z
<SITE> is the name of the directory created in Step 1 to
identify your site.
<SUBM_ID> The submission number (e.g. your first submission
be numbered '1', your second, '2', etc.)
This command creates a single tar file containing all of your results.
Next, ftp to jaguar.ncsl.nist.gov giving the username 'anonymous' and
your e-mail address as the password. After you are logged in, issue
the following set of commands, (the prompt will be 'ftp>'):
ftp> cd /pub/benchmark/tdt2_sep98
ftp> put <SITE>-<SUBM_ID>.tar.Z
You've now submitted your recognition results to NIST. The last thing
you need to do is send an e-mail message to Jon Fiscus at
'email@example.com' to notify NIST of your submission. Please include
the name of your submission file in the message.
Note: If you choose to submit your results in multiple shipments,
submit ONLY one set of results for a given test condition run
unless you've made other arrangements with NIST.
Filename and Directory Structure BNF Grammar
The following is the BNF directory and file name format for TDT2
SITE ::= bbn | cmu | dragon | ibm | ...
SYSID ::= <DESC>_<TASK>
DESC ::= (short system description ID, preferably <= 8 characters)
TASK ::= seg_<SEG_PARAMS> | trk_<TRK_PARAMS> | det_<DET_PARAMS>
SEG_PARAMS :== <SEG-DEFERRAL>_
TRK_PARAMS :== <BOUNDARY>_<NTRAIN>
DET_PARAMS :== <BOUNDARY>_<DET-DEFERRAL>
SEG-DEFERRAL :== DEF=100 | DEF=1K | DEF=10K
DET-DEFERRAL :== DEF=1 | DEF=10 | DEF=100
BOUNDARY := boundary | noboundary
NTRAIN :== Nt=1 | Nt=2 | Nt=4 | Nt=8 | NT=16
FILES ::= sys-desc.txt | (system description including reference to
paper if applicable)
<INDEX>.sys (file containing TDT2 system output)
INDEX ::= The TDT2 index filename corresponding to the system
Phone: (301) 975-3182
(142) previous ~ index ~ next
Last updated Wed Sep 9 09:40:56 1998