(194) previous ~ index ~ next
To: Rich Schwartz <schwartz@bbn.com>
From: George Doddington <doddington@nist.gov>
Subject: Re: Plans for full train+devtest TDT release
Date: Tue, 06 Oct 1998 16:20:22 -0400
--------------1D752339E8743DC064E294E3
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
> In all the discussions about these new lists, etc., I'm assuming
> you're just talking about the new data that sites will be able to use to
> develop/tune their systems. The actual evaluations will ONLY involve the
> new data (collected mostly from May-June), right? That is, the topics to
> track will have the first 4 training tokens (within that period) marked,
> and the detection/clustering will be only on that data.
>
> The reason I'm asking is that some of the descriptions in the last
> two messages sounded like you were talking about the evaluation, because
> you talks about "formal" definitions, etc.
Yes, the discussion of the division of the TDT2 corpus into training and test
that you refer to pertains just to the new release of the training and devset
data. The situation is somewhat confusing because there are two uses of the
term "training". The first is the standard use of the term to mean data with
which a system is created/designed/developed. The second use applies to the
tracking task, for "training" a system for a particular topic. In this sense
the "training" data comprise that portion of the test data for which NIST
supplies topic labels to the system.
So, the answer is yes, the NIST-supplied index files serve to formally divide
the corpus into training and test. And NIST will supply a different set of
index files for the EvalSet (the May-June data) at the beginning of the
formal evaluation. These will serve to formally define the evaluation test.
--
George Doddington at NIST: doddington@nist.gov or 301/975-3261
--------------1D752339E8743DC064E294E3
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML>
<BLOCKQUOTE TYPE=CITE><TT> In
all the discussions about these new lists, etc., I'm assuming</TT>
<BR><TT>you're just talking about the new data that sites will be able
to use to</TT>
<BR><TT>develop/tune their systems. The actual evaluations will ONLY
involve the</TT>
<BR><TT>new data (collected mostly from May-June), right? That is,
the topics to</TT>
<BR><TT>track will have the first 4 training tokens (within that period)
marked,</TT>
<BR><TT>and the detection/clustering will be only on that data.</TT><TT></TT>
<P><TT> The reason I'm asking
is that some of the descriptions in the last</TT>
<BR><TT>two messages sounded like you were talking about the evaluation,
because</TT>
<BR><TT>you talks about "formal" definitions, etc.</TT></BLOCKQUOTE>
<TT></TT>
<P><BR><TT>Yes, the discussion of the division of the TDT2 corpus into
training and test</TT>
<BR><TT>that you refer to pertains just to the new release of the training
and devset</TT>
<BR><TT>data. The situation is somewhat confusing because there are
two uses of the</TT>
<BR><TT>term "training". The first is the standard use of the term
to mean data with</TT>
<BR><TT>which a system is created/designed/developed. The second
use applies to the</TT>
<BR><TT>tracking task, for "training" a system for a particular topic.
In this sense</TT>
<BR><TT>the "training" data comprise that portion of the test data for
which NIST</TT>
<BR><TT>supplies topic labels to the system.</TT><TT></TT>
<P><TT>So, the answer is yes, the NIST-supplied index files serve to formally
divide</TT>
<BR><TT>the corpus into training and test. And NIST will supply a
different set of</TT>
<BR><TT>index files for the EvalSet (the May-June data) at the beginning
of the</TT>
<BR><TT>formal evaluation. These will serve to formally define the
evaluation test.</TT>
<BR><TT>--</TT>
<BR><TT>George Doddington at NIST: doddington@nist.gov or 301/975-3261</TT></HTML>
--------------1D752339E8743DC064E294E3--
(194) previous ~ index ~ next
Last updated Wed Oct 28 14:44:11 1998