(189) previous ~ index ~ next
To: David Graff <graff@unagi.cis.upenn.edu>
From: Jonathan Fiscus <jonathan.fiscus@nist.gov>
Subject: Re: Plans for full train+devtest TDT release
Date: Fri, 02 Oct 1998 15:26:34 -0400
Dave,
> Dave wrote:
>
> I'm not entirely certain which of the following two options will be
> adopted by the sites when running development tests for the tracking
> task:
>
> (1) use all four months of text data, and locate the train/test
> cut-off point for each topic at the "Nt-max"th on-topic story
> starting from 19980104
>
> (2) use only March-April text data, and locate the cut-off points at
> the "Nt-max"th on-topic story starting from 19980301
>
> I will assume that option (2) is the more likely one, which means that
> there will need to be separate topic-relevance tables for the
> within-set and across-set annotations. If option (1) is adopted, the
> two devtest topic tables can simply be combined to get the overall
> coverage.
The tracking evaluation index creation program takes as an argument the
list of source files to generate the data for, so by controlling that
list, researchers can do any number divisions of the data. So (2) is
actually what will be done.
(I've also added an option to the index maker that restricts topics for
which tracking indexes to be made... Mike S.'s comment)
However, the assumption that we should divide data up in to within-set
and accross-set annotations complicates that programatic treatment of
the corpus.
The names that you've proposed for the relevance tables,
(trntop-trntxt.rel, trntop-devtxt.rel, devtop-trntxt.rel and
devtop-devtxt.rel), are not automatically derivable from the corpus.
The file name subparts, devtop, trntxt are defined elsewhere.
I understand the desire to create smaller files, but their names need to
be derivable from the from the corpus, like using the year and month as
the filename.
Also, designations of trn and dev are true for TDT2, but TDT3 may
consider trn, dev and eval to be training data, and then you're in a
hole.
I'd favor a single file for the relavance judgements, it's simpler, and
scalable.
Jon
(189) previous ~ index ~ next
Last updated Fri Oct 2 19:04:21 1998