(249) previous ~ index ~ next
To: "Satya Dharanipragada" <dsatya@watson.ibm.com>,
From: "George Doddington" <doddington@msn.com>
Subject: Re: Proper use of training and dev sets
Date: Wed, 25 Nov 1998 10:14:58 -0800
>It is possible to use both the training and dev sets to train a system,
>and results will vary depending on what one uses. A clear ruling on
>this is needed to make the evaluation consistent and meaningful.
Traditionally, "training data" is used to estimate model parameters
and "development data" is used to tune decision parameters and to
gain an understanding of the issues needed to address problems,
modify algorithms and create new algorithms. Traditionally, the
training data and the development data have been kept separate
in an attempt to maintain the integrity of the understanding gained
from observed devset performance.
Lately there has been a trend toward allowing system training on
devset data in addition to training data. Contributing to this trend
is the much larger size of data sets (which tend to reduce the effect
of a data set on performance) and the eternal need for ever larger
amounts of training data.
>From a dispassionate point of view, and because performance
is measured on a separate and logically independent data set,
it really is the researchers' responsibility to decide how to treat
the development data. Therefore, for the upcoming TDT2 formal
evaluation, training on the development data WILL be permitted.
It will be required, however, to describe fully the research and
procedures used. This will include information on how the devset
data were used and, in particular, whether the system was trained
on devset data.
--
George Doddington in Orinda, CA. doddington@nist.gov 925/631-6628
(249) previous ~ index ~ next
Last updated Fri Dec 4 12:05:50 1998