(120) previous ~ index ~ next
To: Steve Lowe <steve@dragonsys.com>
From: George Doddington <doddington@nist.gov>
Subject: Re: Training data for dry run?
Date: Mon, 14 Jun 1999 08:58:21 -0700
Steve Lowe wrote:
>
> George,
>
> I just received your message after sending my reply to Jaime. Would
> you agree that a "tainted" experiment is acceptable, namely if we run
> the test with our existing background models that were previously
> trained from what is now test data? That is, our "dry run" submission
> would exercise the full protocol for 1999 evaluation, but would not
> constitute a scientifically valid experiment because of the partial
> conflation of training and test data.
Yes. The June dry run is not intended to calibrate research progress.
It is intended to debug the task definitions and the research support
tools and infrastructure. The research sites are expected to perform
research and make progress, of course, and the creation of evaluation
index files and procedures is intended to support research progress.
But calibration of this progress is not the purpose of the June dry
run. Also, I hope that the June meeting will provide an interesting
interchange of research ideas and findings and not just a discussion
of evaluation procedures and infrastructure issues. I think that we
have prepared as well as possible for this dry run, all things
considered, and I'm very much looking forward to an interesting and
productive meeting.
--
George Doddington in Orinda, CA: doddington@nist.gov or 925/631-6628
(120) previous ~ index ~ next
Last updated Mon Jun 21 11:18:50 1999