(264) previous ~ index ~ next
To: Yiming Yang <yiming@cs.cmu.edu>
From: Jonathan Fiscus <jonathan.fiscus@nist.gov>
Subject: Re: detection evaluation: no-deferral not allowed?
Date: Fri, 11 Dec 1998 16:12:31 -0500
Yiming,
Yiming Yang wrote:
>
> All,
>
> I just failed when using the NIST's procedure, "TDT2det.pl", to
> evaluate the output of our detection system with the deferral period
> set to 0. Here is the error message:
>
> yiming@moscow 19:37 [203] [dec98.expr] Illegal deferral period '0' != 1, 10 or 100 at /usr/local/bin/TDT2det.pl line 370, <SYS> chunk 1.
> Illegal deferral period '0' != 1, 10 or 100 at /usr/local/bin/TDT2det.pl line 370, <SYS> chunk 1.
> Illegal deferral period '0' != 1, 10 or 100 at /usr/local/bin/TDT2det.pl line 370, <SYS> chunk 1.
>
> The question here is, are we allowed to compare our system under the
> conditions of with and without decision deferral? Without such a
> comparison, how would we know whether the deferral makes any
> improvement?
> Any clarification would be helpful.
According to the spec. the definition of the deferral period is the
"number of source files, including the currently processed source file"
so a deferral period of 0 doesn't exist.
The eval software only accepts the deferral periods defined by the
spec. One of the suggestions from the last meeting was to relax this
constraint, but I haven't had time to do that.
If you want to experiment with different deferrals, either I can send
you a fix, or comment out that one test and line 370.
> Moreover, do we have the choice of submiting detection
> results without deferral if it turns out with a better result than
> using deferral?
I need to think about this one over the weekend.
Regards,
Jon
--
Jonathan Fiscus Snailmail: Nat'l Inst. of Stds. and Tech.
NIST 100 Bureau Dr. Stop 8940
Phone: (301) 975-3182 Gaithersburg, MD 20899-8940
Email: jfiscus@nist.gov
(264) previous ~ index ~ next
Last updated Mon Dec 14 09:26:56 1998