(243) previous ~ index ~ next
To: Rich Schwartz <schwartz@bbn.com>
From: George Doddington <doddington@nist.gov>
Subject: Re: Tracking eval software
Date: Tue, 17 Nov 1998 15:16:07 -0500
> When you say:
>
> > The previous version excluded empty stories from the evaluation.
> > The current version INCLUDES empty stories in the evaluation.
> > This change was made to prevent excusing ASR systems from gross
> > failure to output text.
>
> Do you mean that if we use the Dragon ASR and it has an empty story due
> to speech recognition errors, then all of the systems will be scored as
> missing this story?
Yes.
--------
> I understand that it is TRUE that the story was, in fact, missed.
> But given that we are all using the ASR input, I'm not sure what we're
> learning here, unless we separately count how often this happened.
We are learning how well errorful ASR output can support TDT tasks.
Some TDT system may perform better than others with ASR output as
input, but certainly there's not much that any system can do with
no text. If this case is significant, then I agree that we should
keep a separate tally of such stories.
--
George Doddington at NIST: doddington@nist.gov or 301/975-3261
(243) previous ~ index ~ next
Last updated Fri Dec 4 12:05:49 1998