(268) previous ~ index ~ next
To: Thomas C Pierce <tp26+@andrew.cmu.edu>
From: Jonathan Fiscus <jonathan.fiscus@nist.gov>
Subject: Re: index file choices
Date: Mon, 14 Dec 1998 08:58:29 -0500
Tom,
Sorry, this is incorrect. The fdch is a variation of the manual
transcriptions, so real mapping is this.
manual transcription == "ccap" "fdch"
automatic transcription == "asr"
sampled data signal ==
Explaintion 1: (Why categorize this way?)
There are a couple of reasons for not specifying asr as sampled data.
1) The evaluation of sample data involves using times in the
evaluation. We agreed a couple meeting ago that in order to have
meaningful comparisons between manual transcription and ASR
transcription, they should use the same units of evaluation.
2) No one is directly tracking from the audio data, they're using that
text generated by an ASR system. It's argueable that it is directly
generated from audio source, but reason 1) takes precidence.
Explaination 2: (Which manual source to use?)
The default evalaution conditions for the tracking and detection evals
are the newswire text and ASR transcripts, (index files trk_nwt+asr* and
det_nwt+asr*). There's no ambiguity here....
There are two alternative source file conditions that can be thought of
as "control" conditions, 1) Newswire text and Closed Captioning
transcripts and 2) Newswire text and FDCH transcripts. (I think of
these as control conditions in the sense that errors due to ASR are
removed from the evluation.)
The preferred control condition is the Newswire text and Close
Captioning (index files trk_nwt+man-ccap* and det_nwt+man-ccap*).
Jon
Thomas C Pierce wrote:
>
> hello,
>
> while i don't really have an answer, i can say how we've been interpreting it:
>
> > manual transcription == "fdch"
> > automatic transcription == "ccap"
> > sampled data signal == "asr"
>
> i'd agree that these interpretations are a bit of a stretch, but given
> the 3 classes listed in the eval plan, and the three sources of
> audio-derived in the data we have, the mapping above seems to be the
> closest match.
>
> -tom
--
Jonathan Fiscus Snailmail: Nat'l Inst. of Stds. and Tech.
NIST 100 Bureau Dr. Stop 8940
Phone: (301) 975-3182 Gaithersburg, MD 20899-8940
Email: jfiscus@nist.gov
(268) previous ~ index ~ next
Last updated Mon Dec 14 09:26:56 1998