(002) previous ~ index ~ next
To: tdt-distrib@unagi.cis.upenn.edu
From: James Allan <allan@cs.umass.edu>
Subject: TDT3, a variety of issues
Date: Sat, 06 Feb 1999 10:43:22 -0500
I guess I'll get the discussion going on TDT3 by raising a couple of
issues from Charles' message of last week. I hope others will choose
some of the comments to respond to.
I have a general concern about creating an additional six tasks over
the three we already have. I don't mind playing with very similar
tasks, though when there is formal evaluation it is discomfitting to
submit results that have not been "tuned" for that task. Perhaps some
of these tasks should be dropped, or converted into trial runs that
are viewed less as evaluation and more as exploration?
> b. Do Tracking without labeled background stories.
I'm not completely clear on what this task means. Presumably I still
have my 1-16 on-topic stories that start the tracking. Is the idea
that I no longer have vast amounts of KNOWN off-topic stories? What
about including a small number (N_t?) of known off-topic stories for
contrast? Ideally, ones that are similar in nature. As if to say,
"I'm interested in the event discussed here, not the similar event
that's discussed in these".
> 2. Languages: We will use both English and Mandarin data. (We had
> earlier talked about using Spanish as well, but decided to stick to a
> single non-English language for this year.)
Could the LDC chime in with a list of Mandarin resources that they
have? For example, if the cross-language task is used, it'd be very
useful to have a Mandarin-English-Mandarin dictionary. In all cases,
it would be good to have a tool that can be used to do word
segmentation or a version of the corpus that has been word segmented
(so we don't all invest weeks of research time doing something that is
tangential). Any other tools for manipulating/displaying/translating
the Mandarin would obviously be nice, too.
> annotated with respect to 50 target topics. (We should revisit the
> question about how topics are chosen and whether 50 are enough.)
Obviously, the more topics the better. I view 50 as a decent number.
If there are resources to do more, I strongly encourage that. A major
reason is that doing so would (presumably) provide a larger number of
each *type* of topic--small, large, event-focused, atmosphere-focused,
etc.
> 4. Training Material: We will not provide any additional English
> language training material. Sites may use all of the TDT2 data (in
> fact, any data prior to October 1998) for training, but may wish to
> preserve TDT2 test material for internal baselines. The LDC will
> provide Mandarin data from the first six months of 1998 and will
> annotate it with respect to TDT2 topics.
What is the chance of getting the missing months' data (July-Sept),
even if it isn't annotated. I'm not entirely certain that we have a
use for it, but it might be helpful for thing like idf.
> 3. The LDC has found some promising Mandarin transcription
> agencies and is hiring Mandarin annotators. They plan to begin doing
> Mandarin segmentation and annotation later this month.
Here's a big question: what is the chance of getting it all
translated? Would it be worthwhile having a "truth" English version
of the Mandarin as a baseline comparison? I'm not certain of the
value of doing this--we know that Mandarin IR works reasonably
well--but I think it's worth raising.
-- james
(002) previous ~ index ~ next
Last updated Thu May 13 09:28:12 1999