(003) previous ~ index ~ next
To: tdt-distrib@unagi.cis.upenn.edu, "'James Allan'" <allan@cs.umass.edu>
From: "Strzalkowski, Tomek (CRD)" <strzalkowski@crd.ge.com>
Subject: RE: TDT3, a variety of issues
Date: Sun, 7 Feb 1999 11:59:37 -0500
> ----------
> From: James Allan[SMTP:allan@cs.umass.edu]
> Sent: Saturday, February 06, 1999 10:43 AM
> To: tdt-distrib@unagi.cis.upenn.edu
> Subject: TDT3, a variety of issues
>
> > b. Do Tracking without labeled background stories.
>
> I'm not completely clear on what this task means. Presumably I still
> have my 1-16 on-topic stories that start the tracking. Is the idea
> that I no longer have vast amounts of KNOWN off-topic stories? What
> about including a small number (N_t?) of known off-topic stories for
> contrast? Ideally, ones that are similar in nature. As if to say,
> "I'm interested in the event discussed here, not the similar event
> that's discussed in these".
>
-------------------
My understanding is that one gets N_t on-topic stories, and that's it.
Makes it tougher to track, but is also more realistic. James suggestion
of carefully selected off-topic stories (as opposed to a vast amount
of "random" off-topic stories) is a good one too. I think this would make
a slightly different evaluation though (not unrealistic). This is in fact similar
to TREC topic narratives that gave negative clues (and which were
mostly disregarded). I can see adding negatives as part of interactive
tuning: first you say: track these, the you realize that you get some
F/A, so you thow in "but not these".
A separate issue I have is the False Alarm measure that we use.
Should we report both FA and precision (and have both as part
of cost function)? For example (taking GE1 run) topics 75 and 98
get similar (very small) precision (approx 2-4%), but their FA rates
differ by the factor of 5 (5% vs. 0.9%). The difference is the number
of stories tracked (ratio 5 to 1).
---- Tomek
(003) previous ~ index ~ next
Last updated Thu May 13 09:28:12 1999