(340) previous ~ index ~ next
To: tdt-distrib@unagi.cis.upenn.edu
From: Jaime Carbonell <Jaime_Carbonell@lti.cs.cmu.edu>
Subject: Re: First Story Detection -- implications of the new decision rule
Date: Mon, 19 Nov 2001 00:07:28 -0500
I basically agree with Victor, for the well-articulated reasons in
his email, and also:
5. We want to be able to track progress, which implies a comparison
baseline and/or results from earlier years. If we score the same
as in earlier TDT cycles, we can do a more direct comparison (in
addition to other forms of scoring). George always tells us that
evaluations are to measure progress of the technology.
6. If, in addition to standard FSD (or NED as per Charles's renaming)
we want to score also 2nd or 3rd stories (mistaken for first story)
with declining value, we certainly can do so.
7. The inroduction of a larger number of confusible events (as per
LDC's objective agreed at the meeting) will make it even more
challenging for NED/FSD. Given that, we really want to work on
the problem, rather than worry about retooling for different
evaluation/optimization metrics.
Underscoring point 4. F-skip will let us test more data and therefore
allows us to measure statistically more meaningful results for
NED/FSD. We really don't want to preclude that by introducing
an incomatible format, as James and Victor point out.
Our message is: Don't take away NED/FSD -- it is a very tough problem
and we want to try hard to make more (measurable) progress on it.
Best,
--Jaime
> I'd like to point out that perhaps we don't really need to change FSD
> in the way George proposed:
>
> 1. I see no reason why the format of the FSD output needs to change.
> It seems that outputting <DOCID YES/NO SCORE> will work just fine,
> since every story is viewed individually as a possible alarm. The
> stories will just be scored differently from how they used to be,
> that's all...
>
> 2. An FSD system does not really need to maintain affiliations between
> stories. It will help to maintain these affiliations, so the system
> does not generate multiple alarms per topic, but it is not required.
> For instance a system based on counting the number of novel words in
> a story will still be able to detect novelty (albeit poorly). Such a
> system does not maintain a notion of a topic.
>
> 3. If we switch to detection output format, we will lose the ability to
> plot DET curves (DET logs don't have scores). Note that the new task
> definition does not actually rule out DET curves, it is still possible
> to construct them...
>
> 4. Furthermore, introduction of F_skip will require changes to FSD output
> format that will make it incompatible with DETECTION output format
> (thanks James!)
>
> Let me know what you think...
>
> -- Victor
>
> ________________________________________________________
> Victor Lavrenko mail: lavrenko@cs.umass.edu
> (413) 545-0728 / 259-1655 http: cs.umass.edu/~lavrenko
> "If you can fill the unforgiving munite..." [R.Kipling]
>
>
> ---------- Forwarded message ----------
> Date: Fri, 16 Nov 2001 10:39:01 -0800
> From: George Doddington <doddington@nist.gov>
> To: TDT distribution <tdt-distrib@ldc.upenn.edu>
> Subject: First Story Detection -- implications of the new decision rule
>
> I hope that most TDTer's now accept that FSD requires topic detection
> as an underlying technology. Essentially, FSD merely skims the first
> story from each topic (aka event). When we first implemented FSD, we
> changed the output format to conform with the evaluation, so that only
> the first story of each topic was output. Now, however, since we want
> to soften the evaluation to allow credit for detecting on-topic stories
> that are "close" to the first story, we are forced to change the format
> for FSD output. FSD systems must now output multiple early on-topic
> stories, and topic affiliation among these stories must be maintained.
> For this reason, FSD output format must revert to the same format that
> is used for topic detection. Thus each FSD output record must contain
> a topic ID, just as in topic detection. Note, however, that FSD output
> need not include ALL on-topic stories. Only those involved in scoring
> must be output.
> --
> George Doddington in Orinda, CA: doddington@nist.gov or 925/250-8346
-------------------------------------------------------------
To unsubscribe from tdt-distrib, email majordomo@ldc.upenn.edu
with "unsubscribe tdt-distrib" in the body of the message.
(340) previous ~ index ~ next
Last updated Mon Nov 19 09:14:10 2001