(217) previous ~ index ~ next
To: tdt-distrib <tdt-distrib@unagi.cis.upenn.edu>
From: Sreenivasa Sista <ssista@bbn.com>
Subject: Re: Are these stories on topic ?
Date: Tue, 3 Nov 1998 11:27:20 -0500 (EST)
Thank you for your response James, Jaime and Christopher.
On Mon, 2 Nov 1998, James Allan wrote:
>
> Getting these judgements to the point that everyone will agree is not
> a useful task. We should focus on removing the obviously incorrect
> judgements (for example, those off-by-one errors). Things that are
> close but not quite right (in my view) are probably also right but
> almost wrong (in someone else's view). Ultimately, relevance is
> subjective and that is particularly true in the finer points.
I wanted to refer to inconsistency in judging a story to be on-topic,
BRIEF and off-topic. All the three stories I've mentioned were
somewhat/not related to the topics. Considering a similar situation, i.e.
topics 44 - national tobacco settlement, 56 - James Earl Ray's Retrial,
the rules are said to be rigid and on the target (most of them were judged
to be BRIEF).
It is difficult for a system to work with the two sets of rules (I donot
know if the systems can adapt to these conditions without the set of
rules given to annotators, I donot know if annotators can do
classification without the rules):
1) All the stories about and surrounding the topic are allowed
example stories on topic 39.
2) Only the stories which talk about the topic are allowed, i.e., topic 56
allows only those stories which talk about "James Earl Ray's Retrial"
only.
Sreenivasa Sista
(217) previous ~ index ~ next
Last updated Fri Nov 6 15:29:21 1998