(218) previous ~ index ~ next
To: Sreenivasa Sista <ssista@bbn.com>
From: James Allan <allan@cs.umass.edu>
Subject: Re: Are these stories on topic ?
Date: Tue, 03 Nov 1998 16:59:21 -0500
Sreenivasa et al,
> It is difficult for a system to work with the two sets of rules (I donot
> know if the systems can adapt to these conditions without the set of
> rules given to annotators, I donot know if annotators can do
> classification without the rules):
> 1) All the stories about and surrounding the topic are allowed
> example stories on topic 39.
> 2) Only the stories which talk about the topic are allowed, i.e., topic 56
> allows only those stories which talk about "James Earl Ray's Retrial"
> only.
I am confident that we are going to find numerous problems like this.
I agree that those rules may not end up being useful to a computer.
That's where the research is....
Of course, if we just accepted that different people have different
ideas of what is "on topic" then the rules wouldn't be necessary and
we'd be closer to the IR notion of relevance. It's our insistance on
high-quality (across many people) judgements that is the source of
this confusion. (This has been the position Ellen Voorhees and Donna
Harman have argued a couple of times in the meetings.)
I like exploring the notion of high-quality and highly-agreed-upon
judgements, but I'm not confident it's going to be a big help in
solving the problem.
-- james
(218) previous ~ index ~ next
Last updated Fri Nov 6 15:29:21 1998