(307) previous ~ index ~ next
To: tdt-distrib@unagi.cis.upenn.edu
From: David Graff <graff@unagi.cis.upenn.edu>
Subject: Another pass of adjudication
Date: Thu, 14 Jan 1999 16:50:33 EST
Folks,
I must apologize for the fact that last week's adjudication of the TDT-2
eval-set judgments involved one significant error on our part. The
person who adjudicated the results for topic 70 had unwittingly applied a
criterion for relevance that differed from what was used during the
initial annotation and the QC pass back in October. In other words, the
criteria for relevance on topic 70 changed abruptly at adjudication time,
and this affected the set of "corrections" that went into the topic table
that we sent to NIST last week.
Topic 70 had to do with nuclear tests conducted by India; it was
determined earlier in 1998 that Pakistani nuclear test were a direct
consequence of India's testing, so stories about Pakistani tests were
ruled as being ON-TOPIC, up to the time that eval data was sent to NIST.
When the adjudication was done last week, the person most knowledgeable
about this issue was still out on vacation, and the person who
adjudicated this topic was not fully informed on the details; this person
decided that stories mentioning just the Pakistani tests, and not
mentioning India at all, would NOT be on-topic.
Once the full annotation crew was back from holidays, we realized the
nature of the problem (in fact, we had anticipated it last week, when we
saw the number of annotation false-alarms for this topic). Today, we did
a complete readjudication of topic 70, and have sent a newer version of
the adjudicated table to NIST. I have also replaced the adjudicated table
file in our members_only directory.
The new results from this pass on topic 70 are:
1 false alarm (we removed an erroneous "on-topic" entry)
81 misses (we added this many new "on-topic" entries for 70)
I am sorry about the confusion. I have spoken with Jon Fiscus, and he is
preparing a new release of scores.
To get the current (correct) version of the adjudicated table, please
follow the same directions I presented in my announcement last week.
Last week's incorrect table is no longer present on our ftp site.
Dave Graff
(307) previous ~ index ~ next
Last updated Wed Feb 3 10:44:21 1999