(171) previous ~ index ~ next
To: tdt-distrib@ldc.upenn.edu
From: Ralf Brown <Ralf_Brown@v.gp.cs.cmu.edu>
Subject: more probable misses on training set
Date: Wed, 09 Sep 1998 14:43:23 -0400
After looking at just a portion of the false alarms I get when running with
all January/February docs as both training and test data (FAs for two events
and 1/3 of a third one), I find the following probably-missed tags:
Event 1:
APW19980108.0631 BRIEF
CNN19980108.1600.1005 BRIEF
ABC19980112.1830.0331 YES
NYT19980114.0891 YES (maybe BRIEF)
NYT19980115.0002 YES (maybe BRIEF)
NYT19980115.0045 at least BRIEF
NYT19980115.0901 YES (maybe BRIEF)
CNN19980116.1130.1023 BRIEF
NYT19980118.0099 at least BRIEF
CNN19980120.1600.1032 BRIEF
APW19980121.0631 BRIEF
Event 11:
VOA19980104.2300.1338 at least BRIEF
ABC19980128.1830.0095 BRIEF
VOA19980128.2100.0592 BRIEF
CNN19980129.0130.0327 BRIEF
CNN19980129.0130.0392 at least BRIEF
VOA19980129.2100.0036 BRIEF
VOA19980129.2300.0013 BRIEF
Event 21:
CNN19980203.1130.0955 at least BRIEF
VOA19980224.2300.1482 YES
PRI19980227.2000.0000 BRIEF
The following are more questionable, but need to be checked:
CNN19980112.1600.1009 for Event 1
VOA19980113.2300.0918 for Event 1
VOA19980203.2100.2469 for Event 21
Once again, roughly 1/6 of the supposed false alarms that I checked were in
fact (in my opinion) missed tags, and I'm sure it isn't a definitive list of
misses, since the tracker probably failed to detect some on-topic stories.
These labeling misses significantly affect Ctrack, since the cost function
is essentially equal to the false alarm rate and a missed label is all the
more likely to show up as a false alarm the better the tracker is. There's
also the more indirect harm from having positive training instances labeled
as negative instances, which may cause the tracker to make unnecessarily-
fine distinctions.
[To forestall one possible objection: I do realize that a number of the
stories I've listed in this and prior messages are MISC rather than NEWS,
but if we're supposed to track everything, I feel that everything should
be labeled as well. That also permits further experiments where MISC
articles are scored and/or used for training.]
Ralf
(171) previous ~ index ~ next
Last updated Fri Sep 11 13:52:54 1998