(166) previous ~ index ~ next
To: Jonathan Fiscus <jonathan.fiscus@nist.gov>
From: Ron Papka <papka@dandenong.cs.umass.edu>
Subject: Re: Tracking Results question
Date: Wed, 9 Sep 1998 11:42:30 -0400 (EDT)
Jon,
My bad. I did verify 1,2, and 3 below, and we are "using" in all cases.
I also noticed that the results file I was providing TDTeval_v0.3 for
the performance below was complete, but not sorted in the correct
file/recid order. It seems like a few additional checks on the
input could have saved me from this mistake. However, the numbers
I'm getting now for tracking look very similar to the UMASS software.
Ron
On Wed, 9 Sep 1998, Jonathan Fiscus wrote:
> Ron,
>
> This does look wrong. Let's try to get to the bottom of this. Here are
> the things off the top of my head.
>
> 1: verify that you're using TDTeval_v0.3 by execution the TDT2trk.pl
> command with no arguments. The usage should indicate you're using
> version 0.3.
>
> 2: verify that you're using version2 of the devtest index files.
>
> 3: verify that you're using a pristine version of the devtest corpus.
>
> 4: send me your results file and index file so I can score it here.
>
> There's got to be an explaination...
>
> Jon
>
>
> Ron Papka wrote:
> >
> > Jon,
> >
> > I was looking at our tracking results, and perhaps I'm confusing
> > the values in some of the columns. For topic 39, which contains
> > 61 YES or BRIEF judgements for all sources I get the following:
> >
> > Tracking Performance Calculations:
> >
> > Filename Topic Train Test Corr Corr Miss F/A Pct. Pct. Ctrack
> > Story Story Det. ! Det. Story Story Miss F/A
> > -------- ----- ----- ------ ------ ------ ------ ------ ------ ------ ------
> > tmp.39.trk 39 4 33661 3 33575 83 0 0.9651 0.0000 0.0193
> >
> > If I correctly detected 3 stories, how could I miss 83 stories for a
> > topic with at most 61 stories?
> >
> > The number of Test Stories appears to be quite large i.e. 33K vs 20K,
> > Could this be the source of the strange number in the Miss Story column ?
> >
> > Ron
>
> --
> Jon Fiscus
> NIST
> Email: jfiscus@nist.gov
> Phone: (301) 975-3182
>
(166) previous ~ index ~ next
Last updated Fri Sep 11 13:52:53 1998