(170) previous ~ index ~ next

---

To: Ron Papka <papka@dandenong.cs.umass.edu>
From: Jonathan Fiscus <jonathan.fiscus@nist.gov>
Subject: Re: Tracking Results question
Date: Wed, 09 Sep 1998 14:09:54 -0400

---

Ron,

Ah, I put in a check to verify ascending order within sources files, but
not to make sure there files were in ascending order, (because the
program doesn't pay attention to file order)

I'll add a check to the software.

Jon


Ron Papka wrote:
>
> Jon,
>
> My bad. I did verify 1,2, and 3 below, and we are "using" in all cases.
> I also noticed that the results file I was providing TDTeval_v0.3 for
> the performance below was complete, but not sorted in the correct
> file/recid order. It seems like a few additional checks on the
> input could have saved me from this mistake. However, the numbers
> I'm getting now for tracking look very similar to the UMASS software.
>
> Ron
>
> On Wed, 9 Sep 1998, Jonathan Fiscus wrote:
>
> > Ron,
> >
> > This does look wrong. Let's try to get to the bottom of this. Here are
> > the things off the top of my head.
> >
> > 1: verify that you're using TDTeval_v0.3 by execution the TDT2trk.pl
> > command with no arguments. The usage should indicate you're using
> > version 0.3.
> >
> > 2: verify that you're using version2 of the devtest index files.
> >
> > 3: verify that you're using a pristine version of the devtest corpus.
> >
> > 4: send me your results file and index file so I can score it here.
> >
> > There's got to be an explaination...
> >
> > Jon
> >
> >
> > Ron Papka wrote:
> > >
> > > Jon,
> > >
> > > I was looking at our tracking results, and perhaps I'm confusing
> > > the values in some of the columns. For topic 39, which contains
> > > 61 YES or BRIEF judgements for all sources I get the following:
> > >
> > > Tracking Performance Calculations:
> > >
> > > Filename Topic Train Test Corr Corr Miss F/A Pct. Pct. Ctrack
> > > Story Story Det. ! Det. Story Story Miss F/A
> > > -------- ----- ----- ------ ------ ------ ------ ------ ------ ------ ------
> > > tmp.39.trk 39 4 33661 3 33575 83 0 0.9651 0.0000 0.0193
> > >
> > > If I correctly detected 3 stories, how could I miss 83 stories for a
> > > topic with at most 61 stories?
> > >
> > > The number of Test Stories appears to be quite large i.e. 33K vs 20K,
> > > Could this be the source of the strange number in the Miss Story column ?
> > >
> > > Ron
> >
> > --
> > Jon Fiscus
> > NIST
> > Email: jfiscus@nist.gov
> > Phone: (301) 975-3182
> >

--
Jon Fiscus
NIST
Email: jfiscus@nist.gov
Phone: (301) 975-3182

---

(170) previous ~ index ~ next

Last updated Fri Sep 11 13:52:54 1998