(142) previous ~ index ~ next
To: TDT distribution <tdt-distrib@ldc.upenn.edu>
From: George Doddington <doddington@nist.gov>
Subject: TDT3 cost function changes
Date: Tue, 06 Jul 1999 13:27:00 -0700
This is a multi-part message in MIME format.
--------------E3B7BB71635EDD4223BA706B
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
The TDT evaluation cost functions provide a linear combination of miss and false alarm probabilities for the various TDT tasks. These cost functions serve to represent the overall system performance in terms of a single performance cost, or "score". This score is a function of the application and cost parameters, namely the prior probability of the target and the costs of miss and false alarm errors. It became clear during the recent TDT workshop that it would be desirable to normalize the cost so as to represent system performance in a way that minimizes the effect of cost parameters. This would provide a stable value that would be intuitively meaningful. To this end, I have modified the TDT3 evaluation plan to provide such a normalized value. The normalization yields a normalized cost of 1.0 for "most favorable guessing", i.e., for always guessing either yes or no, whichever gives minimum cost.
There was also substantial discussion at the last TDT workshop regarding the appropriateness of the cost model for the topic tracking and detection tasks. The gist of this discussion suggested an imbalance in the relative costs of misses and false alarms. In the past, these two costs have been the same. To correct this imbalance, I've changed the evaluation plan to reduce the cost of a false alarm by a factor of 10.
The attached postscript file summarizes the changes and illustrates the result of changing the cost parameters with constant-cost curves on a DET plot.
--
George Doddington in Orinda, CA: doddington@nist.gov or 925/631-6628
(142) previous ~ index ~ next
Last updated Mon Jul 12 17:16:50 1999