EARS 2004
Data Selection and Annotation
LDC Current Plan V1.12

Updated 7/15/2004

EVALUATION DATA

Broadcast News

STT English BN

• Amount: 180 minutes
• Epoch: December 2003
• Sources: up to 6, from EARS 2003 collection
• IPR: secure for 2 sources
• Status: selection underway
• Annotation requirements: careful transcription
• Annotation due:
• Transcripts finished (internal) on 6/16/2004
• Ready-to-distribute transcripts to NIST no later than 9/1/2004
• Annotation manager: Stephanie Strassel

STT Chinese BN

• Amount: 60 minutes
• Epoch: being renegotiated with NIST

• Sources: up to 3, from EARS 2003 collection
• IPR: secure for 1 source
• Status: selection underway
• Annotation requirements: careful transcription
• Annotation due:
• Transcripts finished (internal) on 6/30/2004
• Ready-to-distribute transcripts to NIST no later than 9/1/2004
• Annotation manager: Stephanie Strassel (Shudong Huang)

STT Arabic BN

• Amount: 60 minutes
• Epoch: all December 2003

• Sources: up to 3, from EARS 2003 collection
• IPR: none
• Status: selection underway
• Annotation requirements: careful transcription
• Annotation due:
• Transcripts finished (internal) on 7/16/2004
• Ready-to-distribute transcripts to NIST no later than 9/1/2004
• Annotation manager: Mohamed Maamouri

MDE English BN

• Identical to STT English BN
• Status: selection underway
• Annotation requirements: SimpleMDE V6 annotation
• Annotation due:
• Annotations finished (internal) on 8/15/2004
• Ready-to-distribute annotations to NIST no later than 9/1/2004
• Annotation manager: Stephanie Strassel (Christopher Walker)

Telephone Speech

STT English CTS

• Amount: 180 minutes
• # of calls, excerpts: 36 calls; 5 mins each
• Source: new Fisher data
• Status: selection underway
• Annotation requirements: careful transcription
• Annotation due:
• Transcripts finished (internal) on 6/16/2004
• Ready-to-distribute transcripts to NIST no later than 9/1/2004
• Annotation manager: Stephanie Strassel

STT Chinese CTS

• Amount: 60 minutes
• # of calls, excerpts: 12 calls; 5 mins each
• Source: HKUST collection
• Status: selection not yet begun; selection criteria to be defined by NIST
• Annotation requirements: careful transcription
• Annotation due:
• Transcripts finished (internal) on 6/30/2004
• Ready-to-distribute transcripts to NIST no later than 9/1/2004
• Annotation manager: Shudong Huang

STT Arabic CTS

• Amount: 60 minutes
• # of calls, excerpts: 12 calls; 5 mins each
• Source: new Levantine collection
• Status: selection not yet begun; selection criteria to be defined by NIST
• Annotation requirements: careful transcription
• Annotation due:
• Transcripts finished (internal) on 7/16/2004
• Ready-to-distribute transcripts to NIST no later than 9/1/2004
• Annotation manager: Mohamed Maamouri

MDE English CTS

• Identical to STT English CTS
• Status: selection not yet begun
• Annotation requirements: SimpleMDE V6 annotation
• Annotation due:
• Annotations finished (internal) on 8/15/2004
• Ready-to-distribute annotations to NIST no later than 9/1/2004
• Annotation manager: Stephanie Strassel (Christopher Walker)

DEVELOPMENT DATA

Broadcast News

STT English BN

• Part 1
• Amount: 180 minutes
• Epoch: existing RT-03 evaluation data – February 2001
• Sources: TDT-4
• Status: complete (developed for RT-03)
• Annotation requirements: none
• Part 2
• Amount: 180 minutes
• Epoch: January 2001
• Sources: TDT-4
• Status: selection and development by STT sites
• Annotation requirements: none      
• Part 3
• Amount: 180 minutes
• Epoch: November 15-December 1, 2003
• Sources
• some sources have no IPR
• Status: Delivered 4/2/2004, revised 4/16/2004 (LDC2004E19)
• Annotation requirements: careful transcription
• Annotation due: 4/1/2004
• Annotation manager: Stephanie Strassel (Meghan Glenn)         

STT Chinese BN

• Part 1

• Amount: 60 minutes
• Epoch: existing RT-03 evaluation data (February 2001)
• Sources: TDT-4
• Status: complete (selected & developed for RT-03)
• Annotation requirements: none

• Part 2

• Amount: 30 minutes
• Epoch: November 2003
• Sources: EARS 2003 Collection

• Status: Delivered 4/2/2004, revised 4/16/2004 (LDC2004E19)
• Annotation requirements: careful transcription
• Annotation due: 4/1/2004
• Annotation manager: Shudong Huang

STT Arabic BN

• Amount: 60 minutes
• Epoch: existing RT-03 evaluation data (February 2001)
• Sources: TDT-4
• Status: complete (developed for RT-03); corrected version released by LDC by 4/2/2004
• Annotation requirements: none

MDE English BN

• Part 1  = MDE DevTest #1
• Amount: 180 minutes
• Epoch: existing RT-03 dev/eval data
• Sources: TDT-4
• Status: Delivered 4/2/2004 (LDC2004E16), re-released V1.1 5/14/2004
• Annotation requirements: Re-annotate to V6 spec
• Annotation due: 4/1/2004
• Annotation manager: Stephanie Strassel (Christopher Walker)
• Part 2 = MDE DevTest #2
• Amount: 180 minutes
• Epoch: November 15-December 1, 2003
• Sources: Same as new STT DevTest data (STT English BN Part 3 above)
• Status: Delivered 5/21/2004 (LDC2004E29)
• Annotation requirements: SimpleMDE V6 annotation
• Annotation due: 5/1/2004
• Annotation manager: Stephanie Strassel (Christopher Walker)

Telephone Speech

STT English CTS

• Part 1
• Amount: 180 minutes
• # of calls, excerpts: 36 calls; 5 mins each 
• Source: RT-03 Fisher evaluation data
• Status: complete (developed for RT-03)
• Annotation requirements: none

• Part 2
• Amount: 180 minutes
• # of calls, excerpts: 36 calls; 5 mins each 
• Source: new Fisher data
• Status: Delivered 4/2/2004, revised 4/16/2004 (LDC2004E19)
• Annotation requirements: careful transcription
• Annotation due: 4/1/2004
• Annotation manager: Stephanie Strassel (Meghan Glenn)

STT Chinese CTS

• Part 1

• Amount: 60 minutes
• # of calls, excerpts: 12 calls; 5 mins each 
• Source: RT-03 evaluation data (CallFriend)
• Status: complete (selected & developed for RT-03)
• Annotation requirements: none

• Part 2

• Amount: 120 minutes
• Source: HKUST collection
• Status: delivered (Transcripts: LDC2004E68; Speech: LDC2004E67)
• Annotation requirements: careful transcription
• Annotation due: 5/2004 (pending NIST selection)
• Delivered: 6/30/2004
• Annotation manager: Shudong Huang

STT Arabic CTS

• Amount: 120 minutes
• # of calls, excerpts: 24 calls; 5 mins each 
• Source: new Levantine collection
• Annotation requirements:
• careful transcription
• Annotation due: 4/1/2004
• Status: Delivered 4/2/2004, revised 4/16/2004 (LDC2004E19)
• Annotation manager: Mohamed Maamouri

MDE English CTS

• Part 1 = MDE DevTest #1
• Amount: 360 minutes
• # of calls, excerpts: FISHER: 36 calls; 5 mins each

SWB: 36 calls; 5 mins each

• Source: existing RT-03 dev/eval data
• Status: Delivered 4/2/2004 (LDC2004E16) rereleased as LDC2004E16 V1.1 on 5/14/2004
• Annotation requirements: Re-annotate to V6 spec
• Annotation due: 4/1/2004
• Annotation manager: Stephanie Strassel (Christopher Walker)

• Part 2 =  MDE DevTest #2
• Amount: 180 minutes
• # of calls, excerpts: 36 FISHER calls; 5 mins each 
• Source: Same as new STT DevTest data (STT English CTS Part 2 above)
• Status: Delivered 5/21/2004 (LDC 2004E29)
• Annotation requirements: SimpleMDE V6 annotation
• Annotation due: 5/1/2004
• Annotation manager: Stephanie Strassel (Christopher Walker)

TRAINING DATA

Broadcast News

STT English BN

• Part 1: Audio
• Amount: 10K hours audio
• Epoch:
• Audio 1: March–November 2003 (7080 hours)
• Audio 2: March-July 2001 (530 hours)
• Sources (no IPR, no licenses)
• Audio 1: ABC, CBS, CNN, NBC, CSPAN, FOX, NWI, PBS, WBN
• Audio 2: ABC, NBC, MSNBC, PRI, VOA, CNN
• Status: incremental deliveries from 10/2003 – 1/5/2004; additional VOA and PRI data (340 hours) sent 5/19/04
• Annotation requirements: none

• Part 2: Transcripts
• Amount: 250 hours transcribed QTR-style
• Epoch: Dec 2000-Jan 2001
• Sources: TDT-4 (ABC, NBC, MSNBC, PRI, VOA, CNN)
• Status: incremental deliveries in progress
• 94 hours shipped 1/19/04 
• Remaining 156 hours shipped 2/9/04

STT Chinese BN

• Amount: 1600+ hours audio
• Epoch: March-November 2003 and March-July 2001
• Sources (no IPR, no licenses): CCTV, SCOLA, NTDTV, VOA
• Status: Shipped 5/19/04
• Annotation requirements: none

STT Arabic BN

• Amount: 2300+ hours audio
• Epoch: March-November 2003 and March-July 2001
• Sources (no IPR, no licenses): Al Jazeera, Dubai TV, LBC, VOA
• Status: Shipped 5/19/04
• Annotation requirements: none

MDE English BN

• Amount: ~20 hours
• Sources: Hub4
• Annotation requirements: SimpleMDE V6 annotation
• Annotation due:
• V1.0 – 6/4/2004 ~10 hours delivered, LDC2004E31
• V1.1 – 7/9/2004 remainder delivered LDC2004E31
• Annotation manager: Stephanie Strassel (Christopher Walker)

Telephone Speech

STT English CTS

• Part 1 – Transcribed files
• Amount:
• 200 hours from LDC plus 1720 hours from BBN
• Source: new Fisher data
• Status: Complete
• Annotation requirements: QTR-style transcription
• Annotation manager: David Miller (Nii Martey)
• Part 2 – Audio only
• Amount:
• 1920 hours from LDC
• Source: new Fisher data
• Status: Complete

STT Chinese CTS

• Amount: 200 hours
• Source: new collection
• Status: In progress
• Annotation requirements: verbatim transcription (pin-yin with tones)
• Collection, annotation coordinated by Pascale Fung
• LDC oversight: Chris Cieri, Shudong Huang
• Delivery schedule:
• First 50 hours delivered 6/30/04 (Transcripts: LDC2004E70; Speech: LDC2004E69)
• Incremental deliveries of remaining 150 hours by 8/15/2004

STT Arabic CTS

• Part 1
• Amount: 18 hours
• Source: new Levantine collection
• Status: delivered April 1, 2004
• Annotation requirements: careful transcription
• Annotation due: 3/31/2004
• Annotation manager: Mohamed Maamouri

Part 2

• Amount: 50 hours
• Source: new Levantine collection
• Status: delivered 6/30/2004 (Transcripts: LDC2004E66; Speech: LDC2004E65)
• Annotation requirements: careful transcription
• Annotation due: 6/2004
• Annotation manager: Mohamed Maamouri

• Part 3
• Amount: 50 hours
• Source: new Levantine collection
• Status: selection not yet begun
• Annotation requirements: careful transcription
• Annotation due: 12/2004
• Annotation manager: Mohamed Maamouri

MDE English CTS

• Amount: ~40 hours
• Sources: SWB
• Annotation requirements: SimpleMDE V6 annotation
• Annotation due:
• V1.0 – 6/4/2004 ~3 hours delivered, LDC2004E31
• V1.1 – 7/9/2004 – remainder delivered LDC2004E31
• Annotation manager: Stephanie Strassel (Christopher Walker)

OTHER DELIVERABLES

STT

Transcription consistency study

• Amount: Up to entire RT-03 English evaluation data: 3 hours BN, 6 hours CTS
• Sources: RT-03 evaluation data
• Completed week of 6/7

Waiting for NIST scoring to release

MDE

SimpleMDE V6 Annotation Guidelines

• Distributed 02/03/2004

Annotation consistency study

• Amount:  72 CTS files (6 hours), 30 BN files (3 hours)
• Source: Same as Dev#1 MDE release
• Status: Distributed 4/9/2004 (LDC2004E24)

English pilot annotation to test task changes

• Amount: TBD, based on budget
• Details to be negotiated

Non-English SimpleMDE pilot annotation

• Amount: 1 hour each Arabic, Chinese CTS and BN data (4 hours total)
• Sources: TBD
• Purpose: to test SimpleMDE annotation in Arabic and Chinese in preparation for future evaluation
• Timeline: Summer 2004

Non-English careful transcripts

• Currently on hold

LDC still requires details of this data set, to resolve issues of IPR and distribution