Querying Linguistic Databases

Principal investigators: Steven Bird | Susan Davidson | Mark Liberman | Beatrice Santorini

Research associates and PhD students: Yi Chen | Baden Hughes | Catherine Lai | Haejoong Lee | Yifeng Zheng

Papers and drafts

• Graphical Query for Linguistic Treebanks (Bird & Lee)
• Designing and Evaluating an XPath Dialect for Linguistic Queries (Bird, Chen, Davidson, Lee & Zheng)
• LPath⁺: A First-Order Complete Language for Linguistic Tree Query (Lai & Bird)
• A Formal Framework for Linguistic Tree Query (Lai)
• Extending XPath to Support Linguistic Queries (Bird, Chen, Davidson, Lee & Zheng)
• Querying and Updating Treebanks: A Critical Survey and Requirements Analysis (Lai & Bird)

Presentations

• PACLING 2007: 0th Conference of the Pacific Association for Computational Linguistics (2007-09-21)
• LingEvid 2006: International Conference on Linguistic Evidence (2006-02-02)
• PACLIC 2005: Pacific Asia Conference on Language, Information and Communication (2005-12-01)
• PLAN-X 2005: Programming Language Technologies for XML (2005-01-11)
• School of Informatics, Edinburgh (2005-02-10)
• DFKI, Saarbrücken (2005-02-17)
• SFB 441, Tübingen (2005-02-18)

Software

A graphical query-by-example tool based on LPath is available with the Natural Language Toolkit. Pre-built packages for the tool on Linux and Windows are available here. Complex LPath expressions can be built up by annotating an existing tree with a graphical query. Queries are overlaid on result trees, and can be refined iteratively. Here is a screenshot and a video:

Background materials

• NSF Award
• Annotation graphs - labelled directed acyclic graphs which respect the underlying timeflow of texts and signals.
  • paper: A formal framework for linguistic annotation - general introduction (Bird & Liberman)
  • paper: Towards a query language for annotation graphs - semistructured QL; query examples (Bird, Buneman & Tan)
  • tool: agtk.sourceforge.net - implementation, the Annotation Graph Toolkit
  • data: timit.xml | timit.gif

Data models

1. Trees
   • paper: An Integrated Framework for Treebanks and Multilayer Annotations - inventory of elementary linguistic operations on trees (Cotton & Bird)
   • data: treebank data sample
   • notes: tree operations used by Beatrice Santorini
   • tree QLs: CorpusSearch | EMU | Gsearch | LSE | Netgraph | NXT | Q4M | TGrep2 | TIGERSearch | VIQTORYA
   • Question: can the elementary tree operations be implemented in XQuery?
   • Question: can Beatrice's tree operations be expressed in terms of the elementary operations?
   • Question: how do we optimize query processing for large treebanks?
2. Interlinear Texts
   • paper: A Four-Level Model for Interlinear Text - data model, XML schema, XSL stylesheets (Bow, Hughes & Bird)
   • model: jpg | xml
   • data: scanned images, XML files, stylesheets
   • note: the model has four levels of nesting, and at each level there can be multiple sequences of content in parallel, so this is a generalization of the tree model
3. Lexicons
   • paper: A Preliminary Study of the Structure of Lexicon Entries (Bell & Bird)
   • data: xml | image
   • data: many more lexicon data samples (zip file)
4. Linguistic Paradigms
   • paper: Towards a General Model for Linguistic Paradigms (Penton, Bow, Bird & Hughes)
   • data: scanned images, XML files, stylesheets
   • interface: apply the stylesheets to the XML data
   • Question: How do we easily generate this data type as the result of querying any of the other data types?

Related work

These papers provide many useful query examples to consider:

• XQuery as an annotation query language: a use-case analysis (Cassidy)
• Generalising XPath for directed graphs (Cassidy)
• Querying databases of annotated speech (Cassidy & Bird)
• XSLT as a Linguistic Query Language (Taylor)