Manually Annotated Sub-Corpus

MASC is a balanced subset of 500K words of written texts and transcribed speech drawn primarily from the Open American National Corpus (OANC). The OANC is a 15 million word (and growing) corpus of American English produced since 1990, all of which is in the public domain or otherwise free of usage and redistribution restrictions.

All of MASC includes manually validated annotations for logical structure (headings, sections, paragraphs, etc.), sentence boundaries, three different tokenizations with associated part of speech tags, shallow parse (noun and verb chunks), named entities (person, location, organization, date and time), and Penn Treebank syntax. Additional manually produced or validated annotations have been produced by the MASC project for portions of the sub-corpus, including full-text annotation for FrameNet frame elements and a 100K+ sentence corpus with WordNet 3.1 sense tags, of which one-tenth are also annotated for FrameNet frame elements. Annotations of all or portions of the sub-corpus for a wide variety of other linguistic phenomena have been contributed by other projects, including PropBank, TimeBank, MPQA opinion, and several others. Co-reference annotations and clause boundaries of the entire MASC corpus are scheduled to be released by the end of 2016.

WordNet sense annotations for all occurrences of 114 words are also included in the MASC distribution, as well as FrameNet annotations for 50-100 occurrences of each of the 114 words. The sentences with WordNet and FrameNet annotations are also distributed as a part of the MASC Sentence Corpus.

Genres

Unlike most freely available corpora including a wide variety of linguistic annotations, MASC contains a balanced selection of texts from a broad range of genres:

Genre No. files No. words Pct corpus
Court transcript 2 30052 6%
Debate transcript 2 32325 6%
Email 78 27642 6%
Essay 7 25590 5%
Fiction 5 31518 6%
Gov’t documents 5 24578 5%
Journal 10 25635 5%
Letters 40 23325 5%
Newspaper 41 23545 5%
Non-fiction 4 25182 5%
Spoken 11 25783 5%
Technical 8 27895 6%
Travel guides 7 26708 5%
Twitter 2 24180 5%
Blog 21 28199 6%
Ficlets 5 26299 5%
Movie script 2 28240 6%
Spam 110 23490 5%
Jokes 16 26582 5%
TOTAL 376 506768

Annotations

At present, MASC includes seventeen different types of linguistic annotation (* = in production; ** currently available in original format only):

Annotation type No. words
Logical 506768
Token 506768
Sentence 506768
POS/lemma (GATE) 506768
POS (Penn Treebank) 506768
POS (FrameNet) 506768
Noun chunks 506768
Verb chunks 506768
Named Entities (person, org, loc, date) 506768
Penn Treebank syntax 506768
Coreference *506768
Clause boundaries, nucleus/satellite distinctions, discourse markers *506768
FrameNet frames/frame elements 39160
PropBank **88530
Opinion 51243
TimeBank *55599
Committed Belief 4614
Event 4614
Dependency treebank **5434
Lexical substitution **35,547

All MASC annotations, whether contributed or produced in-house, are transduced to the Graph Annotation Format (GrAF) defined by ISO TC37 SC4’s Linguistic Annotation Framework (LAF). The online tool ANC2Go can transduce annotations over all or parts of MASC to any of several other formats, including CONLL IOB format and formats for use in UIMA and General Architecture for Text Engineering.

Distribution

MASC is an open data resource that can be used by anyone for any purpose. At the same time, it is a collaborative community resource that is sustained by community contributions of annotations and derived data. It is freely downloadable from the MASC download page or through the Linguistic Data Consortium.

MASC is also distributed in part-of-speech-tagged form with the Natural Language Toolkit.

See also

References

This article is issued from Wikipedia - version of the 11/29/2016. The text is available under the Creative Commons Attribution/Share Alike but additional terms may apply for the media files.