[Corpora-List] Call for Participation: Temporal Information Extraction (TempEval-3) - SemEval task 1

Leon Derczynski leon at dcs.shef.ac.uk
Tue Dec 11 10:04:46 CET 2012

(apologies for cross-posting)



as part of


International Workshop on Semantic Evaluations

an ACL-SIGLEX event

Second Call for Participation


The aim of TempEval is to advance research on temporal information processing, which could eventually help NLP applications like question answering, textual entailment, summarization, etc. TempEval-3 follows on from previous TempEval events, incorporating: a three-part task structure covering event, temporal expression and temporal relation extraction; the use of the complete set of TimeML temporal relations, that was simplified in previous editions; a 10-times larger dataset; and single overall performance scores, which allow the ranking of the participating systems in each task and also in general.

Introduction: -------------

Temporal annotation is a time-consuming task for humans, which has limited the size of annotated data in previous TempEvals. Current systems, however, are performing close to the inter-annotator reliability, which suggests that larger corpora could be built starting with automatically annotated data. One of the main goals of this TempEval edition is to explore whether there is value in adding a large automatically created silver standard to a hand-crafted gold standard. It might be that for some tasks an auto-annotated larger corpus might be more useful than a hand annotated small corpus.

TempEval-3, a temporal evaluation task, is a follow-up to TempEval-1 and 2. TempEval-3 differs from its ancestors in the following respects:

(i) size of the corpus: the dataset used comprises about 500K tokens of silver standard data and about 100K tokens of gold standard data for training, compared to the corpus of roughly 50K tokens corpus used in TempEval 1 and 2;

(ii) temporal relation task: the temporal relation classification tasks are to be performed from raw text, i.e. participants need to extract events and temporal expressions first, determine which ones to link and then obtain the relation types;

(iii) tasks not independent: participants must annotate temporal expressions and events in order to do the relation task;

(iv) temporal relation types: the full set of temporal interval relations in TimeML is used, rather than the reduced set used in earlier TempEvals;

(v) annotation: most of the corpus was automatically annotated by the stateof-the-art systems from TempEval-2, a portion of the corpus, including the test dataset, that is human reviewed;

(vi) evaluation: we will report a temporal awareness score for evaluating temporal relations, to help to rank systems with a single score.

TempEval 3 Tasks: ---------------- The tasks proposed for TempEval-3 are related to each one of the main TimeML tags. These are:

* Task A: Temporal expression extraction and normalization Determine the extent of the time expressions in a text as defined by the TimeML TIMEX3 tag. In addition, determine the value of the features TYPE and VAL. The possible values of TYPE are time, date, duration, and set; the value of VAL is a normalized value as defined by the TIMEX3 standard. The main attribute to annotate is VAL.

* Task B: Event extraction As in TempEval-2, participants will determine the extent of the events in a text as defined by the TimeML EVENT tag. In addition, systems may determine the value of the features CLASS, TENSE, ASPECT, POLARITY, MODALITY and also identify if the event is a main event or not. The main attribute to annotate is CLASS.

* Task C: Annotating temporal relations Identify the pairs of temporal entities (events or temporal expressions) that have a temporal link and classify the temporal relation between them as a TLINK. Possible pairs of entities that can have a temporal link are: (i) event and temporal expressions in the same sentence, (ii) event and document creation time, (iii) main events of consecutive sentences and (iv) pairs of events in the same sentence. For this task, we now require that the participating systems determine which entities need to be linked. The relation labels will be same as in TimeML, i.e.: before, after, includes, is-included, during, simultaneous, immediately after, immediately before, identity, begins, ends, begun-by and ended-by.

Task selection Participants may choose to do task A, B, or C. Choosing task C (relation annotation) entails doing tasks A and B (interval annotation). However, a participant may perform only task C by applying existing tools to carry out tasks A and B.

Dataset Creation ---------------- In TempEval-3, we release new data, as well as significantly reviewing and modifying existing corpora.

A large portion of the TempEval-3 data is automatically generated, using a temporal merging system. We include over half a million temporally-annotated tokens from English Gigaword, as well as 40,000 tokens of new gold-standard data.

Task Organizers: ----------------

James Allen, University of Rochester Leon Derczynski, University of Sheffield Hector Llorens, University of Alicante James Pustejovsky, Brandeis University Naushad UzZaman, University of Rochester [Primary Contact] Marc Verhagen, Brandeis University

Important Dates: ----------------

September 12, 2012 First Call for participation November 1, 2012 onwards Full Training Data available for participants February 15, 2013 Test set ready February 15, 2013 Registration Deadline [for Task Participants] March 1, 2013 onwards Start of evaluation period [Task Dependent] March 15, 2013 End of evaluation period April 9, 2013 Paper submission deadline [TBC] April 23, 2013 Reviews Due [TBC] May 4, 2013 Camera ready Due [TBC]

Summer 2013 Workshop co-located with NAACL, as part of SemEval-2013

More infomation: ----------------

The TempEval-3 website, for signup and details, is:


For details, check the task description paper here: http://arxiv.org/pdf/1206.5333v1.pdf Naushad UzZaman, Hector Llorens, James F. Allen, Leon Derczynski, Marc Verhagen, James Pustejovsky. 2012. TempEval-3: Evaluating Events, Time Expressions, and Temporal Relations. arXiv:1206.5333v1.

-- Leon R A Derczynski NLP Research Group

Department of Computer Science University of Sheffield Regent Court, 211 Portobello Sheffield S1 4DP, UK

+45 5157 4948 http://www.dcs.shef.ac.uk/~leon/

More information about the Corpora mailing list