[Corpora-List] Deadline Extension and Final Call: TempEval-3 / Temporal Information Extraction (SemEval 2012, Task 1)

Leon Derczynski leon at dcs.shef.ac.uk
Wed Feb 20 14:50:02 CET 2013

(apologies for cross-posting)

Deadline extended: Register by 22 February

New subtask: linking-only



as part of


International Workshop on Semantic Evaluations

an ACL-SIGLEX event

Second Call for Participation


The aim of TempEval is to advance research on temporal information processing, which could eventually help NLP applications like question answering, textual entailment, summarization, etc. TempEval-3 follows on from previous TempEval events, incorporating: a three-part task structure covering event, temporal expression and temporal relation extraction; the use of the complete set of TimeML temporal relations, that was simplified in previous editions; a 10-times larger dataset; and single overall performance scores, which allow the ranking of the participating systems in each task and also in general.

Introduction: -------------

Temporal annotation is a time-consuming task for humans, which has limited the size of annotated data in previous TempEvals. Current systems, however, are performing close to the inter-annotator reliability, which suggests that larger corpora could be built starting with automatically annotated data. One of the main goals of this TempEval edition is to explore whether there is value in adding a large automatically created silver standard to a hand-crafted gold standard. It might be that for some tasks an auto-annotated larger corpus might be more useful than a hand annotated small corpus.

TempEval-3, a temporal evaluation task, is a follow-up to TempEval-1 and 2. TempEval-3 differs from its ancestors in the following respects:

(i) size of the corpus: the dataset used comprises about 500K tokens of silver standard data and about 100K tokens of gold standard data for training, compared to the corpus of roughly 50K tokens corpus used in TempEval 1 and 2;

(ii) temporal relation task: the temporal relation classification tasks are to be performed from raw text, i.e. participants need to extract events and temporal expressions first, determine which ones to link and then obtain the relation types;

(iii) tasks not independent: participants must annotate temporal expressions and events in order to do the relation task;

(iv) temporal relation types: the full set of temporal interval relations in TimeML is used, rather than the reduced set used in earlier TempEvals;

(v) annotation: most of the corpus was automatically annotated by the stateof-the-art systems from TempEval-2, a portion of the corpus, including the test dataset, that is human reviewed;

(vi) evaluation: we will report a temporal awareness score for evaluating temporal relations, to help to rank systems with a single score.

TempEval 3 Tasks: ---------------- The tasks proposed for TempEval-3 are related to each one of the main TimeML tags. These are:

* Task A: Temporal expression extraction and normalization Determine the extent of the time expressions in a text as defined by the TimeML TIMEX3 tag. In addition, determine the value of the features TYPE and VAL. The possible values of TYPE are time, date, duration, and set; the value of VAL is a normalized value as defined by the TIMEX3 standard. The main attribute to annotate is VAL.

* Task B: Event extraction As in TempEval-2, participants will determine the extent of the events in a text as defined by the TimeML EVENT tag. In addition, systems may determine the value of the features CLASS, TENSE, ASPECT, POLARITY, MODALITY and also identify if the event is a main event or not. The main attribute to annotate is CLASS.

* Task C: Annotating temporal relations Identify the pairs of temporal entities (events or temporal expressions) that have a temporal link and classify the temporal relation between them as a TLINK. For this task, we now require that the participating systems determine which entities need to be linked. The relation labels will be same as in TimeML, i.e.: before, after, includes, is-included, during, simultaneous, immediately after, immediately before, identity, begins, ends, begun-by and ended-by.

Task selection Participants may choose to do task A, B, or C. Choosing task C (relation annotation) entails doing tasks A and B (interval annotation). However, a participant may perform only task C by applying existing tools to carry out tasks A and B.

Further, the linking-only task is now available; gold-standard timex and event annotations will be release and a separate one-week evaluation window is available for teams only wishing to do the temporal relation labelling task.

Dataset Creation ---------------- In TempEval-3, we release new data, as well as significantly reviewing and modifying existing corpora.

A large portion of the TempEval-3 data is automatically generated, using a temporal merging system. We include over half a million temporally-annotated tokens from English Gigaword, as well as 40,000 tokens of new gold-standard data.

Task Organizers: ----------------

James Allen, University of Rochester Leon Derczynski, University of Sheffield Hector Llorens, University of Alicante James Pustejovsky, Brandeis University Naushad UzZaman, University of Rochester [Primary Contact] Marc Verhagen, Brandeis University

Important Dates: ----------------

*February 22, 2013: Team registration deadline*

March 1, 2013 (mid-day EST): Start of evaluation period for Task A; for Task B; for Combined Task ABC, from raw text March 15, 2013 (23.59 UTC-10 Hawaii): End of evaluation period for Task A; for Task B; and for Combined Task ABC, from raw text March 16, 2013 (mid-day EST): Start of evaluation period for Linking-only Task C using reference timex and event annotations March 23, 2013 (23.59 UTC-10 Hawaii): End of evaluation period for Linking-only Task C April 14, 2013 (23.59 UTC-10 Hawaii): System description & analysis paper submission deadline April 24, 2013: Description paper notifications / reviews May 4, 2013: Camera-ready deadline June 13-14: *SEM conference co-hosted with NAACL June 14-15: SemEval workshop (tentative)

More infomation: ----------------

The TempEval-3 website, for signup and details, is:


For details, check the task description paper here: http://arxiv.org/pdf/1206.5333v1.pdf Naushad UzZaman, Hector Llorens, James F. Allen, Leon Derczynski, Marc Verhagen, James Pustejovsky. 2012. TempEval-3: Evaluating Events, Time Expressions, and Temporal Relations. arXiv:1206.5333v1.

-- Leon R A Derczynski NLP Research Group

Department of Computer Science University of Sheffield Regent Court, 211 Portobello Sheffield S1 4DP, UK

+45 5157 4948 http://www.dcs.shef.ac.uk/~leon/

-- Leon R A Derczynski Research Associate, NLP Group

Department of Computer Science University of Sheffield Regent Court, 211 Portobello Sheffield S1 4DP, UK

+45 5157 4948 http://www.dcs.shef.ac.uk/~leon/ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 9086 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20130220/c6501b63/attachment.txt> -------------- next part -------------- ************************************************ This SIGIR-IRList message and the SIG-IRList Digest (a moderated IR newsletter), are brought to you by SIGIR, distributed from the University of Sheffield and edited by Claudia Hauff (irlist-editor at acm.org). o To submit an article, e-mail IRList at lists.shef.ac.uk o To subscribe, send mail to sympa at lists.shef.ac.uk , with the subject: SUBSCRIBE irlist firstname lastname o To unsubscribe, send mail to sympa at lists.shef.ac.uk, with the subject: UNSUBSCRIBE irlist YourEmailAddressHere [The email address is required only if you want to unsubscribe with an address other than the address with which you send the message]


More information about the Corpora mailing list