FIRST CALL FOR PARTICIPATION
GENERATION OF REFERENCES IN CONTEXT (GREC) TASKS 2010 -----------------------------------------------------
Part of Generation Challenges 2010, in conjunction with INLG 2010.
Generation Challenges is a shared-task evaluation initiative for research that involves the generation of language, in the widest sense. Generation Challenges 2010 is the fourth event in this intiative, and aims to continue to provide a common forum for a number of different Shared Tasks (see http://www.nltg.brighton.ac.uk/research/genchal10/).
As part of Generation Challenges 2010, we are organising three GREC Shared Task Competitions. The first is the GREC-NEG (Named Entity Generation) Task which uses the GREC-People Corpus of 1,000 Wikipedia introduction sections about people in which single and plural references to all people mentioned in the text have been annotated; the task in GREC-NEG is to select appropriate referential expressions for all mentions (singular and plural) of people from a given list. The second task, GREC-NER, is a straightforward Named Entity Recognition task, using the new GREC-People-II extension to the GREC-People Corpus. The third task is GREC-FULL which requires participating systems to combine NER and NEG in order to improve references to people in texts, and this also uses the GREC-People-II Corpus.
Submissions to all three tasks will be evaluated using a range of intrinsic and extrinsic measures, some assessed automatically, some manually. Submitted systems and evaluation results will be presented in a special session at INLG'10 in Dublin in July 2010, and published in the INLG'10 proceedings.
1. Background --------------
There has been increasing interest recently among text summarisation researchers in postprocessing techniques to improve the referential clarity and coherence of extractive summaries, and among language generation researchers in generating referential expressions in context. The GREC tasks are aimed at researchers in both of these groups, and the objective is the development of methods for generating chains of referential expressions for discourse entities in the context of a written discourse, as is useful e.g. for postprocessing extractive summaries and repeatedly edited texts (such as Wikipedia articles).
2. Data --------
The GREC data resources consist of introduction sections collected from Wikipedia articles in which several broad syntactic categories of reference to named entities have been annotated. The annotations include features encoding syntactic and semantic information.
The GREC-People and GREC-People-II corpora consist of 1,000 texts each in one domain, Wikipedia articles about people. Here, all references, singular and plural, to all people mentioned in a text have been annotated. Each text in these corpora therefore includes explicit coreference annotation for at least one coreference chain.
For GREC-People and GREC-People-II we have test sets of 100 texts each, where referential expressions have been selected by participants in an elicitation experiment. In these test sets, there are three versions for each corpus text, in each of which the referential expressions have been manually selected by a single participant in the experiment.
Full details of the evaluation methods for GREC'10 will be given in the Participants' Pack that will be distributed to registered participants.
6. Participation ----------------
Registration is now open at the GREC'10 homepage (http://www.nltg.brighton.ac.uk/research/genchal10/grec). Once registered, participants in the GREC-NEG Task will receive the complete training and development set, evaluation software and detailed documentation (collectively known as the Participants' Pack) for this task. Participants in the other two tasks will first receive a sample of the training and development data, to enable them to start building systems; they will receive the complete Participants' Pack for GREC-NER/FULL by the end of 2009.
7. Proceedings and Presentations --------------------------------
The Generation Challenges 2010 meeting will be held as a special session at INLG 2010 in July 2010 in Dublin. The session will include overviews of all the shared tasks, including the GREC'10 Tasks. The participating systems will additionally be presented as papers in the INLG'10 proceedings, and as posters during the INLG'10 poster session.
GREC'10 papers will not undergo a selection procedure with multiple reviews, but the organisers reserve the right to reject material which is not appropriate given the participation guidelines.
8. Important Dates ------------------
Oct 2009 First Call for Participation in GREC'10 Tasks;
GREC-NEG Participants' Pack available;
GREC-NER/FULL Data Sample available Dec 31, 2009 GREC-NER/FULL Participants Pack available Apr 5-May 2, 2010 Test data submission period; 4-step submission process:
1. Fill in submission form (available on website from
Apr 5).
2. Upload 2-page paper describing approach and reporting
development set results.
3. Download test data (inputs only).
4. Submit test data outputs at the latest 48 hours after
download, but in any case no later than May 2. May 2, 2010 Final deadline for submission of test data outputs. May 2-23, 2010 GREC'10 Evaluation period Jul 7-9, 2010 Generation Challenges meeting at INLG'10
9. Organisation ---------------
Anja Belz, NLTG, University of Brighton, UK Eric Kow, NLTG, University of Brighton, UK Jette Viethen, Macquarie University, Australia
GREC'10 homepage: http://www.nltg.brighton.ac.uk/research/genchal10/grec Generation Challenges homepage: http://www.nltg.brighton.ac.uk/research/genchal10 Generation Challenges email: nlg-stec at itri.brighton.ac.uk