[Corpora-List] ResPubliQA 2009:Call for Participation and Final Track Guidelines

Pamela Forner forner at celct.it
Mon Feb 2 18:02:04 CET 2009

Apologies for multiple postings

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ResPubliQA exercise at the MULTILINGUAL QUESTION ANSWERING TRACK AT CLEF 2009 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

NEW: The Final Guidelines of the ResPubliQA exercize are now available on the website!

------------------------------------------------------------------ Call for Participation ------------------------------------------------------------------

We are glad to announce that a NEW EXERCISE will be proposed this year within the Question Answering track at the Cross Language Evaluation Forum (CLEF). For more information and updates visit the new ResPubliQA website at:


We invite participation, both from academic institutions and industrial organizations, on this new task. Guidelines describing the task are distributed among the participants and are downloadable from the ResPubliQA website. http://celct.isti.cnr.it/ResPubliQA/Documents Training data have also been made available to participants in order to test the systems with procedures to be used in the formal evaluation campaign. http://celct.isti.cnr.it/ResPubliQA/Downloads The results of the evaluation will be disseminated at the final workshop which will be organized in Corfu in conjunction with ECDL 2009.


TASK DESCRIPTION: Systems receive natural language questions as input, and must return one paragraph containing the answer from the document collection. No exact answer is required neither multiple responses.

DOCUMENT COLLECTION: The subset of JRC-Acquis documents that have parallel aligned translations into all the languages involved will be used, namely Bulgarian, Dutch, English, French, German, Italian, Portuguese, Romanian and Spanish. The sub-collection is available at the ResPubliQA website http://celct.isti.cnr.it/ResPubliQA/Downloads

QUESTIONS: a pool of 500 independent questions (factoid, definition, reason, purpose and procedure) is provided

o NO LIST questions o NO topic related questions (questions linked to the same topic) o NO NIL questions

ANSWERS: one of the following two responses must be returned

a) one single paragraph containing the candidate answer. Multi-paragraph answers are not considered in this task

b) the string NOA to indicate that the system prefers not to answer the question.

Systems that give no answers (NOA) instead of wrong answers will be rewarded by the evaluation measure. Answer Validation techniques (including Machine Learning) are expected to be used for taking this final decision.

LANGUAGES INVOLVED: Basque (EU), Bulgarian (BG), Dutch (NL), English (EN), French (FR), German (DE), Italian (IT), Portuguese (PT), Romanian (RO) and Spanish (ES).

A monolingual English (EN) task will also be activated this year, as both the exercize and the collection are different from TREC. Basque has been included exclusively as a source language, as there is no Basque collection available.


Registration Open: February 4, 2009

Final Track Guidelines: February 2, 2009

Test Sets Release: May 25, 2009

Submissions of Runs by Participants: June 5, 2009

Release of Individual Results: from July 15, 2009

Submission of Papers for Working Notes: August 14, 2009

CLEF Workshop (in Corfu, Greece): 30 September - 02 October 2009

The participants will have 5 DAYS to upload their submissions, starting from the moment when the questions are downloaded, and not later than June 5, 2009.


- UNED (coordinator)

Spanish Distance Learning University, Spain

Anselmo Peñas

- CELCT (coordinator)

Center for the Evaluation of Language and Communication Technology, Italy

Pamela Forner and Danilo Giampiccolo


Evaluations and Language Resources Distribution Agency, France

Nicolas Moreau

- University of Limerick, Ireland

Richard Sutcliffe


Bulgarian Academy of Science, Bulgaria

Petya Osenova

- UAIC and RACAI, Romania

Alexandru Ioan Cuza University and Romanian Academy Research Institute for Artificial Intelligence, Romania

Corina Forascu


University of Basque Country, Spain

Iñaki Alegria


- Donna Harman (National Institute for Standards and Technology (NIST), USA)

- Maarten de Rijke (University of Amsterdam, The Netherlands)

- Dominique Laurent (Synapse Développement, France.)

================================ Pamela Forner CELCT (web: www.celct.it<http://www.celct.it/>) Center for the Evaluation of Language and Communication Technologies Via alla Cascata 56/c 38100 Povo - TRENTO -Italy

email: forner at celct.it<mailto:forner at celct.it> tel.: +39 0461 314 804 fax: +39 0461 314 846

Secretary Phone: +39 0461 314 870

-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 34760 bytes Desc: not available Url : https://mailman.uib.no/public/corpora/attachments/20090202/34447385/attachment.txt

More information about the Corpora mailing list