[Corpora-List] Call for Participation: The Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge

Myroslava Dzikovska mdzikovs
Tue Jan 22 17:44:18 CET 2013

Please distribute widely - Apologies for cross-posting





- TASK REGISTRATION DEADLINE: February 15, 2013, at http://bit.ly/XUcQ0v - CHALLENGE DURATION: from February 25, 2013 to MARCH 15, 2013 - JOINT CHALLENGE EMAIL DISCUSSION GROUP: http://bit.ly/XwcwCR - DETAILED TASK DESCRIPTION: http://bit.ly/10nQTGJ - UPDATED EVALUATION CODE AND BASELINES: http://bit.ly/140DHMH - SemEval WORKSHOP: June 13-14, 2013, co-located with NAACL

(http://bit.ly/140uIuN) & *Sem (http://bit.ly/WknhYY),

Atlanta, Georgia, USA


We are pleased to invite participants to the Joint Student Response Analysis and 8th Recognizing Textual Entailment Challenge at SemEval 2013, a common effort of both educational technology and textual inference communities in order to present a unified scientific challenge addressing researchers in both fields. It will be run together with the SEMEVAL-2013 Semantic Evaluation Exercise, co-located with the *SEM and NAACL-2013 conferences.


The goal of the task is to produce an assessment of student answers to explanation and definition questions typically asked in practice exercises, tests or tutorial dialogue. It is linked to the tasks of semantic analysis and recognizing textual entailment in the textual inference community, and essay grading and short answer assessment in the educational NLP community.

Specifically, given a question, a known correct "reference answer" and a 1- or 2-sentence "student answer", the Main task consists of assessing the correctness of a student's answer at different levels of granularity - either 5-way (correct, partially correct, contradictory, irrelevant, not in the domain), 3-way task (correct, contradictory, incorrect), or 2-way task (correct, incorrect). Participants can opt to carry out the task at any level of granularity.

This task is similar to essay grading and short answer assessment. However, meaningful labels are used instead of numerical scores as a way to extract information necessary to provide effective tutorial feedback with respect to the specific problems that can be found in student answers. Educational NLP and essay grading system developers are encouraged to adapt their scoring techniques to the categorical labeling problem.

The Student Response Analysis task is also closely related to the notion of textual entailment. In most correct answers at least a substantial portion of the reference answer will be entailed by the student answer text, even if not all of it. Thus textual entailment engine developers are encouraged to exploit textual entailment techniques to test the potential contributions of their systems to the student response assessment problem.


An additional pilot task on partial entailment is offered as part of the challenge, where systems may recognize that specific parts of the hypothesis are entailed by the text, even though entailment might be not recognized for the hypothesis as a whole. Such recognition of partial entailment may have various uses in the educational setting based on identifying the missing parts in the student answer, and may similarly have value in other applications such as summarization or question answering.


- February 15, 2013 REGISTRATION DEADLINE *** CHALLENGE - MAIN TASK test set release: Feb. 25 - MAIN TASK submissions: March 4 - PILOT TASK test set release: March 5 - PILOT TASK submissions: March 12 - Results to participants: March 15 *** - April 9, 2013 Paper submission deadline [TBC] - April 23, 2013 Reviews Due [TBC] - May 4, 2013 Camera ready Due [TBC] *** SemEval WORKSHOP - June 13-14, 2013, co-located with NAACL & *Sem, Atlanta, Georgia, USA


More details on the Main and Pilot tasks are available at the SEMEVAL 2013 website: http://bit.ly/10nQTGJ.

Future updates will be distributed through the Joint Challenge Google Discussion Group, please join to ensure that you will receive announcements and status updates, at http://bit.ly/XwcwCR

To help us with forward planning, please register now at SemEval 2013 website if you are intending to enter the challenge (http://bit.ly/XUcQ0v). All teams are required to register by 15 February 2013.


Myroslava Dzikovska (m.dzikovska at ed.ac.uk), University of Edinburgh, UK [Primary contact] Rodney Nielsen (Rodney.Nielsen at UNT.edu), University of North Texas, USA Chris Brew (christopher.brew at gmail.com), Educational Testing Service (ETS), USA Claudia Leacock (claudia_leacock at mcgraw-hill.com), CTB McGraw-Hill; USA

Luisa Bentivogli (bentivo at fbk.eu), CELCT and FBK, Italy Peter Clark, (peterc at vulcan.com) Vulcan Inc., USA Ido Dagan (dagan at cs.biu.ac.il) Bar-Ilan University, Israel Hoa Trang Dang, (hoa.dang at nist.gov) NIST, USA Danilo Giampiccolo, (giampiccolo at celct.it), CELCT, Italy

-- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

More information about the Corpora mailing list