[Corpora-List] Final call for paper offerings for LREC shared task Reproducibility

Gertjan van Noord g.j.m.van.noord at rug.nl
Wed Jan 9 13:25:58 CET 2019

FINAL CALL FOR PAPER OFFERINGS for LREC Shared Task on Reproducibility

Did you author a recent paper in the area of Natural Language Processing and Computational Linguistics, and are you interested in further visibility of your paper, and wondering how well other researchers are able to reproduce your results?

Please consider offering your paper for the upcoming Shared Task on Reproducibility!


Scientific knowledge is grounded on falsifiable predictions and thus its credibility and raison d’être relies on the possibility of repeating experiments and getting similar results as originally obtained and reported. In many young scientific areas, including ours, acknowledgment and promotion of reproduction of results need very much to be increased.


For this reason, a special reproducibility session will be included into the LREC 2020 regular program (side by side with other sessions on other topics) for papers on reproducibility, and a specific community-wide exercise will be launched to elicit and motivate the spread of work on reproduction: a Shared Task on Reproducibility.


We are adhering to the terminology assumed in the introductory note of the special section on replicability and reproducibility of The Language Resources and Evaluation journal (March 2017, Volume 51, Issue 1, pp 1-5), which follows Stodden (2014): "Replication, the practice of independently implementing scientific experiments to validate specific findings, is the cornerstone of discovering scientific truth. Related to replication is reproducibility, which is the calculation of quantitative scientific results by independent scientist using the original datasets and methods." For the ease of reference and when there is no need to be more explicit, we will be using the terms "reproduction" and "reproducibility".


The shared task is of a new type: it is partly similar to the usual competitive shared tasks --- in the sense that all participants share a common goal; but it is partly different to previous shared tasks --- in the sense that the focus is on seeking support and confirmation of previous results, rather than on overcoming those previous results with superior ones. Thus instead of a competitive shared task, with participant struggling for a top system that scores as far as possible from a rough and ready baseline, this will be a "cooperative shared task", struggling for a system that reproduces as close as possible the results of an original research experiment.

Accepted papers reporting the findings of shared task participants will be published in the regular LREC Proceedings.


For this cooperative shared task, the organizing committee will select a small set of research papers. Since this is the first time such shared task is organized, it seems advisable to enlist only original papers whose authors explicitly accept that their paper is the subject of this exercise.

We therefore are asking authors to offer their paper as a candidate target paper for the Shared Task on Reproducibility. This is a great possibility for your paper to receive additional attention and recognition by the research community. We are seeking offerings by authors of papers that meet the following requirements:

* paper published in an established, peer reviewed, conference proceedings or journal in Natural Language Processing or Computational Linguistics (CL, ACL, TACL, LREC, LRE journal, SEMEVAL, EMNLP, NAACL, EACL, IJCNLP, Coling, ...) in 2015 or later.

* experiments reported in the paper only need access to resources (data sets, corpora, software tools) that are easily available for other researchers (e.g., there are no IPR restrictions, expensive software is not required, no excessive computational resources are required).

* authors of the paper are happy to be the target of the shared task, and will be prepared to answer simple questions from shared task participants (a forum will be set up in such a way that all participants share the same information).


The selection committee will select a small number of target papers, based on the criteria listed above, and taking into account aspects of balance so that there will be enough variety with respect to research field, computational and algorithmic techniques, target language(s), etc.


If you want your paper to be considered for the Shared Task on Reproducibility, please submit a short note (200-400 words) stating

* your motivation for offering your paper

* an indication what resources are required for reproduction, and under what conditions these resources are available

Please email your note as a PDF attachment to the following address:

ReproSharedTaskSelection at gmail.com

Deadline for paper offerings:

January 15, 2019


António Branco (chair) Nicoletta Calzolari (co-chair) Piek Vossen (Program Committee chair) Gertjan van Noord (Selection Committee Chair) Kevin Cohen


Tim Baldwin António Branco Nicoletta Calzolari Çağrı Çöltekin Nancy Ide Malvina Nissim Gertjan van Noord (chair) Stephan Oepen Barbara Plank Piek Vossen Dan Zeman

The latest version of this call is available at http://www.let.rug.nl/vannoord/Repro/

More information about the Corpora mailing list