[Corpora-List] HumEval Workshop 2022: Second Call for Papers

Maja Popovic maja.popovic.166 at gmail.com
Thu Jan 20 17:53:57 CET 2022

Website: https://humeval.github.io/

Previous edition (at EACL 2021): https://humeval.github.io/2021/

Workshop overview

The Second Workshop on Human Evaluation of NLP Systems (HumEval’22) invites the submission of long and short papers on substantial, original, and unpublished research on all aspects of human evaluation of NLP systems with a focus on NLP systems which produce language as output. We welcome work on any quality criteria relevant to NLP, on both intrinsic evaluation (which assesses systems and outputs directly) and extrinsic evaluation (which assesses systems and outputs indirectly in terms of its impact on an external task or system), on quantitative as well as qualitative methods, score-based (discrete or continuous scores) as well as annotation-based (marking, highlighting).

Invited speakers

Samira Shaikh, University of North Carolina and Ally Financial Inc.

Markus Freitag, Google Research

Important dates


28 February 2022: Submission Deadline


22 March 2022: Commitment deadline for ARR submissions with reviews


26 March 2022: Notification of Acceptance


10 April 2022: Camera-ready papers due


27 May 2022: Workshop at ACL

All deadlines are 23:59h UTC-12


We invite papers on topics including, but not limited to, the following:


Experimental design and methods for human evaluations


Reproducibility of human evaluations


Approaches to validating human evaluation methods


Work on inter-evaluator and intra-evaluator agreement


Ethical considerations in human evaluation of computational systems


Quality assurance for human evaluation


Crowdsourcing for human evaluation


Issues in meta-evaluation of automatic metrics by correlation with human



Alternative forms of meta-evaluation and validation of human evaluations


Comparability of different human evaluations


Methods for assessing the quality and the reliability of human



Role of human evaluation in the context of Responsible and Accountable AI

We welcome work from any subfield of NLP (and ML/AI more generally), with a particular focus on evaluation of systems that produce language as output.


Long papers

Long papers must describe substantial, original, completed and unpublished work. Wherever appropriate, concrete evaluation and analysis should be included. Long papers may consist of up to eight (8) pages of content, plus unlimited pages of references. Final versions of long papers will be given one additional page of content (up to 9 pages) so that reviewers’ comments can be taken into account. Long papers will be presented orally or as posters as determined by the programme committee. Decisions as to which papers will be presented orally and which as posters will be based on the nature rather than the quality of the work. There will be no distinction in the proceedings between long papers presented orally and as posters.

Short papers

Short paper submissions must describe original and unpublished work. Short papers should have a point that can be made in a few pages. Examples of short papers are a focused contribution, a negative result, an opinion piece, an interesting application nugget, a small set of interesting results. Short papers may consist of up to four (4) pages of content, plus unlimited pages of references. Final versions of short papers will be given one additional page of content (up to 5 pages) so that reviewers’ comments can be taken into account. Short papers will be presented orally or as posters as determined by the programme committee. While short papers will be distinguished from long papers in the proceedings, there will be no distinction in the proceedings between short papers presented orally and as posters.

Multiple submission policy

HumEval22 allows multiple submissions. However, if a submission has already been, or is planned to be, submitted to another event, this must be clearly stated in the submission form.

Submission procedure and templates

Submission is electronic, through the OpenReview portal for the workshop: https://openreview.net/group?id=aclweb.org/ACL/2022/Workshop/HumEval

with the deadline on 28 February 2022.

Both long and short papers must be anonymised for double-blind reviewing, must follow the ACL Author Guidelines <https://www.aclweb.org/adminwiki/index.php?title=ACL_Author_Guidelines>, and must use the ACL 2022 templates available here https://aclrollingreview.org/cfp.

The submitting author must have an OpenReview profile. Please ensure profiles are complete before submission. This tutorial <https://docs.google.com/presentation/d/1kJeoAfwbnFapUN0ySLSoOm11-2odz48DGS1DEzNs03k/edit#slide=id.gcfa2063058_0_0> from the ACL Rolling Review might be helpful.

In addition to standard submissions, submissions which were previously submitted to ARR and reviewed there can be committed to the workshop (together with the reviews) not later than 22 March 2022. We suppose that all papers submitted to ARR until 15 February 2022 will be reviewed by 22 March, however we cannot guarantee because it depends on ARR editors and not on us.

Optional Supplementary Materials: Appendices, Software and Data

ARR encourages the submission of these supplementary materials to improve the reproducibility of results, and to enable authors to provide additional information that does not fit in the paper. Supplementary materials may include appendices, software or data. For example, pre processing decisions, model parameters, feature templates, lengthy proofs or derivations, pseudocode, sample system inputs/outputs, and other details that are necessary for the exact replication of the work described in the paper can be put into appendices. However, if the pseudo-code or derivations or model specifications are an important part of the contribution, or if they are important for the reviewers to assess the technical correctness of the work, they should be a part of the main paper, and not appear in appendices. Reviewers are not required to consider material in appendices.

Appendices should come after the references in the submitted pdf, but do not count towards the page limit. Software should be submitted as a single .tgz or .zip archive, and data as a separate single .tgz or .zip archive. Supplementary materials must be fully anonymized to preserve the two-way anonymized reviewing policy.


Anya Belz, ADAPT Centre, Dublin City University, Ireland

Maja Popović, ADAPT Centre, Dublin City University, Ireland

Ehud Reiter, University of Aberdeen, UK

Anastasia Shimorina, Orange, Lannion, France

For questions and comments regarding the workshop please contact the organisers at humeval.ws at gmail.com.

Programme committee

Eleftherios Avramidis, DFKI, Germany

Sheila Castilho, ADAPT, Dublin City University, Ireland

Sandipan Dandapat, Microsoft, India

Ondrej Dušek, Charles University, Czechia

Markus Freitag, Google, USA

Albert Gatt, Malta University, Malta

Behnam Hedayatnia, Amazon, USA

David Howcroft, Heriot Watt University, UK

Tom Kocmi, Microsoft, Germany

Filip Klubička, ADAPT, Technological University of Dublin, Ireland

Samuel Läubli, University of Zurich, Switzerland

Chris van der Lee, Tilburg University, Netherlands

Saad Mahamood, Trivago, Germany

Nitika Mathur, University of Melbourne, Australia

Margot Mieskes, UAS Darmstadt, Germany

Emiel van Miltenburg, Tilburg University, Netherlands

Mathias Mueller, University of Zurich, Switzerland

Sergiu Nisioi, University of Bucharest, Romania

Juri Opitz, University of Heidelberg, Germany

Maike Paetzel-Prüsmann, University Potsdam, Germany

Maxime Peyrard, EPFL, Switzerland

Tim Polzehl, TU Berlin, Germany

Martin Popel, UFAL, Czechia

Verena Rieser, Heriot Watt University, UK

Samira Shaikh, UNC, USA

Wei Zhao, TU Darmstadt, Germany -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 39298 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20220120/78ed8e7c/attachment.txt>

More information about the Corpora mailing list