===== Summary ===== Following the success of the 1st IWPT Enhanced Universal Dependency Parsing in 2020, a second edition of the IWPT Parsing Shared task is launched with a focus an emphasis on the parsing of Enhanced Universal Dependencies (often reflecting deeper syntactic structures, represented as more complex graphs, than regular surface dependencies).
https://iwpt21.sigparse.org/sharedtask Webpage: https://universaldependencies.org/iwpt21/
Release of training and dev. data : April 6 Test data release: May 1 Test submission deadline: May 10 System paper deadline: May 31 Camera Ready Deadline: June 30 Timezone: Anytime on Earth (CET-12)
Interested parties are encouraged to subscribe to the shared task mailing list at http://sympa.inria.fr/sympa/info/iwptsharedtask. contact: iwptsharedtask at gmail.com
===== Introduction =====
The IWPT 2021 Shared Task will be on Multilingual Parsing into Enhanced Universal Dependencies (EUD). In recent years, Universal Dependencies (UD)—the de-facto standard target representations in surface-syntactic dependency parsing—have grown a second layer of structure, called enhanced dependencies, where grammatical relations that cannot be adequately represented in pure rooted trees are encoded, for example control relations and argument sharing in relative clauses, shared dependencies involving coordinate structures, and dependencies involving ellipsis. Enhanced dependencies call for non-tree graphs with reentrancies, cycles, and empty nodes.
Data for the shared task consists of at least the treebanks in UD release 2.5. that contain enhanced annotation, and potentially one or more additional languages/treebanks. The task will be parsing from raw strings into EUD according to the guidelines at https://universaldependencies.org/u/overview/enhanced-syntax.html. On top of a classic F-measure metrics, evaluation will measure performance per phenomenon and will take into account the fact that not all treebanks cover all of the phenomena listed in the EUD guidelines.
===== Task Description and Evaluation =====
We invite participants to develop a system for parsing raw text into enhanced universal dependencies for all of the languages included in the training data. The task is similar to that of the CoNLL 2017 and 2018 shared tasks on parsing into UD, except that the prime evaluation metric now is the enhanced dependency annotation. Participants are encouraged to consider all enhancements listed in the UD guidelines, even if some of these enhancements might be absent in some of the treebanks included in the training data. Evaluation will take into account the fact that some treebanks are incomplete in this respect. Participants are also encouraged to predict all lower levels of annotation (lemma, tag, morphological features, basic dependency tree). These annotations will be evaluated as secondary metrics. Also, it is possible to train a pre-existing parser (such as UDPipe), use it to predict the lower levels of annotation and then develop one’s own system that focuses on the transition from the basic UD tree to the enhanced UD graph.
== Training Data ==
The evaluation will be done on 17 languages from 4 language families: Arabic, Bulgarian, Czech, Dutch, English, Estonian, Finnish, French, Italian, Latvian, Lithuanian, Polish, Russian, Slovak, Swedish, Tamil, Ukrainian. The language selection is driven simply by the fact that at least partial enhanced representation is available for the given language. Training and development data in the CoNLL-U format are available on the shared task website. These datasets are based on the UD release 2.7 but the annotation is often not identical to the corresponding treebank in UD 2.7. Nevertheless, the participants are also allowed to use the training and development data from the official UD 2.7 release package on Lindat, even in languages that are not part of the shared task evaluation. No other version of UD (either previous releases or Github repositories or other copies and clones online) can be used in the shared task; this is to avoid the danger of incompatible training-test splits.
== Evaluation Metric ==
The prime evaluation metric is LAS on enhanced dependencies (DEPS), where LAS is defined as F1-score over the set of enhanced dependencies in the system output and the gold standard. Complete edge labels are taken into account, i.e. conj:and differs from conj. While some effort has gone into ensuring that the data in the various treebanks is annotated consistently w.r.t. the level of enhancements and the format of enhanced labels, not all treebanks include all of the enhancements listed in the UD guidelines. For those treebanks, an additional evaluation will be carried out, where dependencies that are the consequence of including enhancement E, where E is not included in the training data of that treebank, are ignored during evaluation.
===== Shared Task Schedule =====
Release of training and dev. data April 6 Release of blind test data May 1 Submission of parsed test data May 10 Announcement of results May 12 Shared task papers due (provisional) May 31 Camera ready papers due June 30 Presentation of results at IWPT August 1
Shared Task papers will be published in the "Working notes of the IWPT 2021 Enhanced Dependency Parsing Shared Task".
===== Shared task Organizers =====
Gosse Bouma (University of Groningen, Netherlands) Djamé Seddah (Inria Paris, France) Daniel Zeman (Charles University, Czechia)
===== IWPT 2021 Organizers =====
Yuji Matsumoto Stephan Oepen Kenji Sagae Weiwei Sun Anders Søgaard Reut Tsarfaty
===== Contact details ===== - Mail: iwptsharedtask at gmail.com - Webpage: https://universaldependencies.org/iwpt21/ - Mailing list: https://sympa.inria.fr/sympa/info/iwptsharedtask - IWPT 2020 website: https://iwpt21.sigparse.org