[Corpora-List] Word Sense Disambiguation: a Unified Evaluation Framework and Empirical Comparison

Alessandro Raganato raganato at di.uniroma1.it
Mon Jan 16 15:12:29 CET 2017

The unified evaluation framework for Word Sense Disambiguation (WSD) is available at http://lcl.uniroma1.it/wsdeval .

We have gathered together five popular all-words WSD evaluation datasets and two training datasets, standardizing their format and sense inventory, providing a unified evaluation framework. WSD is a long-standing task in Natural Language Processing, lying at the core of human language understanding. However, the field seems to be slowing down due to the lack of groundbreaking improvements. We argue that this is partly due to the lack of a standard benchmark, which prevents new approaches to be easily compared with old approaches. Current benchmarks tend to differ in format, construction guidelines and underlying sense inventory.

In our work we used this framework to perform an empirical comparison among a set of heterogeneous approaches, including latest advances based on neural networks. All supervised approaches were trained on the same preprocessed corpora, ensuring a fair comparison among all systems. Additionally, we have enabled a competition in CodaLab <https://competitions.codalab.org/competitions/15984> for testing new models (or models not considered in our empirical comparison).

If you would like to contribute to the framework with sense-annotated training data or other evaluation datasets, you can share it with us (instructions in the website <http://lcl.uniroma1.it/wsdeval/share-your-data>).

Let’s make WSD great again! :)

For more information, please read the reference paper:

Alessandro Raganato, Jose Camacho-Collados and Roberto Navigli.

Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison <http://lcl.uniroma1.it/wsdeval/data/EACL17_WSD_EvaluationFramework.pdf>. Proceedings of EACL 2017, Valencia, Spain

-- ===================================== Alessandro Raganato Dipartimento di Informatica Sapienza University of Rome Viale Regina Elena 295 00161 Roma Italy Home Page: http://wwwusers.di.uniroma1.it/~raganato ===================================== -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 6357 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20170116/f4a05478/attachment.txt>

More information about the Corpora mailing list