[Corpora-List] WSD / # WordNet senses / Mechanical Turk

Kilian Evang maschinenraum at texttheater.net
Tue Jul 16 00:15:03 CEST 2013


Off the top of my head, here's two relevant studies on inter-rater reliability for WSD, one for the case of expert annotators and one for the case of non-experts:

http://link.springer.com/article/10.1023/A:1002693207386#page-1

http://dl.acm.org/citation.cfm?id=1613751

Cheers, Kilian

On 15/07/13 23:59, Mark Davies wrote:
> Sorry if this is a basic question for computational linguists; I'm a
> corpus linguist.
>
> I'm wondering if there has been much research on inter-rater
> reliability of word sense disambiguation by raters on something like
> Mechanical Turk. For example:
>
> -- Given some verbs that have 5 word senses each in WordNet (e.g. the
> words tag, tame, taste, temper), how well do native speakers agree on
> the word sense for these verbs in context -- How does this
> inter-rater reliability change for words that might have just two
> senses (e.g. the verbs taint, tamper, tan, tank) or maybe 10 senses
> (e.g. the verbs shift, spread, stop, trim). (In other words,
> intuition suggests that for words with two WordNet senses, there
> might be higher inter-rater reliability than those words with five
> senses, and that for words with 10 WN senses, inter-rate reliability
> would be pretty bad.) -- Semantically, which kinds of 2 / 5 / 10 WN
> entry words have the best inter-rater reliability, and which have the
> worst?
>
> Thanks in advance.
>
> Mark Davies
>
> ============================================ Mark Davies Professor of
> Linguistics / Brigham Young University
> http://davies-linguistics.byu.edu/
>
> ** Corpus design and use // Linguistic databases ** ** Historical
> linguistics // Language variation ** ** English, Spanish, and
> Portuguese ** ============================================
>
> _______________________________________________ UNSUBSCRIBE from this
> page: http://mailman.uib.no/options/corpora Corpora mailing list
> Corpora at uib.no http://mailman.uib.no/listinfo/corpora
>



More information about the Corpora mailing list