[Corpora-List] Syntactic parsing performance by humans?

Koos Wilt kooswilt at gmail.com
Fri May 13 21:31:36 CEST 2016


It is not so much an overview of parser performance as it is a bibliography. Many of the entries, however, do contain parser evaluations. Hope it's still useful.

-K

2016-05-13 20:26 GMT+02:00 Koos Wilt <kooswilt at gmail.com>:


> Also please allow me to give a plug for the Stanford Parser. I cannot
> claim if performs worse or better than Google's, but it's become my trusty
> of war-horse.
>
> 2016-05-13 20:24 GMT+02:00 Koos Wilt <kooswilt at gmail.com>:
>
>> Bob Berwick
>> 20:13 (10 minuten geleden)
>> aan mij
>> would be useful for the list and the community. please do.
>> Koos Wilt <kooswilt at gmail.com>
>> 20:14 (9 minuten geleden)
>> aan Bob
>> Coming up tomorrow or so.
>>
>> 2016-05-13 19:51 GMT+02:00 Koos Wilt <kooswilt at gmail.com>:
>>
>>> I wrote an overview of the performance of parsers about 4 years ago.
>>> Would sending it somewhere (e.g. to Mr Brew) be helpful to anyone? It's on
>>> my other laptop so I have to dig for it.
>>>
>>> Best,
>>>
>>>
>>> -K
>>>
>>> 2016-05-13 19:30 GMT+02:00 chris brew <cbrew at acm.org>:
>>>
>>>> It is an unarguable fact that Google's parser gets a higher score, on
>>>> the metrics chosen, which are completely standard in the NLP community.
>>>> What is really being measured is what percentage of the links in a graph
>>>> that links words to words via labeled links are correct. If, as is common,
>>>> there are many words in the sentence, there will be many links too, and
>>>> many opportunities for mistakes. You could get a 90% score and still have a
>>>> mistake or two in nearly every sentence.
>>>>
>>>> Whether this quality level is OK depends entirely on what use you plan
>>>> to make of the graph that has been produced.
>>>>
>>>> The Penn Treebank was made many years ago, with version 2 coming out in
>>>> 1995. We have learnt a lot about how to annotate corpora and evaluate
>>>> parsing since then. The Web Treebank is much newer, and reflects painfully
>>>> learned best practices, so should be good quality, but is on the other hand
>>>> dealing with much messier language, so performance scores are lower.
>>>>
>>>>
>>>> The current practice of evaluating individual dependencies was
>>>> introduced as a result of major deficiencies in the first evaluation
>>>> metrics that were used. It has the major plus of being transparent and
>>>> straightforward. I believe that improvements in the metric will usually
>>>> translate into improvements for downstream tasks that use parsing as
>>>> inputs, and I wasn't so sure using earlier metrics. This is progress, but
>>>> quite modest progress.
>>>>
>>>>
>>>> On 13 May 2016 at 12:55, Darren Cook <darren at dcook.org> wrote:
>>>>
>>>>> Google have trained a neural net (part of publicizing their
>>>>> open-source
>>>>> TensorFlow framework?) to parse syntax, claiming it is the world's
>>>>> best:
>>>>>
>>>>>
>>>>> http://googleresearch.blogspot.co.uk/2016/05/announcing-syntaxnet-worlds-most.html
>>>>>
>>>>> I just wanted to quote this bit, on performance: (they've called in
>>>>> Parsey McParseface)
>>>>>
>>>>> "Parsey McParseface recovers individual dependencies between words
>>>>> with over 94% accuracy, ... While there are no explicit studies in the
>>>>> literature about human performance, we know from our in-house
>>>>> annotation
>>>>> projects that linguists trained for this task agree in 96-97% of the
>>>>> cases ... Sentences drawn from the web are a lot harder to analyze,
>>>>> ...[it] achieves just over 90% of parse accuracy on this dataset. "
>>>>>
>>>>> Are there really no studies of human performance?! Surely some
>>>>> professor
>>>>> has hinted to their PhD students that it is a nice bit of relatively
>>>>> easy linguistics research, that should also get them cited a lot...
>>>>>
>>>>> (I was mainly curious what the human performance gap between Penn
>>>>> Treebank and Google WebTreebank would be; if it would be more or less
>>>>> than the 4% gap for the deep learning algorithm.)
>>>>>
>>>>> Darren
>>>>>
>>>>> _______________________________________________
>>>>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>>>>> Corpora mailing list
>>>>> Corpora at uib.no
>>>>> http://mailman.uib.no/listinfo/corpora
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
>>>> Corpora mailing list
>>>> Corpora at uib.no
>>>> http://mailman.uib.no/listinfo/corpora
>>>>
>>>>
>>>
>>
>
-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 9293 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20160513/1c25ced1/attachment.txt> -------------- next part -------------- A non-text attachment was scrubbed... Name: Bibliography (1).pdf Type: application/pdf Size: 70303 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20160513/1c25ced1/attachment-0001.pdf>



More information about the Corpora mailing list