[Corpora-List] A question about co-reference scorer program

Joel Nothman joel at it.usyd.edu.au
Mon Feb 2 12:20:12 CET 2015


Perhaps this requires clarification. The MUC coreference score that Sapena implements is that of Vilain et al. (1995) as stated in the code. That paper, "A model-theoretic coreference scoring scheme", states that it is the scheme used from MUC6 on, and that "Coreference task definition ., version 2.0 and earlier." had used a different metric, the pairwise F1 you are expecting.

I think I've seen the meaning of that term confused elsewhere in the literature, so it is best to specify MUC scoring with reference to Vilain et al (1995), and otherwise refer to Pairwise F1 (from among a family of pairwise clustering evaluation metrics, upon which BLANC extends).

On 2 February 2015 at 22:08, shohreh tabatabaee <shohreh_taba at yahoo.com> wrote:


>
>
> Here is an example:
>
> Gold keys:
> 1->2->3->4->5
> 6->7
> 8->9->A->B->c
>
> System key:
> 1->2->3->4->
> 6->7->8->9->A->B-C
>
> MUC precision in Scorer is 8.75/9.75
> but it should be 9/11 Since 9 out of 11 links are correct
>
> ------------------------------
> *From:* Joel Nothman <joel at it.usyd.edu.au>
> *To:* shohreh tabatabaee <shohreh_taba at yahoo.com>
> *Sent:* Monday, February 2, 2015 2:57 AM
> *Subject:* Re: [Corpora-List] A question about co-reference scorer program
>
> Could you provide an example?
>
> I've reimplemented the MUC scorer with testcases from the paper, and
> checked on real datasets that it equates to the implementation at
> https://code.google.com/p/reference-coreference-scorers/, which is Emili
> Sapena's. Without an example, I don't think it's possible to understand
> what you mean.
>
> Cheers,
>
> Joel
>
> On 1 February 2015 at 16:26, shohreh tabatabaee <shohreh_taba at yahoo.com>
> wrote:
>
>
>
> Hello,
>
> I am trying to use the scorer program for evaluating my co-reference
> implementation.
>
> It is written in Perl Language by Emili Sapena
>
> But it seems that scorer use different formulation for MUC.
> It has some toy-examples but the score which it produces for MUC does not
> match with the ones in the literature for the very same example.
>
> Anybody know why?
>
>
>
> _______________________________________________
> UNSUBSCRIBE from this page: http://mailman.uib.no/options/corpora
> Corpora mailing list
> Corpora at uib.no
> http://mailman.uib.no/listinfo/corpora
>
>
>
>
>
-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/html Size: 5534 bytes Desc: not available URL: <https://mailman.uib.no/public/corpora/attachments/20150202/3a7505ec/attachment.txt>



More information about the Corpora mailing list