[Corpora-List] semi-automatic annotation tool evaluation

Eric Atwell E.S.Atwell at leeds.ac.uk
Tue May 1 14:00:39 CEST 2012


To evaluate a system for annotating dialogue acts, you could take a Discourse Corpus where the text has been manually annotated with discourse connectives and realtions, apply your system to the same text, and compare your system's analysis with the manual analysis. For example, if you work with Arabic, you could try the Arabic Disoourse Corpus http://www.arabicdiscourse.net/ built by Leeds PhD student Amal Alsaif.

One problem you may find is that analyses are only directly comparable if you use the same tag-set of discourse connectives and relations

- eg for the Arabic Discourse Corpus, see

http://www.arabicdiscourse.net/connectives/ and


I assume you want to evaluate accuracy; you might also want to evaluate ease-of-use, portability, speed etc of your annotation tool, and for this you could compare it to other annotation tools eg Amal Alsaif's READ tool: http://www.arabicdiscourse.net/annotation-tool/

I hope this is useful

Eric Atwell, Leeds University http://www.comp.leeds.ac.uk/nlp/

On Tue, 1 May 2012, samira ben dbabis wrote:

> Hi,
> I am developing a semi-automatic annotation tool specific for Dialogue Acts
> and I want to evaluate my system.
> So, does anyone knows how to evaluate the performance of semi-automatic
> annotation tools (like GATE, MATE, XDML and DAT tools).
> Best ,
> Samira
> **************************************************************************
> Samira Ben Dbabis
> PhD Computer Science Student
> Faculty of Economic Sciences and management of Sfax
> MIRACL Laboratory
> www.miracl.rnu.tn
> ANLP Research Group
> http://sites.google.com/site/anlprg
> ***************************************************************************

-- Eric Atwell, Associate Professor, Language research group,

I-AIBS Institute for Artificial Intelligence and Biological Systems

School of Computing, Faculty of Engineering, UNIVERSITY OF LEEDS

Leeds LS2 9JT, England. TEL: 0113-3435430 FAX: 0113-3435468

WWW: http://www.comp.leeds.ac.uk/eric



More information about the Corpora mailing list