[developers] developers Digest, Vol 101, Issue 2

[António Branco]

Dear Emily,

In his MA thesis (2011), Sérgio Castro developed reliability
metrics (ITA) for deep treebanking on the basis of discriminants,
and associated tools, that sought to discount chance agreement,
to handle the "reject all trees" option, and to circumvent
the "redundancy" issues mentioned by Dan.

There was a presentation of his results in that year,
in the Suquamish Summit: http://www.delph-in.net/2011/sergio.pdf

You can find his dissertation here:
http://www.di.fc.ul.pt/~ahb/SergioCastro2011_MA_Dissertation.pdf

We are in the process of further pursuing these results
in writing a paper to be submited. Comments will be very
much appreciated.

Best,

António





> Date: Tue, 15 Oct 2013 09:56:02 -0700
> From: "Emily M. Bender" <ebender at uw.edu>
> To: developers <developers at delph-in.net>
> Subject: [developers] Inter-annotator agreement for Redwoods treebanks
> Message-ID:
> 	<CAMype6f2wuLhe5VbNvAAcqeethfofckNtE77HpCqLPrdy1LuLA at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi all,
>
> Is there an accepted, chance-corrected measure for inter-annotator
> agreement with Redwoods treebanks?  It seems to me that measuring
> chance agreement over discriminants would make more sense than
> measuring over trees, and I'm not quite sure how to conceptualize the
> chance agreement for the "reject all trees" option...
>
> Emily
>