[developers] Inter-annotator agreement for Redwoods treebanks

[Emily M. Bender]

Thanks, Dan.  For measuring with discriminants I was imagining that we
could use both the manually selected and inferred constituents.  It
looks like Tanaka et al use a random choice among *trees* for the
baseline, but that significantly underestimates (I think) the
probability of agreeing by chance.

Emily

On Tue, Oct 15, 2013 at 10:38 AM, Dan Flickinger <danf at stanford.edu> wrote:
> I think this Tanaka et al (2005) paper describes the nearest to an accepted set of measures for inter-annotator agreement with our treebanks, computing a harmonic mean of precision across labeled constituents, rather than measuring agreement of discriminants.  Here is a link to that paper:
> http://acl.ldc.upenn.edu/P/P05/P05-1041.pdf
> MacKinlay et al (2010) adopts this approach as well, in this paper:
> http://aclweb.org/anthology//I/I11/I11-1028.pdf
>
> The problem with measuring agreement on discriminants is that there is a lot of redundancy in the list of discriminants presented to the annotator, and two annotators can arrive at exactly the same tree by making almost completely disjoint manual discriminant choices, for example with one annotator working "top-down" (initially making spanning-level choices of phrasal constructions) and the other working "bottom-up" (initially making lexical disambiguation choices).
>
>  Dan
>
> ----- Original Message -----
> From: "Emily M. Bender" <ebender at uw.edu>
> To: "developers" <developers at delph-in.net>
> Sent: Tuesday, October 15, 2013 9:56:02 AM
> Subject: [developers] Inter-annotator agreement for Redwoods treebanks
>
> Hi all,
>
> Is there an accepted, chance-corrected measure for inter-annotator
> agreement with Redwoods treebanks?  It seems to me that measuring
> chance agreement over discriminants would make more sense than
> measuring over trees, and I'm not quite sure how to conceptualize the
> chance agreement for the "reject all trees" option...
>
> Emily
>
> --
> Emily M. Bender
> Associate Professor
> Department of Linguistics
> Check out CLMS on facebook! http://www.facebook.com/uwclma



-- 
Emily M. Bender
Associate Professor
Department of Linguistics
Check out CLMS on facebook! http://www.facebook.com/uwclma