[developers] Comparing a profile with a grammar output

Stephan Oepen oe at ifi.uio.no
Fri Aug 21 17:35:26 CEST 2020


hi alexandre:

> After having a profile disambiguated with FFTB, my first question is how to compare it with the grammar output for the same set of sentences? This comparison should give me an evaluation of the current ranking model with my data. It should tell me if it is worth to train a new model. In particular, if I compare the semantic structure, it would also allow me to ignore variations on syntactic analysis that doesn't impact semantic representation.

one used to do this kind of evaluation in [incr tsdb()], initiated
through the 'Trees | Score' command.  i am not sure this will work out
of the box for comparing an FFTB-based treebank to an ACE-generated
parsing profile ... although it should, in principle!  i expect you
would want to select 'Result Equivalence' and 'Score All Items' in
'Trees | Switches', and just give it a shot :-).

if you get the above to work (which should give exact match
accuracies), the [incr tsdb()] scorer can also compute a range of
additional metrics, including EDM and ParsEval, but these would have
to be activated programmatically:

(setf *redwoods-score-counts* '(:ta :parseval :edma :edmp))

it should also be possible to batch-score a selection of profiles,
with a little bit of coding in the high-level [incr tsdb()] scripting
language; for your inspiration, i have put on-line an archive of
working files from the (as of yet unpublished) manuscript on robust
parsing and unification with, among others, yi zhang:

http://nlpl.eu/oe/edm.tgz

> I also tried to use the mtools from Stephan (https://github.com/cfmrp/mtool) but I am probably not using it right, since even with two different sentences I am getting the same output below:
>
> % echo "It is rainning today." | ace -g ../wn/terg-mac.dat -T -n 1 | delphin convert --from ace --to eds > 1.eds
> % echo "It is rainning today." | ace -g ../wn/terg-mac.dat -T -n 1 | delphin convert --from ace --to eds > 2.eds
> % ./main.py --read eds --score mrp --framework eds --gold 1.eds 2.eds
> {"n": 0,

you end up scoring zero items, which either suggests your EDS input
files are not considered valid by mtool, or the '--framework eds'
selection fails.  the latter should not be necessary (it may only work
with MRP input files; its purpose is to select a sub-set of graphs,
explicitly marked for a specific framework, from a multi-framework
input file).  equally likely, your EDS input files may be missing the
identifier prefix; please see 'data/score/eds/' in mtool for the
expected syntax.

cheers, oe



More information about the developers mailing list