[developers] Comparing a profile with a grammar output

[Alexandre Rademaker]

Hi Stephan,

I tried the mtool again. Same problem.

% echo "It is rainning today." | ace -g ../wn/terg-mac.dat -T -n 1 | delphin convert --from ace --to eds > 2.eds
% echo "It is rainning today." | ace -g ../wn/terg-mac.dat -T -n 1 | delphin convert --from ace --to eds > 2.eds

I added the #XXXX right before the EDS serialization. The only different between these files in the https://github.com/cfmrp/mtool/blob/master/data/score/eds/wsj.pet.eds is that these files are not formatted with one predicate per line, instead, the EDS is serialised in a single line without line breaks.

% ./main.py --read eds --score smatch --gold ../sick/1.eds ../sick/2.eds
{"n": 0,
 "g": 0,
 "s": 0,
 "c": 0,
 "p": 0.0,
 "r": 0.0,
 "f": 0.0,
 "time": 1.4781951904296875e-05,
 "cpu": 4.4000000000044004e-05}

% ./main.py --read eds --score mrp --gold ../sick/1.eds ../sick/2.eds
{"n": 0,
 "null": 0,
 "exact": 0,
 "tops": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "labels": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "properties": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "anchors": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "edges": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "attributes": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "all": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "time": 3.0994415283203125e-05,
 "cpu": 9.099999999995223e-05}


But… I found the repo that Michael presented in the Summit https://github.com/delph-in/delphin.edm:

% delphin edm 1.eds 2.eds
Precision:	1.0
   Recall:	1.0
  F-score:	1.0

It works! But I need to remove the prefix (#NNNNN) before the EDS serializations. Even better, it works directly with profiles although the verbose option didn’t show anything interesting (I would like to see results per item):

% delphin edm -v golden parsed
Precision:	0.9637710992177851
   Recall:	0.9683557394002068
  F-score:	0.9660579799855565

Thank you Michael! I am not very confident on these numbers, I was expecting more differences, but… Anyway, it would be nice to double-check with mtool if I can.

Best,
Alexandre

> On 21 Aug 2020, at 12:35, Stephan Oepen <oe at ifi.uio.no> wrote:
> 
>> I also tried to use the mtools from Stephan (https://github.com/cfmrp/mtool) but I am probably not using it right, since even with two different sentences I am getting the same output below:
>> 
>> % echo "It is rainning today." | ace -g ../wn/terg-mac.dat -T -n 1 | delphin convert --from ace --to eds > 1.eds
>> % echo "It is rainning today." | ace -g ../wn/terg-mac.dat -T -n 1 | delphin convert --from ace --to eds > 2.eds
>> % ./main.py --read eds --score mrp --framework eds --gold 1.eds 2.eds
>> {"n": 0,
> 
> you end up scoring zero items, which either suggests your EDS input
> files are not considered valid by mtool, or the '--framework eds'
> selection fails.  the latter should not be necessary (it may only work
> with MRP input files; its purpose is to select a sub-set of graphs,
> explicitly marked for a specific framework, from a multi-framework
> input file).  equally likely, your EDS input files may be missing the
> identifier prefix; please see 'data/score/eds/' in mtool for the
> expected syntax.