[developers] Comparing a profile with a grammar output

[Alexandre Rademaker]

Hi,

After having a profile disambiguated with FFTB, my first question is how to compare it with the grammar output for the same set of sentences? This comparison should give me an evaluation of the current ranking model with my data. It should tell me if it is worth to train a new model. In particular, if I compare the semantic structure, it would also allow me to ignore variations on syntactic analysis that doesn't impact semantic representation.

I remember that Michael mentioned in the last Summit that PyDelphin has some support for comparing semantic representation, am I right? I didn't find it in the documentation.

I also tried to use the mtools from Stephan (https://github.com/cfmrp/mtool) but I am probably not using it right, since even with two different sentences I am getting the same output below:

% echo "It is rainning today." | ace -g ../wn/terg-mac.dat -T -n 1 | delphin convert --from ace --to eds > 1.eds
% echo "It is rainning today." | ace -g ../wn/terg-mac.dat -T -n 1 | delphin convert --from ace --to eds > 2.eds
% ./main.py --read eds --score mrp --framework eds --gold 1.eds 2.eds
{"n": 0,
 "null": 0,
 "exact": 0,
 "tops": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "labels": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "properties": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "anchors": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "edges": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "attributes": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "all": {"g": 0, "s": 0, "c": 0, "p": 0.0, "r": 0.0, "f": 0.0},
 "time": 6.985664367675781e-05,
 "cpu": 0.00020100000000000673}

What am I missing? Should I use any other method to compare the profile with the grammar output? Comments and suggestions are welcome! :-)	

Best,

Alexandre Rademaker
http://arademaker.github.com/
http://researcher.ibm.com/person/br-alexrad