[developers] EDM implementations

[goodman.m.w at gmail.com]

Thanks again, Bec.

I just want to make sure my implementation gets the same scores for the
same inputs under the same assumptions as the original implementation. For
this to work, its behavior concerning the points I've sought clarification
for should be intentional. In light of your responses, I've separated the
CARG triples from other properties and have given it its own weight. Thus I
should be able to get the same scores as your code by setting the weights
of CARGs (but not properties) and graph-tops to zero. Similarly, I'll add
an option to ignore missing test items and otherwise treat them as
mismatches.

On Fri, Jan 17, 2020 at 6:14 PM Bec Dridan <bec.dridan at gmail.com> wrote:

>
>
> On Fri, Jan 17, 2020 at 5:39 PM goodman.m.w at gmail.com <
> goodman.m.w at gmail.com> wrote:
>
>>
>> One more detail is what to do when the two sides (gold and test) have
>> different numbers of items. Currently my code stops as soon as either a
>> gold or test item is missing, which is what smatch (the similar metric made
>> for AMR) does, but I think that may be wrong because parsing profiles are
>> likely to have missing or extra (overgeneration) items in the middle. So
>> the question is whether we ignore it or count it as a full mismatch.
>>
>
> If you are asking what is 'correct', I guess that depends on why you are
> evaluating. The perl implementation wouldn't have noticed missing gold
> parses, because it used the gold set as the definition of the set. A
> missing test item, on the other hand, by default counts as a full mismatch,
> but there is a command line option to ignore any gold parse with no
> corresponding test parse. The ignore option is useful when the purpose of
> the evaluation is assessing the system you are working on (and you consider
> coverage separately). For comparing across systems, I imagine you probably
> want to count parse failure as a full mismatch. It was useful for me to
> have both options.
>
> Bec
>
>
>>
>> On Thu, Jan 16, 2020 at 6:33 PM Bec Dridan <bec.dridan at gmail.com> wrote:
>>
>>> Wow, that is some old code... From memory, export was a wrapper around
>>> `parse --export`, where I could add :ltriples to the tsdb::*redwoods-export-values*
>>> set.
>>>
>>> I don't know the mtool code at all, but re-reading the paper and looking
>>> at the perl code, I don't think the original implementation evaluated CARG
>>> at all. We only checked that the correct character span had a pred name
>>> of`named`.
>>>
>>> I think you are right that the triple export at the time did not produce
>>> a triple for TOP and it hence would not have been counted.
>>>
>>> That match your memory Stephan?
>>>
>>> Bec
>>>
>>>
>>> On Thu, Jan 16, 2020 at 8:34 PM goodman.m.w at gmail.com <
>>> goodman.m.w at gmail.com> wrote:
>>>
>>>> Hello developers,
>>>>
>>>> Recently I wanted to try out Elementary Dependency Match (EDM) but I
>>>> did not find an easy way to do it. I saw lisp code in the LKB's repository
>>>> and Bec's Perl code, but I'm not sure how to call the former from the
>>>> command line and the latter seems outdated (I don't see the "export"
>>>> command required by its instructions).
>>>>
>>>> The Dridan & Oepen, 2011 algorithm was simple enough so I though I'd
>>>> implement it on top of PyDelphin. The result is here:
>>>> https://github.com/delph-in/delphin.edm. It requires the latest
>>>> version of PyDelphin (v1.2.0). It works with MRS, EDS, and DMRS, and it
>>>> reads text files or [incr tsdb()] profiles.
>>>>
>>>> When I nearly had my version working I found that Stephan et al.'s
>>>> mtool (https://github.com/cfmrp <https://github.com/cfmrp/mtool>The
>>>> paper example
>>>> /mtool <https://github.com/cfmrp/mtool>) also had an implementation of
>>>> EDM, so I used that to compare with my outputs (as I couldn't get the
>>>> previous implementations to work). In this process I think I found some
>>>> differences from Dridan & Oepen, 2011's description, and this email is to
>>>> confirm those findings. Namely, that mtool's (and now my) implementation do
>>>> the following:
>>>>
>>>> * CARGs are treated as property triples ("class 3 information").
>>>> Previously they were combined with the predicate name. This change means
>>>> that predicates like 'named' will match even if their CARGs don't and the
>>>> CARGs are a separate thing that needs to be matched.
>>>>
>>>> * The identification of the graph's TOP counts as a triple.
>>>>
>>>> One difference between mtool and delphin.edm is that mtool does not
>>>> count "variable" properties from EDS, but that's just because its EDS
>>>> parser does not yet handle them while PyDelphin's does.
>>>>
>>>> Can anyone familiar with EDM confirm the above? Or can anyone explain
>>>> how to call the Perl or LKB code so I can compare?
>>>>
>>>> --
>>>> -Michael Wayne Goodman
>>>>
>>>
>>
>> --
>> -Michael Wayne Goodman
>>
>

-- 
-Michael Wayne Goodman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.delph-in.net/archives/developers/attachments/20200120/00ade632/attachment.html>