From paul at haleyai.com Sat Jan 4 20:34:59 2020 From: paul at haleyai.com (paul at haleyai.com) Date: Sat, 4 Jan 2020 14:34:59 -0500 Subject: [developers] issues building PET w/ Boost, trigrams, etc. Message-ID: <004201d5c336$0e7e82c0$2b7b8840$@haleyai.com> Greetings Folks, I found the new quoting convention in the ERG?s TDL (lextypes) when upgrading and that I needed a more recent version of PET to process the current ERG. In the course of upgrading I found a few issues with the build process and its instructions. I?ve done many builds of all the above over recent years, so hopefully this will be of assistance... FYI, I typically build using Ubuntu inside Docker, so any of this is completely repeatable and has nothing to do with my system, per se. The first problem I encountered involved the lack of support for the current version of GCC supported on 16.04 (or 18.04 which I upgraded to in this process). This problem arises from the (outdated?) version of boost.m4 cached in the PET repository. I overcame it by inserting the current versions around line 1419, FYI. The second problem was that additional modules of Boost appear to be required (as reported by errors during configure). These included system, filesystem, and iostream. After this, compilation proceeded but failed for 2 reasons. First, the addition of the trigram subdirectory under the cheap directory seems not properly reflected in cheap/Makefile.am (it omits the relative path when building from the release directory, for example). The same was true for the (?new??) repp subdirectory of cheap. Both were resolved by the following edit to line 12 of cheap/Makefile.am, FYI. I?m not sure that?s the right approach, but ?works for me?! CPPFLAGS += -I$(top_srcdir)/common -I$(top_srcdir)/fspp -I$(top_srcdir)/cheap/repp -I$(top_srcdir)/cheap/trigram @CHEAPCPPFLAGS@ Regards, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodman.m.w at gmail.com Thu Jan 16 10:33:54 2020 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Thu, 16 Jan 2020 17:33:54 +0800 Subject: [developers] EDM implementations Message-ID: Hello developers, Recently I wanted to try out Elementary Dependency Match (EDM) but I did not find an easy way to do it. I saw lisp code in the LKB's repository and Bec's Perl code, but I'm not sure how to call the former from the command line and the latter seems outdated (I don't see the "export" command required by its instructions). The Dridan & Oepen, 2011 algorithm was simple enough so I though I'd implement it on top of PyDelphin. The result is here: https://github.com/delph-in/delphin.edm. It requires the latest version of PyDelphin (v1.2.0). It works with MRS, EDS, and DMRS, and it reads text files or [incr tsdb()] profiles. When I nearly had my version working I found that Stephan et al.'s mtool ( https://github.com/cfmrp/mtool) also had an implementation of EDM, so I used that to compare with my outputs (as I couldn't get the previous implementations to work). In this process I think I found some differences from Dridan & Oepen, 2011's description, and this email is to confirm those findings. Namely, that mtool's (and now my) implementation do the following: * CARGs are treated as property triples ("class 3 information"). Previously they were combined with the predicate name. This change means that predicates like 'named' will match even if their CARGs don't and the CARGs are a separate thing that needs to be matched. * The identification of the graph's TOP counts as a triple. One difference between mtool and delphin.edm is that mtool does not count "variable" properties from EDS, but that's just because its EDS parser does not yet handle them while PyDelphin's does. Can anyone familiar with EDM confirm the above? Or can anyone explain how to call the Perl or LKB code so I can compare? -- -Michael Wayne Goodman -------------- next part -------------- An HTML attachment was scrubbed... URL: From bec.dridan at gmail.com Thu Jan 16 11:33:17 2020 From: bec.dridan at gmail.com (Bec Dridan) Date: Thu, 16 Jan 2020 21:33:17 +1100 Subject: [developers] EDM implementations In-Reply-To: References: Message-ID: Wow, that is some old code... From memory, export was a wrapper around `parse --export`, where I could add :ltriples to the tsdb::*redwoods-export-values* set. I don't know the mtool code at all, but re-reading the paper and looking at the perl code, I don't think the original implementation evaluated CARG at all. We only checked that the correct character span had a pred name of`named`. I think you are right that the triple export at the time did not produce a triple for TOP and it hence would not have been counted. That match your memory Stephan? Bec On Thu, Jan 16, 2020 at 8:34 PM goodman.m.w at gmail.com wrote: > Hello developers, > > Recently I wanted to try out Elementary Dependency Match (EDM) but I did > not find an easy way to do it. I saw lisp code in the LKB's repository and > Bec's Perl code, but I'm not sure how to call the former from the command > line and the latter seems outdated (I don't see the "export" command > required by its instructions). > > The Dridan & Oepen, 2011 algorithm was simple enough so I though I'd > implement it on top of PyDelphin. The result is here: > https://github.com/delph-in/delphin.edm. It requires the latest version > of PyDelphin (v1.2.0). It works with MRS, EDS, and DMRS, and it reads text > files or [incr tsdb()] profiles. > > When I nearly had my version working I found that Stephan et al.'s mtool ( > https://github.com/cfmrp The paper > example > /mtool ) also had an implementation of > EDM, so I used that to compare with my outputs (as I couldn't get the > previous implementations to work). In this process I think I found some > differences from Dridan & Oepen, 2011's description, and this email is to > confirm those findings. Namely, that mtool's (and now my) implementation do > the following: > > * CARGs are treated as property triples ("class 3 information"). > Previously they were combined with the predicate name. This change means > that predicates like 'named' will match even if their CARGs don't and the > CARGs are a separate thing that needs to be matched. > > * The identification of the graph's TOP counts as a triple. > > One difference between mtool and delphin.edm is that mtool does not count > "variable" properties from EDS, but that's just because its EDS parser does > not yet handle them while PyDelphin's does. > > Can anyone familiar with EDM confirm the above? Or can anyone explain how > to call the Perl or LKB code so I can compare? > > -- > -Michael Wayne Goodman > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ebender at uw.edu Fri Jan 17 01:42:08 2020 From: ebender at uw.edu (Emily M. Bender) Date: Thu, 16 Jan 2020 16:42:08 -0800 Subject: [developers] Skipping non-parsed items in fftb Message-ID: Dear all, We are doing some treebanking here at UW with fftb with grammars that have very low coverage over their associated test corpora. The current behavior of fftb with these profiles is to include all items for treebanking, but give a 404 for each one with no parse forest stored. This necessitates clicking the back button and tracking which one is next (since nothing changes color). In that light, two questions: (1) Is there some option we can pass fftb so that it just doesn't present items with no parses? (2) Failing that, is it fairly straightforward with pydelphin, [incr tsdb()] or something else to export a version of the profiles that only includes items which the grammar successfully parsed? Thanks, Emily -- Emily M. Bender (she/her) Howard and Frances Nostrand Endowed Professor Department of Linguistics Faculty Director, CLMS University of Washington Twitter: @emilymbender -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodman.m.w at gmail.com Fri Jan 17 03:33:03 2020 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Fri, 17 Jan 2020 10:33:03 +0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Hi Emily, For (2), here is how you could do it with PyDelphin: delphin process -g grm.dat original-profile/ delphin mkprof --full --where 'readings > 0' --source original-profile/ new-profile/ delphin process -g grm.dat --full-forest new-profile/ Note that original-profile/ is first parsed in regular (non-forest) mode, because in full-forest mode the number of readings is essentially unknown until they are enumerated and thus the 'readings' field is always 0. The second command not only prunes lines in the 'parse' file with readings == 0, but also lines in the 'item' file which correspond to those 'parse' lines. Once you have created new-profile/, you can parse again with --full-forest for use with FFTB (and of course you don't have to use PyDelphin for the parsing steps, if you prefer other means). Also note that this results in a profile with no edges for partial parses. I think this is what you want. There should be a way to prune the full-forest profile directly while keeping partial parses, but while investigating this use case I found a bug, so I don't recommend it yet. Try `delphin mkprof --help` to see descriptions of these and other options. They map fairly directly to the function documented here: https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html#mkprof On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender wrote: > Dear all, > > We are doing some treebanking here at UW with fftb with grammars that have > very low coverage over their associated test corpora. The current behavior > of fftb with these profiles is to include all items for treebanking, but > give a 404 for each one with no parse forest stored. This necessitates > clicking the back button and tracking which one is next (since > nothing changes color). In that light, two questions: > > (1) Is there some option we can pass fftb so that it just doesn't present > items with no parses? > (2) Failing that, is it fairly straightforward with pydelphin, [incr > tsdb()] or something else to export a version of the profiles that only > includes items which the grammar successfully parsed? > > Thanks, > Emily > > > -- > Emily M. Bender (she/her) > Howard and Frances Nostrand Endowed Professor > Department of Linguistics > Faculty Director, CLMS > University of Washington > Twitter: @emilymbender > -- -Michael Wayne Goodman -------------- next part -------------- An HTML attachment was scrubbed... URL: From ebender at uw.edu Fri Jan 17 03:36:51 2020 From: ebender at uw.edu (Emily M. Bender) Date: Thu, 16 Jan 2020 18:36:51 -0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Thanks, Mike! I will give this a try. On Thu, Jan 16, 2020 at 6:33 PM goodman.m.w at gmail.com wrote: > Hi Emily, > > For (2), here is how you could do it with PyDelphin: > > delphin process -g grm.dat original-profile/ > delphin mkprof --full --where 'readings > 0' --source > original-profile/ new-profile/ > delphin process -g grm.dat --full-forest new-profile/ > > Note that original-profile/ is first parsed in regular (non-forest) mode, > because in full-forest mode the number of readings is essentially unknown > until they are enumerated and thus the 'readings' field is always 0. The > second command not only prunes lines in the 'parse' file with readings == > 0, but also lines in the 'item' file which correspond to those 'parse' > lines. Once you have created new-profile/, you can parse again with > --full-forest for use with FFTB (and of course you don't have to use > PyDelphin for the parsing steps, if you prefer other means). > > Also note that this results in a profile with no edges for partial parses. > I think this is what you want. There should be a way to prune the > full-forest profile directly while keeping partial parses, but while > investigating this use case I found a bug, so I don't recommend it yet. > > Try `delphin mkprof --help` to see descriptions of these and other > options. They map fairly directly to the function documented here: > https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html > #mkprof > > > On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender wrote: > >> Dear all, >> >> We are doing some treebanking here at UW with fftb with grammars that >> have very low coverage over their associated test corpora. The current >> behavior of fftb with these profiles is to include all items for >> treebanking, but give a 404 for each one with no parse forest stored. This >> necessitates clicking the back button and tracking which one is next (since >> nothing changes color). In that light, two questions: >> >> (1) Is there some option we can pass fftb so that it just doesn't present >> items with no parses? >> (2) Failing that, is it fairly straightforward with pydelphin, [incr >> tsdb()] or something else to export a version of the profiles that only >> includes items which the grammar successfully parsed? >> >> Thanks, >> Emily >> >> >> -- >> Emily M. Bender (she/her) >> Howard and Frances Nostrand Endowed Professor >> Department of Linguistics >> Faculty Director, CLMS >> University of Washington >> Twitter: @emilymbender >> > > > -- > -Michael Wayne Goodman > -- Emily M. Bender (she/her) Howard and Frances Nostrand Endowed Professor Department of Linguistics Faculty Director, CLMS University of Washington Twitter: @emilymbender -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodman.m.w at gmail.com Fri Jan 17 03:45:49 2020 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Fri, 17 Jan 2020 10:45:49 +0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Let me know how it goes. And a clarification: the --full option on `mkprof` doesn't hurt, but it's unnecessary since you're re-parsing the created profile. Also here's the bug report for the other thing, if you're interested in that use case: https://github.com/delph-in/pydelphin/issues/273 On Fri, Jan 17, 2020 at 10:37 AM Emily M. Bender wrote: > Thanks, Mike! I will give this a try. > > On Thu, Jan 16, 2020 at 6:33 PM goodman.m.w at gmail.com < > goodman.m.w at gmail.com> wrote: > >> Hi Emily, >> >> For (2), here is how you could do it with PyDelphin: >> >> delphin process -g grm.dat original-profile/ >> delphin mkprof --full --where 'readings > 0' --source >> original-profile/ new-profile/ >> delphin process -g grm.dat --full-forest new-profile/ >> >> Note that original-profile/ is first parsed in regular (non-forest) mode, >> because in full-forest mode the number of readings is essentially unknown >> until they are enumerated and thus the 'readings' field is always 0. The >> second command not only prunes lines in the 'parse' file with readings == >> 0, but also lines in the 'item' file which correspond to those 'parse' >> lines. Once you have created new-profile/, you can parse again with >> --full-forest for use with FFTB (and of course you don't have to use >> PyDelphin for the parsing steps, if you prefer other means). >> >> Also note that this results in a profile with no edges for partial >> parses. I think this is what you want. There should be a way to prune the >> full-forest profile directly while keeping partial parses, but while >> investigating this use case I found a bug, so I don't recommend it yet. >> >> Try `delphin mkprof --help` to see descriptions of these and other >> options. They map fairly directly to the function documented here: >> https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html >> #mkprof >> >> >> On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender wrote: >> >>> Dear all, >>> >>> We are doing some treebanking here at UW with fftb with grammars that >>> have very low coverage over their associated test corpora. The current >>> behavior of fftb with these profiles is to include all items for >>> treebanking, but give a 404 for each one with no parse forest stored. This >>> necessitates clicking the back button and tracking which one is next (since >>> nothing changes color). In that light, two questions: >>> >>> (1) Is there some option we can pass fftb so that it just doesn't >>> present items with no parses? >>> (2) Failing that, is it fairly straightforward with pydelphin, [incr >>> tsdb()] or something else to export a version of the profiles that only >>> includes items which the grammar successfully parsed? >>> >>> Thanks, >>> Emily >>> >>> >>> -- >>> Emily M. Bender (she/her) >>> Howard and Frances Nostrand Endowed Professor >>> Department of Linguistics >>> Faculty Director, CLMS >>> University of Washington >>> Twitter: @emilymbender >>> >> >> >> -- >> -Michael Wayne Goodman >> > -- > Emily M. Bender (she/her) > Howard and Frances Nostrand Endowed Professor > Department of Linguistics > Faculty Director, CLMS > University of Washington > Twitter: @emilymbender > -- -Michael Wayne Goodman -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodman.m.w at gmail.com Fri Jan 17 07:39:26 2020 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Fri, 17 Jan 2020 14:39:26 +0800 Subject: [developers] EDM implementations In-Reply-To: References: Message-ID: Thanks, Bec! I manually put in the :ltriples in the parse script and was able to produce some output that edm_eval.pl could read. Regarding the CARG being combined with the predicate name, that was what I guessed by looking at the Lisp code. Thanks for correcting my mistake. One more detail is what to do when the two sides (gold and test) have different numbers of items. Currently my code stops as soon as either a gold or test item is missing, which is what smatch (the similar metric made for AMR) does, but I think that may be wrong because parsing profiles are likely to have missing or extra (overgeneration) items in the middle. So the question is whether we ignore it or count it as a full mismatch. On Thu, Jan 16, 2020 at 6:33 PM Bec Dridan wrote: > Wow, that is some old code... From memory, export was a wrapper around > `parse --export`, where I could add :ltriples to the tsdb::*redwoods-export-values* > set. > > I don't know the mtool code at all, but re-reading the paper and looking > at the perl code, I don't think the original implementation evaluated CARG > at all. We only checked that the correct character span had a pred name > of`named`. > > I think you are right that the triple export at the time did not produce a > triple for TOP and it hence would not have been counted. > > That match your memory Stephan? > > Bec > > > On Thu, Jan 16, 2020 at 8:34 PM goodman.m.w at gmail.com < > goodman.m.w at gmail.com> wrote: > >> Hello developers, >> >> Recently I wanted to try out Elementary Dependency Match (EDM) but I did >> not find an easy way to do it. I saw lisp code in the LKB's repository and >> Bec's Perl code, but I'm not sure how to call the former from the command >> line and the latter seems outdated (I don't see the "export" command >> required by its instructions). >> >> The Dridan & Oepen, 2011 algorithm was simple enough so I though I'd >> implement it on top of PyDelphin. The result is here: >> https://github.com/delph-in/delphin.edm. It requires the latest version >> of PyDelphin (v1.2.0). It works with MRS, EDS, and DMRS, and it reads text >> files or [incr tsdb()] profiles. >> >> When I nearly had my version working I found that Stephan et al.'s mtool ( >> https://github.com/cfmrp The paper >> example >> /mtool ) also had an implementation of >> EDM, so I used that to compare with my outputs (as I couldn't get the >> previous implementations to work). In this process I think I found some >> differences from Dridan & Oepen, 2011's description, and this email is to >> confirm those findings. Namely, that mtool's (and now my) implementation do >> the following: >> >> * CARGs are treated as property triples ("class 3 information"). >> Previously they were combined with the predicate name. This change means >> that predicates like 'named' will match even if their CARGs don't and the >> CARGs are a separate thing that needs to be matched. >> >> * The identification of the graph's TOP counts as a triple. >> >> One difference between mtool and delphin.edm is that mtool does not count >> "variable" properties from EDS, but that's just because its EDS parser does >> not yet handle them while PyDelphin's does. >> >> Can anyone familiar with EDM confirm the above? Or can anyone explain how >> to call the Perl or LKB code so I can compare? >> >> -- >> -Michael Wayne Goodman >> > -- -Michael Wayne Goodman -------------- next part -------------- An HTML attachment was scrubbed... URL: From bec.dridan at gmail.com Fri Jan 17 11:14:28 2020 From: bec.dridan at gmail.com (Bec Dridan) Date: Fri, 17 Jan 2020 21:14:28 +1100 Subject: [developers] EDM implementations In-Reply-To: References: Message-ID: On Fri, Jan 17, 2020 at 5:39 PM goodman.m.w at gmail.com wrote: > > One more detail is what to do when the two sides (gold and test) have > different numbers of items. Currently my code stops as soon as either a > gold or test item is missing, which is what smatch (the similar metric made > for AMR) does, but I think that may be wrong because parsing profiles are > likely to have missing or extra (overgeneration) items in the middle. So > the question is whether we ignore it or count it as a full mismatch. > If you are asking what is 'correct', I guess that depends on why you are evaluating. The perl implementation wouldn't have noticed missing gold parses, because it used the gold set as the definition of the set. A missing test item, on the other hand, by default counts as a full mismatch, but there is a command line option to ignore any gold parse with no corresponding test parse. The ignore option is useful when the purpose of the evaluation is assessing the system you are working on (and you consider coverage separately). For comparing across systems, I imagine you probably want to count parse failure as a full mismatch. It was useful for me to have both options. Bec > > On Thu, Jan 16, 2020 at 6:33 PM Bec Dridan wrote: > >> Wow, that is some old code... From memory, export was a wrapper around >> `parse --export`, where I could add :ltriples to the tsdb::*redwoods-export-values* >> set. >> >> I don't know the mtool code at all, but re-reading the paper and looking >> at the perl code, I don't think the original implementation evaluated CARG >> at all. We only checked that the correct character span had a pred name >> of`named`. >> >> I think you are right that the triple export at the time did not produce >> a triple for TOP and it hence would not have been counted. >> >> That match your memory Stephan? >> >> Bec >> >> >> On Thu, Jan 16, 2020 at 8:34 PM goodman.m.w at gmail.com < >> goodman.m.w at gmail.com> wrote: >> >>> Hello developers, >>> >>> Recently I wanted to try out Elementary Dependency Match (EDM) but I did >>> not find an easy way to do it. I saw lisp code in the LKB's repository and >>> Bec's Perl code, but I'm not sure how to call the former from the command >>> line and the latter seems outdated (I don't see the "export" command >>> required by its instructions). >>> >>> The Dridan & Oepen, 2011 algorithm was simple enough so I though I'd >>> implement it on top of PyDelphin. The result is here: >>> https://github.com/delph-in/delphin.edm. It requires the latest version >>> of PyDelphin (v1.2.0). It works with MRS, EDS, and DMRS, and it reads text >>> files or [incr tsdb()] profiles. >>> >>> When I nearly had my version working I found that Stephan et al.'s mtool >>> (https://github.com/cfmrp The paper >>> example >>> /mtool ) also had an implementation of >>> EDM, so I used that to compare with my outputs (as I couldn't get the >>> previous implementations to work). In this process I think I found some >>> differences from Dridan & Oepen, 2011's description, and this email is to >>> confirm those findings. Namely, that mtool's (and now my) implementation do >>> the following: >>> >>> * CARGs are treated as property triples ("class 3 information"). >>> Previously they were combined with the predicate name. This change means >>> that predicates like 'named' will match even if their CARGs don't and the >>> CARGs are a separate thing that needs to be matched. >>> >>> * The identification of the graph's TOP counts as a triple. >>> >>> One difference between mtool and delphin.edm is that mtool does not >>> count "variable" properties from EDS, but that's just because its EDS >>> parser does not yet handle them while PyDelphin's does. >>> >>> Can anyone familiar with EDM confirm the above? Or can anyone explain >>> how to call the Perl or LKB code so I can compare? >>> >>> -- >>> -Michael Wayne Goodman >>> >> > > -- > -Michael Wayne Goodman > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ebender at uw.edu Sat Jan 18 00:41:16 2020 From: ebender at uw.edu (Emily M. Bender) Date: Fri, 17 Jan 2020 15:41:16 -0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Dear Mike, Alas, I'm hitting this error: (run_agg) ebender at patas:~$ delphin process -g ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ Traceback (most recent call last): File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in sys.exit(main()) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", line 42, in main args.func(args) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", line 135, in call_process gzip=args.gzip) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", line 540, in process source = itsdb.TestSuite(source) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/itsdb.py", line 644, in __init__ '*schema* argument is required for new test suites') delphin.itsdb.ITSDBError: *schema* argument is required for new test suites I'll poke around and see where the schema requirement is coming from (nothing in the bit on "process" in the documentation page mentions it), but thought I'd post here too in the meantime. Emily On Thu, Jan 16, 2020 at 6:46 PM goodman.m.w at gmail.com wrote: > Let me know how it goes. > > And a clarification: the --full option on `mkprof` doesn't hurt, but it's > unnecessary since you're re-parsing the created profile. > > Also here's the bug report for the other thing, if you're interested in > that use case: https://github.com/delph-in/pydelphin/issues/273 > > On Fri, Jan 17, 2020 at 10:37 AM Emily M. Bender wrote: > >> Thanks, Mike! I will give this a try. >> >> On Thu, Jan 16, 2020 at 6:33 PM goodman.m.w at gmail.com < >> goodman.m.w at gmail.com> wrote: >> >>> Hi Emily, >>> >>> For (2), here is how you could do it with PyDelphin: >>> >>> delphin process -g grm.dat original-profile/ >>> delphin mkprof --full --where 'readings > 0' --source >>> original-profile/ new-profile/ >>> delphin process -g grm.dat --full-forest new-profile/ >>> >>> Note that original-profile/ is first parsed in regular (non-forest) >>> mode, because in full-forest mode the number of readings is essentially >>> unknown until they are enumerated and thus the 'readings' field is always >>> 0. The second command not only prunes lines in the 'parse' file with >>> readings == 0, but also lines in the 'item' file which correspond to those >>> 'parse' lines. Once you have created new-profile/, you can parse again with >>> --full-forest for use with FFTB (and of course you don't have to use >>> PyDelphin for the parsing steps, if you prefer other means). >>> >>> Also note that this results in a profile with no edges for partial >>> parses. I think this is what you want. There should be a way to prune the >>> full-forest profile directly while keeping partial parses, but while >>> investigating this use case I found a bug, so I don't recommend it yet. >>> >>> Try `delphin mkprof --help` to see descriptions of these and other >>> options. They map fairly directly to the function documented here: >>> https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html >>> #mkprof >>> >>> >>> On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender wrote: >>> >>>> Dear all, >>>> >>>> We are doing some treebanking here at UW with fftb with grammars that >>>> have very low coverage over their associated test corpora. The current >>>> behavior of fftb with these profiles is to include all items for >>>> treebanking, but give a 404 for each one with no parse forest stored. This >>>> necessitates clicking the back button and tracking which one is next (since >>>> nothing changes color). In that light, two questions: >>>> >>>> (1) Is there some option we can pass fftb so that it just doesn't >>>> present items with no parses? >>>> (2) Failing that, is it fairly straightforward with pydelphin, [incr >>>> tsdb()] or something else to export a version of the profiles that only >>>> includes items which the grammar successfully parsed? >>>> >>>> Thanks, >>>> Emily >>>> >>>> >>>> -- >>>> Emily M. Bender (she/her) >>>> Howard and Frances Nostrand Endowed Professor >>>> Department of Linguistics >>>> Faculty Director, CLMS >>>> University of Washington >>>> Twitter: @emilymbender >>>> >>> >>> >>> -- >>> -Michael Wayne Goodman >>> >> -- >> Emily M. Bender (she/her) >> Howard and Frances Nostrand Endowed Professor >> Department of Linguistics >> Faculty Director, CLMS >> University of Washington >> Twitter: @emilymbender >> > > > -- > -Michael Wayne Goodman > -- Emily M. Bender (she/her) Howard and Frances Nostrand Endowed Professor Department of Linguistics Faculty Director, CLMS University of Washington Twitter: @emilymbender -------------- next part -------------- An HTML attachment was scrubbed... URL: From ebender at uw.edu Sat Jan 18 00:51:48 2020 From: ebender at uw.edu (Emily M. Bender) Date: Fri, 17 Jan 2020 15:51:48 -0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Apologies --- that error meant I hadn't given the right path to the testsuite. Correcting that, I now see: (run_agg) ebender at patas:/home2/kphowell/run_aggregation/output/emb_treebank$ delphin process -g ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ Traceback (most recent call last): File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in sys.exit(main()) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", line 42, in main args.func(args) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", line 135, in call_process gzip=args.gzip) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", line 542, in process column, tablename, condition = _interpret_selection(select, source) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", line 562, in _interpret_selection if len(queryobj['tables']) == 1: KeyError: 'tables' On Fri, Jan 17, 2020 at 3:41 PM Emily M. Bender wrote: > Dear Mike, > > Alas, I'm hitting this error: > > (run_agg) ebender at patas:~$ delphin process -g > ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ > Traceback (most recent call last): > File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in > sys.exit(main()) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", > line 42, in main > args.func(args) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", > line 135, in call_process > gzip=args.gzip) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", > line 540, in process > source = itsdb.TestSuite(source) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/itsdb.py", > line 644, in __init__ > '*schema* argument is required for new test suites') > delphin.itsdb.ITSDBError: *schema* argument is required for new test suites > > I'll poke around and see where the schema requirement is coming from > (nothing in the bit on "process" in the documentation page mentions it), > but thought I'd post here too in the meantime. > > Emily > > On Thu, Jan 16, 2020 at 6:46 PM goodman.m.w at gmail.com < > goodman.m.w at gmail.com> wrote: > >> Let me know how it goes. >> >> And a clarification: the --full option on `mkprof` doesn't hurt, but it's >> unnecessary since you're re-parsing the created profile. >> >> Also here's the bug report for the other thing, if you're interested in >> that use case: https://github.com/delph-in/pydelphin/issues/273 >> >> On Fri, Jan 17, 2020 at 10:37 AM Emily M. Bender wrote: >> >>> Thanks, Mike! I will give this a try. >>> >>> On Thu, Jan 16, 2020 at 6:33 PM goodman.m.w at gmail.com < >>> goodman.m.w at gmail.com> wrote: >>> >>>> Hi Emily, >>>> >>>> For (2), here is how you could do it with PyDelphin: >>>> >>>> delphin process -g grm.dat original-profile/ >>>> delphin mkprof --full --where 'readings > 0' --source >>>> original-profile/ new-profile/ >>>> delphin process -g grm.dat --full-forest new-profile/ >>>> >>>> Note that original-profile/ is first parsed in regular (non-forest) >>>> mode, because in full-forest mode the number of readings is essentially >>>> unknown until they are enumerated and thus the 'readings' field is always >>>> 0. The second command not only prunes lines in the 'parse' file with >>>> readings == 0, but also lines in the 'item' file which correspond to those >>>> 'parse' lines. Once you have created new-profile/, you can parse again with >>>> --full-forest for use with FFTB (and of course you don't have to use >>>> PyDelphin for the parsing steps, if you prefer other means). >>>> >>>> Also note that this results in a profile with no edges for partial >>>> parses. I think this is what you want. There should be a way to prune the >>>> full-forest profile directly while keeping partial parses, but while >>>> investigating this use case I found a bug, so I don't recommend it yet. >>>> >>>> Try `delphin mkprof --help` to see descriptions of these and other >>>> options. They map fairly directly to the function documented here: >>>> https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html >>>> #mkprof >>>> >>>> >>>> On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender wrote: >>>> >>>>> Dear all, >>>>> >>>>> We are doing some treebanking here at UW with fftb with grammars that >>>>> have very low coverage over their associated test corpora. The current >>>>> behavior of fftb with these profiles is to include all items for >>>>> treebanking, but give a 404 for each one with no parse forest stored. This >>>>> necessitates clicking the back button and tracking which one is next (since >>>>> nothing changes color). In that light, two questions: >>>>> >>>>> (1) Is there some option we can pass fftb so that it just doesn't >>>>> present items with no parses? >>>>> (2) Failing that, is it fairly straightforward with pydelphin, [incr >>>>> tsdb()] or something else to export a version of the profiles that only >>>>> includes items which the grammar successfully parsed? >>>>> >>>>> Thanks, >>>>> Emily >>>>> >>>>> >>>>> -- >>>>> Emily M. Bender (she/her) >>>>> Howard and Frances Nostrand Endowed Professor >>>>> Department of Linguistics >>>>> Faculty Director, CLMS >>>>> University of Washington >>>>> Twitter: @emilymbender >>>>> >>>> >>>> >>>> -- >>>> -Michael Wayne Goodman >>>> >>> -- >>> Emily M. Bender (she/her) >>> Howard and Frances Nostrand Endowed Professor >>> Department of Linguistics >>> Faculty Director, CLMS >>> University of Washington >>> Twitter: @emilymbender >>> >> >> >> -- >> -Michael Wayne Goodman >> > > > -- > Emily M. Bender (she/her) > Howard and Frances Nostrand Endowed Professor > Department of Linguistics > Faculty Director, CLMS > University of Washington > Twitter: @emilymbender > -- Emily M. Bender (she/her) Howard and Frances Nostrand Endowed Professor Department of Linguistics Faculty Director, CLMS University of Washington Twitter: @emilymbender -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodman.m.w at gmail.com Sat Jan 18 01:17:13 2020 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Sat, 18 Jan 2020 08:17:13 +0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Hi Emily, Yes those error messages are not very clear. But the second one looks like old code, as 'tables' is no longer a key in the object it's being looked up on. I suggest making sure that your run_agg environment has an updated version of PyDelphin. While the environment is active, try `pip install -U pydelphin` and make sure it has a 1.0 or newer version (`delphin --version`), then try again. On Sat, Jan 18, 2020 at 7:52 AM Emily M. Bender wrote: > Apologies --- that error meant I hadn't given the right path to the > testsuite. Correcting that, I now see: > > (run_agg) ebender at patas:/home2/kphowell/run_aggregation/output/emb_treebank$ > delphin process -g ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ > Traceback (most recent call last): > File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in > sys.exit(main()) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", > line 42, in main > args.func(args) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", > line 135, in call_process > gzip=args.gzip) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", > line 542, in process > column, tablename, condition = _interpret_selection(select, source) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", > line 562, in _interpret_selection > if len(queryobj['tables']) == 1: > KeyError: 'tables' > > On Fri, Jan 17, 2020 at 3:41 PM Emily M. Bender wrote: > >> Dear Mike, >> >> Alas, I'm hitting this error: >> >> (run_agg) ebender at patas:~$ delphin process -g >> ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ >> Traceback (most recent call last): >> File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in >> sys.exit(main()) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >> line 42, in main >> args.func(args) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >> line 135, in call_process >> gzip=args.gzip) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >> line 540, in process >> source = itsdb.TestSuite(source) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/itsdb.py", >> line 644, in __init__ >> '*schema* argument is required for new test suites') >> delphin.itsdb.ITSDBError: *schema* argument is required for new test >> suites >> >> I'll poke around and see where the schema requirement is coming from >> (nothing in the bit on "process" in the documentation page mentions it), >> but thought I'd post here too in the meantime. >> >> Emily >> >> On Thu, Jan 16, 2020 at 6:46 PM goodman.m.w at gmail.com < >> goodman.m.w at gmail.com> wrote: >> >>> Let me know how it goes. >>> >>> And a clarification: the --full option on `mkprof` doesn't hurt, but >>> it's unnecessary since you're re-parsing the created profile. >>> >>> Also here's the bug report for the other thing, if you're interested in >>> that use case: https://github.com/delph-in/pydelphin/issues/273 >>> >>> On Fri, Jan 17, 2020 at 10:37 AM Emily M. Bender wrote: >>> >>>> Thanks, Mike! I will give this a try. >>>> >>>> On Thu, Jan 16, 2020 at 6:33 PM goodman.m.w at gmail.com < >>>> goodman.m.w at gmail.com> wrote: >>>> >>>>> Hi Emily, >>>>> >>>>> For (2), here is how you could do it with PyDelphin: >>>>> >>>>> delphin process -g grm.dat original-profile/ >>>>> delphin mkprof --full --where 'readings > 0' --source >>>>> original-profile/ new-profile/ >>>>> delphin process -g grm.dat --full-forest new-profile/ >>>>> >>>>> Note that original-profile/ is first parsed in regular (non-forest) >>>>> mode, because in full-forest mode the number of readings is essentially >>>>> unknown until they are enumerated and thus the 'readings' field is always >>>>> 0. The second command not only prunes lines in the 'parse' file with >>>>> readings == 0, but also lines in the 'item' file which correspond to those >>>>> 'parse' lines. Once you have created new-profile/, you can parse again with >>>>> --full-forest for use with FFTB (and of course you don't have to use >>>>> PyDelphin for the parsing steps, if you prefer other means). >>>>> >>>>> Also note that this results in a profile with no edges for partial >>>>> parses. I think this is what you want. There should be a way to prune the >>>>> full-forest profile directly while keeping partial parses, but while >>>>> investigating this use case I found a bug, so I don't recommend it yet. >>>>> >>>>> Try `delphin mkprof --help` to see descriptions of these and other >>>>> options. They map fairly directly to the function documented here: >>>>> https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html >>>>> #mkprof >>>>> >>>>> >>>>> On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender >>>>> wrote: >>>>> >>>>>> Dear all, >>>>>> >>>>>> We are doing some treebanking here at UW with fftb with grammars that >>>>>> have very low coverage over their associated test corpora. The current >>>>>> behavior of fftb with these profiles is to include all items for >>>>>> treebanking, but give a 404 for each one with no parse forest stored. This >>>>>> necessitates clicking the back button and tracking which one is next (since >>>>>> nothing changes color). In that light, two questions: >>>>>> >>>>>> (1) Is there some option we can pass fftb so that it just doesn't >>>>>> present items with no parses? >>>>>> (2) Failing that, is it fairly straightforward with pydelphin, [incr >>>>>> tsdb()] or something else to export a version of the profiles that only >>>>>> includes items which the grammar successfully parsed? >>>>>> >>>>>> Thanks, >>>>>> Emily >>>>>> >>>>>> >>>>>> -- >>>>>> Emily M. Bender (she/her) >>>>>> Howard and Frances Nostrand Endowed Professor >>>>>> Department of Linguistics >>>>>> Faculty Director, CLMS >>>>>> University of Washington >>>>>> Twitter: @emilymbender >>>>>> >>>>> >>>>> >>>>> -- >>>>> -Michael Wayne Goodman >>>>> >>>> -- >>>> Emily M. Bender (she/her) >>>> Howard and Frances Nostrand Endowed Professor >>>> Department of Linguistics >>>> Faculty Director, CLMS >>>> University of Washington >>>> Twitter: @emilymbender >>>> >>> >>> >>> -- >>> -Michael Wayne Goodman >>> >> >> >> -- >> Emily M. Bender (she/her) >> Howard and Frances Nostrand Endowed Professor >> Department of Linguistics >> Faculty Director, CLMS >> University of Washington >> Twitter: @emilymbender >> > > > -- > Emily M. Bender (she/her) > Howard and Frances Nostrand Endowed Professor > Department of Linguistics > Faculty Director, CLMS > University of Washington > Twitter: @emilymbender > -- -Michael Wayne Goodman -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodman.m.w at gmail.com Mon Jan 20 02:14:53 2020 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Mon, 20 Jan 2020 09:14:53 +0800 Subject: [developers] EDM implementations In-Reply-To: References: Message-ID: Thanks again, Bec. I just want to make sure my implementation gets the same scores for the same inputs under the same assumptions as the original implementation. For this to work, its behavior concerning the points I've sought clarification for should be intentional. In light of your responses, I've separated the CARG triples from other properties and have given it its own weight. Thus I should be able to get the same scores as your code by setting the weights of CARGs (but not properties) and graph-tops to zero. Similarly, I'll add an option to ignore missing test items and otherwise treat them as mismatches. On Fri, Jan 17, 2020 at 6:14 PM Bec Dridan wrote: > > > On Fri, Jan 17, 2020 at 5:39 PM goodman.m.w at gmail.com < > goodman.m.w at gmail.com> wrote: > >> >> One more detail is what to do when the two sides (gold and test) have >> different numbers of items. Currently my code stops as soon as either a >> gold or test item is missing, which is what smatch (the similar metric made >> for AMR) does, but I think that may be wrong because parsing profiles are >> likely to have missing or extra (overgeneration) items in the middle. So >> the question is whether we ignore it or count it as a full mismatch. >> > > If you are asking what is 'correct', I guess that depends on why you are > evaluating. The perl implementation wouldn't have noticed missing gold > parses, because it used the gold set as the definition of the set. A > missing test item, on the other hand, by default counts as a full mismatch, > but there is a command line option to ignore any gold parse with no > corresponding test parse. The ignore option is useful when the purpose of > the evaluation is assessing the system you are working on (and you consider > coverage separately). For comparing across systems, I imagine you probably > want to count parse failure as a full mismatch. It was useful for me to > have both options. > > Bec > > >> >> On Thu, Jan 16, 2020 at 6:33 PM Bec Dridan wrote: >> >>> Wow, that is some old code... From memory, export was a wrapper around >>> `parse --export`, where I could add :ltriples to the tsdb::*redwoods-export-values* >>> set. >>> >>> I don't know the mtool code at all, but re-reading the paper and looking >>> at the perl code, I don't think the original implementation evaluated CARG >>> at all. We only checked that the correct character span had a pred name >>> of`named`. >>> >>> I think you are right that the triple export at the time did not produce >>> a triple for TOP and it hence would not have been counted. >>> >>> That match your memory Stephan? >>> >>> Bec >>> >>> >>> On Thu, Jan 16, 2020 at 8:34 PM goodman.m.w at gmail.com < >>> goodman.m.w at gmail.com> wrote: >>> >>>> Hello developers, >>>> >>>> Recently I wanted to try out Elementary Dependency Match (EDM) but I >>>> did not find an easy way to do it. I saw lisp code in the LKB's repository >>>> and Bec's Perl code, but I'm not sure how to call the former from the >>>> command line and the latter seems outdated (I don't see the "export" >>>> command required by its instructions). >>>> >>>> The Dridan & Oepen, 2011 algorithm was simple enough so I though I'd >>>> implement it on top of PyDelphin. The result is here: >>>> https://github.com/delph-in/delphin.edm. It requires the latest >>>> version of PyDelphin (v1.2.0). It works with MRS, EDS, and DMRS, and it >>>> reads text files or [incr tsdb()] profiles. >>>> >>>> When I nearly had my version working I found that Stephan et al.'s >>>> mtool (https://github.com/cfmrp The >>>> paper example >>>> /mtool ) also had an implementation of >>>> EDM, so I used that to compare with my outputs (as I couldn't get the >>>> previous implementations to work). In this process I think I found some >>>> differences from Dridan & Oepen, 2011's description, and this email is to >>>> confirm those findings. Namely, that mtool's (and now my) implementation do >>>> the following: >>>> >>>> * CARGs are treated as property triples ("class 3 information"). >>>> Previously they were combined with the predicate name. This change means >>>> that predicates like 'named' will match even if their CARGs don't and the >>>> CARGs are a separate thing that needs to be matched. >>>> >>>> * The identification of the graph's TOP counts as a triple. >>>> >>>> One difference between mtool and delphin.edm is that mtool does not >>>> count "variable" properties from EDS, but that's just because its EDS >>>> parser does not yet handle them while PyDelphin's does. >>>> >>>> Can anyone familiar with EDM confirm the above? Or can anyone explain >>>> how to call the Perl or LKB code so I can compare? >>>> >>>> -- >>>> -Michael Wayne Goodman >>>> >>> >> >> -- >> -Michael Wayne Goodman >> > -- -Michael Wayne Goodman -------------- next part -------------- An HTML attachment was scrubbed... URL: From ebender at uw.edu Wed Jan 22 02:19:18 2020 From: ebender at uw.edu (Emily M. Bender) Date: Tue, 21 Jan 2020 17:19:18 -0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Updating PyDelphin caused the error to change, at least: Traceback (most recent call last): File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in sys.exit(main()) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", line 40, in main args.func(args) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/cli/process.py", line 46, in call_process gzip=args.gzip) File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", line 602, in process with processor(grammar, cmdargs=options, **kwargs) as cpu: File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/ace.py", line 110, in __init__ self._open() File "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/ace.py", line 143, in _open raise ACEProcessError('Process closed on startup; see .') delphin.ace.ACEProcessError: Process closed on startup; see . ... I'm a little puzzled as to how I got about "seeing " as I didn't myself do anything to redirect it somewhere else, but it's not printing to the console... Emily On Fri, Jan 17, 2020 at 4:17 PM goodman.m.w at gmail.com wrote: > Hi Emily, > > Yes those error messages are not very clear. But the second one looks like > old code, as 'tables' is no longer a key in the object it's being looked up > on. I suggest making sure that your run_agg environment has an updated > version of PyDelphin. While the environment is active, try `pip install -U > pydelphin` and make sure it has a 1.0 or newer version (`delphin > --version`), then try again. > > On Sat, Jan 18, 2020 at 7:52 AM Emily M. Bender wrote: > >> Apologies --- that error meant I hadn't given the right path to the >> testsuite. Correcting that, I now see: >> >> (run_agg) ebender at patas:/home2/kphowell/run_aggregation/output/emb_treebank$ >> delphin process -g ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ >> Traceback (most recent call last): >> File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in >> sys.exit(main()) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >> line 42, in main >> args.func(args) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >> line 135, in call_process >> gzip=args.gzip) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >> line 542, in process >> column, tablename, condition = _interpret_selection(select, source) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >> line 562, in _interpret_selection >> if len(queryobj['tables']) == 1: >> KeyError: 'tables' >> >> On Fri, Jan 17, 2020 at 3:41 PM Emily M. Bender wrote: >> >>> Dear Mike, >>> >>> Alas, I'm hitting this error: >>> >>> (run_agg) ebender at patas:~$ delphin process -g >>> ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ >>> Traceback (most recent call last): >>> File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in >>> sys.exit(main()) >>> File >>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>> line 42, in main >>> args.func(args) >>> File >>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>> line 135, in call_process >>> gzip=args.gzip) >>> File >>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >>> line 540, in process >>> source = itsdb.TestSuite(source) >>> File >>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/itsdb.py", >>> line 644, in __init__ >>> '*schema* argument is required for new test suites') >>> delphin.itsdb.ITSDBError: *schema* argument is required for new test >>> suites >>> >>> I'll poke around and see where the schema requirement is coming from >>> (nothing in the bit on "process" in the documentation page mentions it), >>> but thought I'd post here too in the meantime. >>> >>> Emily >>> >>> On Thu, Jan 16, 2020 at 6:46 PM goodman.m.w at gmail.com < >>> goodman.m.w at gmail.com> wrote: >>> >>>> Let me know how it goes. >>>> >>>> And a clarification: the --full option on `mkprof` doesn't hurt, but >>>> it's unnecessary since you're re-parsing the created profile. >>>> >>>> Also here's the bug report for the other thing, if you're interested in >>>> that use case: https://github.com/delph-in/pydelphin/issues/273 >>>> >>>> On Fri, Jan 17, 2020 at 10:37 AM Emily M. Bender >>>> wrote: >>>> >>>>> Thanks, Mike! I will give this a try. >>>>> >>>>> On Thu, Jan 16, 2020 at 6:33 PM goodman.m.w at gmail.com < >>>>> goodman.m.w at gmail.com> wrote: >>>>> >>>>>> Hi Emily, >>>>>> >>>>>> For (2), here is how you could do it with PyDelphin: >>>>>> >>>>>> delphin process -g grm.dat original-profile/ >>>>>> delphin mkprof --full --where 'readings > 0' --source >>>>>> original-profile/ new-profile/ >>>>>> delphin process -g grm.dat --full-forest new-profile/ >>>>>> >>>>>> Note that original-profile/ is first parsed in regular (non-forest) >>>>>> mode, because in full-forest mode the number of readings is essentially >>>>>> unknown until they are enumerated and thus the 'readings' field is always >>>>>> 0. The second command not only prunes lines in the 'parse' file with >>>>>> readings == 0, but also lines in the 'item' file which correspond to those >>>>>> 'parse' lines. Once you have created new-profile/, you can parse again with >>>>>> --full-forest for use with FFTB (and of course you don't have to use >>>>>> PyDelphin for the parsing steps, if you prefer other means). >>>>>> >>>>>> Also note that this results in a profile with no edges for partial >>>>>> parses. I think this is what you want. There should be a way to prune the >>>>>> full-forest profile directly while keeping partial parses, but while >>>>>> investigating this use case I found a bug, so I don't recommend it yet. >>>>>> >>>>>> Try `delphin mkprof --help` to see descriptions of these and other >>>>>> options. They map fairly directly to the function documented here: >>>>>> https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html >>>>>> #mkprof >>>>>> >>>>>> >>>>>> On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender >>>>>> wrote: >>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> We are doing some treebanking here at UW with fftb with grammars >>>>>>> that have very low coverage over their associated test corpora. The current >>>>>>> behavior of fftb with these profiles is to include all items for >>>>>>> treebanking, but give a 404 for each one with no parse forest stored. This >>>>>>> necessitates clicking the back button and tracking which one is next (since >>>>>>> nothing changes color). In that light, two questions: >>>>>>> >>>>>>> (1) Is there some option we can pass fftb so that it just doesn't >>>>>>> present items with no parses? >>>>>>> (2) Failing that, is it fairly straightforward with pydelphin, [incr >>>>>>> tsdb()] or something else to export a version of the profiles that only >>>>>>> includes items which the grammar successfully parsed? >>>>>>> >>>>>>> Thanks, >>>>>>> Emily >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Emily M. Bender (she/her) >>>>>>> Howard and Frances Nostrand Endowed Professor >>>>>>> Department of Linguistics >>>>>>> Faculty Director, CLMS >>>>>>> University of Washington >>>>>>> Twitter: @emilymbender >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -Michael Wayne Goodman >>>>>> >>>>> -- >>>>> Emily M. Bender (she/her) >>>>> Howard and Frances Nostrand Endowed Professor >>>>> Department of Linguistics >>>>> Faculty Director, CLMS >>>>> University of Washington >>>>> Twitter: @emilymbender >>>>> >>>> >>>> >>>> -- >>>> -Michael Wayne Goodman >>>> >>> >>> >>> -- >>> Emily M. Bender (she/her) >>> Howard and Frances Nostrand Endowed Professor >>> Department of Linguistics >>> Faculty Director, CLMS >>> University of Washington >>> Twitter: @emilymbender >>> >> >> >> -- >> Emily M. Bender (she/her) >> Howard and Frances Nostrand Endowed Professor >> Department of Linguistics >> Faculty Director, CLMS >> University of Washington >> Twitter: @emilymbender >> > > > -- > -Michael Wayne Goodman > -- Emily M. Bender (she/her) Howard and Frances Nostrand Endowed Professor Department of Linguistics Faculty Director, CLMS University of Washington Twitter: @emilymbender -------------- next part -------------- An HTML attachment was scrubbed... URL: From goodman.m.w at gmail.com Wed Jan 22 02:36:07 2020 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Wed, 22 Jan 2020 09:36:07 +0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Ok, that's progress. This error shows up when ACE exited abnormally (recall that PyDelphin calls ACE in a subprocess in order to process profiles). Since I don't capture ACE's stderr, it should be printed in the terminal just above the stacktrace. The PyDelphin error is directing you to look for that message to fix the problem. Most likely, the path to the grammar image is incorrect or the grammar image was compiled with a different version of ACE. The stacktrace and error message are both printed to stderr so if you see one you should see both (unless ACE exited abnormally without printing anything). By the way, I've recently pushed some commits to suppress the stacktrace when encountering anticipated errors from the `delphin` command, as I don't think the stacktrace is useful except for me (it can be shown again when called in DEBUG mode). In addition I tried to provide more useful messages for common situations. These changes will be part of the next release. On Wed, Jan 22, 2020 at 9:19 AM Emily M. Bender wrote: > Updating PyDelphin caused the error to change, at least: > > Traceback (most recent call last): > File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in > sys.exit(main()) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", > line 40, in main > args.func(args) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/cli/process.py", > line 46, in call_process > gzip=args.gzip) > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", > line 602, in process > with processor(grammar, cmdargs=options, **kwargs) as cpu: > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/ace.py", > line 110, in __init__ > self._open() > File > "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/ace.py", > line 143, in _open > raise ACEProcessError('Process closed on startup; see .') > delphin.ace.ACEProcessError: Process closed on startup; see . > > > ... I'm a little puzzled as to how I got about "seeing " as I > didn't myself do anything to redirect it somewhere else, but it's not > printing to the console... > > Emily > > On Fri, Jan 17, 2020 at 4:17 PM goodman.m.w at gmail.com < > goodman.m.w at gmail.com> wrote: > >> Hi Emily, >> >> Yes those error messages are not very clear. But the second one looks >> like old code, as 'tables' is no longer a key in the object it's being >> looked up on. I suggest making sure that your run_agg environment has an >> updated version of PyDelphin. While the environment is active, try `pip >> install -U pydelphin` and make sure it has a 1.0 or newer version (`delphin >> --version`), then try again. >> >> On Sat, Jan 18, 2020 at 7:52 AM Emily M. Bender wrote: >> >>> Apologies --- that error meant I hadn't given the right path to the >>> testsuite. Correcting that, I now see: >>> >>> (run_agg) ebender at patas:/home2/kphowell/run_aggregation/output/emb_treebank$ >>> delphin process -g ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ >>> Traceback (most recent call last): >>> File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in >>> sys.exit(main()) >>> File >>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>> line 42, in main >>> args.func(args) >>> File >>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>> line 135, in call_process >>> gzip=args.gzip) >>> File >>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >>> line 542, in process >>> column, tablename, condition = _interpret_selection(select, source) >>> File >>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >>> line 562, in _interpret_selection >>> if len(queryobj['tables']) == 1: >>> KeyError: 'tables' >>> >>> On Fri, Jan 17, 2020 at 3:41 PM Emily M. Bender wrote: >>> >>>> Dear Mike, >>>> >>>> Alas, I'm hitting this error: >>>> >>>> (run_agg) ebender at patas:~$ delphin process -g >>>> ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ >>>> Traceback (most recent call last): >>>> File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in >>>> sys.exit(main()) >>>> File >>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>>> line 42, in main >>>> args.func(args) >>>> File >>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>>> line 135, in call_process >>>> gzip=args.gzip) >>>> File >>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >>>> line 540, in process >>>> source = itsdb.TestSuite(source) >>>> File >>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/itsdb.py", >>>> line 644, in __init__ >>>> '*schema* argument is required for new test suites') >>>> delphin.itsdb.ITSDBError: *schema* argument is required for new test >>>> suites >>>> >>>> I'll poke around and see where the schema requirement is coming from >>>> (nothing in the bit on "process" in the documentation page mentions it), >>>> but thought I'd post here too in the meantime. >>>> >>>> Emily >>>> >>>> On Thu, Jan 16, 2020 at 6:46 PM goodman.m.w at gmail.com < >>>> goodman.m.w at gmail.com> wrote: >>>> >>>>> Let me know how it goes. >>>>> >>>>> And a clarification: the --full option on `mkprof` doesn't hurt, but >>>>> it's unnecessary since you're re-parsing the created profile. >>>>> >>>>> Also here's the bug report for the other thing, if you're interested >>>>> in that use case: https://github.com/delph-in/pydelphin/issues/273 >>>>> >>>>> On Fri, Jan 17, 2020 at 10:37 AM Emily M. Bender >>>>> wrote: >>>>> >>>>>> Thanks, Mike! I will give this a try. >>>>>> >>>>>> On Thu, Jan 16, 2020 at 6:33 PM goodman.m.w at gmail.com < >>>>>> goodman.m.w at gmail.com> wrote: >>>>>> >>>>>>> Hi Emily, >>>>>>> >>>>>>> For (2), here is how you could do it with PyDelphin: >>>>>>> >>>>>>> delphin process -g grm.dat original-profile/ >>>>>>> delphin mkprof --full --where 'readings > 0' --source >>>>>>> original-profile/ new-profile/ >>>>>>> delphin process -g grm.dat --full-forest new-profile/ >>>>>>> >>>>>>> Note that original-profile/ is first parsed in regular (non-forest) >>>>>>> mode, because in full-forest mode the number of readings is essentially >>>>>>> unknown until they are enumerated and thus the 'readings' field is always >>>>>>> 0. The second command not only prunes lines in the 'parse' file with >>>>>>> readings == 0, but also lines in the 'item' file which correspond to those >>>>>>> 'parse' lines. Once you have created new-profile/, you can parse again with >>>>>>> --full-forest for use with FFTB (and of course you don't have to use >>>>>>> PyDelphin for the parsing steps, if you prefer other means). >>>>>>> >>>>>>> Also note that this results in a profile with no edges for partial >>>>>>> parses. I think this is what you want. There should be a way to prune the >>>>>>> full-forest profile directly while keeping partial parses, but while >>>>>>> investigating this use case I found a bug, so I don't recommend it yet. >>>>>>> >>>>>>> Try `delphin mkprof --help` to see descriptions of these and other >>>>>>> options. They map fairly directly to the function documented here: >>>>>>> https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html >>>>>>> #mkprof >>>>>>> >>>>>>> >>>>>>> On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender >>>>>>> wrote: >>>>>>> >>>>>>>> Dear all, >>>>>>>> >>>>>>>> We are doing some treebanking here at UW with fftb with grammars >>>>>>>> that have very low coverage over their associated test corpora. The current >>>>>>>> behavior of fftb with these profiles is to include all items for >>>>>>>> treebanking, but give a 404 for each one with no parse forest stored. This >>>>>>>> necessitates clicking the back button and tracking which one is next (since >>>>>>>> nothing changes color). In that light, two questions: >>>>>>>> >>>>>>>> (1) Is there some option we can pass fftb so that it just doesn't >>>>>>>> present items with no parses? >>>>>>>> (2) Failing that, is it fairly straightforward with pydelphin, >>>>>>>> [incr tsdb()] or something else to export a version of the profiles that >>>>>>>> only includes items which the grammar successfully parsed? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Emily >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Emily M. Bender (she/her) >>>>>>>> Howard and Frances Nostrand Endowed Professor >>>>>>>> Department of Linguistics >>>>>>>> Faculty Director, CLMS >>>>>>>> University of Washington >>>>>>>> Twitter: @emilymbender >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> -Michael Wayne Goodman >>>>>>> >>>>>> -- >>>>>> Emily M. Bender (she/her) >>>>>> Howard and Frances Nostrand Endowed Professor >>>>>> Department of Linguistics >>>>>> Faculty Director, CLMS >>>>>> University of Washington >>>>>> Twitter: @emilymbender >>>>>> >>>>> >>>>> >>>>> -- >>>>> -Michael Wayne Goodman >>>>> >>>> >>>> >>>> -- >>>> Emily M. Bender (she/her) >>>> Howard and Frances Nostrand Endowed Professor >>>> Department of Linguistics >>>> Faculty Director, CLMS >>>> University of Washington >>>> Twitter: @emilymbender >>>> >>> >>> >>> -- >>> Emily M. Bender (she/her) >>> Howard and Frances Nostrand Endowed Professor >>> Department of Linguistics >>> Faculty Director, CLMS >>> University of Washington >>> Twitter: @emilymbender >>> >> >> >> -- >> -Michael Wayne Goodman >> > > > -- > Emily M. Bender (she/her) > Howard and Frances Nostrand Endowed Professor > Department of Linguistics > Faculty Director, CLMS > University of Washington > Twitter: @emilymbender > -- -Michael Wayne Goodman -------------- next part -------------- An HTML attachment was scrubbed... URL: From ebender at uw.edu Wed Jan 22 02:39:57 2020 From: ebender at uw.edu (Emily M. Bender) Date: Tue, 21 Jan 2020 17:39:57 -0800 Subject: [developers] Skipping non-parsed items in fftb In-Reply-To: References: Message-ID: Nice -- got it working! (In fact, it was a permissions error on the .dat file.) Yes, I think that the trace isn't that helpful, especially given that it functionally hid the actionable bit of information from me. Thanks for your help! Emily On Tue, Jan 21, 2020 at 5:36 PM goodman.m.w at gmail.com wrote: > Ok, that's progress. This error shows up when ACE exited abnormally > (recall that PyDelphin calls ACE in a subprocess in order to process > profiles). Since I don't capture ACE's stderr, it should be printed in the > terminal just above the stacktrace. The PyDelphin error is directing you to > look for that message to fix the problem. Most likely, the path to the > grammar image is incorrect or the grammar image was compiled with a > different version of ACE. The stacktrace and error message are both printed > to stderr so if you see one you should see both (unless ACE exited > abnormally without printing anything). > > By the way, I've recently pushed some commits to suppress the stacktrace > when encountering anticipated errors from the `delphin` command, as I don't > think the stacktrace is useful except for me (it can be shown again when > called in DEBUG mode). In addition I tried to provide more useful messages > for common situations. These changes will be part of the next release. > > On Wed, Jan 22, 2020 at 9:19 AM Emily M. Bender wrote: > >> Updating PyDelphin caused the error to change, at least: >> >> Traceback (most recent call last): >> File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in >> sys.exit(main()) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >> line 40, in main >> args.func(args) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/cli/process.py", >> line 46, in call_process >> gzip=args.gzip) >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >> line 602, in process >> with processor(grammar, cmdargs=options, **kwargs) as cpu: >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/ace.py", >> line 110, in __init__ >> self._open() >> File >> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/ace.py", >> line 143, in _open >> raise ACEProcessError('Process closed on startup; see .') >> delphin.ace.ACEProcessError: Process closed on startup; see . >> >> >> ... I'm a little puzzled as to how I got about "seeing " as I >> didn't myself do anything to redirect it somewhere else, but it's not >> printing to the console... >> >> Emily >> >> On Fri, Jan 17, 2020 at 4:17 PM goodman.m.w at gmail.com < >> goodman.m.w at gmail.com> wrote: >> >>> Hi Emily, >>> >>> Yes those error messages are not very clear. But the second one looks >>> like old code, as 'tables' is no longer a key in the object it's being >>> looked up on. I suggest making sure that your run_agg environment has an >>> updated version of PyDelphin. While the environment is active, try `pip >>> install -U pydelphin` and make sure it has a 1.0 or newer version (`delphin >>> --version`), then try again. >>> >>> On Sat, Jan 18, 2020 at 7:52 AM Emily M. Bender wrote: >>> >>>> Apologies --- that error meant I hadn't given the right path to the >>>> testsuite. Correcting that, I now see: >>>> >>>> (run_agg) ebender at patas:/home2/kphowell/run_aggregation/output/emb_treebank$ >>>> delphin process -g ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ >>>> Traceback (most recent call last): >>>> File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in >>>> sys.exit(main()) >>>> File >>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>>> line 42, in main >>>> args.func(args) >>>> File >>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>>> line 135, in call_process >>>> gzip=args.gzip) >>>> File >>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >>>> line 542, in process >>>> column, tablename, condition = _interpret_selection(select, source) >>>> File >>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >>>> line 562, in _interpret_selection >>>> if len(queryobj['tables']) == 1: >>>> KeyError: 'tables' >>>> >>>> On Fri, Jan 17, 2020 at 3:41 PM Emily M. Bender wrote: >>>> >>>>> Dear Mike, >>>>> >>>>> Alas, I'm hitting this error: >>>>> >>>>> (run_agg) ebender at patas:~$ delphin process -g >>>>> ctn1_grammar_fixed/ace/ctn1.dat ctn_orig/ >>>>> Traceback (most recent call last): >>>>> File "/home2/kphowell/Envs/run_agg/bin/delphin", line 11, in >>>>> sys.exit(main()) >>>>> File >>>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>>>> line 42, in main >>>>> args.func(args) >>>>> File >>>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/main.py", >>>>> line 135, in call_process >>>>> gzip=args.gzip) >>>>> File >>>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/commands.py", >>>>> line 540, in process >>>>> source = itsdb.TestSuite(source) >>>>> File >>>>> "/home2/kphowell/Envs/run_agg/lib/python3.6/site-packages/delphin/itsdb.py", >>>>> line 644, in __init__ >>>>> '*schema* argument is required for new test suites') >>>>> delphin.itsdb.ITSDBError: *schema* argument is required for new test >>>>> suites >>>>> >>>>> I'll poke around and see where the schema requirement is coming from >>>>> (nothing in the bit on "process" in the documentation page mentions it), >>>>> but thought I'd post here too in the meantime. >>>>> >>>>> Emily >>>>> >>>>> On Thu, Jan 16, 2020 at 6:46 PM goodman.m.w at gmail.com < >>>>> goodman.m.w at gmail.com> wrote: >>>>> >>>>>> Let me know how it goes. >>>>>> >>>>>> And a clarification: the --full option on `mkprof` doesn't hurt, but >>>>>> it's unnecessary since you're re-parsing the created profile. >>>>>> >>>>>> Also here's the bug report for the other thing, if you're interested >>>>>> in that use case: https://github.com/delph-in/pydelphin/issues/273 >>>>>> >>>>>> On Fri, Jan 17, 2020 at 10:37 AM Emily M. Bender >>>>>> wrote: >>>>>> >>>>>>> Thanks, Mike! I will give this a try. >>>>>>> >>>>>>> On Thu, Jan 16, 2020 at 6:33 PM goodman.m.w at gmail.com < >>>>>>> goodman.m.w at gmail.com> wrote: >>>>>>> >>>>>>>> Hi Emily, >>>>>>>> >>>>>>>> For (2), here is how you could do it with PyDelphin: >>>>>>>> >>>>>>>> delphin process -g grm.dat original-profile/ >>>>>>>> delphin mkprof --full --where 'readings > 0' --source >>>>>>>> original-profile/ new-profile/ >>>>>>>> delphin process -g grm.dat --full-forest new-profile/ >>>>>>>> >>>>>>>> Note that original-profile/ is first parsed in regular (non-forest) >>>>>>>> mode, because in full-forest mode the number of readings is essentially >>>>>>>> unknown until they are enumerated and thus the 'readings' field is always >>>>>>>> 0. The second command not only prunes lines in the 'parse' file with >>>>>>>> readings == 0, but also lines in the 'item' file which correspond to those >>>>>>>> 'parse' lines. Once you have created new-profile/, you can parse again with >>>>>>>> --full-forest for use with FFTB (and of course you don't have to use >>>>>>>> PyDelphin for the parsing steps, if you prefer other means). >>>>>>>> >>>>>>>> Also note that this results in a profile with no edges for partial >>>>>>>> parses. I think this is what you want. There should be a way to prune the >>>>>>>> full-forest profile directly while keeping partial parses, but while >>>>>>>> investigating this use case I found a bug, so I don't recommend it yet. >>>>>>>> >>>>>>>> Try `delphin mkprof --help` to see descriptions of these and other >>>>>>>> options. They map fairly directly to the function documented here: >>>>>>>> https://pydelphin.readthedocs.io/en/latest/api/delphin.commands.html >>>>>>>> #mkprof >>>>>>>> >>>>>>>> >>>>>>>> On Fri, Jan 17, 2020 at 8:44 AM Emily M. Bender >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Dear all, >>>>>>>>> >>>>>>>>> We are doing some treebanking here at UW with fftb with grammars >>>>>>>>> that have very low coverage over their associated test corpora. The current >>>>>>>>> behavior of fftb with these profiles is to include all items for >>>>>>>>> treebanking, but give a 404 for each one with no parse forest stored. This >>>>>>>>> necessitates clicking the back button and tracking which one is next (since >>>>>>>>> nothing changes color). In that light, two questions: >>>>>>>>> >>>>>>>>> (1) Is there some option we can pass fftb so that it just doesn't >>>>>>>>> present items with no parses? >>>>>>>>> (2) Failing that, is it fairly straightforward with pydelphin, >>>>>>>>> [incr tsdb()] or something else to export a version of the profiles that >>>>>>>>> only includes items which the grammar successfully parsed? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Emily >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Emily M. Bender (she/her) >>>>>>>>> Howard and Frances Nostrand Endowed Professor >>>>>>>>> Department of Linguistics >>>>>>>>> Faculty Director, CLMS >>>>>>>>> University of Washington >>>>>>>>> Twitter: @emilymbender >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> -Michael Wayne Goodman >>>>>>>> >>>>>>> -- >>>>>>> Emily M. Bender (she/her) >>>>>>> Howard and Frances Nostrand Endowed Professor >>>>>>> Department of Linguistics >>>>>>> Faculty Director, CLMS >>>>>>> University of Washington >>>>>>> Twitter: @emilymbender >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> -Michael Wayne Goodman >>>>>> >>>>> >>>>> >>>>> -- >>>>> Emily M. Bender (she/her) >>>>> Howard and Frances Nostrand Endowed Professor >>>>> Department of Linguistics >>>>> Faculty Director, CLMS >>>>> University of Washington >>>>> Twitter: @emilymbender >>>>> >>>> >>>> >>>> -- >>>> Emily M. Bender (she/her) >>>> Howard and Frances Nostrand Endowed Professor >>>> Department of Linguistics >>>> Faculty Director, CLMS >>>> University of Washington >>>> Twitter: @emilymbender >>>> >>> >>> >>> -- >>> -Michael Wayne Goodman >>> >> >> >> -- >> Emily M. Bender (she/her) >> Howard and Frances Nostrand Endowed Professor >> Department of Linguistics >> Faculty Director, CLMS >> University of Washington >> Twitter: @emilymbender >> > > > -- > -Michael Wayne Goodman > -- Emily M. Bender (she/her) Howard and Frances Nostrand Endowed Professor Department of Linguistics Faculty Director, CLMS University of Washington Twitter: @emilymbender -------------- next part -------------- An HTML attachment was scrubbed... URL: From oe at ifi.uio.no Mon Jan 27 18:42:15 2020 From: oe at ifi.uio.no (Stephan Oepen) Date: Mon, 27 Jan 2020 18:42:15 +0100 Subject: [developers] EDM implementations In-Reply-To: References: Message-ID: hi mike, belatedly, thanks (once again) for pushing forward standardization! and also my apologies for returning to this thread a little late! regarding EDM, i used to think of the Common-Lisp implementation (which it appears i produced in early 2012, i.e. more recently than the Perl version by bec) as the reference until recently. last year, when comparing its scores to my re-implementation in Python as part of mtool, that comparison also turned up the two questions you raised, viz. the treatment of the TOP property and how to score parameterized predicates. regarding the first, this appears to be one of the better-kept secrets in meaning representation comparison: in my view, it is a semantically highly relevant property (marking the contrast between e.g. 'all fierce dogs bark' vs. 'all barking dogs are fierce'), but neither the original EDM paper nor its derivative in the AMR world (Cai & Knight, 2013) discuss it. yet, both the Lisp implementation of EDM and SMATCH seem to always have scored the TOP node as an additional tuple (counted among the 'argument' tuples for EDM, while considered among the 'attribute' tuples in SMATCH). the Perl implementation of EDM, on the other hand, worked off my 'ltriples' export format for EDS, which appears to not include a separate TOP tuple. i confirmed the nature of those triples by reminding myself of what became of the 'export' script mentioned in the original EDM wiki notes you had found. it was folded into the LOGON 'redwoods' script, so something like the following actually works today to prepare the input for the Perl implementation of EDM: $LOGONROOT/redwoods --erg --export ltriples --target /tmp mrs i attach the output for item #21 from the MRS test suite, for reference. so, i agree with the conclusion bec and you have already reached: the original Perl implementation of EDM did not consider TOP tuples. the Lisp implementation, on the other hand, appears to have had TOP tuples from its very beginning. regarding the second design choice you raise, parameterized relations (involving one or more constant arguments), it appears that both the Lisp and Perl implementations of EDM do the same thing, viz. assume that there can be at most one constant argument in a relation and 'inline' its value (if present) with the predicate itself, e.g. internally using node label shorthands like 'named(Abrams)'. in this regard, i suspect bec and you actually may have arrived at the wrong conclusion about historic behavior; thus, personally, i see no reason for pyDelphin to provide a special-cased version of EDM that wholly ignores constant arguments. looking at this particular design choice today, however, it seems too limiting an assumption and meshing together two things that arguably should be considered separate. even though ERG versions for the past 15 or more years have not used predicates with multiple (constant) parameters, there would be nothing wrong with representing, say, the fraction '2/3' as involving two constant arguments, e.g. something like fraction [ CARG1 "2", CARG2 "3" ]. this is, for example, what AMR does for complex proper names. thus, even though our two historic EDM implementations appear to agree on the 'inlining' treatment of constant arguments, i would be prepared to argue that CARG et al. values should rather be treated as separate node properties, i.e. for the above example the 'named' predicate and the 'CARG' == 'Abrams' value should be treated as two distinct tuples. in part for cross-framework compatibility, this is what we ended up doing in mtool, including in its re-implementation of EDM, see: http://mrp.nlpl.eu/index.php?page=5 in summary, it sounds as if your EDM re-implementation, mike, had arrived at the same conclusions: TOP tuples should be scored, and constant arguments considered as separate properties. i would expect your implementation and mtool should then come to the exact same results (on EDSs stripped of MRS variable properties, which the current mtool EDS reader deliberately discards; see below)? seeing as we have identified two ways in which this way of computing EDM differs from the original publication and the two earlier implementations (in Perl and Lisp), i would like to suggest we formally coin this refinement of the metric EDM 2.0. regarding how to deal with missing graphs on either the gold or system side of the comparison: it appears the Lisp implementation of EDM provides a toggle *redwooods-score-all-p*, which selects between two modes of computing EDM over two sets of corresponding items, either on the intersection of items only; or on their union, treating gaps on either side of the comparison as empty graphs (thus, incurring recall or precision penalties). in practice, i believe we used to near-exclusively compute EDM over sets of items for which there was both a gold and a system graph. but that can of course only give comparable results when fixing that very set of items. thus, the setup of scoring 'all' items seems more general, robust to attempts at gaming, and in my view should be considered the default. finally, regarding variable properties in mtool: for the 2019 CoNLL shared task on meaning representation parsing (MRP 2019), we had agreed with other framework developers to keep morpho-semantic decorations out of the comparison. hence, the MRP 2019 graphs did not include tense, aspect, or number information from the full ERSs. but technically, i would consider that a property of the EDS used in MRP 2019, not a design decision in mtool. for the re-run of the MRP task at CoNLL 2020, we are currently preparing to throw these properties back into the mix (also in other frameworks, where annotations are available), which means the EDS reader in mtool in the near future will no longer discard (underlying) variable properties by default. best wishes, oe On Mon, Jan 20, 2020 at 2:15 AM goodman.m.w at gmail.com wrote: > > Thanks again, Bec. > > I just want to make sure my implementation gets the same scores for the same inputs under the same assumptions as the original implementation. For this to work, its behavior concerning the points I've sought clarification for should be intentional. In light of your responses, I've separated the CARG triples from other properties and have given it its own weight. Thus I should be able to get the same scores as your code by setting the weights of CARGs (but not properties) and graph-tops to zero. Similarly, I'll add an option to ignore missing test items and otherwise treat them as mismatches. > > On Fri, Jan 17, 2020 at 6:14 PM Bec Dridan wrote: >> >> >> >> On Fri, Jan 17, 2020 at 5:39 PM goodman.m.w at gmail.com wrote: >>> >>> >>> One more detail is what to do when the two sides (gold and test) have different numbers of items. Currently my code stops as soon as either a gold or test item is missing, which is what smatch (the similar metric made for AMR) does, but I think that may be wrong because parsing profiles are likely to have missing or extra (overgeneration) items in the middle. So the question is whether we ignore it or count it as a full mismatch. >> >> >> If you are asking what is 'correct', I guess that depends on why you are evaluating. The perl implementation wouldn't have noticed missing gold parses, because it used the gold set as the definition of the set. A missing test item, on the other hand, by default counts as a full mismatch, but there is a command line option to ignore any gold parse with no corresponding test parse. The ignore option is useful when the purpose of the evaluation is assessing the system you are working on (and you consider coverage separately). For comparing across systems, I imagine you probably want to count parse failure as a full mismatch. It was useful for me to have both options. >> >> Bec >> >>> >>> >>> On Thu, Jan 16, 2020 at 6:33 PM Bec Dridan wrote: >>>> >>>> Wow, that is some old code... From memory, export was a wrapper around `parse --export`, where I could add :ltriples to the tsdb::*redwoods-export-values* set. >>>> >>>> I don't know the mtool code at all, but re-reading the paper and looking at the perl code, I don't think the original implementation evaluated CARG at all. We only checked that the correct character span had a pred name of`named`. >>>> >>>> I think you are right that the triple export at the time did not produce a triple for TOP and it hence would not have been counted. >>>> >>>> That match your memory Stephan? >>>> >>>> Bec >>>> >>>> >>>> On Thu, Jan 16, 2020 at 8:34 PM goodman.m.w at gmail.com wrote: >>>>> >>>>> Hello developers, >>>>> >>>>> Recently I wanted to try out Elementary Dependency Match (EDM) but I did not find an easy way to do it. I saw lisp code in the LKB's repository and Bec's Perl code, but I'm not sure how to call the former from the command line and the latter seems outdated (I don't see the "export" command required by its instructions). >>>>> >>>>> The Dridan & Oepen, 2011 algorithm was simple enough so I though I'd implement it on top of PyDelphin. The result is here: https://github.com/delph-in/delphin.edm. It requires the latest version of PyDelphin (v1.2.0). It works with MRS, EDS, and DMRS, and it reads text files or [incr tsdb()] profiles. >>>>> >>>>> When I nearly had my version working I found that Stephan et al.'s mtool (https://github.com/cfmrpThe paper example >>>>> /mtool) also had an implementation of EDM, so I used that to compare with my outputs (as I couldn't get the previous implementations to work). In this process I think I found some differences from Dridan & Oepen, 2011's description, and this email is to confirm those findings. Namely, that mtool's (and now my) implementation do the following: >>>>> >>>>> * CARGs are treated as property triples ("class 3 information"). Previously they were combined with the predicate name. This change means that predicates like 'named' will match even if their CARGs don't and the CARGs are a separate thing that needs to be matched. >>>>> >>>>> * The identification of the graph's TOP counts as a triple. >>>>> >>>>> One difference between mtool and delphin.edm is that mtool does not count "variable" properties from EDS, but that's just because its EDS parser does not yet handle them while PyDelphin's does. >>>>> >>>>> Can anyone familiar with EDM confirm the above? Or can anyone explain how to call the Perl or LKB code so I can compare? >>>>> >>>>> -- >>>>> -Michael Wayne Goodman >>> >>> >>> >>> -- >>> -Michael Wayne Goodman > > > > -- > -Michael Wayne Goodman -------------- next part -------------- A non-text attachment was scrubbed... Name: 21.gz Type: application/gzip Size: 271 bytes Desc: not available URL: From goodman.m.w at gmail.com Tue Jan 28 02:57:27 2020 From: goodman.m.w at gmail.com (goodman.m.w at gmail.com) Date: Tue, 28 Jan 2020 09:57:27 +0800 Subject: [developers] EDM implementations In-Reply-To: References: Message-ID: Thanks for the reply, Stephan, > [...] it appears that both the > Lisp and Perl implementations of EDM do the same thing, viz. assume > that there can be at most one constant argument in a relation and > 'inline' its value (if present) with the predicate itself, e.g. > internally using node label shorthands like 'named(Abrams)'. in this > regard, i suspect bec and you actually may have arrived at the wrong > conclusion about historic behavior; Thanks for confirming how the Lisp implementation works. I took your 21.gz file and created a version that replaced "Abrams" with "Brown", then used edm_eval.pl to compare; it reports a full match (1.0), so based on this limited test I think Bec was correct about the Perl version. > thus, personally, i see no reason > for pyDelphin to provide a special-cased version of EDM that wholly > ignores constant arguments. Me too, and that's not the case. I separated CARGs into their own category and callers of the script can give the category a weight of zero to ignore them, which allows them to recreate the results of the Perl implementation. Otherwise, the default weight for all categories (arguments (-A), names/predicates (-N), morphosemantic properties (-P), constants (-C), and tops (-T)) is 1.0. > i would expect > your implementation and mtool should then come to the exact same > results (on EDSs stripped of MRS variable properties, [...]) Yes, but there's no need to strip the properties; just give the category a weight of zero. I've confirmed on a few test items that my implementation gets the exact same scores as mtool with -P0. Furthermore, I think the following option configurations for my re-implementation cover all current and historical use cases except for the inlined constants of the Lisp version, which interact with node names in a way that isn't reproducible with weights alone. * Perl: `delphin edm -C0 -T0 --ignore-missing=gold` * Perl with -i option: `delphin edm -C0 -T0 --ignore-missing=both` * Lisp where *redwooods-score-all-p* is true: `delphin edm` * Lisp where *redwooods-score-all-p* is false: `delphin edm --ignore-missing=both` * mtool (MRP 2019): `delphin edm -P0` * mtool (MRP 2020? or EDM 2.0): `delphin edm` On Tue, Jan 28, 2020 at 1:42 AM Stephan Oepen wrote: > hi mike, > > belatedly, thanks (once again) for pushing forward standardization! > and also my apologies for returning to this thread a little late! > > regarding EDM, i used to think of the Common-Lisp implementation > (which it appears i produced in early 2012, i.e. more recently than > the Perl version by bec) as the reference until recently. last year, > when comparing its scores to my re-implementation in Python as part of > mtool, that comparison also turned up the two questions you raised, > viz. the treatment of the TOP property and how to score parameterized > predicates. > > regarding the first, this appears to be one of the better-kept secrets > in meaning representation comparison: in my view, it is a semantically > highly relevant property (marking the contrast between e.g. 'all > fierce dogs bark' vs. 'all barking dogs are fierce'), but neither the > original EDM paper nor its derivative in the AMR world (Cai & Knight, > 2013) discuss it. yet, both the Lisp implementation of EDM and SMATCH > seem to always have scored the TOP node as an additional tuple > (counted among the 'argument' tuples for EDM, while considered among > the 'attribute' tuples in SMATCH). the Perl implementation of EDM, on > the other hand, worked off my 'ltriples' export format for EDS, which > appears to not include a separate TOP tuple. > > i confirmed the nature of those triples by reminding myself of what > became of the 'export' script mentioned in the original EDM wiki notes > you had found. it was folded into the LOGON 'redwoods' script, so > something like the following actually works today to prepare the input > for the Perl implementation of EDM: > > $LOGONROOT/redwoods --erg --export ltriples --target /tmp mrs > > i attach the output for item #21 from the MRS test suite, for > reference. so, i agree with the conclusion bec and you have already > reached: the original Perl implementation of EDM did not consider TOP > tuples. the Lisp implementation, on the other hand, appears to have > had TOP tuples from its very beginning. > > regarding the second design choice you raise, parameterized relations > (involving one or more constant arguments), it appears that both the > Lisp and Perl implementations of EDM do the same thing, viz. assume > that there can be at most one constant argument in a relation and > 'inline' its value (if present) with the predicate itself, e.g. > internally using node label shorthands like 'named(Abrams)'. in this > regard, i suspect bec and you actually may have arrived at the wrong > conclusion about historic behavior; thus, personally, i see no reason > for pyDelphin to provide a special-cased version of EDM that wholly > ignores constant arguments. > > looking at this particular design choice today, however, it seems too > limiting an assumption and meshing together two things that arguably > should be considered separate. even though ERG versions for the past > 15 or more years have not used predicates with multiple (constant) > parameters, there would be nothing wrong with representing, say, the > fraction '2/3' as involving two constant arguments, e.g. something > like fraction [ CARG1 "2", CARG2 "3" ]. this is, for example, what > AMR does for complex proper names. > > thus, even though our two historic EDM implementations appear to agree > on the 'inlining' treatment of constant arguments, i would be prepared > to argue that CARG et al. values should rather be treated as separate > node properties, i.e. for the above example the 'named' predicate and > the 'CARG' == 'Abrams' value should be treated as two distinct tuples. > in part for cross-framework compatibility, this is what we ended up > doing in mtool, including in its re-implementation of EDM, see: > > http://mrp.nlpl.eu/index.php?page=5 > > in summary, it sounds as if your EDM re-implementation, mike, had > arrived at the same conclusions: TOP tuples should be scored, and > constant arguments considered as separate properties. i would expect > your implementation and mtool should then come to the exact same > results (on EDSs stripped of MRS variable properties, which the > current mtool EDS reader deliberately discards; see below)? seeing as > we have identified two ways in which this way of computing EDM differs > from the original publication and the two earlier implementations (in > Perl and Lisp), i would like to suggest we formally coin this > refinement of the metric EDM 2.0. > > regarding how to deal with missing graphs on either the gold or system > side of the comparison: it appears the Lisp implementation of EDM > provides a toggle *redwooods-score-all-p*, which selects between two > modes of computing EDM over two sets of corresponding items, either on > the intersection of items only; or on their union, treating gaps on > either side of the comparison as empty graphs (thus, incurring recall > or precision penalties). in practice, i believe we used to > near-exclusively compute EDM over sets of items for which there was > both a gold and a system graph. but that can of course only give > comparable results when fixing that very set of items. thus, the > setup of scoring 'all' items seems more general, robust to attempts at > gaming, and in my view should be considered the default. > > finally, regarding variable properties in mtool: for the 2019 CoNLL > shared task on meaning representation parsing (MRP 2019), we had > agreed with other framework developers to keep morpho-semantic > decorations out of the comparison. hence, the MRP 2019 graphs did not > include tense, aspect, or number information from the full ERSs. but > technically, i would consider that a property of the EDS used in MRP > 2019, not a design decision in mtool. for the re-run of the MRP task > at CoNLL 2020, we are currently preparing to throw these properties > back into the mix (also in other frameworks, where annotations are > available), which means the EDS reader in mtool in the near future > will no longer discard (underlying) variable properties by default. > > best wishes, oe > > > > > > On Mon, Jan 20, 2020 at 2:15 AM goodman.m.w at gmail.com > wrote: > > > > Thanks again, Bec. > > > > I just want to make sure my implementation gets the same scores for the > same inputs under the same assumptions as the original implementation. For > this to work, its behavior concerning the points I've sought clarification > for should be intentional. In light of your responses, I've separated the > CARG triples from other properties and have given it its own weight. Thus I > should be able to get the same scores as your code by setting the weights > of CARGs (but not properties) and graph-tops to zero. Similarly, I'll add > an option to ignore missing test items and otherwise treat them as > mismatches. > > > > On Fri, Jan 17, 2020 at 6:14 PM Bec Dridan wrote: > >> > >> > >> > >> On Fri, Jan 17, 2020 at 5:39 PM goodman.m.w at gmail.com < > goodman.m.w at gmail.com> wrote: > >>> > >>> > >>> One more detail is what to do when the two sides (gold and test) have > different numbers of items. Currently my code stops as soon as either a > gold or test item is missing, which is what smatch (the similar metric made > for AMR) does, but I think that may be wrong because parsing profiles are > likely to have missing or extra (overgeneration) items in the middle. So > the question is whether we ignore it or count it as a full mismatch. > >> > >> > >> If you are asking what is 'correct', I guess that depends on why you > are evaluating. The perl implementation wouldn't have noticed missing gold > parses, because it used the gold set as the definition of the set. A > missing test item, on the other hand, by default counts as a full mismatch, > but there is a command line option to ignore any gold parse with no > corresponding test parse. The ignore option is useful when the purpose of > the evaluation is assessing the system you are working on (and you consider > coverage separately). For comparing across systems, I imagine you probably > want to count parse failure as a full mismatch. It was useful for me to > have both options. > >> > >> Bec > >> > >>> > >>> > >>> On Thu, Jan 16, 2020 at 6:33 PM Bec Dridan > wrote: > >>>> > >>>> Wow, that is some old code... From memory, export was a wrapper > around `parse --export`, where I could add :ltriples to the > tsdb::*redwoods-export-values* set. > >>>> > >>>> I don't know the mtool code at all, but re-reading the paper and > looking at the perl code, I don't think the original implementation > evaluated CARG at all. We only checked that the correct character span had > a pred name of`named`. > >>>> > >>>> I think you are right that the triple export at the time did not > produce a triple for TOP and it hence would not have been counted. > >>>> > >>>> That match your memory Stephan? > >>>> > >>>> Bec > >>>> > >>>> > >>>> On Thu, Jan 16, 2020 at 8:34 PM goodman.m.w at gmail.com < > goodman.m.w at gmail.com> wrote: > >>>>> > >>>>> Hello developers, > >>>>> > >>>>> Recently I wanted to try out Elementary Dependency Match (EDM) but I > did not find an easy way to do it. I saw lisp code in the LKB's repository > and Bec's Perl code, but I'm not sure how to call the former from the > command line and the latter seems outdated (I don't see the "export" > command required by its instructions). > >>>>> > >>>>> The Dridan & Oepen, 2011 algorithm was simple enough so I though I'd > implement it on top of PyDelphin. The result is here: > https://github.com/delph-in/delphin.edm. It requires the latest version > of PyDelphin (v1.2.0). It works with MRS, EDS, and DMRS, and it reads text > files or [incr tsdb()] profiles. > >>>>> > >>>>> When I nearly had my version working I found that Stephan et al.'s > mtool (https://github.com/cfmrpThe paper example > >>>>> /mtool) also had an implementation of EDM, so I used that to compare > with my outputs (as I couldn't get the previous implementations to work). > In this process I think I found some differences from Dridan & Oepen, > 2011's description, and this email is to confirm those findings. Namely, > that mtool's (and now my) implementation do the following: > >>>>> > >>>>> * CARGs are treated as property triples ("class 3 information"). > Previously they were combined with the predicate name. This change means > that predicates like 'named' will match even if their CARGs don't and the > CARGs are a separate thing that needs to be matched. > >>>>> > >>>>> * The identification of the graph's TOP counts as a triple. > >>>>> > >>>>> One difference between mtool and delphin.edm is that mtool does not > count "variable" properties from EDS, but that's just because its EDS > parser does not yet handle them while PyDelphin's does. > >>>>> > >>>>> Can anyone familiar with EDM confirm the above? Or can anyone > explain how to call the Perl or LKB code so I can compare? > >>>>> > >>>>> -- > >>>>> -Michael Wayne Goodman > >>> > >>> > >>> > >>> -- > >>> -Michael Wayne Goodman > > > > > > > > -- > > -Michael Wayne Goodman > -- -Michael Wayne Goodman -------------- next part -------------- An HTML attachment was scrubbed... URL: