<div dir="ltr">Thank you Francis, Olga and Woodley! With these additions I was able to parse the unknown word "baac" and the inflected form of the unknown root "ha-baac". </div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 30, 2018 at 11:01 PM, Woodley Packard <span dir="ltr"><<a href="mailto:sweaglesw@sweaglesw.org" target="_blank">sweaglesw@sweaglesw.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Hi Olga and Kristin,<div><br></div><div>You were close. As Francis mentioned, you need to define some generic lexical entries. You managed to declare types for generic lexical entries, but not the entries themselves. Add the following to abz-pet.tdl, near the main lexicon section:</div><div><br></div><div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">:begin :instance :status generic-lex-entry.</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">:include "generic-lexicon".</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">:end :instance.</span></div></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">and then create the generic-lexicon.tdl file containing a single statement:</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures"><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">generic-verb := generic_verb_lex_etry & [ STEM < string > ].</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div></span></div><div>With those changes, I was able to successfully parse using the YY lattice Kristin gave for "baac". I noticed a message about a loopy optional complement rule, so the generic verb lexical type may be a bit too underspecified, e.g. regarding its opinion about its valence (in addition to being shy one ’N’).</div><div><br></div><div>I apologize for not getting back to you about this quicker; somehow I missed Olga’s March 23rd email (though I see it in my mailbox now when I go back and look).</div><div><br></div><div>Good luck, and let me know if you run into more trouble!</div><span class="HOEnZb"><font color="#888888"><div>Woodley</div></font></span><div><div class="h5"><div><br><div><blockquote type="cite"><div>On Mar 30, 2018, at 7:35 PM, Olga Zamaraeva <<a href="mailto:olzama@uw.edu" target="_blank">olzama@uw.edu</a>> wrote:</div><br class="m_-1017746772779000886Apple-interchange-newline"><div><div dir="ltr">Here are some relevant types. When I started working on it, I mostly copied things which seemed relevant, over from ERG. (I see now that the #pred identity is present twice).<div><br></div><div><div><div>generic_verb_lex_etry := unknown_word & basic-verb-lex &</div><div> [ SYNSEM.LKEYS.KEYREL.PRED #pred,</div><div> ORTH < "_generic_vb_" >,</div><div> TOKENS.+LIST < [ +TNT.+TAGS.FIRST "VB", +PRED #pred ] > ].</div><div><br></div></div><div>unknown_word := norm_unknown_word.<br></div><div><br></div><div><div>norm_unknown_word := basic_unknown_word &</div><div> [ SYNSEM [ LOCAL.CONT.HOOK.LTOP #ltop,</div><div> LKEYS.KEYREL [ LBL #ltop,</div><div><span style="white-space:pre-wrap"> </span> PRED #pred ] ],</div><div> TOKENS.+LIST.FIRST.+PRED #pred ].</div></div><div><br></div><div><div>basic_unknown_word := basic_generic_lex_entry.</div><br class="m_-1017746772779000886inbox-inbox-Apple-interchange-newline"></div><div><div>generic_lex_entry := basic_generic_lex_entry &</div><div> [ TOKENS.+LIST < [ +TNT null_tnt ] > ].</div></div><div><br></div><div><div>basic_generic_lex_entry := word &</div><div> [ SYNSEM.PHON.ONSET unk_onset ].</div></div><div><br></div><div><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Mar 30, 2018 at 7:18 PM Francis Bond <<a href="mailto:bond@ieee.org" target="_blank">bond@ieee.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">G'day,<div><br></div><div>if you run ace in a more verbose mode (I think -vv should be enough) it tells you a bit more about what it is doing with the tokens.</div><div><br></div><div>In addition to yy-mode, you must also have some generic lexical entries for unknown words. </div><div><br></div><div>You can find some nice examples by Sanghoun in: <a href="https://github.com/delph-in/zhong/blob/master/cmn/gle.tdl" target="_blank">https://github.com/delph-<wbr>in/zhong/blob/master/cmn/gle.<wbr>tdl</a></div><div>(I think easier to follow than Jacy).</div><div><br></div><div>Can you show the lexical type you want to instantiate?</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Mar 31, 2018 at 2:36 AM, Kristen Howell <span dir="ltr"><<a href="mailto:kphowell@uw.edu" target="_blank">kphowell@uw.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm picking this up for Olga. I've followed the same steps and am encountering the same issue, where I can parse a known word in YY mode, but not an unkown word. I've attached the toy grammar we are using. If anyone has insight on what we are missing, we'd appreciate it. Here is an example, where "baab" is a known word and "baac" is not.<div><br></div><div><div>[kphowell@patas ace-0.9.26]$ ./ace -g ../aggregation/analyses/<wbr>unknown-roots-morphology/data/<wbr>abz-modified/ace/abz.dat -y</div><div>(42, 0, 1, <0:4>, 1, "baab", 0, "null", "VB" 1.0)</div><span><div>SENT: (yy mode)</div><div>[ LTOP: h0 INDEX: e2 [ e SF: prop-or-ques E.TENSE: tense E.ASPECT: aspect E.MOOD: mood ] RELS: < [ "_strike.pfv_v_rel"<-1:-1> LBL: h1 ARG0: e2 ARG1: x3 [ x SPECI: bool COG-ST: in-foc PNG.PER: person PNG.NUM: number PNG.GEND: gender ] ARG2: x4 [ x SPECI: bool COG-ST: cog-st PNG.PER: person PNG.NUM: number PNG.GEND: gender ] ] > HCONS: < h0 qeq h1 > ICONS: < e2 non-focus x4 e2 non-focus x3 > ] ; (10 decl-head-opt-subj 0.000000 0 1 (9 basic-head-opt-comp 0.000000 0 1 (2 baab 0.000000 0 1 ("baab" 1 "token [ +FORM \"baab\" +FROM \"0\" +TO \"4\" +ID diff-list [ LIST list LAST list ] +TNT tnt [ +TAGS cons [ FIRST \"VB\" REST null ] +PRBS cons [ FIRST \"1.000000\" REST null ] +MAIN tnt_main [ +TAG string +PRB string ] ] +CLASS token_class +TRAIT token_trait [ +UW bool +IT italics +LB bracket_list +RB bracket_list +HD token_head [ +LL ctype [ -CTYPE- string ] +TG string +TI string ] ] +PRED predsort +CARG string +TICK bool ]"))))</div><div>NOTE: 1 readings, added 6 / 2 edges to chart (3 fully instantiated, 2 actives used, 2 passives used) RAM: 41k</div><div><br></div><div><br></div></span><div>(42, 0, 1, <0:4>, 1, "baac", 0, "null", "VB" 1.0)</div><div>NOTE: lexemes do not span position 0 `baac'!</div><span><div>NOTE: post reduction gap</div><div>SKIP: (yy mode)</div><div><br></div></span><div>Best,</div><div>Kristen</div></div></div><div class="m_-1017746772779000886m_-2376498202399916878HOEnZb"><div class="m_-1017746772779000886m_-2376498202399916878h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 23, 2018 at 1:55 PM, Olga Zamaraeva <span dir="ltr"><<a href="mailto:olzama@uw.edu" target="_blank">olzama@uw.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">OK, I can run ACE in yy mode and I've attempted to enable token mapping and to map tags to generic entries, but apparently I am missing some step(s).<div><br></div><div><b>On an existing word, it works:</b></div><div><br></div><div><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-s1">$cat ../../yy.txt | ace -g abz.dat -y</span></p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-s1">SENT: (yy mode)</span></p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-s1">[ LTOP: h0 INDEX: e2 [ e SF: prop-or-ques E.TENSE: tense E.ASPECT: aspect E.MOOD: mood ] RELS: < [ "_strike.pfv_v_rel"<-1:-1> LBL: h1 ARG0: e2 ARG1: x3 [ x SPECI: bool COG-ST: in-foc PNG.PER: person PNG.NUM: number PNG.GEND: gender ] ARG2: x4 [ x SPECI: bool COG-ST: cog-st PNG.PER: person PNG.NUM: number PNG.GEND: gender ] ] > HCONS: < h0 qeq h1 > ICONS: < e2 non-focus x4 e2 non-focus x3 > ] ;<span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-Apple-converted-space"> </span>(10 decl-head-opt-subj 0.000000 0 1 (9 basic-head-opt-comp 0.000000 0 1 (2 baab 0.000000 0 1 ("baab" 1 "token [ +FORM \"baab\" +FROM \"0\" +TO \"4\" +ID diff-list [ LIST list LAST list ] <b>+TNT tnt [ +TAGS cons [ FIRST \"VB\" REST null ]</b> +PRBS cons [ FIRST \"1.000000\" REST null ] +MAIN tnt_main [ +TAG string +PRB string ] ] +CLASS token_class +TRAIT token_trait [ +UW bool +IT italics +LB bracket_list +RB bracket_list +HD token_head [ +LL ctype [ -CTYPE- string ] +TG string +TI string ] ] +PRED predsort +CARG string +TICK bool ]"))))</span></p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-s1">NOTE: 1 readings, added 6 / 2 edges to chart (3 fully instantiated, 2 actives used, 2 passives used)<span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-Apple-tab-span"> </span>RAM: 41k</span></p><div><b>But on an unknown word it does not still:</b></div><div><br></div><div> ace Murka$ cat ../../yy.txt | ace -g abz.dat -y</div><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-s1">NOTE: lexemes do not span position 0 `baabb'!</span></p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-s1">NOTE: post reduction gap</span></p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-s1">SKIP: (yy mode)</span></p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><b>Does anyone have an idea what I have likely failed to define/enable?</b></p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1"><b>I've defined token paths like in the ERG, because that's where I copied other types from:</b></p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">token-mapping := enabled.</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">lexicon-tokens-path := TOKENS +LIST.</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">lexicon-last-token-path := TOKENS +LAST.</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">token-type<span style="white-space:pre-wrap"> </span>:= token.</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">token-form-path := +FORM. ; [required] string for lexical lookup</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">token-id-path := +ID. ; [optional] list of external ids</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">token-from-path := +FROM. ; [optional] surface start position</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">token-to-path := +TO. ; [optional] surface end position</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">token-postags-path := +TNT +TAGS. ; [optional] list of POS tags</p><p class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833inbox-inbox-p1">token-posprobs-path := +TNT +PRBS. ; [optional] list of POS probabilities</p></div><div><b>Thank you,</b></div><div><b>Olga</b></div></div><div class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377HOEnZb"><div class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377h5"><br><div class="gmail_quote"><div dir="ltr">On Fri, Mar 23, 2018 at 10:38 AM Olga Zamaraeva <<a href="mailto:olzama@uw.edu" target="_blank">olzama@uw.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks very much, Paul, Woodley, and Michael. Michael, thanks especially for the detailed explanation!<div><br></div><div>I did not notice that YY mode has a field for a POS tag. I will try that then.</div><div><br></div><div>Best,</div><div>Olga</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Mar 22, 2018 at 4:11 PM Michael Wayne Goodman <<a href="mailto:goodmami@uw.edu" target="_blank">goodmami@uw.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Following Woodley's suggestion, for YY-mode I can point you to a few things.<br><br></div>In Jacy, we use POS tags from an external morphological analyzer (previously Chasen; recently MeCab). We have a script that takes the output of MeCab and transforms it into the YY format. Note the definition of the pos_info variable---it holds POS data that is slightly more complex than a simple, e.g., NNS or VBG tag.<br><br> <a href="https://github.com/delph-in/jacy/blob/develop/utils/jpn2yy" target="_blank">https://github.com/delph-in/<wbr>jacy/blob/develop/utils/jpn2yy</a><br><br>Then see gle.tdl in Jacy, which maps the POS "tags" to generic lexical entries:<br><br> <a href="https://github.com/delph-in/jacy/blob/develop/gle.tdl" target="_blank">https://github.com/delph-in/<wbr>jacy/blob/develop/gle.tdl</a>.<br><br>For ACE (and presumably other processors) you might also need to define paths to the token info:<br><br> <a href="https://github.com/delph-in/jacy/blob/develop/ace/config.tdl#L143-L151" target="_blank">https://github.com/delph-in/<wbr>jacy/blob/develop/ace/config.<wbr>tdl#L143-L151</a><br></div><br></div>When you call ACE you'll need to tell it to expect YY input. I think it's the -y option. There might be some other pieces to this that Woodley or Francis can probably fill in for you. In my experiments, YY mode did help a bit for getting parses where the standard machinery for unknowns failed.<br></div><br></div>If you're working in Python, then PyDelphin's 'tokens' module can help with constructing YY input. This section of the relevant unit tests might be informative:<br><br> <a href="https://github.com/delph-in/pydelphin/blob/develop/tests/tokens_test.py#L40-L59" target="_blank">https://github.com/delph-in/<wbr>pydelphin/blob/develop/tests/<wbr>tokens_test.py#L40-L59</a><br></div></div><div class="gmail_extra"></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 22, 2018 at 3:40 PM, Woodley Packard <span dir="ltr"><<a href="mailto:sweaglesw@sweaglesw.org" target="_blank">sweaglesw@sweaglesw.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div></div><div>Hi Olga,</div><div><br></div><div>Since you are interested primarily in a demonstration rather than a real world system from what I understand, why not specify the POS tags as part of the input, using YY mode?</div><span class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833m_-2348585205083705892m_6731330848249017968HOEnZb"><font color="#888888"><div><br></div><div>Woodley</div></font></span><div><div class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833m_-2348585205083705892m_6731330848249017968h5"><div><br>On Mar 22, 2018, at 11:42 AM, Olga Zamaraeva <<a href="mailto:olzama@uw.edu" target="_blank">olzama@uw.edu</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">Dear developers!<div><div><br></div><div>I am looking into the problem of handling unknown roots with LKB and ACE in a situation where we want to first be able to analyze the word morphologically (apply lexical rules). </div><div><br></div><div>I had already sent an email about that a year ago, and Francis and I actually sat down and went through the process of constructing a minimal example which showed that there was a problem of some sort preventing us from analyzing the word morphologically and using the unknown word handling machinery at the same time.</div><div><br></div><div>Alas, I cannot recover any record of this. It is possible that we did that on Francis's computer,...</div><div><br></div><div>Anyway, I want to reconstruct this minimal example one more time, this time hopefully understanding more and producing some actual documentation.</div><div><br></div><div>I would like to start from recreating what e.g. the ERG does: treating the words as full-form, relying on a POS tag which maps the word to a specific unknown_type.</div><div><br></div><div>I have a small grammar to which I added what I was able to detect as relevant in the ERG (generic lexical entries, unknown onset etc). I also included mtr.tdl and I included it into the script.</div><div><br></div><div>Next thing I need to understand (I think) is what does it actually mean to "mock the POS tagger". How do I make the system aware of that information? </div></div><div><br></div><div>I can see that the tags can be mapped to the generic lexical entries as described in <a href="http://moin.delph-in.net/PetInput" target="_blank">http://moin.delph-in.net/<wbr>PetInput</a>. But how do I get the tags in the first place? Suppose I just want to consider everything the same POS, for starters. </div><div><br></div><div>Thank you!</div><div>Olga</div></div> </div></blockquote></div></div></div></blockquote></div><br><br clear="all"><br></div><div class="gmail_extra">-- <br><div class="m_-1017746772779000886m_-2376498202399916878m_8786428506328098377m_-5241082563590051833m_-2348585205083705892m_6731330848249017968gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Michael Wayne Goodman<div>Ph.D. Candidate, UW Linguistics</div></div></div> </div></blockquote></div></blockquote></div> </div></div></blockquote></div><br></div> </div></div></blockquote></div><br><br clear="all"><div><br></div></div><div class="gmail_extra">-- <br><div class="m_-1017746772779000886m_-2376498202399916878gmail_signature" data-smartmail="gmail_signature">Francis Bond <<a href="http://www3.ntu.edu.sg/home/fcbond/" target="_blank">http://www3.ntu.edu.sg/home/<wbr>fcbond/</a>><br>Division of Linguistics and Multilingual Studies<br>Nanyang Technological University<br></div> </div></blockquote></div></div></div></div> </div></blockquote></div><br></div></div></div></div></blockquote></div><br></div>