<div dir="ltr"><div dir="auto">Thank your for your response! Sorry but just to be clear, my language model was trained on the wikiwoods corpus, and I used the leaf nodes of the derivation to decide how a word would appear in the LM's training data. The rules were as follows</div><div dir="auto"><br></div><div dir="auto">1. If the word has a native entry, the word's original form was kept.</div><div dir="auto">2. If the word was recognized as a POS generic, the part of speech tag was used to "unk" the word ("JJ_u_unknown" etc.).</div><div dir="auto">3. Otherwise, the word was replaced with its class attribute (these include "generic_proper_ne", "generic_card_ne" etc.)</div><div dir="auto"><br></div><div dir="auto">To my understanding, this encompasses all the unknown word handling in the ERG. A sample from my language model might look like "generic_proper_ne had VBP_u_unknown a cat ." I want to see if these sequences can be parsed by the ERG. </div><div dir="auto"><br></div><div dir="auto">One way I thought might be to proceed is to add a lexical rule into the grammar that parses the surface form "generic_proper_ne" into the generic_proper_ne. Would this be the easiest way? What would the easiest way for making such a change be? I would really appreciate some pointers, thank you!</div><div dir="auto"><div><br><div id="m_6239955583475133452AppleMailSignature">--<div>Johnny Wei</div></div><div><br>On Jun 11, 2018, at 6:39 PM, Michael Wayne Goodman <<a href="mailto:goodmami@uw.edu" target="_blank">goodmami@uw.edu</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr"><div>Hello developers,</div><div><br></div><div>See the forwarded message below for a question about parsing unknowns using the ERG, asked by Johnny Wei at the University of Massachusetts, Amherst.</div><div><br></div><div>Johnny: others on the list are more qualified to talk about parsing unknown tokens using ACE or PET with the ERG, but I'll attempt a response:</div><div><br></div><div>Parsing unknowns with DELPH-IN grammars is generally the task of matching tokens that couldn't be analyzed to a defined lexical entry (i.e., "lexical gaps") to some generic lexical entry. To avoid the explosion of ambiguity caused by attempting every generic lexical entry for every gap, filters are used to block some generic entries. One such filter, which it seems you are aware of, is the TNT POS tags assigned to the unknown token. These tags can be assigned using a trained POS tagger which is employed by PET or ACE during the parsing process, or they can be passed in via structured input to the parser (e.g., "yy-tokens"). In both of these cases, the POS tag is paired with the input token. What your language model is outputting looks like predicate symbols, and I'm not sure how to use those to directly influence the parser, but others on this list might. Also see this wiki page for more information: <a href="http://moin.delph-in.net/PetInput" target="_blank">http://moin.delph-in.net/<wbr>PetInput</a><br></div><div><br></div><div>There are also other methods of robust parsing, such as a PCFG backoff ("csaw"), but maybe these are not what you're looking for right now.<br></div><div><br></div><div>Also note that, in addition to PET and ACE, the LKB system can parse using DELPH-IN grammars, and it has a bit more robust support for unknown tokens (e.g., regarding morphological inflection of unknowns), although its Lisp-based implementation can make it tricky to interface with external programs, and it tends to run a bit slower than the so-called "efficient implementations" (but work is being done on improving the Lisp code's performance).</div><div><br></div><div> i hope this helps!<br></div><div><br></div><div><div><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Michael Wayne Goodman</b> <span dir="ltr"><<a href="mailto:goodman.m.w@gmail.com" target="_blank">goodman.m.w@gmail.com</a>></span><br>Date: Mon, Jun 11, 2018 at 1:39 PM<br>Subject: Fwd: Question on using PET/ACE for parsing<br>To: <a href="mailto:goodmami@uw.edu" target="_blank">goodmami@uw.edu</a><br><div id="m_6239955583475133452gmail-m_7675815109010958386m_646230504454141855m_2742573610091825846geary-quote"><br>---------- Forwarded message ----------<br>From: Johnny Wei <<a href="mailto:jwei@umass.edu" target="_blank">jwei@umass.edu</a>><br>Subject: Question on using PET/ACE for parsing<br>Date: Mon, 11 Jun 2018 14:46:45 -0400<br>To: <a href="mailto:goodman.m.w@gmail.com" target="_blank">goodman.m.w@gmail.com</a><br><br><div dir="ltr">Dear Michael,<div><br></div><div>My name is Johnny Wei, an undergraduate from the University of Massachusetts, Amherst. Deep grammars are very interesting to me, and I am looking to use the ERG with PET/ACE for parsing language model output. I have a few questions on parsing I was wondering whether you could answer. The questions are below and I really would appreciate your help!</div><div><br></div><div>To my understanding there are two ways that ERG handles unknown words, using TNT POS tags and certain regex matching for classes. Is this correct? The way I have my language model set up is that it can generate a ''JJ_u_unknown" or "_generic_proper_ne" for each of the unknown word classe! s. To parse, what would be the easiest way to proceed? For some of the generic classes, I have been able to replace them with some word such as card_ne -> 9, but I do not know of an easy way to incorporate the part of speech unknown words.<br></div><div><br></div><div>Again, I really appreciate your help. If anything is not clear please let me know, thanks!</div><span class="m_6239955583475133452gmail-m_7675815109010958386m_646230504454141855HOEnZb"><font color="#888888"><div><div><br></div>-- <br><div class="m_6239955583475133452gmail-m_7675815109010958386m_646230504454141855m_2742573610091825846gmail_signature"><div dir="ltr">Johnny Wei</div></div> </div></font></span></div> </div></div><br><br clear="all"><br>-- <br><div class="m_6239955583475133452gmail-m_7675815109010958386m_646230504454141855gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Michael Wayne Goodman</div></div></div></div></div></div> </div></div></div> </div></blockquote></div></div></div>