<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Hi Francis, Stephan, and all,</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Sorry to leave you in the dark. John and I were continuing the discussion off-list to avoid spamming the list with our back-and-forth about some very minor details (e.g., bugs in the syntax description at the TdlRfc wiki). Our discussion eventually lead to identifiers, which we should have put it back on the list. I'll try to summarize.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"> We found ourselves playing whack-a-mole with various punctuation characters in identifiers as we turned up ambiguities with other parts of the syntax (which led to five exclusions from the previous set: ; ' " % and |), and recognized that we only had a handful of printable punctuation characters in the lower 128 ASCII set that weren't excluded: @ \ ` { } and ~. So we thought we'd go with a whitelist instead of a blacklist, in spirit with the Krieger & Schäfer 1994 description but allowing for unicode alphanumerics (\w) instead of just [a-zA-Z0-9_] (plus the limited punctuation set [*+?-]). John is testing his new TDL parser for the LKB, and I just released PyDelphin v0.9.0 (perhaps too hastily) which used this whitelist pattern. It seems we may have been too aggressive in restricting identifier patterns, so I appreciate the feedback.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Yesterday John turned up issues with the middle dot in the KRG as well as numeric characters like <span class="gmail-box">Ⅲ. The middle dot is considered punctuation by unicode, so I was in favor of excluding it (considering our strategy for dealing with spaces in identifiers, e.g., ad_hoc_a1, a+bit_det, etc.). The numeric character was allowed by \w in Python's regular expressions with the re.UNICODE flag, but the Lisp regex engine (cl-ppcre?) does not allow it (as I understand). Stephan makes a good point about \w being potentially locale-dependent. In Python, \w matches alphanumeric characters in any locale when it's used to match strings of unicode codepoints, but it *is* locale-dependent if it is used with encoded byte strings. We also don't mandate unicode normalization, meaning that <span class="gmail-box">Ⅰ (U+2160) and I (U+0049) could be different identifiers (or <span class="gmail-charSample">Ω </span>(U+03A9) and <span class="gmail-box">Ω</span> (U+2126), <span class="gmail-box">Å (U+00C5) and <span class="gmail-box">Å (U+0041 + U+003A), etc)</span></span>.<br></span></span></div><br><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span class="gmail-box">And all this work to produce a complete and definitive syntax of DELPH-IN-style TDL is precisely to help ensure our tools are consistent. The current LKB, for instance, does not really have a notion of TDL identifiers because it parses them as Lisp identifiers (see the lkb-read function called by read-tdl-type). John had some nice examples (on-list) of identifiers that the LKB treated as equivalent despite differences in form. I gave an example identifier that ACE parsed but the LKB did not. We are trying to prevent these inconsistencies. Once we've all agreed on something final, I'd like to see all processors updated to reflect our decisions.<br></span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span class="gmail-box"><br></span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span class="gmail-box">If the whitelist approach is too restrictive, then the following is the most recent blacklist we had, which I believe avoids ambiguity with other parts of the TDL syntax:</span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span class="gmail-box"><br></span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span class="gmail-box"><span> [^\s!"#$%&'(),.\/:;<=>[\]^|]+</span></span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span class="gmail-box"><span><br></span></span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span class="gmail-box">Francis: the difference <span class="gmail-box">in PyDelphin </span>between regular identifiers and coreferences (and now-deprecated single-quoted 'symbols) was unintentional. I'll make a v0.9.1 patch release to fix this, once the issue regarding identifier syntax is resolved.<br></span></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span class="gmail-box"><br></span></div><span class="gmail-box"><span></span></span></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Oct 23, 2018 at 4:45 AM Francis Bond <<a href="mailto:bond@ieee.org" target="_blank">bond@ieee.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Thanks for the feedback! <div dir="auto"><br></div><div dir="auto">I tried to list the characters I thought should be excluded. Did I miss any?</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, 23 Oct 2018, 19:37 Stephan Oepen, <<a href="mailto:oe@ifi.uio.no" target="_blank">oe@ifi.uio.no</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">personally, i like all of your identifiers, francis, maybe even<br> including the one with a non-breaking space :-).<br> <br> i would think whether or not they are accepted by ‘\w’ depends on the<br> specific interpretation of ‘word characters’, which in turn may well<br> depend on your local set-up, i.e. the current locale. from the<br> perlre(1) man page:<br> <br> [...]Thus, under<br> this modifier, the ASCII platform effectively becomes a Unicode<br> platform; and hence, for example, "\w" will match any of the more than<br> 100,000 word characters in Unicode.<br> <br> Unlike most locales, which are specific to a language and country pair,<br> Unicode classifies all the characters that are letters somewhere in the<br> world as "\w".<br> <br> to avoid such dependencies on locale context, it might indeed be<br> simpler to define the syntax in terms of everything except a small<br> list of characters (that have operator-like status in TDL). this is<br> more or less how the current lexers in the LKB and PET work, so might<br> also be easier to make consistent across platforms.<br> <br> cheers, oe<br> <br> On Tue, Oct 23, 2018 at 12:30 PM Francis Bond <<a href="mailto:bond@ieee.org" rel="noreferrer" target="_blank">bond@ieee.org</a>> wrote:<br> ><br> > G'day,<br> ><br> > currently Zhong has several identifiers which Mike's TDL code<br> > considers invalid, but which the LKB and ACE are fine with:<br> ><br> > ＊-marker := symbol &<br> > ，_c_1 := conj_-_e_le &<br> > _n_1 := n_-_pn_le &<br> > 和_c_⚠ := conj_-_e_le &<br> > 格里姆斯比•罗伊洛特_n_1 := n_-_h_pn_le &<br> ><br> > full width *<br> > full width ,<br> > nonbreakspace [our bad, I will remove]<br> > warning sign (which I like to use for mal-rules).<br> > dot (often used in foreign names)<br> ><br> > And in Jacy:<br> > ザ・ベスト_n_1-tc := ordinary-nohon-n-lex &<br> > full width dot (often used in foreign names)<br> ><br> > PyDelphin defines identifiers to be: ([\w_+*?-]+),<br> > and coreference to be \#([^\s!"#$&'(),./:;<=>[\]^]+)<br> ><br> > It would be nice to at least include: ・•⚠, in identifiers, but maybe<br> > better to have a list of disallowed things (like coreference, now I<br> > guess with |):<br> ><br> > ([^\s!"#$&'(),./:;<=>[\]^|]+)<br> ><br> > and even better if the LKB, PET, ACE, AGREE and PyDelphin are consistent.<br> ><br> > What do people think?<br> ><br> > --<br> > Francis Bond <<a href="http://www3.ntu.edu.sg/home/fcbond/" rel="noreferrer noreferrer" target="_blank">http://www3.ntu.edu.sg/home/fcbond/</a>><br> > Division of Linguistics and Multilingual Studies<br> > Nanyang Technological University<br> ><br> </blockquote></div> </blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_7105169789550740615m_1432946470837825731gmail_signature" data-smartmail="gmail_signature">-Michael Wayne Goodman</div>