<div dir="ltr">What Woodley described is I think what's going on. It looks like the START/END offsets returned by the lkb:repp call are different to that of the +FROM/+TO offsets found in the derivation. The derivation I'm using is from the gold tree in the logon repository: i-id 10032820 in $LOGONROOT/lingo/erg/tsdb/gold/ws01/result.gz. (also, for some reason I just get NIL when I try to evaluate that repp function call in my lisp buffer after loading logon)<div> <br></div><div>From comparing derivations and the relevant portions of the p-token field, it looks to me like using the +FORM feature of the has the same effect as extracting the string from the p-tokens field for the token that was ultimately used? But as you say, this still leaves the issue of the correct casing. Angelina's workaround is a good suggestion, but definitely feels like a hack. It seems like it would be desirable to be keeping track of the value of tokens after REPP normalization but before downcasing for lexicon lookup. I was talking about this with Bec also, and while her problem was slightly different in that she only needed features for ubertagging rather than the original surface form, she said she was also struggling with this limitation.</div> <div><br></div><div>Ned</div><div><br></div><div><div><div> <br></div><div><br></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Feb 6, 2014 at 11:26 AM, Woodley Packard <span dir="ltr"><<a href="mailto:sweaglesw@sweaglesw.org" target="_blank">sweaglesw@sweaglesw.org</a>></span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I believe what happened in this particular case is that "control" and the following "," punctuation token got combined, resulting in a contiguous CFROM/CTO span of 137 to 147, which includes not only "control" and "," but also the deleted text in the middle.<br> <span class="HOEnZb"><font color="#888888"><br> -Woodley<br> </font></span><div class="HOEnZb"><div class="h5"><br> On Feb 5, 2014, at 2:24 PM, Stephan Oepen <<a href="mailto:oe@ifi.uio.no">oe@ifi.uio.no</a>> wrote:<br> <br> > hi ned,<br> ><br> > the practical challenge you are facing is deeply interesting. i<br> > believe both angelina (in conversion to bi-lexical dependencies)<br> > and bec (working on ubertagging in PET) have looked at this.<br> ><br> > the derivation trees include the identifiers of internal tokens<br> > (an integer immediately preceding the token feature structure),<br> > and these tokens you can retrieve from the :p-tokens field in<br> > reasonably up-to-date [incr tsdb()] profiles. this will give you<br> > the strings that were used for lexical lookup. capitalization is<br> > lost at this point, more often than not, hence one needs to do<br> > something approximative in addition to finding the token. for<br> > all i recall, angelina compares the actual token to others that<br> > have the same position in the chart (by start and end vertex,<br> > as recorded in the :p-tokens format); in case she finds one<br> > whose orthography differs from the downcased string, then<br> > she uses that token instead. bec, on the other hand, i think<br> > consults the +CASE feature in the token feature structure.<br> ><br> > underlying all this, i suspect there is a question of what the<br> > characterization of initial tokens really should be, e.g. when<br> > we strip wiki markup at the REPP level. but i seem unable<br> > to reproduce the particular example you give:<br> ><br> > TSNLP(88): (setf string<br> > "Artificial intelligence has successfully been used in a<br> > wide range of fields including [[medical diagnosis]], [[stock<br> > trading]], [[robot control]], [[law]], scientific discovery and<br> > toys.")<br> > "Artificial intelligence has successfully been used in a wide range of<br> > fields including [[medical diagnosis]], [[stock trading]], [[robot<br> > control]], [[law]], scientific discovery and toys."<br> ><br> > TSNLP(89): (pprint (lkb::repp string :calls '(:xml :wiki :lgt :ascii<br> > :quotes) :format :raw))<br> > ...<br> > #S(LKB::TOKEN :ID 20 :FORM "control" :STEM NIL :FROM 20 :TO 21 :START<br> > 137 :END 144 :TAGS NIL :ERSATZ NIL)<br> > ...<br> ><br> > TSNLP(90): (subseq string 137 144)<br> > "control"<br> ><br> > i don't doubt the problem is real, but out of curiosity: how did<br> > you produce your derivations?<br> ><br> > all best, oe<br> ><br> > On Wed, Feb 5, 2014 at 5:02 AM, Ned Letcher <<a href="mailto:nletcher@gmail.com">nletcher@gmail.com</a>> wrote:<br> >> Hi all,<br> >><br> >> I'm trying to export DELPH-IN derivation trees for use in the Fangorn<br> >> treebank querying tool (which uses PTB style trees for importing) and have<br> >> run into a hiccup extracting the string to use for the leaves of the trees.<br> >> Fangorn does not support storing the original input string alongside the<br> >> derivation, with the string used for displaying the original sentence being<br> >> reconstructed by concatenating the leaves of the tree together.<br> >><br> >> I've been populating the leaves of the exported PTB tree by extracting the<br> >> relevant slice of the i-input string using the +FROM +TO offsets in the<br> >> token information (if token mapping was used). One case I've found where<br> >> this doesn't work so well (and there may be more), is where characters which<br> >> have been stripped by REPP occur within a token, so these characters are<br> >> then included in the slice. Wikipedia markup, for instance, results in these<br> >> artefacts:<br> >><br> >> "Artificial intelligence has successfully been used in a wide range of<br> >> fields including medical diagnosis]], stock trading]], robot control]],<br> >> law]], scientific discovery and toys."<br> >><br> >> I also tried using the value of the +FORM feature, but it seems that this<br> >> doesn't always preserve the casing of the original input string.<br> >><br> >> Does anyone have any ideas for combating this problem?<br> >><br> >> Ned<br> >><br> >> --<br> >> <a href="http://nedned.net" target="_blank">nedned.net</a><br> ><br> ><br> ><br> > --<br> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br> > +++ Universitetet i Oslo (IFI); Boks 1080 Blindern; 0316 Oslo; <a href="tel:%28%2B47%29%202284%200125" value="+4722840125">(+47) 2284 0125</a><br> > +++ --- <a href="mailto:oe@ifi.uio.no">oe@ifi.uio.no</a>; <a href="mailto:stephan@oepen.net">stephan@oepen.net</a>; <a href="http://www.emmtee.net/oe/" target="_blank">http://www.emmtee.net/oe/</a> ---<br> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++<br> <br> </div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><a href="http://nedned.net" target="_blank">nedned.net</a> </div>