[developers] preprocessor implementations

[R. Bergmair]

...it seems like none of the FSPP implementations are
currently 100% compatible with the Lisp implementation.

On Thu, 27 Nov 2008, Stephan Oepen wrote:

> in addition to tokenizing your data, do you also plan on parsing it?
> if so, efficiency of the tokenization step may not matter that much.

No, actually I was really only going to run the tokenization.

> just another heads-up on ongoing developments in this area: we are in
> the process of re-working tokenization assumptions in the ERG, coupled
> with a full re-design of the token lattice interface into PET.  there
> was a quite general discussion of the motivation for this work in LREC
> 2008 (by Peter Adolphs et al.), and in the meantime the chart mapping
> implementation in PET and revisions in the ERG have matured to a point
> where we anticipate a first release early next year.

Ah, thanks for the hint. That's good to know. I've read the LREC paper
now. I'll probably keep playing around with the LISP FSPP, then, until
you have the chart-mapping thing ready for general use.

Thanks!

Richard