[developers] token-level ambiguity in the LKB parser

[Ben Waldron]

Ann Copestake wrote:
>Just as a follow-up - it would be good to have some artificial test cases for 
>input token-level ambiguity which exercised all the possible cases.  The 
>potentially problematic issue for the LKB is the interaction with multi word 
>entries (in the sense of entries with an orth value that can cover more than 
>one of the initial tokens).
>  
A suite of tokenisation test cases is on my "to do" list -- along with 
integration with itsdb too, perhaps.

- Ben