[developers] Perl FSR tokenisation tool

[Rebecca Dridan]

Hi all,

As I mentioned in Japan, I have Perl code that I use to tokenise 
sentences as expected by the grammars (or as defined by the 
preprocessor.fsr files) before I feed them to PET.  I believe this code 
will be obsoleted by the chart-mapping changes, but I've made it 
available to anyone who needs that functionality now, before the 
chart-mapping version of PET is mainstream release.

As we also discussed in Japan, I've created a page on the wiki 
(http://wiki.delph-in.net/moin/ToolsTop) to list tools and scripts of 
this sort. If anyone else has scripts they want to share, maybe they 
could link them from here. The wiki page should also be linked from 
somewhere I guess.

Rebecca