[developers] Scoring ranking results in the Logon System

[Bill McNeill (UW)]

I am running ERG parse ranking experiments using the logon framework.  I am
following the instructions on the LogonModeling wiki page:
http://wiki.delph-in.net/moin/LogonModeling.  I have been able to run the
steps outlined in the "Automating Experiments" section of that page, but I
have a few questions that I hope someone with experience with this system
can answer.

   1. Can the grid search be run in parallel?  I understand that the feature
   cachine and training steps have to be run in a single process, but it seems
   like the grid search (the step performed by the grid.lisp) could be run in
   parallel, with one grid point per computer.  I have the parallel computing
   resources to do this, so if parallelization works in this framework it would
   be a huge time saver.
   2. Are there scripts to perform scoring?  After grid.lisp has completed,
   it's not clear to me how to collect accuracy scores or other statistics.  Of
   course I could write code to do this myself by groveling the score and
   preference files, but it seems like someone would have already written this.
   3. What is the state of the art for ERG parse ranking?  What are the
   learning method, features, the data sets, and the accuracy?  Is this
   documented anywhere on the logon tree?  Is there a paper that outlines it?

Thanks.

-- 
Bill McNeill
http://staff.washington.edu/billmcn/index.shtml
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.delph-in.net/archives/developers/attachments/20090202/ed73bfb1/attachment.html>