[developers] Scoring ranking results in the Logon System
Bill McNeill (UW)
billmcn at u.washington.edu
Tue Feb 3 01:18:01 CET 2009
I am running ERG parse ranking experiments using the logon framework. I am
following the instructions on the LogonModeling wiki page:
http://wiki.delph-in.net/moin/LogonModeling. I have been able to run the
steps outlined in the "Automating Experiments" section of that page, but I
have a few questions that I hope someone with experience with this system
1. Can the grid search be run in parallel? I understand that the feature
cachine and training steps have to be run in a single process, but it seems
like the grid search (the step performed by the grid.lisp) could be run in
parallel, with one grid point per computer. I have the parallel computing
resources to do this, so if parallelization works in this framework it would
be a huge time saver.
2. Are there scripts to perform scoring? After grid.lisp has completed,
it's not clear to me how to collect accuracy scores or other statistics. Of
course I could write code to do this myself by groveling the score and
preference files, but it seems like someone would have already written this.
3. What is the state of the art for ERG parse ranking? What are the
learning method, features, the data sets, and the accuracy? Is this
documented anywhere on the logon tree? Is there a paper that outlines it?
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the developers