[developers] Why am I running parse ranking with only 2 folds when the config file specifies 10?

[Stephan Oepen]

hi bill,

:niterations controls how many of the folds actually are executed.   
the default values call for two-out-of-ten folds, i.e. a relatively  
quick indication of expected performance.  change :niterations to five  
or ten, and yoy'll get more folds.

best, oe


On Jul 9, 2009, at 3:18 AM, "W.P. McNeill (UW)" <billmcn at u.washington.edu 
 > wrote:

> I am running parse ranking experiments on the jhpstg profile in  
> logon/lingo/redwoods.  I want to measure how stable the accuracy  
> numbers are across different train/test partitions.  To do this I am  
> reading the accuracy values for individual folds out of the fold  
> table in the output profiles.
>
> In my configuration file I have nfold=10.  However, the fold files  
> in the output profiles contain only 2 lines.  I assume this means  
> only 2 folds were run instead of 10.
>
> 1. Am I correct in thinking that the experiment only used 2 folds?
> 2. Is this expected behavior?
> 3. How do I make the experiment do 10 folds?
>
> I see that there is code in rank-profile that limits the number of  
> folds to be less than the number of item sets (redwoods.lisp:3226).   
> I don't think that is the issue here because I am using the default  
> jhpstg item-set file which appears to contain 3492 unique item sets.
>
> I also have niterations=2 in this file. I don't understand what this  
> setting does, since I don't see it documented in Velldal 2008 or on  
> the Logon wiki, and in the source code for rank-profile it appears  
> that the nfold value is assignment to niterations.
>
> 4. What is niterations?
>
> As usual, I've done due diligence looking for the answers in Velldal  
> 2008, on the Logon Wiki, reading the Logon source code, running  
> experiments to try and figure out what the code is supposed to be  
> doing etc.
>
> The full content of my grid.lisp file is shown below:
>
> (in-package :tsdb)
>
> (load "parsing.lisp")
>
> (batch-experiment
>  :source "jhpstg" :skeleton "jhpstg"
>  :nfold 10 :niterations 2 :type :mem
>  :prefix "jhpstg"
>  :score-similarities nil
>  :grandparenting '(3 4)
>  :active-edges-p '(nil t)
>  :lexicalization-p nil
>  :constituent-weight '(1 2 0)
>  :ngram-size '(3 4) :ngram-back-off-p '(nil t)
>  :lm-p nil
>  :random-sample-size nil
>  :counts-absolute 0 :counts-contexts 0 :counts-events 0 :counts- 
> relevant 1
>  :variance 1e0
>  :relative-tolerance 1e-8)
>
>
> -- 
> W.P. McNeill
> http://staff.washington.edu/billmcn/index.shtml
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.delph-in.net/archives/developers/attachments/20090709/0ddfcedd/attachment.html>