[developers] Why am I running parse ranking with only 2 folds when the config file specifies 10?

[W.P. McNeill (UW)]

I am running parse ranking experiments on the jhpstg profile in
logon/lingo/redwoods.
 I want to measure how stable the accuracy numbers are across
different train/test partitions.
 To do this I am reading the accuracy values for individual folds out of the
fold table in the output profiles.
In my configuration file I have nfold=10.  However, the fold files in the
output profiles contain only 2 lines.  I assume this means only 2 folds were
run instead of 10.
1. Am I correct in thinking that the experiment only used 2 folds?
2. Is this expected behavior?
3. How do I make the experiment do 10 folds?

I see that there is code in rank-profile that limits the number of folds to
be less than the number of item sets (redwoods.lisp:3226).  I don't think
that is the issue here because I am using the default jhpstg item-set file
which appears to contain 3492 unique item sets.

I also have niterations=2 in this
file. I don't understand what this setting does, since I don't see it
documented in Velldal 2008 or on the Logon wiki, and in the source
code for rank-profile it appears that the nfold value is assignment to
niterations.

4. What is niterations?

As usual, I've done due diligence looking for the answers in Velldal 2008,
on the Logon Wiki, reading the Logon source code, running experiments to try
and figure out what the code is supposed to be doing etc.

The full content of my grid.lisp file is shown below:

(in-package :tsdb)

(load "parsing.lisp")

(batch-experiment
 :source "jhpstg" :skeleton "jhpstg"
 :nfold 10 :niterations 2 :type :mem
 :prefix "jhpstg"
 :score-similarities nil
 :grandparenting '(3 4)
 :active-edges-p '(nil t)
 :lexicalization-p nil
 :constituent-weight '(1 2 0)
 :ngram-size '(3 4) :ngram-back-off-p '(nil t)
 :lm-p nil
 :random-sample-size nil
 :counts-absolute 0 :counts-contexts 0 :counts-events 0 :counts-relevant 1
 :variance 1e0
 :relative-tolerance 1e-8)


-- 
W.P. McNeill
http://staff.washington.edu/billmcn/index.shtml
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.delph-in.net/archives/developers/attachments/20090708/04319134/attachment.html>