I am running parse ranking experiments on the jhpstg profile in logon/lingo/redwoods. I want to measure how stable the accuracy numbers are across different train/test partitions. To do this I am reading the accuracy values for individual folds out of the fold table in the output profiles.<div>
<br></div><div>In my configuration file I have nfold=10. However, the fold files in the output profiles contain only 2 lines. I assume this means only 2 folds were run instead of 10.<div><br></div><div>1. Am I correct in thinking that the experiment only used 2 folds?</div>
<div>2. Is this expected behavior?</div><div>3. How do I make the experiment do 10 folds?</div><div><br></div><div>I see that there is code in rank-profile that limits the number of folds to be less than the number of item sets (redwoods.lisp:3226). I don't think that is the issue here because I am using the default jhpstg item-set file which appears to contain 3492 unique item sets.</div>
<div><br></div><div>I also have niterations=2 in this file. I don't understand what this setting does, since I don't see it documented in Velldal 2008 or on the Logon wiki, and in the source code for rank-profile it appears that the nfold value is assignment to niterations.</div>
<div><br></div><div>4. What is niterations?</div><div><br></div><div>As usual, I've done due diligence looking for the answers in Velldal 2008, on the Logon Wiki, reading the Logon source code, running experiments to try and figure out what the code is supposed to be doing etc.</div>
<div><br></div><div>The full content of my grid.lisp file is shown below:</div><div><br></div><div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;">(in-package :tsdb)</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"><br></span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;">(load "parsing.lisp")</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"><br></span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;">(batch-experiment</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :source "jhpstg" :skeleton "jhpstg"</span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :nfold 10 :niterations 2 :type :mem</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :prefix "jhpstg"</span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :score-similarities nil</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :grandparenting '(3 4)</span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :active-edges-p '(nil t)</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :lexicalization-p nil</span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :constituent-weight '(1 2 0)</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :ngram-size '(3 4) :ngram-back-off-p '(nil t)</span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :lm-p nil</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :random-sample-size nil</span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :counts-absolute 0 :counts-contexts 0 :counts-events 0 :counts-relevant 1</span></font></div>
<div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :variance 1e0</span></font></div><div><font class="Apple-style-span" face="'courier new', monospace"><span class="Apple-style-span" style="font-size: large;"> :relative-tolerance 1e-8)</span></font></div>
<div><br></div></div><div><br>-- <br>W.P. McNeill<br><a href="http://staff.washington.edu/billmcn/index.shtml">http://staff.washington.edu/billmcn/index.shtml</a><br>
</div></div>