[developers] Why am I running parse ranking with only 2 folds when the config file specifies 10?
stephan.oepen at gmail.com
Thu Jul 9 06:35:27 CEST 2009
:niterations controls how many of the folds actually are executed.
the default values call for two-out-of-ten folds, i.e. a relatively
quick indication of expected performance. change :niterations to five
or ten, and yoy'll get more folds.
On Jul 9, 2009, at 3:18 AM, "W.P. McNeill (UW)" <billmcn at u.washington.edu
> I am running parse ranking experiments on the jhpstg profile in
> logon/lingo/redwoods. I want to measure how stable the accuracy
> numbers are across different train/test partitions. To do this I am
> reading the accuracy values for individual folds out of the fold
> table in the output profiles.
> In my configuration file I have nfold=10. However, the fold files
> in the output profiles contain only 2 lines. I assume this means
> only 2 folds were run instead of 10.
> 1. Am I correct in thinking that the experiment only used 2 folds?
> 2. Is this expected behavior?
> 3. How do I make the experiment do 10 folds?
> I see that there is code in rank-profile that limits the number of
> folds to be less than the number of item sets (redwoods.lisp:3226).
> I don't think that is the issue here because I am using the default
> jhpstg item-set file which appears to contain 3492 unique item sets.
> I also have niterations=2 in this file. I don't understand what this
> setting does, since I don't see it documented in Velldal 2008 or on
> the Logon wiki, and in the source code for rank-profile it appears
> that the nfold value is assignment to niterations.
> 4. What is niterations?
> As usual, I've done due diligence looking for the answers in Velldal
> 2008, on the Logon Wiki, reading the Logon source code, running
> experiments to try and figure out what the code is supposed to be
> doing etc.
> The full content of my grid.lisp file is shown below:
> (in-package :tsdb)
> (load "parsing.lisp")
> :source "jhpstg" :skeleton "jhpstg"
> :nfold 10 :niterations 2 :type :mem
> :prefix "jhpstg"
> :score-similarities nil
> :grandparenting '(3 4)
> :active-edges-p '(nil t)
> :lexicalization-p nil
> :constituent-weight '(1 2 0)
> :ngram-size '(3 4) :ngram-back-off-p '(nil t)
> :lm-p nil
> :random-sample-size nil
> :counts-absolute 0 :counts-contexts 0 :counts-events 0 :counts-
> relevant 1
> :variance 1e0
> :relative-tolerance 1e-8)
> W.P. McNeill
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the developers