[logon] The highest accuracy on the JHPSTG corpus is 40.6. Is this a bug?

[Bill McNeill (UW)]

I ran a set of parse ranking experiments on the JHPSTG corpus.  I chose a
reasonable set of learning parameters and ranged over the machine learning
priors.  The highest accuracy I got was 40.6.  This seems very low, and way
beneath the state of the art.  Do I have a bug?
Here's the batch-experiment call I use:

(in-package :tsdb)

(load "parsing.lisp")

(batch-experiment
 :source "jhpstg" :skeleton "jhpstg"
 :nfold 10 :niterations 2 :type :mem
 :prefix "jhpstg"
 :score-similarities nil
 :grandparenting 4
 :active-edges-p t
 :lexicalization-p nil
 :constituent-weight 0
 :ngram-size 4 :ngram-back-off-p t
 :lm-p nil
 :random-sample-size nil
 :counts-absolute 0 :counts-contexts 0 :counts-events 0 :counts-relevant 1
 :variance '(nil 1e4 1e2 1e0 1e-2 1e-4 1e-6)
 :relative-tolerance '(1e-6 1e-8 1e-10))

I ran scoring with the following lisp script:

(setf *tsdb-home* "/home/billmcn/logon/lingo/redwoods/tsdb/home")
(summarize-folds :output "/home/billmcn/temp/jhpstg.results" :pattern
"\\[jhpstg\\]")

I got the following results:

39.449802 9.746283 13.783325 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-6]
AT[1.0e-20] VA[1.0e-6] PC[100]'
38.287010 11.390718 16.108908 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-6]
AT[1.0e-20] VA[1.0e-4] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-6]
AT[1.0e-20] VA[1.0e+4] PC[100]'
38.173570 4.652545 6.579693 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-10]
AT[1.0e-20] VA[1.0e-2] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-8]
AT[1.0e-20] VA[1.0e+0] PC[100]'
38.287010 11.390718 16.108908 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-10]
AT[1.0e-20] VA[] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-10]
AT[1.0e-20] VA[1.0e+0] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-10]
AT[1.0e-20] VA[1.0e+4] PC[100]'
40.612595 8.101849 11.457745 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-10]
AT[1.0e-20] VA[1.0e-4] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-6]
AT[1.0e-20] VA[1.0e+2] PC[100]'
38.173570 4.652545 6.579693 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-6]
AT[1.0e-20] VA[1.0e-2] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-6]
AT[1.0e-20] VA[1.0e+0] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-8]
AT[1.0e-20] VA[1.0e+4] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-8]
AT[1.0e-20] VA[1.0e+2] PC[100]'
38.287010 11.390718 16.108908 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-8]
AT[1.0e-20] VA[] PC[100]'
34.571754 2.847678 4.027225 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-10]
AT[1.0e-20] VA[1.0e+2] PC[100]'
38.287010 11.390718 16.108908 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-10]
AT[1.0e-20] VA[1.0e-6] PC[100]'
38.173570 4.652545 6.579693 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-8]
AT[1.0e-20] VA[1.0e-2] PC[100]'
40.612595 8.101849 11.457745 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-8]
AT[1.0e-20] VA[1.0e-4] PC[100]'
39.449802 9.746283 13.783325 `/[jhpstg] GP[4] +PT -LEX CW[] +AE NS[4]
NT[type] +NB LM[0] FT[:::1] RS[] MM[tao_lmvm] MI[5000] RT[1.0e-8]
AT[1.0e-20] VA[1.0e-6] PC[100]'


-- 
W.P. McNeill
http://staff.washington.edu/billmcn/index.shtml
Sent from Seattle, WA, United States
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.emmtee.net/archives/logon/attachments/20090410/4a3e5c3c/attachment.html>