[logon] end-to-end reranking

[Stephan Oepen]

howdy,

> If we train an end-to-end reranking model (as Eric has: jaen.r.mem),
> where do we actually read it?  I couldn't seem to find any such
> setting for noen.  Or do we just set the individual parameters
> somewhere?

the use of a (proper) MaxEnt re-ranker is completely separate from the
original, naive re-ranker.  for NoEn, the MaxEnt model is activated in
`logon.lisp' (aka `no2en.lisp'):

  ;;
  ;; as of the HandOn release, the re-ranker code is fully integrated, so load
  ;; the MaxEnt re-ranking model and set up feature selection appropriately.
  ;;
  (funcall (symbol-function (find-symbol "READ-MODEL" :tsdb))
           (format nil "~a/lingo/lkb/src/mt/jh.r.mem" logon))
  (set (intern "*FEATURE-FLAGS*" :tsdb)
       '((0 :ascore) (1 :tscore) (2 :rscore) 
         (3 :lm)
         (5 :lfn) (6 :lnf)
         (10 :distortion)
         (11 :distance)
         (12 :nmtrs) (13 :tratio)
         (14 :ratio)
         (15 :nfragments)))
  (set (intern "*FEATURE-GRANDPARENTING*" :tsdb) -1)
  (set (intern "*FEATURE-NGRAM-SIZE*" :tsdb) 0)
  (set (intern "*FEATURE-LM-P*" :tsdb) nil)

as soon as the global %model% is set (the effect of read-model()), the
code will compute the re-ranking score using the MaxEnt model instead
of the clunky *mt-analysis-weight* et al.  this setting goes into the
:translate Lisp process, the parent process to analysis, transfer, and
generation.

just now, i think the model files should probably go into the transfer
grammar directory, i.e. `uio/noen/jh.r.mem' in our case.  once i copy
the HandOn release into the SVN trunk (any day now), i will apply that
change to NoEn.

--- out of curiosity, are you getting tangible improvements out of the
MaxEnt re-ranker?  for NoEn, several of us feel that it clearly helped
with BLEU, but in terms of subjective quality we were a bit uncertain.

                                                     all best  -  oe

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++ Universitetet i Oslo (IFI); Boks 1080 Blindern; 0316 Oslo; (+47) 2284 0125
+++     CSLI Stanford; Ventura Hall; Stanford, CA 94305; (+1 650) 723 0515
+++       --- oe at ifi.uio.no; oe at csli.stanford.edu; stephan at oepen.net ---
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++