[developers] [incr() tsdb] batch batch processing

[Michael Wayne Goodman]

(Retrying with the correct From: address so it doesn't get bounced.)

On Sat, Dec 4, 2010 at 4:41 PM, Michael Wayne Goodman <goodmami at uw.edu> wrote:
> Thanks for the responses, Francis and Emily,
>
> On Mon, Nov 29, 2010 at 4:26 PM, Emily M. Bender
> <ebender at u.washington.edu> wrote:
>> Thanks, Francis.
>> Emily
>>
>> On Mon, Nov 29, 2010 at 4:13 PM, Francis Bond <bond at ieee.org> wrote:
>>> G'day,
>>>
>>>>> If you point the skeletons to a directory with none, then it will be
>>>>> quicker :-).
>>>>
>>>> But I think we need to point the skeletons to a directory with
>>>> skeletons, because we are using them to create profiles.
>>>
>>> If you are pointing itsdb just to the Index.lsp you want, then it will
>>> be as quick as it can be.
>
> I was unable to stop the default set from loading unless I moved
> logon/dot.tsdbrc to some other name so it couldn't be found. I was
> able to break up the Index.lisp file for our tests into a separate
> file for each test, and that helped a bit. Thanks for the tips.
>
>>>> They are different grammars, but they all have the same collateral
>>>> files, since they're all generated by the customization system.
>>>>
>>>> The question is how do we do it?
>>>
>>> logon/batch or logon/parse are just creating a lisp file and piping it to lisp.
>>>
>>> You can see the file being created by prepending --cat:
>>> ---
>>> ~/logon/batch --cat --jaen mrs
>>> (setf (system:getenv "DISPLAY") nil)
>>> (pushnew :logon *features*)
>>> (load "/home/bond/logon/lingo/lkb/src/general/loadup.lisp")
>>> (pushnew :lkb *features*)
>>> (compile-system "tsdb")
>>> (tsdb:tsdb :skeleton
>>>             "/home/bond/logon/dfki/jacy/tsdb/skeletons")
>>> (setf tsdb::*tsdb-maximal-number-of-analyses* 5)
>>> (setf tsdb::*tsdb-exhaustive-p* nil)
>>> (tsdb:tsdb :cpu :ja2en :file t :count 1 :error :exit :wait 1200)
>>> (setf target
>>>         (format
>>>          nil "~a~a"
>>>          (tsdb::suggest-test-run-directory "mrs" :absolute nil)
>>>          ""))
>>> (tsdb:tsdb :create target :skeleton "mrs")
>>> (tsdb::tsdb-do-process target :type :translate :overwrite t)
>>> (pvm::pvm_quit)
>>> (sleep 1)
>>> (excl:exit)
>>> ---
>>>
>>> You can copy this file, and add in extra bits of parsing:
>>>
>>> For example, call another cpu and parse again:
>>> (tsdb:tsdb :cpu :ja2en3 :file t :count 1 :error :exit :wait 1200)
>>> (setf target
>>>         (format
>>>          nil "~a~a"
>>>          (tsdb::suggest-test-run-directory "mrs2" :absolute nil)
>>>          ""))
>>> (tsdb:tsdb :create target :skeleton "mrs")
>>> (tsdb::tsdb-do-process target :type :translate :overwrite t)
>>>
>>> This could go just before the (pvm::pvm_quit).
>>>
>>> You then pass this to lisp:
>>>
>>> ${LOGONROOT}/bin/logon --binary  yourlispfile -I base -locale
>>> ja_JP.UTF-8 -qq 2>&1 | tee logfile
>
> You were right. It "worked", but had weird differences in the outputs
> after, say, the 10th grammar. Guess I won't be able to get away with
> this after all.
>
>>>
>>> --
>>> Francis Bond <http://www3.ntu.edu.sg/home/fcbond/>
>>> Division of Linguistics and Multilingual Studies
>>> Nanyang Technological University
>>>
>>
>>
>>
>> --
>> Emily M. Bender
>> Associate Professor
>> Department of Linguistics
>> Check out CLMA on facebook! http://www.facebook.com/uwclma
>>
>
> Thanks again!
>
> --
> -Michael Wayne Goodman
>



-- 
-Michael Wayne Goodman