[developers] [incr() tsdb]/LKB memory allocation error

[Olga Zamaraeva]

Dear developers!

I am trying to use the LKB and [incr() tsdb] to parse a list of verbs by a
grammar of Chintang [ctn]. The language is polysynthetic, plus the grammar
was created automatically using k-means clustering for the morphology
section, so some of the position classes have lots and lots of inputs and
lots and lots of lexical rule types and instances.

I am running into a problem when  [incr() tsdb] crashes because of a memory
allocation error. If I don't use itsdb and just go with LKB batch parsing,
it is more robust as it can catch the error and continue parsing, having
reported a failure on the problematic item, but the problem is still there
and the parses still fail.

I am a fairly inexperienced user of both systems, so right now I am trying
to understand what is the best way for me to:

 1) debug the grammar with respect to the problem, i.e. what is it about
the grammar exactly that causes the issues;
2) do something with itsdb so that perhaps this does not happen? Limit it
somehow so that it doesn't try as much?

Currently I am mostly just trying to filter out the problematic items... I
also tried limiting the chart size to 30K, and that seems to have helped a
little, but the crashes still happen on some items. If I limit the chart
size to 20K, then it seems like maybe I can go through the test suite, but
then my coverage suffers when I think it shouldn't: I think there are items
which I can parse with 30K limit but not 20K... Is this the route I should
be going in any case? Just optimizing for the chart size?.. Maybe 25K is my
number :). The chart is the parse chart, is that correct? I need to
understand what exactly makes the chart so huge in my case; how should I
approach debugging that?..

One specific question: what does "tenuring" mean with respect to garbage
collection? Google doesn't know (nor does the manual, I think).

Does anyone have any comment on any of these issues? The (very helpful)
chapter on errors and debugging in Copestake (2002) book mostly talks about
other types of issues such as type loading problems etc.. I also looked at
what I found in ItsdbTop (http://moin.delph-in.net/ItsdbTop), and it does
mention that on 32-bit systems memory problems are possible, but I think
that note has to do with treebanking, and it doesn't really tell me much
about what I should try in my case... I also looked thorough the itsdb
manual (http://www.delph-in.net/itsdb/publications/manual.pdf) -- but it
looks like some of the sections, specifically about debugging and options
and parameters, are empty?

Anyway, I would greatly appreciate any advice! I attach a picture of a
running testsuite processing, to give an idea about the memory usage and
the chart size, and of the error. It is possible that the grammar that I
have is just not a usage scenario as far as itsdb is concerned, but I don't
yet have a clear understanding of whether that's the case.

Thanks!
Olga
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.delph-in.net/archives/developers/attachments/20160315/200dbfdf/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2016-03-15 at 12.41.48 PM.png
Type: image/png
Size: 526078 bytes
Desc: not available
URL: <http://lists.delph-in.net/archives/developers/attachments/20160315/200dbfdf/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Screen Shot 2016-03-15 at 12.08.33 PM.png
Type: image/png
Size: 574199 bytes
Desc: not available
URL: <http://lists.delph-in.net/archives/developers/attachments/20160315/200dbfdf/attachment-0003.png>