[developers] PET code changes; new 64-bit testing binary in the LOGON tree

[Stephan Oepen]

dear colleagues,

in preparing a fresh release of the Redwoods treebanks and the
WikiWoods treecache, dan and i wanted to explore somewhat more
generous resource limits than we have traditionally used in PET.  up
until this summer, we have always made do with 32-bit binaries of PET,
which by nature were limited to a maximum process size of around 3.5
gbytes.

while experimenting with a 64-bit binary of PET, it emerged that there
were two deficiencies in the implementation of its ‘-memlimit’ command
line option, however.  i have just committed some patches into the PET
‘trunk’ to address these issues; these changes have user-visible
consequences, hence please read on.

first, the data type traditionally used to monitor memory usage in PET
(a signed ‘long int’eger) would overflow for values in excess of 2048
mbytes, making any ‘-memlimit’ specification larger than that have no
effect.  fixing this issue can only be good news to anyone running PET
with potentially large process sizes.

second, for at least the past several years (maybe always), PET has
only counted (what in its code is called) _dynamic_ dag storage (as
used e.g. inside the unification operation) against the memory limit;
however, so-called _static_ dag storage can also take substantial
memory.  hence, as of source revision 920, both types of dag memory
will now be considered against the memory limit.  this latter change
means that users will need to adjust ‘-memlimit’ values accordingly,
typically increase values you might have used in the past by a factor
between 1.5 and 2.

there is a new 64-bit testing binary in the LOGON tree now, which (on
64-bit installations) can be activated by passing the ‘--test’ (or
‘-t’) option to the LOGON wrapper script for ‘cheap’.

finally, even with the rationalized treatment of ‘-memlimit’ above, a
typical ‘cheap’ process will be several hundred megabytes larger than
the ‘-memlimit’; this is plausible behavior, as there are other
(non-dag) data structures allocated in the data segment, and other
components of the process image (e.g. its code segment and stack)
further contribute to total process size.

to give PET users (in LOGON) a way of enforcing a set absolute limit
on total process size, the ‘parse’ (and ‘cheap’ wrapper) scripts now
allow an additional option, ‘--size’, which takes an argument in
megabytes and translates into a call to ulimit(3) prior to invocation
of ‘cheap’.  note that this latter functionality will only work for
users of the LOGON run-time binaries, once i rebuild these sometime
later this week.

i would be grateful if PET and LOGON users could test these
modifications and let me know in case you see anything surprising!

best, oe