[developers] [oe at csli.stanford.edu: tuning image size and the garbage collector]

[Stephan Oepen]

g'day,

attached below is a query i (think i) sent to Franz earlier this week,
but have not heard back about yet.  i am wondering whether anyone on
this list has knowledge about gc() internals and the interactions with 
memory layout (possibly specific to the AMD64 architecture)?  also, i
am curious as to whether our 2 -- 10 gbyte process sizes are unique,
or whether others are operating in a comparable space (well, maybe we
should just learn how to code memory-efficiently)?  

the pieces of ACL documentation that discuss tuning gc() performance,
memory layout, et al. have been routine literature for me over the past
fifteen or so years, but still i get a sense of `black art' when trying
to optimize in that neighborhood :-).  any words of wisdom or pointers
to related materials would be very much appreciated!

                                         many thanks in advance  -  oe

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++ Universitetet i Oslo (IFI); Boks 1080 Blindern; 0316 Oslo; (+47) 2285 7989
+++     CSLI Stanford; Ventura Hall; Stanford, CA 94305; (+1 650) 723 0515
+++       --- oe at csli.stanford.edu; oe at ifi.uio.no; stephan at oepen.net ---
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


[--- start of forwarded message ---]

Date: Mon, 5 Jun 2006 20:46:15 +0200
From: oe at csli.stanford.edu (Stephan Oepen)
To: support at franz.com
Cc: developers at delph-in.net
Subject: tuning image size and the garbage collector

hei!

i am wondering about gc() performance: every now and again and recently
with distressing frequency, i experience a global gc() that requires on
the order of seven minutes to complete.  here are some gc() messages:

  [20:18:39] gc-after-hook(): local; new: 911819488; old: 0; efficiency: 81.

  Indexing completescavenging...done
  Mark Pass...done(1,270+0), marked 6303240 objects, overflows = 0, \
    max depth (minus ovflow) = 16.
  Weak-vector Pass...done(0+0).
  Cons-cell swap...done(40+0), 696311 cons cells moved
  Oldarea break chain...done(100+0), 68252 holes totalling 40041472 bytes
  Page-compaction data...done(0+0).
  Address adjustment...done(409,990+50).
  Compacting other objects...done(190+0).
  Page compaction...done(0+0), 0 pages moved
  New rootset...done(500+0), 58773 rootset entries
  Building new pagemap...done(110+0).
  Merging empty oldspaces...done, 0 oldspaces merged.
  global gc recovered 53360048 bytes of old space.
  scavenging...done
   eff: 2%, copy new: 243086224 + old: 0 = 243086224
    Page faults: non-gc = 0 major + 39011 minor, gc = 0 major + 47676 minor

  [20:25:45] gc-after-hook(): global; new: 243086224; old: 0; efficiency: 2.

the one time-consuming step appears to be `address adjustment', so now
i wonder what exactly is done there and how i get it to be so bad?  it
seems ACL while in that phase is interacting badly with my Linux: load
goes up to above 4, even though there is only one active process.  ACL
itself has a process size of around 1.6 gbyte at this point (and there
is lots more physical memory available).

i also have the impression that the number of `cons cells moved' has a
direct impact on the time taken in address adjustment.  furthermore, i
believe that ACL 8.0 made this problem worse.  i have tried a range of
image creation parameters (initial new- and oldspace sizes, lisp heap, 
incremental growth parameters, et al.), but so far to no avail.

at this point, i believe i would benefit from a developer looking over
the gc() messages above and pointing out (a) whether there is anything
odd-looking there and (b) summarizing what exactly goes on in address
adjustment and how i can affect the amount of work done in that phase?

                                        many thanks in advance  -  oe

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++ Universitetet i Oslo (IFI); Boks 1080 Blindern; 0316 Oslo; (+47) 2285 7989
+++     CSLI Stanford; Ventura Hall; Stanford, CA 94305; (+1 650) 723 0515
+++       --- oe at csli.stanford.edu; oe at ifi.uio.no; stephan at oepen.net ---
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[--- end of forwarded message ---]