[developers] [spr31716] tuning image size and the garbage collector

[Mikel Bancroft]

Stephan,

> i am wondering about gc() performance: every now and again and recently
> with distressing frequency, i experience a global gc() that requires on
> the order of seven minutes to complete.  here are some gc() messages:
> 
>   [20:18:39] gc-after-hook(): local; new: 911819488; old: 0; efficiency: 81.
> 
>   Indexing completescavenging...done
>   Mark Pass...done(1,270+0), marked 6303240 objects, overflows = 0, \
>     max depth (minus ovflow) = 16.
>   Weak-vector Pass...done(0+0).
>   Cons-cell swap...done(40+0), 696311 cons cells moved
>   Oldarea break chain...done(100+0), 68252 holes totalling 40041472 bytes
>   Page-compaction data...done(0+0).
>   Address adjustment...done(409,990+50).
>   Compacting other objects...done(190+0).
>   Page compaction...done(0+0), 0 pages moved
>   New rootset...done(500+0), 58773 rootset entries
>   Building new pagemap...done(110+0).
>   Merging empty oldspaces...done, 0 oldspaces merged.
>   global gc recovered 53360048 bytes of old space.
>   scavenging...done
>    eff: 2%, copy new: 243086224 + old: 0 = 243086224
>     Page faults: non-gc = 0 major + 39011 minor, gc = 0 major + 47676 minor
> 
>   [20:25:45] gc-after-hook(): global; new: 243086224; old: 0; efficiency: 2.
> 
> the one time-consuming step appears to be `address adjustment', so now
> i wonder what exactly is done there and how i get it to be so bad?  it
> seems ACL while in that phase is interacting badly with my Linux: load
> goes up to above 4, even though there is only one active process.  ACL
> itself has a process size of around 1.6 gbyte at this point (and there
> is lots more physical memory available).
> 
> ...
> 
> at this point, i believe i would benefit from a developer looking over
> the gc() messages above and pointing out (a) whether there is anything
> odd-looking there and (b) summarizing what exactly goes on in address
> adjustment and how i can affect the amount of work done in that phase?

One of the steps that occurs during a global gc is to determine what
objects are no longer live. The space each of these objects occupied
is noted and collected as a series of 'holes', which you can see
mentioned in the global GC output on the line beginning with 'Oldarea
break chain'. Later in the GC, these holes are collapsed, with the
remaining live objects moving down to fill the space they
occupied. The address adjustment phase is responsible for calculating
the address where objects above any number of these holes will reside
once the compaction phase is complete. As you've observed, the time a
global gc takes is highly dependent upon the number of holes.

Your global GC suggest that a large number of scattered objects are
being freed from oldspace. There are a few different options you can
take.

1. If you can exert some control over your allocation pattern, it is much
easier on the global gc to generate fewer large holes than many small
holes. 

2. Set the gsgc parameter :open-old-area-fence to instruct the GC not
to scan some number of your oldspaces. The documentation on this
parameter can be found at:

   http://www.franz.com/support/documentation/8.0/doc/gc.htm#generation-and-tenuring-parameters-2

   The stats from the above global gc output shows that ~50MB was
recovered, which is a small percentage of the total heap allocated for
your application. The 'holes' in the closed areas would remain
untouched, but they would not cause any increase in global gc
times. You'd need to experiment to come up with a good value for the
fence, based on the number of oldspaces in your app, and what amount
of time spent in a global gc is acceptable to you.

3. Some amount of ephemeral data is leaking down into oldspace and
being collected in the global gc, creating the holes. You can reduce
this to some extent by increasing the generation-spread to keep this
data in newspace longer. The tradeoff is that your newspaces will grow
larger as a result, though it appears yours are already pushing 1GB in
size based on the first gc-after-hook output.

The balance you'll want to strike is between finding acceptable times
for your global gcs and remaining within the resource limits of your
machine (namely physical RAM).

If you have any further questions about this let me know.

So that we may better track communication on your inquiry, and ensure
that your questions can be answered if I am unavailable, please address
further comments on this issue to support at franz.com as well as me, with
the identifier [spr31716] in the subject-line of your message.

Regards,

Mikel Bancroft		mikel at Franz.COM		http://www.franz.com/
Franz Inc., 555 12th St., Suite 1450, Oakland, CA  94607, USA
Phone: (510) 452-2000	FAX: (510) 452-0182