VCM - Memory leak check list

VCM - Memory leak check list

STEP 1 – Identify patterns of your system's memory use over time (KB 8381)
==========================================================================

If you suspect your system is leaking memory – or that memory use is growing to
unacceptable levels over a certain timescale – then the first thing to do is
chart this memory growth over time. The longer the time period the better; in
previous cases, a week's worth of data was good enough to see a generalised
picture emerging.

One easy way of tracking memory use over time is to enable Garbage Collection
("GC") statistics for your JVM; how to do this on both 7.3.0.5 (SUN JVM) and
7.3.1 (BEA JVM) is fully documented in KB 8381. KB 8381 also details how to
render the stats into a graphical format (viewing the memory/time data in a
graphical format is very useful: you can get a really good idea of when memory
use is growing or – perhaps more importantly – not shrinking).

Tip: Get as much context around the graph as possible: when do users start
logging on in the morning? When do they go for coffee? When do they leave? Are
there times of the day when there is a lot publishing activity (remember to
include automatic publishing)? Are there any automated batch processes running
at night? When do the backups run? Are data imports running in the middle of
the night?

Ideally you want to mark up your graph with these usage patterns to help figure
out what could be contributing (or not) to the memory growth.

STEP 2 – Analyse the "GC" graph
=========================================

Ok, you've generated a week's worth of "GC" data and you have a good idea of
how and when the system is being used.

"GC VIEWER" TIP1: For viewing days' worth of "GC" information, zoom right out
to 1%.

"GC VIEWER" TIP2: For our purposes here, deselect everything but "Total Heap"
(red area) and "Used Heap" (blue line) – this will unclutter the graph and
speed up the display.

For the sake of argument, let's say we have a system which is used from 9 to 5
every day, with no activity during the night. We have "GC" graphs from
Monday to Friday – the system was started on Monday morning – one hour before
the users start logging in; and for the time being, assume that there is no
publishing going on.

First locate the system startup: it will be reflected in the graph with a
slope upwards and some general noise, and it will then flatten out as the
system becomes ready for use.

As the users start logging in and continue to use the system throughout the
day, the graph will climb upwards ("up ramp").

At the end of the day, the graph will start to climb downwards ("down ramp").

During the night, the graph will level off to a "plateau".

The pattern will repeat for each day.

Ok, so how do you see whether the system's memory is growing – or even leaking?
One possible pattern is a 'staircase' – the "up ramps" will be larger than the
"down ramps" – and each nighttime "plateau" will be higher than the previous
one.

Note, the above is an oversimplication. Sometimes memory will (rightly)
increase some days more than other days, and the plateau will move upwards.
This can be accounted for: there are settings within WebLogic (for instance,
JDBC pools) which are designed to grow, so that they can dynamically match
system demand.

If, however, you see large and regular increases between each plateau – say
50–200 Meg per day – then chances are that the system will run out of memory
within a matter of days and you have a problem. (For instance, if each plateau
is 250 meg greater than the last, after 4 days memory usage will have moved up
by a Gig – this will cause problems.)

STEP 3 (approach 1)
===================

The VCM has a lot of 'moving parts': custom–listeners, custom workflow program
tasks, custom widgets.

Try and isolate (temporarily disable if possible) separate components and take
the "GC" stats again – or force load onto candidates (e.g., construct tests
where particular listeners will fire over and over) and observe the effect on
the "GC" graphs.

Work your way in – start with custom–code and eliminate each bit in turn,
working your way back into the VCM itself.

STEP 4 (approach 2 – )
===================

Profiling is a mechanism whereby the JVM is configured (or temporarily
replaced) to provide extra information about memory (heap) usage, allowing you
to observe which objects are contributing to memory–use.

This process has a severe effect on system performance and should not be done
on a production environment – instead use test users (or scripts) to simulate
real–system usage.

There are a number of profiling tools available, we suggest (but do not
recommend any particular one) these two as possible candidates. You will need
to download and license these products yourselves. Vignette does not supply
these tools.

"Mission Control" – from BEA
"JProbe" – From Quest

Profiling tools allow you to take "snapshots" of heap usage: tables of data,
indicating how many classes are loaded, how much memory the classes are using,
and what classes reference other classes.

Heap profiling tends to show primitive and built–in Java objects (e.g.,
"String", "Char[]", "Vector[]", "int[]", etc.) at the top of the heap; you need
to trace references back to higher–level objects before the profiling makes
sense.

Once you have established a set of candidate high–level objects (e.g., they are
numerous, or memory–heavy – or both), you then need to correlate that to code:
for instance, if you have the candidate class 'MyPDFWriter', then you need to
locate where this object is being used – in a listener? in a workflow task ?

STEP 5 – Retest
===============
Once you have debugged and modified the offending code, you will need to go
back and verify the changes have worked: take more snapshots, take more GC
graphs.

It's possible that at this stage you may uncover another set of objects which
now take on the the role of most numerous or most memory–heavy, and you may
need to iterate the process until you have the memory situation under control.