01166: Excessive Garbage Collection pause times on large BBJ installations.

Title:

Excessive Garbage Collection pause times on large BBj installations.

Description:

Beginning with 1.6.0_18, Oracle Java includes a ‘zero setting’ option that dynamically allocates memory to Java applications, based on the hardware profile of the client/server and the memory needs of the application. If you are unable to take advantage of Oracle Java’s dynamic memory allocation, follow these instructions for manually tuning your memory settings.  

Dynamic Tuning of Memory Allocation

Note: Dynamic tuning of memory allocation is only available with Oracle Java installations.

BASIS highly recommends taking advantage of this setting; versions of BBj 10.03 and higher are pre-configured to run this way.

For further detail see the excerpt below from Oracle’s 1.6.0_18 Release Notes:

Garbage collection improvements

Updated Client JVM heap configuration

In the Client JVM, the default Java heap configuration has been modified to improve the performance of today’s rich client applications. Initial and maximum heap sizes are larger and settings related to generational garbage collection are better tuned.

The default maximum heap size is half of the physical memory up to a physical memory size of 192 MB and otherwise one fourth of the physical memory up to a physical memory size of 1 GB.

For example, if your machine has 128 MB of physical memory, then the maximum heap size is 64 MB, and greater than or equal to 1 GB of physical memory results in a maximum heap size of 256 MB.

The maximum heap size is not actually used by the JVM unless your program creates enough objects to require it. A much smaller amount, termed the initial heap size, is allocated during JVM initialization. This amount is at least 8 MB and otherwise 1/64 of physical memory up to a physical memory size of 1 GB.

The maximum amount of space allocated to the young generation is one third of the total heap size.

The updated heap configuration ergonomics apply to all collectors except Concurrent Mark-Sweep (CMS). CMS heap configuration ergonomics remain the same.

Server JVM heap configuration ergonomics are now the same as the Client, except that the default maximum heap size for 32-bit JVMs is 1 GB, corresponding to a physical memory size of 4 GB, and for 64-bit JVMs is 32 GB, corresponding to a physical memory size of 128 GB.

(end of Oracle’s notes)

Manual Tuning of Memory Allocation

If you are unable to take advantage of Oracle Java’s dynamic memory allocation, please follow the guidelines below to tune your memory settings.

The following are guidelines for manually tuning the JVM on large installations of BBj (approximately 1 GB or more of RAM, two or more processors) using Sun Microsystems HotSpot JVM.

First, determine if long GC times are occurring. Add the following arguments to JVM Arguments in Enterprise Manager (EM ) or to the basis.java.args.BBj Services line in the BBj.properties file to create a log file for the GC times:

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:<basis log dir>/gc.log

If editing the BBj.properties directly, add a ‘\’ before every ‘:’ and ‘=’ in any of the properties. Here is a sample of the modified line in the BBj.properties file located in <bbj_isntall_dir>\cfg directory:

basis.java.args.BBjServices=-Xmx912m -Xms912m -XX\:NewRatio\=4 -XX\:CompileCommandFile\=C\:\\basis\\cfg\\.hotspot_compiler -Dnetworkaddress.cache.ttl\=10 -Dsun.net.inetaddr.ttl\=10 -Xloggc\:\\basis\\gclog -XX\:+PrintGCDetails -XX\:+PrintGCTimeStamps

The last “n.nnn secs” of each line in the file ‘gc.log’ shows how long that line took. Excessive times in this field is indication of long GC times. It can be helpful to import the gc log into Excel and sort by the last column to determine how long the full gc’s are taking.

The default collector on “server-class” machines is the parallel collector (option -XX:+UseParallelGC). The definition of “server-class” may vary over time and difficult to define. This collector is intended to work very efficiently with many processors, but does not guarantee low pause times. With responsive applications, server pause times of more than 1 or 2 seconds are usually quite noticeable.

Another collector, the concurrent low pause collector, or CMS collector (option -XX:+UseConcMarkSweepGC) is intended for this usage pattern. If you decide to switch to this version, here are some basic guidelines:

1. Add about 40% larger -Xmx. This collector does not compact data in the old generation, and memory fragmentation can become an issue.

2. Ensure the following options are enabled:

-XX:+PrintHeapAtGC -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=60 -XX:+CMSClassUnloadingEnabled

This should keep heap fragmentation from becoming an issue, and keep the stop-the-world phases of the CMS collector from being too long.

3. Add the following options:

-XX:MaxNewSize=250m -XX:TargetSurvivorRatio=90 -XX:+UseParallelNewGC -XX:SurvivorRatio=6 -XX:MaxTenuringThreshold=8

This will attempt to keep young generation pause times low, and will attempt to delay promoting objects that don’t last very long to avoid fragmentation. For each core on the system, MaxNewSize may increase by 50m (so for 4 processors, this could be 350m).

Note: For 64-bit systems add 30% to all memory sizes since the overhead of OS structures increases. Ex:

-XX:MaxNewSize=250m

to:

-XX:MaxNewSize=325m

The logs generated by the diagnostic options will greatly help with tuning these parameters further if necessary.

Client-side tuning can be challenging because Java Web start does not support all tuning options. However, a stand alone ThinClientProxyServer and Web Start have nearly identical performance characteristics. So, we can use the above techniques to identify an appropriate amount of memory for a client machine. Client side-virtual machines require too much memory by default on long running applications. Requiring less memory can improve long running client performance problems.

---

Last Modified: 08/11/2014 Product: BBj Operating System: All platforms

BASIS structures five components of their technology into the BBx Generations.