How to identify root cause for Frequent Minor Garbage Collections using DynaTrace

Frequent young generation (minor) GCs have two root causes :

1. Small young generation size for application work load
2. High Object allocation rate

In case your old generation size is growing quickly , it is quite possible that this is linked to small young generation size. This can easily be identified using Jconsole or jstat.

In case you have high object allocation rate then it becomes difficult to identify exact root cause for the same. Performance diagnostic tools such as dynatrace provides capability to identify root cause of such issue. In order to troubleshoot high object allocation rate in dynatrace , you need to add 'RunTime Suspension' dashlet.

RunTime Suspension report a massive statistical concentration of garbage collection in any particular function. This dashlet basically provides following line of information :

1. Method Name
2. Class Name
3. Suspension Count
4. Suspension Sum[ms]
5. Suspension Avg[ms]
6. Reason

Method with high suspension count suggests that method itself allocates enough objects to fill up young generation and thus triggers a GC. This way we can find prime candidate for optimization area.

 

Performance Impact of Data Structure Resizing in Java

Java applications tend to make very frequent use of StringBuilder or StringBuffer for assempling of Strings. Both StringBuilder and StringBuffer use char[] internally for their data storage. As elements are added to StringBuffer or StringBuilder , the underlying char[] may be subject to resizing. As a result a new char[] of larger size (2x size) is allocated and elements from old char[] are copied into new char[]. Old char[] are discarded then and it becomes available for Garbage collection.This whole process results in consumption of extra CPU cycles for new array allocation, copying of elements from old array to new array and at some future point cost of garbage collection cycle.

Above facts also true for other java collections that uses array for its internal data storage such as ArrayList, Vector, ConcurrentHashMap and HashMap. Other Collections such as LinkedList or TreeMap often use one or more object references between elements stored to chain together the elements managed by the Collections. Collections that use array for data storage , their relevant classes also provides constructors that provide optional size arguments , but these constructors are often not used or size provided in application program is not optimal for Collection's use.

For HashMaps , to actually view whether resizing is happening or not, you need to perform memory profiling either run time profiling or offline using dumps. In either case, if you are observing java.util.HashMap.resize(int) method call, then your HashMap is getting resized during application execution.

Collecting and Analyzing Redis Performance Metrics

Redis is an open source, in memory advance key value store with optional persistence to disk. Two common use of Redis is caching and publish-subscribe queues. 

How to access Redis performance metrics

Redis performance metrics is accessible through Redis command line interface (Redis-cli). Use the info command to print all performance metrics for your redis server.

Output of info command is grouped in following 10 groups :

1. server
2. client
3. memory
4. persistence
5. stats
6. replication
7. cpu
8. commandstats
9. cluster
10. keyspace

In case you just want to see metrics for any specific section, use info command. For example to print memory statistics for redis server use info memory :

Top Redis Performance Metrics:

• Memory Usage : used_memory

The used_memory metrics reports the total number of bytes allocated by Redis. This metrics reflects the amount of memory in bytes that Redis has requested to store your data and supporting metadata it needs to run. This metrics doesn't consider memory wastage due to fragmentation , meaning amount of memory reported by this parameter will always be different than total amount of memory that Redis has been allocated by Operating System.

• total_commands_processed

total_commands_processed metric give number of commands processed by Redis server. Tracking number of commands processed is critical for diagnosing latency issue in Redis instance. Because Redis is single threaded so commands are processed sequentially. The typical latency in 1 GB/s network is about 200 micro second. In case you are seeing slow response of commands and latency is greater than 200 micro second, this could be because of high number of commands in command queue.

• mem_fragmentation_ratio

The mem_fragmentation _ratio metrics gives the ratio of memory used as seen by the operating system (used_memory_rss) to memory allocated by Redis (used_memory)

Memory Fragmentation ratio = used_memory_rss/used_memory

If fragmentation ratio is outside range of 1 to 1.5 , it is likely sign of poor memory fragmentation by either Operating system or Redis instance.

• Evictions

The evicted_keys metrics gives number of keys removed by Redis due to hitting maxmemory limit.