Continued from Part 1 of this article.

The Experiment....

As part of this exercise, I wanted to know what really happens when different eviction policies were used for the Ehcache. In the local development environment, purely for the experiment purpose, I configured the maximum elements to be 5 (which means that the cache will store the ARC record collections of 5 web harvests, and will start evicting elements once this limit is reached), and tested the effect of LRU versus LFU as the eviction policy. The results were interesting.

The test started by viewing one of the very large web harvests (over 76,000 ARC records) and then continued to view the smaller ones, up to 8 web harvests. Once I started viewing the 6th web harvest (that is, after when the cache had 5 elements in it), the eviction algorithm kicked in.

• Least Frequently Used (LFU): Since the last-viewed web harvest was the smallest among the web harvests viewed and thus was accessed the least in the cache, the ARC record collection of this web harvest was evicted when LFU was used. Since this was the smallest, it removed only a few hundred ARC records from the cache, providing only a very little gain in heap space.

• Least Recently Used (LRU): The first-viewed web harvest was the least recently used among the web harvests viewed. With LRU, the ARC record collection of this web harvest was evicted, and since it had the largest number of ARC records, the total number of ARC records referenced by the cache dropped hugely by about 76,000 leaving only about 27,000 really live ARC record objects. This was a huge gain.

The tabulated result is given below. The high-lighted cells show the remaining records in the cache after the eviction started.

Shown also is a graph indicating the gain in the heap space for the two eviction policies. The LRU was a much bigger memory-saver compared to LFU in our case.

Does it mean that LFU is bad and LRU is good? Not at all. It was just that LFU was not good for our usage.

Few Recommendations....

Caching is definitely good, provided a disciplined approach is followed while designing, using and configuring them. I would like to suggest few tips to use while designing cache-based solutions:

• Don’t use a single blotted cache to store all kinds of objects to be cached. Instead, use separate cache for storing each kind of objects so that each cache can be configured/tuned differently. For example, separate the caches that store short-lived, session-bound user data from the caches that store long-living application-wide objects.

• It is wiser to stay away from creating your own cache implementations using Java Collections API. There are already plenty of products (open-source and commercial) available in the market, such as Terracotta Ehcache and Apache JCS.

• As in the case of Java Collections API, when doing heap analysis of cache and the objects held within, make sure to inspect the "retained heap" space rather than shallow heap. For example, as seen in the screenshot below of the heap analysis from the production system, while the shallow heap of the cache elements was very small, the retained heap was close to 2GB - And these were the memory used by ARC records indirectly referenced by the cache.

   

  

• Know your objects well. Learn about the objects that are going to be stored in the cache and use this knowledge to choose the expiration/eviction strategy accordingly.
  • If the objects are of uniform size, the eviction strategy based on the number of elements in the cache may be used (such as LRU or LFU).
  • On the other hand, if you can’t predict the size of one object (which was the case of the ARC record collection in web archive viewer), then one may have to experiment with various eviction policies. If the cache framework that you use supports evictions based on the utilised heap size (in absolute value or percentage of maximum heap), then one could use this.

• As in Ehcache, the cache framework may not evict the expired elements from the cache until the limit on the maximum cache elements is reached. If memory is the utmost concern of the application, one could write logic to evict expired elements from the cache and run this at off-peak as a low priority thread. However, care must be taken since such eviction operations can be synchronized and thus may create performance bottlenecks.

• With the introduction of annotations in Java, there has been a tendency to provide the cache configurations as annotations on a cachable object (e.g. Ehcache annotations for Spring framework). While this improves the code readability, this can cause a maintanance nightmare several years later when all developers would have left the organisation and the application support team needs to investigate the reasons for blotted cache!! Thus, prefer to keep the configurations outside the code so that the cache could be reconfigured without editing/recompiling the code.

• Cache frameworks provide facility to have the elements overflow to secondary storage. While this is a useful feature, use it cautiously since the secondary storage access will usually be slower. If the objects to be stored in cache are derived from a time-consuming operation, then it is worth using a cache that overflows to disk. Otherwise, this feature may be as good as not storing the objects in the cache.
  • Could we have used this approach? Probably not, because size of each cache element was unpredictable and thus an overflow of just one cache element to disk could cause 100K’s of ARC Record objects being written onto disk.

Are things like cache tuning a development concern or an architecture concern? The boundary is fuzzy. In the case of web archive viewer, it was indeed an architecture concern, primarily because it shared the very same deployment environment as the highly business-sensitive digital preservation application.