Reach for a cache when it is expensive and repetitive
A cache is worth adding when two conditions hold at once: the underlying operation is genuinely costly, and the same input comes back often enough for the repetition to matter. One without the other is not a cache problem.
Miss both and you have added a layer of invalidation for no gain. Meet both and the win is usually large — a schema lookup, a permission check, a parse of a large file — because what you are really buying is not speed so much as the ability to stop asking.
What LRU actually guarantees, and what it does not
LRU evicts the least recently used entry when the cache is full. That gives you two things:
Bounded memory. The cache cannot grow without limit, which is the whole reason you added it. An unbounded map holding List<ToolResult> will eventually take the process down, and it will do so on the busiest day rather than a quiet one.
Recency, not importance. This is the part people get wrong. LRU is a policy about time, not about value. An entry that is expensive to compute and used twice a year will be evicted before a trivial one that was touched a second ago. If your access pattern is genuinely skewed — a few hot entries and a long cold tail — that is usually fine. If every entry is roughly equally useful, LRU is throwing away work you paid for and an LFU or a plain TTL would serve you better.
The library or the twelve lines
In Java this is a real fork in the road.
A hand-rolled LRU is a LinkedHashMap in access order with one overridden method:
new LinkedHashMap<>(16, 0.75f, true) {
@Override
protected boolean removeEldestEntry(Map.Entry<K, V> eldest) {
return size() > MAX;
}
}
That is correct, dependency-free, and about fifteen lines including the imports. It is also entirely synchronous — get on an access-ordered map can restructure it, which means any concurrent access needs external locking.
Guava’s CacheBuilder gives you the same policy with a great deal more around it:
Cache<String, Result> cache = CacheBuilder.newBuilder()
.maximumSize(512)
.recordStats() // you get hitRate(), evictionCount()
.refreshAfterWrite(15, TimeUnit.MINUTES)
.build(loader);
recordStats() is the reason to choose it. Without it you are running a cache you cannot reason about, and cache tuning without hit-rate data is guessing. With it, cache.stats() gives you the two numbers that matter.
The honest summary: use the library if it is already on your classpath, because the statistics and the concurrency are worth more than the dependency. Write it yourself if it is not, and you have learned something.
The cache key is the whole risk
Here is the failure mode that actually bites people, and no amount of correct eviction policy protects you from it.
A cache returns a wrong answer silently. Nothing throws, nothing logs, the test suite passes, and the bug surfaces weeks later as a report of “it sometimes shows the wrong thing.” Every one of those is a case where the key did not capture every input that affects the output.
So the key needs to include:
- What you are computing. The operation name, not just the arguments — two operations with identical parameters are different results.
- Every parameter, hashed if it is long.
Hashing.sha256().hashString(paramsJson, UTF_8).toString()is the easy version. The discipline is what matters: if a parameter can change the result, it is in the key or you have a bug. - The scope. If the result depends on a project, a tenant, a user or a permission set, that goes in the key too. This is the omission that turns a caching bug into a security incident.
- A version, if the computation itself might change. Cache entries do not survive a deploy that fixes the thing they were caching.
And the test that catches it is cheap: for a handful of inputs, call the function twice with identical parameters and assert the results are equal to a fresh, uncached call. If they are not, the key is wrong.
Size it from the eviction rate, not the hit rate
Hit rate tells you whether the cache is useful. It rarely tells you whether it is the right size, because a cache that is too small and a cache that is too large can both plateau around the same number.
Eviction rate is the better signal, and it is the one people forget to look at. Sustained evictions mean you are paying the full cost of the operation and then throwing the answer away — strictly worse than no cache, because you have added overhead on top. If you see a meaningful eviction rate, the fix is a bigger maximumSize, not a better algorithm.
refreshAfterWrite is the other half. A hard expireAfterWrite makes an entry disappear and the next reader pays full price. A soft expiry keeps serving the stale value while one caller reloads in the background, so a burst of traffic after expiry costs you one miss rather than a hundred.
Measure before you tune
Two numbers, continuously, or you are not tuning:
CacheStats s = cache.stats();
log.info("hitRate={} evictions={}", s.hitRate(), s.evictionCount());
The one that will actually save you is the leading indicator. A rise in misses usually shows up before the latency it causes — often attached to something recognisable: a deploy, a traffic step, a config change, a peak. If you can see the miss rate climb during a deploy, you know the new release is generating keys nothing has cached yet, and you can pre-warm instead of watching the p99 climb.
What a cache will not do
It will not make the underlying operation faster. If a lookup takes 400ms because the thing you are caching is slow, every miss still takes 400ms, and a cache with a bad hit rate makes the average worse.
It does not remove the need to invalidate. LRU evicts on recency, which means it will happily serve a stale-but-recent entry indefinitely if the underlying data changes and the key does not. For anything with a real invalidation signal, a TTL is doing more of the work than the eviction policy is.
And it adds a class of bug that does not exist without it — the silent wrong answer. That is the real cost of caching, and it is worth pricing in before you start.