Caching & the two hard problems

This is the first lesson of the building-blocks stage, so one orientation note. Two of the "core building blocks" — load balancers and CDNs — already have full deep-dives in the networking track; this stage doesn't repeat them. What it adds are the pieces every backend reaches for that aren't on the request's network path: caching, queues, object storage, and rate limiting. We start with caching, because it's the one you'll reach for first and misuse most.

A cache is a fast copy of data kept close to whoever needs it, so you don't have to recompute or re-fetch it from a slower source. That's the entire idea, and its power is that it's fractal — the same pattern appears at every layer you've studied. The CPU caches memory in L1/L2/L3. The OS caches disk in the page cache. Postgres caches disk pages in its buffer pool. A CDN caches origin responses at the edge. The browser caches HTTP responses. Application caching — an app keeping hot data in Redis or memcached instead of hitting Postgres every time — is just this same move at one more layer. Learn the trade-offs once and they transfer everywhere.

Why cache at all: latency and load

Two distinct wins, and it's worth separating them because they justify caching in different situations:

• Latency. Reading from an in-memory cache (a Redis GET over the local network, ~0.2–1ms) is one to two orders of magnitude faster than the query it replaces (a Postgres query that plans, hits indexes, and reads pages — often several ms to tens of ms). The user feels the difference directly.
• Load. Every request served from cache is a request that didn't touch the database. For read-heavy workloads (most apps) this is often the difference between one modest database handling everything and needing read replicas or sharding. The cache absorbs the reads the database would otherwise have to.

The trade-off you accept in return is the whole rest of this lesson: a cached copy can be stale, and keeping it honest is genuinely hard.

Reading and writing through a cache

There's a small vocabulary of patterns, and the choice is about who talks to the database and when.

Cache-aside (a.k.a. lazy loading) — the default, and the one you'll write most. The application owns the logic: on a read, check the cache; on a hit, return it; on a miss, read the database, write the result into the cache, and return it. The cache only ever fills with data someone actually asked for. Its characteristic quirk: the first request for any key is always a miss (a "cold" cache), and there's a window on a miss where two requests can both miss and both query the database (hold that thought — it's the stampede below).

Read-through — same read behavior, but a cache library sits in front of the database and does the miss-fetch transparently, so the application just asks the cache and never sees the database directly. Cache-aside with the plumbing hidden.

Write-through — on a write, update the cache and the database synchronously, together. The cache is never stale for that key, but every write pays for both operations, and you cache data whether or not it's ever read again.

Write-back (write-behind) — on a write, update the cache and acknowledge immediately, then flush to the database asynchronously a little later. Writes are very fast and you can batch many into one database write — but there's now a window where an acknowledged write lives only in the cache, so a crash loses it. This is exactly the trade the WAL/fsync lesson made you suspicious of: fast-but-not-yet-durable. Use it only where losing a few recent writes is acceptable.

Write-around — on a write, skip the cache entirely and go straight to the database; the cache only picks the key up later, on a subsequent read that misses. This is the odd one out among the three, and the contrast is the whole point: write-through and write-back both put every write through the cache, on the bet that it'll be read again soon. Write-around makes the opposite bet — that this particular write probably won't be re-read soon — so pushing it through the cache on the way in would just occupy space with something nobody's about to ask for. Bulk imports, audit logs, and high-volume event ingestion are the classic fits: you write a lot, you rarely re-read what you just wrote through this same path, and pre-warming the cache with it only evicts data that's actually earning its keep. The cost is unavoidable and specific: the very next read of that key is a guaranteed miss, even in the rare case where someone does want it a moment later — write-around never optimizes for that case, only for the common one where nobody does.

Read twice to see a miss then a hit, then write and read again. Under write-around the cache keeps serving the old value — a stale read — while invalidate and write-through stay fresh. The staleness counter is the invalidation problem made visible.

Keeping the cache bounded: TTL and eviction

A cache is finite memory, so entries have to leave. Two mechanisms:

• TTL (time-to-live) — each entry gets an expiry; after it, the entry is gone and the next read is a miss that repopulates. TTL is also the simplest invalidation strategy (below): set a short TTL and the cache is never more than that stale.
• Eviction policy — when the cache hits its memory limit, which entry does it drop to make room? LRU (least-recently-used — evict the entry untouched longest) is the common default and a good fit for the "recently used data will be used again" locality most workloads have. LFU (least-frequently-used) keeps the popular entries even if not just-touched. Redis exposes exactly these as maxmemory-policy. The policy matters: a bad one evicts your hottest keys and your hit rate collapses.

Neither LRU nor LFU is safe by default, and the failure modes are worth naming because they're the reason the other two policies below exist at all. LRU's specific weakness is a sequential scan: a batch job or a full-table export that touches your entire dataset once makes every single item look "just used," so LRU happily evicts your actually-hot, repeatedly-accessed keys to make room for a one-time pass that will never be read again — a failure mode with its own name, cache pollution. LFU's weakness is the mirror image: an item that was extremely popular last month keeps a high accumulated count and squats on cache space today's genuinely popular items need, unless the implementation decays old counts over time.

Two more policies exist specifically to answer those two weaknesses, and they're worth knowing even though Redis doesn't expose either as a maxmemory-policy option:

• FIFO (first-in-first-out) — evict whatever was inserted longest ago, full stop, with no access tracking of any kind. Contrast this with LRU directly: LRU pays a small bookkeeping cost on every access to know what's "recent"; FIFO pays nothing, at the cost of ignoring usage entirely. An item read a thousand times a second gets evicted at the same age as one nobody has touched since it arrived, if the two arrived together — which is exactly why FIFO is rarely the right default for a general-purpose application cache, but is cheap enough that it shows up in simple buffers and streaming windows where age genuinely is the right proxy for relevance.
• ARC (adaptive replacement cache) — runs recency-tracking and frequency-tracking side by side and shifts weight toward whichever has been more predictive recently, instead of committing to one bet the way LRU and LFU each do alone. It's self-tuning, with no policy parameter for you to get wrong, and it's the reason ZFS uses it for its own page cache. The trade is implementation complexity — ARC is genuinely more involved to build correctly than LRU, LFU, or FIFO, which is why you'll typically reach for a mature implementation rather than write your own.

The number that measures all of this is the hit rate — the fraction of reads served from cache. A cache with a 30% hit rate is barely helping; one at 95% is carrying the system. If your hit rate is low, the cache is costing you memory and complexity for little gain — measure it before trusting it. And a low hit rate is not always an eviction-policy problem: no policy, however clever, helps a workload with no locality at all — if access is genuinely uniform and random across your dataset, every policy converges to the same poor hit rate, because there's no pattern left to bet on.

The hard part: invalidation

There's a famous line (Phil Karlton): "There are only two hard things in Computer Science: cache invalidation and naming things." Here's why the first one is hard. The moment you keep a copy, you have two representations of the same fact — the source of truth and the cached copy — and the instant the source changes, the copy is a lie. Invalidation is the discipline of keeping that lie from being served. Three strategies, in increasing precision and cost:

• TTL-based (expire and refresh). Don't actively invalidate at all — just let entries expire. Dead simple, no coordination, but you accept staleness up to the TTL: for a short window after a change, reads serve the old value. Fine for data where being a few seconds stale is harmless (a feed, a count, a rarely-edited profile). This is eventual consistency by another name, and it's the right default far more often than people admit.
• Explicit invalidation on write. When the source changes, actively delete (or overwrite) the cache key so the next read misses and repopulates fresh. Precise, but now your write path has to know every cache key affected by the change — and that coupling is where invalidation bugs breed (forget one key, and it serves stale data indefinitely; the classic "why is this still showing the old value" bug). Deleting rather than updating the key is usually safer — it avoids caching a value you computed in a racy in-between state.
• Write-through, from above — the cache is updated in lockstep with the database, so it's never stale. Precise, at the cost of write latency and coupling.

There is no free option. Every design is a point on the staleness-vs-cost line: TTL trades correctness for simplicity; explicit invalidation buys freshness with fragile write-path coupling; write-through buys it with latency. The senior move is choosing the cheapest strategy the data can tolerate, per kind of data — not reaching for the most precise one everywhere.

Go deeper

• Redis docs — key eviction & maxmemory policies — The concrete mechanics of TTL and LRU/LFU eviction in the cache you'll actually deploy — read it to see how the abstract policies map to real config.
• AWS Builders' Library — "Caching challenges and strategies" — A production-grade tour of stampedes, negative caching, and invalidation from engineers who run caches at enormous scale — the failure-mode callout, expanded.
• Facebook — "Scaling Memcache at Facebook" — The canonical real-world paper on application caching: lease-based stampede prevention, invalidation, and what breaks at scale — dense but foundational.
• Wikipedia — Cache replacement policies — The authoritative catalog of eviction policies beyond LRU/LFU — including FIFO and ARC, and the sequential-scan pollution and stale-popularity weaknesses this lesson names above.

Check yourself

Answer out loud, as if an interviewer asked. If you hand-wave, reread that section.

1. A cache is described as fractal — the same idea at every layer. Name four distinct caches between a CPU register and a user's browser, and state the single idea they all implement.
2. Contrast cache-aside, write-through, write-back, and write-around on two axes: is the cache ever stale, and can an acknowledged write be lost in a crash? Which one echoes the WAL lesson's fast-but-not-durable trade, and which one is betting that this write won't be re-read soon?
3. Explain why invalidation is called one of the two hard problems, in terms of having two representations of one fact. Give the staleness-vs-cost trade-off across TTL, explicit invalidation, and write-through.
4. LRU and LFU each have a specific, named failure mode. Describe cache pollution from a sequential scan under LRU, and the stale-popularity problem under LFU — then explain what FIFO gives up to avoid tracking either, and what ARC does instead of committing to one bet.
5. Walk through a cache stampede on a hot key: why does key popularity make it worse, and what does a single-flight lock or probabilistic early recomputation change?
6. Cache penetration and cache avalanche are different failures with different fixes. Define each, and give the fix (negative caching / Bloom filter for one; TTL jitter for the other).
7. Fable caches media and idempotency responses but reads balances straight from Postgres. Justify that split using the "cheapest strategy the data can tolerate" principle.