Keeping a Legacy Cache Consistent During an Incremental Migration

During an incremental system migration, I ran into a subtle consistency problem: our new service could successfully update the database while the legacy service continued reading stale data from its own in-memory cache.

The obvious solutions such as distributed locks, cache invalidation, or adding another piece of infrastructure didn't fit our constraints.

The solution ended up being much simpler: the legacy system already had a lease mechanism that we could reuse.

This is the story of how I found it, how we used it, and how we handled the failure cases.

Architecture

Our system currently has three main components:

1. Main Service: The client-facing service that sends business requests.
2. Legacy Service: An older, high-performance engine that maintains an embedded in-memory cache. It loads a subset of database tables into its own memory for fast access.
3. New Service: A modern service that is gradually replacing the Legacy Service using the Strangler Fig pattern.

The New Service currently acts as a proxy between the Main Service and the Legacy Service.

If an endpoint has already been migrated, the New Service handles the request directly. Otherwise, it forwards the request to the Legacy Service.

The architecture looks roughly like this:

The Problem

The problem appeared when the New Service handled a migrated request.

The New Service writes updates directly to the database. However, the Legacy Service has its own in-memory copy of the data and has no way of knowing that the database changed.

That creates a dangerous sequence:

1. The Legacy Service caches status = pending.
2. A request is migrated to the New Service.
3. The New Service updates the database to status = completed.
4. A later request reaches the Legacy Service.
5. The Legacy Service reads status = pending from its local cache.

The database is correct, but the Legacy Service is operating on stale state.

We discovered these issues during QA testing. As more endpoints moved to the New Service, the problem would become increasingly common.

So we needed a way to guarantee that the Legacy Service could not continue operating on stale data while a migrated endpoint was modifying the same resource.

The Constraints

The textbook solutions weren't a good fit for this system.

We had several constraints:

1. The Legacy and Main Services should not be modified. The Legacy Service is largely a black box, so modifying its behavior is risky.
2. The Legacy Service could not temporarily operate on stale data. For these operations, we needed to prevent conflicting legacy operations rather than accept eventual consistency.
3. The Legacy Service had limited cache synchronization capabilities. It automatically reloaded its cache only periodically, or through a manual refresh operation.
4. We wanted to avoid introducing new infrastructure. Adding something like Redis purely for distributed locking would introduce another dependency to provision, operate, and maintain.

These constraints ruled out several conventional approaches.

The Naive Solution

My first thought was:

What if the New Service acquires a distributed lock, performs the database update, and then triggers a cache refresh on the Legacy Service?

That sounds reasonable, but there were two problems.

1. The Legacy Service wouldn't know about the lock

An external distributed lock only coordinates operations if the participating systems can use it.

The Legacy Service was a black box and wouldn't check our new lock before accessing its local cache.

We could add that behavior, but that violated one of our main constraints.

2. Refreshing the entire cache was too expensive

The Legacy Service's manual refresh operation reloaded a large amount of data.

Doing that after every migrated request would introduce significant latency and unnecessary work.

We needed something more targeted.

The Solution: Reusing an Existing Lease

While investigating the Legacy Service's existing APIs, I found something useful: it already had a built-in lease mechanism.

The Legacy Service exposed APIs to:

1. Acquire a lease for a specific resource.
2. Release the lease when the operation was complete.

More importantly, releasing the lease had an existing side effect: the Legacy Service refreshed the cached data associated with that resource.

The lease mechanism was already part of how the Legacy Service protected its own critical sections.

This changed the problem completely.

Instead of introducing a new distributed lock, we could use the same coordination mechanism the Legacy Service already understood.

The New Service could wrap a migrated operation like this:

1. Acquire lease for the resource
2. Update the database
3. Perform business logic
4. Release the lease
5. Legacy Service refreshes that resource's cached data

The important part is that the lease wasn't merely an arbitrary lock exposed by the Legacy Service. The Legacy Service's own operations already align with the same lease mechanism.

That gave us two properties we needed:

• Conflicting Legacy Service operations could not proceed while the resource was leased.
• Releasing the lease caused the Legacy Service to refresh the affected resource rather than performing a full cache reload.

So instead of adding Redis, Kafka, or another synchronization system, I reused a primitive that already existed in the system.

What If Releasing the Lease Fails?

The happy path was straightforward.

But distributed systems rarely fail only on the happy path.

Consider this sequence:

New Service
    |
    | acquire lease
    v
Legacy Service
    |
    | lease acquired
    |
    v
New Service
    |
    | update database
    |
    X---- network failure ----> release request never arrives

The database update succeeded, but the release request never reached the Legacy Service.

If leases could remain indefinitely, the resource could become permanently locked.

So I looked at how the Legacy Service handled abandoned leases.

It already had two mechanisms:

• TTL (Time-To-Live): Every lease automatically expires after a configured duration.
• Background reaper: A background process periodically detects expired leases and cleans them up.

Even better, lease cleanup also triggered the same cache refresh behavior.

That meant the failure path became:

Acquire lease
     |
     v
Update database
     |
     X  release request lost
     |
     v
Lease expires
     |
     v
Background reaper
     |
     v
Lease cleared + cache refreshed

We therefore didn't need to build a separate retry system just to recover from a lost release request.

The existing lease lifecycle already provided a recovery mechanism.

An Important Invariant

The key correctness property here is that the resource remains protected by the lease until the Legacy Service releases or expires it.

As long as the database update completes before the lease is released or expires, the Legacy Service cannot resume normal access to the resource using its old cached state during that protected window.

When the lease is eventually cleaned up, the cache refresh brings the Legacy Service back in sync with the database.

This is an important distinction: the TTL isn't what provides consistency by itself. The consistency comes from the lease preventing conflicting access; the TTL and reaper provide failure recovery when the normal release path doesn't happen.

The Tradeoff

The solution wasn't free.

The main cost was latency.

If a resource has many related records that need to be refreshed, releasing the lease and synchronizing the cache can take noticeable time and I verified this through tests.

So I didn't view this as a permanent architecture.

It's a migration bridge.

During the migration, migrated endpoints use the lease mechanism to safely coordinate with the Legacy Service.

As more functionality moves to the New Service, individual resources eventually become fully owned by the New Service.

At that point, the Legacy Service no longer needs to read or modify those resources.

That's our exit condition.

Once a resource is completely owned by the New Service, we can remove the lease and cache-synchronization logic for it.

This is important because migration code has a tendency to become permanent if nobody defines when it can be deleted. So I documented this as well.

What I Learned

1. Start with invariants, not technologies

My first instinct was to think about distributed locks and infrastructure such as Redis.

But the more useful question was:

What must never happen?

In this case, the important invariant was that the Legacy Service must not perform conflicting operations against a resource while the New Service is modifying it.

Once that was clear, the technology choice became much easier.

2. Existing systems often contain useful primitives

At first, I thought of the Legacy Service as something we couldn't modify and therefore mostly had to work around.

But "don't modify the legacy system" doesn't mean "don't understand the legacy system."

By digging into its existing APIs and lease lifecycle, I found a mechanism that already provided most of the coordination and recovery behavior we needed.

Sometimes the best solution isn't introducing another system. It's discovering what the current system can already do.

3. Temporary migration code needs an exit condition

A migration introduces a lot of code that ideally won't exist forever.

The lease wrapper is useful while ownership is split between the Legacy and New Services. But once the New Service completely owns a resource, keeping the wrapper would only add unnecessary latency and complexity.

Defining the exact conditions under which the lease logic can be removed makes it much less likely that temporary migration infrastructure becomes permanent technical debt.

Final Thoughts

This experience changed how I think about solving distributed systems problems.

It's easy to reach for familiar tools like Redis for locking, Kafka for asynchronous communication, an outbox for reliable delivery.

Sometimes those are exactly the right tools.

But before introducing another component, it's worth understanding the guarantees and primitives that already exist in the system.

In this case, the legacy system already had a resource-level lease, automatic expiration, a background reaper, and targeted cache synchronization.

The challenge wasn't building a new distributed system.

In this case, the solution wasn't introducing another system. It was understanding the existing one well enough to find what we could reuse.