The wait queue is just a channel: building a small distributed lock server in Go

Sooner or later you hit the same small problem: two services, on two machines, want to touch the same thing at the same moment — append to a shared file, update a row nobody is fencing, call an API that tolerates one caller at a time. One of them has to wait.

The usual answers feel heavier than the problem. Put a service in front and serialize everything through it — now you are building a queue, and then a second queue to hand results back, because you no longer know the outcome at the moment you asked. Cache the resource in Redis and lock there — fine until the resource does not fit in memory, and you have inherited Redlock's ordering guarantees (there are none) and its debates.

I wanted the lock as its own primitive: lock a key, do the work, unlock the key. Nothing else. That is Locking-Center — a single binary, one dependency, no config file, no consensus layer to operate. This post is about the three ideas that made it small enough to be worth trusting.

1. One channel per key — and the queue comes for free

Every key gets a Go channel with a buffer of exactly one:

type Channel struct {
    key       string
    mutexChan chan bool // buffered, capacity 1
}

func NewChannel(key string) *Channel {
    return &Channel{key: key, mutexChan: make(chan bool, 1)}
}

Sending into it acquires the lock. Receiving from it releases:

c.mutexChan <- true // acquire — blocks if someone already holds the key
// ... critical section ...
<-c.mutexChan       // release

The buffer of one is the whole trick. The first send fills the buffer and returns immediately: that caller holds the key. The second send has nowhere to go, so it blocks — and so does the third, and the fourth. The blocked senders are the wait queue. When the holder releases (a receive frees the slot), the runtime wakes the next blocked sender.

And it wakes them in order. The Go runtime keeps a FIFO wait queue behind every channel, so callers are served roughly in arrival order rather than whoever happens to reschedule first. That single property — blocking, FIFO-fair acquisition — is the thing a Redis SETNX loop cannot give you: SETNX makes every caller spin and retry, with no ordering at all. Here it falls out of the language for free.

No condition variables, no ready-queue bookkeeping, no priority list. A map of keys to channels, and the runtime does the hard part.

2. The catch: you cannot tap a goroutine parked on a send

The elegant version above has a fatal gap. Once a goroutine is blocked on c.mutexChan <- true, it is unreachable. You cannot cancel it, time it out, or tell it "never mind." And you need to, for two reasons:

• An operator runs reset to clear a key a crashed client left held — every waiter on that key has to be let go.
• A client drops its connection while queued. If you hand it the key later, nobody is left to release it — a stuck lock you created yourself.

So next to the channel there is a second structure: a plain map, keyed by each request's id.

func (c *Channel) PushContext(ctx context.Context, r *Request) bool {
    c.pushToQueue(r) // register in queueMap[r.Id] — our only handle on this waiter

    select {
    case c.mutexChan <- true: // won the key
    case <-c.closeChan:       // a reset dropped this whole key
        c.pullFromQueue(r.Id)
        return false
    case <-ctx.Done():        // the requester went away while waiting
        c.pullFromQueue(r.Id)
        return false
    }

    c.setLatest(c.pullFromQueue(r.Id))
    return true
}

The map is not the queue — the channel is. The map is the cancellation registry: the only way to reach a request that is otherwise buried inside a channel send. The select gives a parked waiter three exits instead of one — it wins the key, the key gets reset, or its context is cancelled.

That last exit is what makes a dropped client safe. The server spawns a goroutine that just reads the connection until it ends, and cancels the context when it does:

ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go watchForHangUp(conn, cancel) // reads until EOF, then cancel()

for locked := false; !locked; {
    locked, err = lock.LockContext(ctx, key, sourceAddr, conn.RemoteAddr())
    if errors.Is(err, context.Canceled) {
        return io.EOF // client hung up while queued — withdraw it, answer nobody
    }
    // ...
}

A client that dies in the queue is pulled out of it, and the next live requester gets the key. Without this, the queue slowly fills with ghosts that each get handed a lock they will never release.

3. Persistence without a database — a log that only fails safe

By default the lock table lives in memory, which means a restart — including every deploy — drops every key, and the clients holding them are never told. Two of them can then end up in the same critical section: one from before the restart, one from after.

Set DATA_PATH to a file and held keys are kept in a write-ahead log. The interesting part is not the log; it is the ordering, which is chosen so the system can only ever fail in one direction.

The rule for acquiring: the key goes on disk before the client is told it holds it.

// won the key in memory; make it durable BEFORE acknowledging
if err := l.appendLocked([]Entry{AcquireEntry(report)}); err != nil {
    channel.Pull()          // couldn't persist -> release it again, answer '-'
    return false, err
}
return true, nil            // only now does the client get '+'

Play out the crash windows:

• Crash after the write, before the +. On restart the key is restored as held — but the client was never told, so it never entered its critical section. The key is now held by nobody.
• The write fails. The key is released again and the client gets -. It knows it does not hold the lock.
• Release is recorded before it takes effect. The other order could resurrect a key that nobody holds.

Notice what is not on that list: two clients believing they hold the same key. The design accepts leaving a key held by nobody — which looks exactly like a crashed client, and is cleared the same way, by a reset — in exchange for never handing one key to two owners. A held-by-nobody key even trips the same Prometheus stuck-lock alert, so you find out about it.

The log stays small on its own. Each lock and release is one appended record with a CRC32C checksum; when the log roughly doubles over the set of currently-held keys, it is compacted — rewritten to hold only the live holders, through a temp file and a rename so a crash mid-compaction leaves the old log intact. A torn final record from a crash is dropped on the next start; a record that is whole but fails its checksum is real corruption and refuses to start, rather than silently misreporting the table.

The part that makes nine clients cheap: a deliberately boring protocol

A lock is only useful if every service can speak to it, so the wire protocol is kept trivial. One request per TCP connection:

[action:1][keySize:1][key][sourceSize:1][source]

Five actions — 1 Lock, 2 Unlock, 3 ResetByKey, 4 ResetBySource, 5 TryLock — and a one-byte answer: + success, - failure, # for a try-lock that did not win the key (held by someone else, not an error). Sizes are a single signed byte, so a key is at most 127 bytes.

Writing a client is: open a socket, write a handful of bytes, read one byte. That is the entire reason there are nine of them — Go, Rust, Python, Java, C#, JavaScript, C, C++ and Zig — all offering the same small API (Lock, TryLock, Unlock, Wait, and the two resets). Lock blocks and waits its turn; TryLock takes the key only if it is free at that instant, so you can decide what to do instead of queueing.

There is also a CLI that speaks the same protocol, which makes the server usable straight from a shell — lc-cli lock deploy/prod -- ./deploy.sh acquires the key, runs the command, and releases it whatever the outcome. It is, more or less, a distributed flock(1).

When not to use it

Simple has a price, and the README is blunt about it — which matters more than any feature list:

• It is a single instance with no replication. While it is down, nobody locks anything. If that is unacceptable, you want etcd, Consul, or ZooKeeper.
• There is no authentication. Anyone who can reach the port can lock, unlock someone else's key, or reset everything. Trusted networks only.
• There are no fencing tokens or TTLs. A holder that stalls past its turn is not shut out of the resource, and a crashed client's key stays held until a reset clears it (the alert is how you notice). If a double-acquire would corrupt data or move money, use something with fencing — Martin Kleppmann's "How to do distributed locking" is the piece to read first.
• If you already run PostgreSQL, pg_advisory_lock ties the lock to the database session, so a crashed client releases automatically. On Kubernetes, a coordination.k8s.io Lease covers leader election with no new infrastructure.

Locking-Center earns its place when you want blocking, ordered, sub-millisecond locks without standing up a database or a consensus system — and you are honest with yourself about those four bullets.

Try it

The server, the nine clients, the wire protocol, the Kubernetes manifest and the Prometheus alert are all in the repo:

https://github.com/freakmaxi/locking-center

If you go read the code, the two files worth your time are mutex/common/channel.go (the channel-as-queue and the cancellation registry) and mutex/common/lock.go (the write-ahead-log ordering). I would genuinely like to hear where the design is wrong — especially on the persistence guarantees.