You Can Retry a Write. You Cannot Retry an Email.

I run Upwork Scout, a small SaaS that watches Upwork and emails you only the jobs that fit you. The whole product is one cron endpoint. It wakes up every fifteen minutes, fetches new jobs, matches them against every active user, and sends mail.

The interesting engineering in that sentence is not the matching. It is the sending.

A scheduled job that only writes to a database is easy to get right, because you can run it twice and the second run overwrites the first with the same values. Nobody finds out. Email is different. Email is a side effect that leaves your system and lands in a human being's inbox at seven in the morning. There is no undo, there is no idempotency key I can pass to a person, and the failure mode is not a stack trace, it is someone unsubscribing.

So the real design constraint is not "make the scan correct". It is "make the scan safe to run twice, because sooner or later it will run twice".

Every reason my cron runs twice

• The scheduler retries. Mine is a GitHub Actions workflow that curls the endpoint. If the response is not a 200 it fails the run, and I have re-run failed runs by hand more than once.
• The caller giving up does not stop the server. A timeout on the curl side cancels nothing. The function keeps looping through users and keeps sending.
• The function has a wall. maxDuration is 300 seconds. If a cycle hits it halfway through the user loop, half the users got email and half did not, and the next run has to know which half.
• Anything can ping it. The endpoint is a plain GET guarded by a shared secret, deliberately, so I can point cron-job.org or n8n at it, or hit it from my terminal while debugging. Every one of those is another caller.

None of that is exotic. It is the normal life of a scheduled job. The only question is whether your users can tell.

The ledger is the answer, and the key is the whole trick

There is one collection called deliveries, with one document per user per job:

const dRef = deliveriesCol().doc(`${u.uid}_${j.id}`);
if ((await dRef.get()).exists) continue;

That is the entire duplicate suppression system, and the part doing the work is the document id. It is derived from the data, not from the run. Two different invocations, on different machines, at different times, computing the same user and the same job produce the same id. There is no sequence number to coordinate, no run id to compare, nothing to reconcile afterwards.

The same shape is now everywhere in this codebase. AI verdicts live at matches/${uid}_${jobId}, so a job is never scored twice for the same person. User documents are keyed by a base64url encoding of the lowercased email, so submitting the login form twice cannot produce two accounts.

Deterministic ids turn "did I already do this" from a query into a lookup. It is the cheapest idempotency I know, and it needs no infrastructure: no queue, no exactly-once broker, no dedupe table with a TTL. Just a naming convention that every write in the codebase obeys.

A lock that expires on its own

The ledger protects individual sends. It does not stop two full scans from grinding through the same work at once and doubling my scraping bill, so there is also a lock:

const lock = await lockRef.get();
if (lock.exists && now - (lock.get("ts") || 0) < 10 * 60_000) {
  return NextResponse.json({ skipped: "previous run in flight", ...stats });
}
await lockRef.set({ ts: now });

Two things about it I would defend in review.

First, it stores a timestamp, not a boolean. A boolean lock is a loaded gun pointed at your own product. The one time a process dies between taking the lock and releasing it, the flag stays true forever, no scan ever runs again, and nothing throws. Your monitoring stays green because the failure mode is silence. A timestamp lock heals itself in ten minutes without me being awake. It is released in a finally for the ordinary crash, and the timestamp covers the extraordinary one where finally never runs.

Second, it is not a real mutex. It is a read followed by a write, so two callers landing within the same few milliseconds can both walk through it. I know, and I left it, because the ledger behind it protects the thing users actually see. The lock is a cost optimization. The ledger is the correctness guarantee. Confusing those two is how people end up putting a distributed lock manager in front of a product with a hundred users.

The awkward part: I write the ledger before sending, except when I do not

Here is the tradeoff with no clean answer, and the place where my own codebase is deliberately inconsistent.

Instant alerts write the delivery record first, then send:

await dRef.set({ uid: u.uid, jobId: j.id, status, createdAt: now, sentAt: ... });
// ... later, after collecting this user's batch
await sendEmail({ to: u.email, ...mail });

If that send throws, the record already says sent. The job is gone for that user forever. This is at-most-once, and I picked it on purpose. An instant alert is one job out of many arriving all day, so losing one costs a user very little, while the same job landing twice costs them confidence in the product.

The daily digest does the exact opposite. It sends the mail, and only then flips the queued rows:

await sendEmail({ to: u.email, ...mail });
await Promise.all(q.docs.map((d) => d.ref.update({ status: "sent", sentAt: Date.now() })));

Crash in that gap and the digest goes out again on the next flush. That is at-least-once, and it is also on purpose. A digest is not one job, it is the entire day's value in a single email. Losing it silently is far worse than a rare duplicate.

I did not plan that asymmetry. I found it months later, reading both code paths in the same sitting, and my first instinct was to make them consistent. Then I worked out what consistency would actually cost in each direction and kept them different. At-least-once versus at-most-once is not a house style you choose once and apply everywhere. It is a per-side-effect decision, and the useful question is which direction of being wrong this particular user forgives.

One table, two delivery modes

The ledger also turned out to be the queue I never had to build. The row carries a status:

const status = u.delivery?.mode === "daily" ? "queued" : "sent";

Instant and digest are the same write with a different word in it. The 08:00 UTC digest run is a query for that user's queued rows, sorted by match score, capped at twenty, and then one email. No second table, no job queue, no worker process. When people ask why I did not reach for a proper queue, the honest answer is that a status column on a table I already needed covered the requirement, and every piece of infrastructure you skip is one you never have to operate at 3am.

What this costs me, honestly

That existence check is one read per candidate job per user, inside a loop. So my read count grows with users times matched jobs, and on the day that number gets uncomfortable it will be Firestore reads, not model calls, that show up on the bill. The fix is not clever. It is a batched multi-get before the loop instead of a get() inside it. I have not done it, because at current volume the loop is fine and I would rather ship features, and I would rather write that down than pretend the code is finished.

The ledger also grows forever. One document per user per job, never pruned, while the jobs themselves go stale after 24 hours. The entries stop being useful long before they stop existing.

The test I run on anything scheduled now

Run it twice on purpose, back to back, against production-shaped data, and check whether anyone outside the system can tell. If the second run sends nothing, charges nothing, and leaves the same state behind, you have a cron job. If you have to reason about how often it fires to know whether it is correct, you have a ritual that happens to work.

That test is why the first table I design for a background job is not the interesting one. It is the boring one that says "I already did this".

What is your default when a scheduled job touches the outside world, at-least-once or at-most-once? I am curious whether people pick per side effect the way I ended up doing, or standardize and eat the cost.