Our build wave shipped four scrapers that scraped nothing

Our build wave finished four scrapers in a day. Every test was green. Not one of them scraped anything.

This is what that failure actually looked like, and why every gate we had said the opposite.

The tell was a cloud run, not a test

The first of the four went to the platform, built cleanly, and died on its smoke run:

apify_client.errors.ApifyApiError: Schema validation failed
  File "/usr/src/app/src/main.py", line 69, in main
    await Actor.push_data(batch)

Not a network error. Not a block. The scraper had produced rows and our own dataset schema refused them. So we ran the scraper's entry point locally against the live target and printed what came back:

[
  {
    "placeholder": true,
    "note": "src/scraper.py is a T01 scaffold stub — the real implementation lands in T02-T06",
    "search_terms": ["chanel"]
  }
]

One row. A scaffold stub. The generator that was supposed to page through a search index and yield listings was still the placeholder our scaffolder writes on day one. A grep across the wave found the same marker in all four.

Why nothing caught it

This is the part worth sitting with, because the checks were not weak — they were simply all measuring something else.

• The test suite was green. One of these Actors had 30 passing tests. They passed because they exercised the stub. A test suite written against a placeholder is green forever and means nothing.
• The linters and type checker were clean. A stub is well-formed code.
• Every schema gate passed. Input schema, output schema, dataset schema, manifest references, prefill validation — all valid, because the scaffolder writes valid schemas.
• Our build counter counted them. It marked an Actor "built" once src/models.py was committed. The scaffolder writes models.py.

So the dashboard said 2 of 5 built, tests green. The honest number was zero.

What actually caught it was the dataset schema rejecting a row shape — a check we had added for a completely different reason. That is luck, not process. A more permissive schema would have let a placeholder-emitting scraper straight through to the Store.

The same shape was already live

Once we knew what to look for, we swept the whole fleet and found one that had shipped. A listing that had been public for two weeks, monetized, taking real customer runs — and whose entry point said so in its own docstring:

*** SCAFFOLD-ONLY RUN — 0 dataset rows is the expected, correct outcome. ***

It validated input, charged the start fee, logged, and exited zero. Eleven customer runs in thirty days, every one recorded as SUCCEEDED.

And here is the sting: on a success-rate dashboard it was the healthiest thing we owned. It sat at 100% while the Actors we were actively triaging sat at 69–89%. An Actor that reliably does nothing never fails.

Success is not delivery. Run status and row count are different signals, and we had only been watching the first one. We now check both, and we check something else too — whether an Actor has any runs of our own at all. That one had none. Its mandatory pre-publish QA had never run; all twelve runs belonged to customers.

Two checks, and one that had to be thrown away

The obvious fix is to grep for placeholder. Do not do this. Our fleet is full of legitimate uses — one scraper's comments describe its selectors as "not a placeholder hypothesis", another documents a placeholder-slug URL pattern, a third builds a query string with a {query} placeholder. Those are working scrapers talking about placeholders.

So the check walks the syntax tree instead and flags a stub only when the code is shaped like one: a placeholder-keyed dict literal, a run() that yields a dict whose keys are not the Actor's declared row fields, or an explicit stub banner in the module docstring.

The middle rule is the load-bearing one — it encodes the actual requirement, that the thing you yield must be the row you promised, so it keeps working even if a future scaffolder stops using the word "placeholder".

It also had to be scoped carefully. The first draft walked every return in the module and immediately flagged a dozen perfectly good Actors, because a helper that returns {"Accept": ..., "User-Agent": ...} is, structurally, a dict literal being returned from the row module. Narrowing it to yields inside run() fixed that. A gate that cries wolf gets switched off, which is worse than not having it.

A bug we only found by reading the plumbing

While fixing one of the four, we found the trailing batch flush charging an event the pricing manifest did not declare:

await Actor.push_data(batch)
await _charge("result", count=len(batch))   # manifest declares "result-row"

Apify does not error on this. It logs Ignored attempt to charge for an event and carries on. Since the batch flush threshold is 50 rows, every run smaller than that — which is most runs — would have billed nothing for its rows. We checked the other 48 Actors that charge a similarly-named event; all of them declare it correctly, so this one was genuinely isolated. It is now pinned by a test that asserts every charged event name is a literal key in the pricing manifest.

What shipped

The rebuilt Actors were done properly — wire format confirmed against the live endpoint first, fixtures captured from real responses rather than hand-written, and the row contract driven by the schema instead of the other way round. Two of them turned up defects that fixtures alone would never have surfaced: one had a JSON-LD fallback path that was completely dead against production data (not one live record carried the key it read), and another's spec had assumed the wrong status code for a bad index — the real API answers 404, not 400.

Cameo Talent Listings Scraper came out of that rebuild. It searches Cameo's talent marketplace by name and returns structured rows — video, DM, rush and business price tiers, category tags, star rating and booking activity — from Cameo's own search backend. Targets change their markup and their defences without warning; handling the retries, rotation and the awkward status codes is our job, not yours.

The broader lesson we are keeping: a green test suite tells you your code does what your tests say. It tells you nothing about whether your tests describe reality. For a scraper, only a real request against a real target does that.