Previously

"URL retrieval failed, help me check the logs."

I receive this kind of report about once every week or two. Usually, it's because a website has added another layer of anti-crawling protection. A quick check and adding a fallback method to bypass it usually settles it. This time started the same way: a user sent an acm.org URL, and the summary feature spat out an error.

Upon investigation, I realized it wasn't that simple. The URL retrieval feature had been silently broken for three days. The problem wasn't the target website; it was an unlocked npm package version in my own Docker image. One day, it quietly upgraded to a new version that required a newer Node.js to run, while the Node.js in the container was still stuck on the version installed six months ago.

First, let's look at the logs:

gcloud logging read 'resource.type="cloud_run_revision" AND \
  resource.labels.service_name="linebot-helper-python" AND \
  textPayload:"acm.org"' --limit=50 --freshness=2d \
  --format="table(timestamp,severity,textPayload)"


ERROR:loader.url:All methods failed for URL: https://www.acm.org/articles/people-of-acm/2026/russ-cox
WARNING:loader.url:cloudscraper failed ...: 403 Client Error: Forbidden for url: ...
WARNING:loader.url:httpx failed ...: Client error '403 Forbidden' for url: ...
WARNING:loader.url:singlefile failed ...: SingleFile exited with code 1
ERROR:loader.singlefile:SingleFile loading failed ...: SingleFile exited with code 1

This Bot has a fallback chain for grabbing webpage content: first try singlefile (full rendering with headless Chromium), if that fails switch to httpx, and if that fails switch to cloudscraper (a tool specifically for bypassing Cloudflare). It only reports an error to the user if all three fail. It's understandable for httpx and cloudscraper to get a 403; acm.org has protection. But singlefile shouldn't fail; it's designed as the last line of defense to bypass such protections.

Scrolling down, I found the actual error for singlefile:

Error: WebSocket is not available, Node.js 22.4.0 or later is required
    at file:///usr/local/lib/node_modules/single-file-cli/lib/deno-polyfill.js:132:8
Node.js v18.20.4

The container was running Node 18, but single-file-cli required 22.4 or later. Checking the logs, this error first appeared three days ago, not today. This means for the past three days, if any of the first two methods were blocked for a URL, the whole process would fail. Users would only receive a message saying "Unable to read content from the URL," with no indication of the underlying cause.

Why Run Cloud Build Three Times Instead of Deploying Immediately After Changes

The rest of this article will feature three cycles of "modify Dockerfile → build with Cloud Build → run and see." Initially, I didn't plan on doing this; I thought changing a version number and seeing the build pass would be enough to deploy. I later found this assumption was wrong every single time.

The reason is that for commands like npm install -g single-file-cli, a successful build only means "npm found the package and could install it," not "the package works when it runs." npm's check for the engines field is just a warning by default and won't fail the build. Whether the package actually crashes won't be known until it tries to launch a headless Chromium and establish a WebSocket connection. There's a gap in between that the build log doesn't show at all.

So this time, I didn't stop at "build passed." Every time I changed a version, I ran a shell script directly in Cloud Build to use the installed single-file in the container to actually fetch the acm.org URL that was failing. This habit later saved me twice: if I had deployed just because the build succeeded, the Dockerfile changes would have eventually re-enacted the exact same crash in the production environment, just with a different error message.

System Architecture

After the fix, the URL retrieval failure handling looks like this:

graph TD
    A[User sends URL] --> B[loader/url.py fallback chain]
    B --> C1[singlefile: headless Chromium]
    C1 -->|Content fetched| D{is_challenge_page check}
    D -->|Is Cloudflare challenge page| E[Treat as failure, move to next method]
    D -->|Is real content| F[Return to Gemini]
    C1 -->|Crash or timeout| E
    E --> C2[httpx]
    C2 -->|403 or challenge page| C3[cloudscraper]
    C2 -->|Real content| F
    C3 -->|403 or challenge page| G[All three failed]
    C3 -->|Real content| F
    G --> H[Throw Chinese error message to user]

Originally, only the "crash or timeout" path was considered a failure; the is_challenge_page branch was newly added this time, and I'll explain why later.

Core Implementation

1. Dockerfile: Locking Versions for Both Node.js and single-file-cli

Before the fix, this RUN command looked like this:

RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        nodejs \
        npm \
        git \
        chromium \
        ffmpeg \
    && npm install -g single-file-cli \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

Both problems were in these lines. apt-get install nodejs pulls the version from the built-in Debian apt source, which is stuck on v18. npm install -g single-file-cli had no version lock, so every image rebuild would pull the latest version currently on npm. Separately, these aren't severe, but together they are a ticking time bomb: I have no control over when single-file-cli increases its Node.js requirement, while my container version was fixed.

After the fix:

# Lock both Node.js and single-file-cli versions to avoid silently pulling incompatible new versions during future rebuilds:
# - Node.js: Switched to NodeSource 24.x (Debian's built-in v18 is too old), locking the exact version.
# The ws/simple-cdp dependency of single-file-cli requires a global CloseEvent, which is only available in Node 24 
# (not in Node 22, even with --experimental-websocket, as verified by testing). This was the root cause of the acm.org scraping failure.
# - single-file-cli: Locked at 2.0.83, verified to work correctly under Node 24.
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        gnupg \
        git \
        chromium \
        ffmpeg \
    && mkdir -p /etc/apt/keyrings \
    && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
    && echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_24.x nodistro main" > /etc/apt/sources.list.d/nodesource.list \
    && apt-get update && apt-get install -y --no-install-recommends nodejs=24.19.0-1nodesource1 \
    && npm install -g single-file-cli@2.0.83 \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

The exact version numbers nodejs=24.19.0-1nodesource1 and single-file-cli@2.0.83 weren't chosen at random; they were determined through three rounds of Cloud Build testing. I'll explain how I arrived at these numbers in the "Major Pitfalls" section below.

2. Detecting Cloudflare Challenge Pages

After singlefile stopped crashing, I thought it was over until I actually used it to fetch acm.org:

== single-file fetch acm.org (headless chromium) ==
OK: 18400 bytes
<title>Attention Required! | Cloudflare</title>

It "succeeded," but what it fetched was the Cloudflare human verification page, not the article itself. The problem was that this feature had no detection for this situation. While httpx and cloudscraper would call raise_for_status() on 4xx/5xx and fail immediately, the page rendered by headless Chromium is always an HTTP 200 (the browser displays whatever it gets). This layer of check didn't exist for singlefile. The verification page text would be sent to Gemini, and the user would receive a nonsensical summary completely unrelated to the original article, appearing as a normal response without any error prompts.

I added the detection logic in loader/html.py:

# Common characteristic strings for Cloudflare/WAF human verification pages. 
# singlefile uses headless Chromium to render pages, and unlike httpx/cloudscraper, 
# it doesn't call raise_for_status() for 4xx/5xx. It treats fetching a challenge page 
# as a "successful fetch"—this specifically intercepts that case. If detected, it must 
# be treated as a failure so the fallback chain in loader/url.py moves to the next method, 
# preventing challenge page content from being sent to Gemini.
CHALLENGE_PAGE_MARKERS = (
    "Attention Required! | Cloudflare",
    "Just a moment...",
    "Checking your browser before accessing",
    "cf-browser-verification",
    "Enable JavaScript and cookies to continue",
    "DDoS protection by Cloudflare",
)

def is_challenge_page(text: str) -> bool:
    """Check if the fetched content is a Cloudflare/WAF challenge page rather than real content."""
    return any(marker in text for marker in CHALLENGE_PAGE_MARKERS)

All three loaders (httpx, cloudscraper, singlefile) now use this check. If a challenge page is caught, it's treated as a failure, allowing the fallback chain to try the next method:

if is_challenge_page(resp.text):
    raise RuntimeError(f"httpx got a bot-challenge page instead of real content: {url}")
return parse_html(resp.text, markdown=markdown)

Major Pitfalls and Solutions

Pitfall 1: Don't Take Version Numbers in Error Messages at Face Value

Initially, seeing Node.js 22.4.0 or later is required, I naturally upgraded the container to Node 22, and the build passed smoothly. I thought I was done until I actually ran single-file to fetch acm.org and got a completely different error:

ReferenceError: CloseEvent is not defined
    at #onClose (.../single-file-cli/node_modules/simple-cdp/mod.js:334:36)
Node.js v22.23.2

The ws package that single-file-cli depends on tries to new CloseEvent(...) when handling connection close events, which requires a global CloseEvent class in the execution environment. I ran a probe directly in the container:

node -e "console.log('WebSocket', typeof WebSocket, 'CloseEvent', typeof CloseEvent)"
# WebSocket function CloseEvent undefined

node --experimental-websocket -e "console.log('CloseEvent', typeof CloseEvent)"
# CloseEvent undefined

Node 22.23.2 has a global WebSocket (which is what the "22.4.0 or later" error message was actually checking), but it doesn't have CloseEvent, even with the experimental flag. Testing again with Node 24.19.0:

node -e "console.log('CloseEvent', typeof CloseEvent)"
# CloseEvent function

Cause and Solution: The internal version check in single-file-cli (the "Node 22.4 or later" in deno-polyfill.js) only checks the first API it uses; it doesn't mean all APIs used by its dependencies are present in that version. The real blocker was the downstream CloseEvent, which only became globally available in Node 24. Treat version numbers in error messages as a starting point, not the final answer. You only know where it will actually get stuck by running it.

Pitfall 2: The "Latest Version" on npm is a Moving Target

After upgrading to Node 22, I thought it was over until a user asked to "lock the versions while you're at it to avoid future issues." Checking the current single-file-cli info on npm:

npm view single-file-cli engines
# { deno: '>=2.2', bun: '>=1.2', node: '>=24.0.0' }

The latest version requires Node 24 or higher, not the 22 I had just fixed. Checking the release times, I realized there were two consecutive version jumps within the same week:

npm view single-file-cli time --json | grep -E "2\.0\.83|2\.1\.0"
# "2.0.83": "2025-11-29T23:34:16.671Z"
# "2.1.0": "2026-08-16T15:56:28.819Z"

Version 2.0.83 was from last November, with engines set to node >= 20, and it worked stably for over six months. Version 2.1.0, released on August 16th, suddenly raised the bar to node >= 24. I first observed the crash on August 20th, which aligned perfectly with an image rebuild: npm install -g single-file-cli without a specified version had silently pulled the 2.1.0 version released just days prior.

Cause and Solution: npm install -g <package> (without a version) in CI/CD or a Dockerfile essentially means "every rebuild might install something different." This becomes a problem the moment a package maintainer raises the Node version requirement. Locking versions isn't about being conservative; it's about changing "when this container breaks" from "whenever the package author feels like it" to "whenever I decide to upgrade." I eventually chose 2.0.83—not the latest, but the last stable version explicitly compatible with Node 20—rather than trying to meet the new Node 24 requirement of 2.1.3.

Pitfall 3: Build Success Does Not Equal Runtime Success

I mentioned this earlier in "Why Run Cloud Build Three Times," but here is how it actually looked. After locking Node 24 + single-file-cli 2.0.83, gcloud builds submit finished successfully, and Successfully built was printed. If I had only looked at the build success message, I would have mistakenly thought the problem was solved. But the version that first passed the build was actually the Node 22 version, which would have exploded at runtime due to the CloseEvent issue in Pitfall 1. Build success itself does not prove the content of the container actually works.

I added an extra step to the Cloud Build configuration to run the newly built image immediately:

steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'nodefix-verify', '.']
- name: 'nodefix-verify'
  entrypoint: 'bash'
  args:
    - '-c'
    - |
      node -v
      single-file --version
      single-file --browser-executable-path=/usr/bin/chromium \
        --browser-args='["--no-sandbox","--disable-dev-shm-usage"]' \
        "https://www.acm.org/articles/people-of-acm/2026/russ-cox" /tmp/out.html \
        && echo "OK: $(wc -c < /tmp/out.html) bytes"

The first time I ran this, it caught the CloseEvent is not defined error from Pitfall 1. Only on the second run (Node 24 + 2.0.83) did I finally see OK: 18400 bytes.

Cause and Solution: npm install only prints a warning for engines mismatches by default; it doesn't fail the command. docker build success only means the exit code of every command was 0. Together, this means there is no necessary correlation between "build passed" and "this thing works correctly." Including "actually run the target scenario" in the verification process is the only way to bridge this gap.

Pitfall 4: Fetching Content Doesn't Mean Fetching the Right Content

When the verification in Pitfall 3 printed OK: 18400 bytes, I initially thought the problem was solved until I saw that within those 18,400 bytes, the <title> said Attention Required! | Cloudflare. I realized that even if single-file-cli works perfectly and renders the page with headless Chromium, Cloudflare might still block the request because it detects a headless browser, returning a challenge page. To single-file-cli, this challenge page looks exactly like a normal article: it's HTML, it can be saved, and it counts as a "success."

After adding the is_challenge_page check, the first round of tests didn't all pass:

FAILED test_load_html_with_httpx_raises_on_challenge_page
FAILED test_load_html_with_cloudscraper_raises_on_challenge_page

| The logic seemed correct, yet the httpx/cloudscraper tests failed to catch the challenge page. Upon investigation, I found that load_html_with_httpx in loader/html.py converts HTML to markdown (using markdownify) before returning it. markdownify does not preserve tags like <title> which are in the <head> and not displayed as body text. My detection string "Attention Required! | Cloudflare" happened to be in that discarded <title>, so it wasn't in the text after the markdown conversion. Even worse was cf-browser-verification, which in a real Cloudflare page is a CSS ID (<div id="cf-browser-verification">). It only exists in HTML attributes, so neither markdown conversion nor plain text extraction (get_text()) would include the attribute value as text content. |

Cause and Solution: Move the checkpoint from "processed text" to "raw HTML":

# Some challenge page features (like cf-browser-verification) only appear in HTML attributes, 
# which markdownify/get_text won't catch. So, check the raw HTML before conversion.
if is_challenge_page(resp.text):
    raise RuntimeError(f"httpx got a bot-challenge page instead of real content: {url}")
return parse_html(resp.text, markdown=markdown)

I made the same adjustment for singlefile. Originally, it used BeautifulSoup to convert the file to plain text before checking. I changed it to read the raw bytes and check them first; only if it passes is it handed over to BeautifulSoup:

with open(f, "rb") as fp:
    raw_html = fp.read()

if is_challenge_page(raw_html.decode("utf-8", errors="ignore")):
    raise RuntimeError(
        f"SingleFile got a bot-challenge page instead of real content: {url}")

soup = BeautifulSoup(raw_html, "html.parser")
text = soup.get_text(strip=True)

The lesson here is: when writing protection logic, you must be clear about which layer of data you are checking. The same content, after different conversions (HTML → markdown, HTML → plain text), will lose information asymmetrically: some words are visible, while others only exist in attributes, and both extraction methods might miss them.

Results and Benefits

After the changes, I ran a real end-to-end test, calling the load_url() function actually used in Python within the container against the same acm.org URL:

INFO:loader.url:Trying singlefile for URL: ...
ERROR:loader.singlefile:SingleFile loading failed ...: SingleFile got a bot-challenge page instead of real content: ...
WARNING:loader.url:singlefile failed ...
INFO:loader.url:Trying httpx for URL: ...
WARNING:loader.url:httpx failed ...: 403 Forbidden
INFO:loader.url:Trying cloudscraper for URL: ...
WARNING:loader.url:cloudscraper failed ...: 403 Forbidden
ERROR:loader.url:All methods failed for URL: ...
Raised as expected: Exception 無法從網址讀取內容，請確認網址是否正確或稍後再試 (Unable to read content from the URL, please verify the URL or try again later)

All three methods still failed; the Cloudflare protection behind acm.org was not bypassed. That hasn't changed. What changed is how it failed: previously, the singlefile crash dragged down the entire fallback chain, leaving the user unaware that three methods had even been attempted. Now, all three methods fail cleanly on their own, and the user receives a clear error message instead of a nonsensical summary that is actually a Cloudflare challenge page.

Locking versions solves "will it break again in the future," not "can the acm.org URL be fetched." It's easy to conflate these two at first: once versions are locked and single-file-cli no longer crashes, it's tempting to think the problem is solved. But the Cloudflare hurdle was never the scope of this fix, nor is there a simple solution (bypassing it would likely require browser fingerprint spoofing, which has a poor ROI). What was achieved was the complete separation of "crawler blocked" and "my own container is broken." The latter should no longer recur due to npm version drift.

"Build success" cannot be treated as verification passed. This was the biggest takeaway. If I had only looked for red text in the build log after every Dockerfile change, I would have missed at least two issues: the CloseEvent crash with Node 22, and the fact that Cloudflare challenge pages were being sent as body text even when singlefile worked perfectly. Both are "build okay, crash at runtime" types of problems that only become visible when running against a target URL.

Protection logic must align with the actual data layer received. The issue in Pitfall 4 where markdownify swallowed the <title> would have likely gone unnoticed until a user reported the exact same problem again, had I not written a test and seen it DID NOT RAISE.

In total, four files were modified: Dockerfile, loader/html.py, loader/singlefile.py, plus a new test tests/test_challenge_page_detection.py. All 150 tests in the project passed. The code is available at kkdai/linebot-helper-python.