nginx add_header Appends. It Doesn't Override — And That Can Remove Protection. – Discover

TL;DR — I shipped a baseline of security headers into every generated nginx vhost, meaning it as a fallback for applications that send none of their own. Within the hour, production was answering with two X-Frame-Options values: DENY from the app and SAMEORIGIN from my baseline. Browsers treat a contradictory set as no directive at all, so a header added to increase protection had removed it. The fix is a map on $upstream_http_*, because add_header has no way to ask a question.

The setup

I maintain a system that generates nginx server blocks for applications it deploys. The applications aren't mine. Some are well-built Laravel apps that set their own security headers in middleware; some are ten-year-old PHP that sets nothing at all and is being moved off a managed hosting panel that used to set headers on its behalf.

So the requirement is easy to state: give every vhost a sane baseline, without stepping on an application that already knows what it wants.

The first cut looked like this, rendered into every server block:

server_tokens off;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;

Tests green. Config valid. Deployed.

What production actually answered

I went to check the thing I had just claimed was proven — a habit that has earned its keep — and curl gave me this:

x-content-type-options: nosniff
x-content-type-options: nosniff
x-frame-options: DENY
x-frame-options: SAMEORIGIN
referrer-policy: strict-origin-when-cross-origin
referrer-policy: strict-origin-when-cross-origin

Every header, twice. The application's own middleware was setting X-Frame-Options: DENY. My baseline had added SAMEORIGIN beside it.

Here's the thing I want to be blunt about, because I think it's the part people assume their way past:

Two contradictory X-Frame-Options values is not "the stricter one wins." There is no merge rule. Browsers that receive an unparseable set of directives for that header treat the header as absent. So DENY — the strongest possible value, which the application had correctly set — became nothing, because I added a weaker value next to it.

On that particular host, Content-Security-Policy: frame-ancestors 'none' still covered the gap; CSP takes precedence over X-Frame-Options in every modern browser. But the applications this baseline was written for are exactly the ones relying on X-Frame-Options alone. For all of them, running my "hardening" left them worse off than before it ran.

Three nginx facts worth internalising

The bug wasn't a typo. It was three separate properties of add_header, none of which I had actually sat with.

1. `add_header` appends. Always.

The name is honest and I read it as if it said set_header. It adds a field to the response. It does not replace an existing one, and there is no add_header ... if_not_present flag. Think of it as $response->headers->set() versus ->add() — nginx only gives you ->add().

2. `add_header` cannot ask a question.

This is the real constraint. Inside a location, there is no expression available to add_header that means "unless the upstream already sent this." if won't help you — if inside location is famously not what you want, and it can't inspect response headers anyway, because at request-evaluation time the response doesn't exist yet.

3. `add_header` does not merge across levels — it replaces the whole set.

One add_header in a nested block silently discards every add_header inherited from the parent. Not the matching one. All of them.

This one bit me in a place I would never have looked. My generated config has a fallback location that returns a friendly 503 while an application is starting up:

location @app_unavailable {
    add_header Retry-After 5 always;
    return 503 '<!doctype html>...';
}

That single Retry-After wiped the entire inherited baseline. Which means the response an operator is most likely to see with their own eyes — the "just a moment, starting up" page during a restart — was the one response on the whole node with no protection headers on it. The fix is to repeat the set inside the location, with a comment explaining that the repetition is load-bearing:

location @app_unavailable {
    # Repeated, not inherited: nginx replaces the whole add_header set at the
    # deepest level that defines one, so Retry-After above would otherwise
    # strip the baseline.
    add_header Retry-After 5 always;
    add_header X-Content-Type-Options $fallback_xcto always;
    add_header X-Frame-Options $fallback_xfo always;
    add_header Referrer-Policy $fallback_rp always;
    return 503 '<!doctype html>...';
}

And while we're here: always matters. Without it nginx omits the header from 4xx and 5xx responses — protecting only the responses that were never the risk.

The fix: `map` on `$upstream_http_*`

add_header can't ask a question, but a map can answer one before the question is asked. Two behaviours combine into exactly what I needed:

$upstream_http_<header> holds the value the upstream sent for that header, or the empty string if it sent nothing.
nginx omits an add_header whose value evaluates to an empty string.

Put those together and the fallback writes itself. In http context:

# The baseline security headers are a FALLBACK: each value is empty when the
# application already sent that header, and nginx skips an add_header with an
# empty value.
map $upstream_http_x_content_type_options $fallback_xcto {
    ''      "nosniff";
    default "";
}

map $upstream_http_x_frame_options $fallback_xfo {
    ''      "SAMEORIGIN";
    default "";
}

map $upstream_http_referrer_policy $fallback_rp {
    ''      "strict-origin-when-cross-origin";
    default "";
}

Then the vhost uses the variables instead of literals:

server_tokens off;
add_header X-Content-Type-Options $fallback_xcto always;
add_header X-Frame-Options $fallback_xfo always;
add_header Referrer-Policy $fallback_rp always;

An application that sets its own headers keeps exactly what it set — no duplicate, no contradiction. An application that sets none gets the baseline. That's the behaviour the requirement asked for, and it's what the first cut didn't do.

Two properties of $upstream_http_* that made me comfortable shipping this:

It's populated for fastcgi and uwsgi upstreams too, not just proxy_pass. So a PHP-FPM vhost behaves the same as a reverse-proxied one.
It's empty when nginx serves the file itself, which is the correct answer for a static vhost: nothing upstream spoke, so apply the fallback.

The deployment trap that nearly cost more than the bug

Here's the operational edge case, and it's the one I'd want a reviewer to catch if I'd missed it.

A vhost that references $fallback_xfo when no map defines it is a config nginx refuses outright. Not a warning. Not that vhost failing. nginx -t fails, the reload is rejected, and if a deploy pushes the vhost without the map, you have taken down every site on the node — including the ones you weren't touching.

So the maps and the vhosts have to land together, in all three of the config-writing paths in my system (a systemd driver writing to disk, a container-oriented writer, and a Kubernetes ConfigMap). All three render vhosts from the same shared trait, so all three needed the maps added — the shared trait is exactly what makes it easy to forget, because the change looks like one place.

If you're adding this to an existing setup: deploy the map block first, on its own, and reload. Then deploy the vhosts. A map nothing references is harmless. A reference with no map is a node-wide outage.

What I didn't copy

The panel these applications are migrating from also sends X-XSS-Protection: 1; mode=block. I deliberately left it out. Every current browser has removed the XSS auditor that header names, and on the browsers that kept it, the header introduced a cross-site information leak of its own. Matching the old platform's protection level is the goal; matching its config file is not. Worth writing that distinction into a comment, because "the old system sent it" is a very persuasive argument at review time.

Testing this honestly

The config generator is unit-testable, and the test is worth writing because the shape of the output is the whole contract:

it('renders baseline headers as a fallback, not a second opinion', function () {
    $config = $this->proxy->renderVhost($application);

    expect($config)
        ->toContain('server_tokens off;')
        ->toContain('add_header X-Frame-Options $fallback_xfo always;')
        // The literal is the bug: a hardcoded value cannot defer to the app.
        ->not->toContain('add_header X-Frame-Options "SAMEORIGIN"')
        // Not copied on purpose — the auditor it names no longer exists.
        ->not->toContain('X-XSS-Protection');
});

it('repeats the baseline inside the unavailable location', function () {
    // One add_header at a deeper level discards the entire inherited set,
    // so the 503 page needs its own copy or it ships bare.
    expect($this->proxy->renderVhost($application))
        ->toContain('add_header Retry-After 5 always;')
        ->toContain('add_header X-Frame-Options $fallback_xfo always;');
});

But be clear-eyed about what that proves. It asserts a directive was emitted. It cannot assert what a browser does with the response. The original bug passed a green suite. A test over generated config verifies the generator, not the behaviour — and the difference between "CI-proven" and "live-proven" is exactly the gap this bug lived in for an hour.

The only thing that caught it was hitting the live host and reading the response headers, right after shipping.

Takeaways

add_header appends, can't be conditional, and replaces the entire inherited set at the deepest level that defines one. All three of those will surprise you at least once.
A map on $upstream_http_* plus nginx's empty-value rule is the idiomatic way to express "only if the upstream didn't". No if, no Lua, no duplicated header.
A security header added carelessly can be a downgrade, not an addition. "More headers" isn't a safety property.
Ship the map before the reference. A dangling variable in a vhost fails the whole config, not just that site.
Verify the claim you just made, against the live thing, in the hour you made it. A green suite is evidence about the generator. It is not evidence about the response.

nginx add_header Appends. It Doesn't Override — And That Can Remove Protection.

The setup

What production actually answered

Three nginx facts worth internalising

1. add_header appends. Always.

2. add_header cannot ask a question.

3. add_header does not merge across levels — it replaces the whole set.

The fix: map on $upstream_http_*