Upscaling guest photos with a local model instead of an API

I run Knipsmig, a QR-code photo sharing service for weddings and parties. Guests scan a code and upload straight from the phone, no app. Most of those uploads are 12 MP and print fine. A meaningful slice are not: photobooth captures at 1080x810, WhatsApp forwards at 1600x1200, screenshots, old scans someone re-uploaded. Those end up in the printed photo book looking soft.

So I added an "Improve resolution" option to the editor. It adds up to 4x the pixels, and the whole thing runs on my own server. No API, no vendor, nothing leaves the box. This post is about why I went local and what it took to make that work inside a Rails app.

Why not just call an image API

I already have Gemini and OpenAI keys configured in the app for other things, so the lazy path was obvious. I did not take it, for three reasons.

The generative models redraw the image. They don't upscale, they regenerate. Faces drift. These are guests' faces at someone's wedding, and "your aunt looks slightly different now" is not a feature. A super-resolution network stays faithful to the input: it only adds pixels consistent with the ones already there.

Privacy paperwork. Every third-party processor I add has to go into the DPA. Guests' photos leaving the server to be fetched by a vendor is a real change, not a footnote. Running locally means the data processing agreement doesn't change and the existing opt-out for third-party AI stays about third parties.

Cost. Per-image API pricing on a bulk action over hundreds of photos adds up fast. CPU time on a job lane I already pay for is free at the margin.

The model

I went with realesr-general-x4v3 from the Real-ESRGAN project (BSD-3-Clause). It's the compact SRVGGNet variant: about 1.2M parameters, roughly 5 MB as an ONNX file, and around 10x faster on CPU than the full RRDBNet x4plus. Quality is more than fine for event snapshots.

Getting it into a usable shape was a one-off: export the release weights with the repo's pytorch2onnx.py script using dynamic H/W axes, in a throwaway Python venv. Torch never touches the app. The exported graph takes [1,3,H,W] float RGB in 0..1 and returns [1,3,4H,4W]. The .onnx file lives in vendor/models/ next to the face detector I already ship, and both run through the onnxruntime gem.

Running it from Ruby

The service is ImageUpscaler.call(vips_image) -> vips_image. It slots into the existing editor pipeline, which already holds a Vips::Image, as the last step after rotate, flip, crop and tone. That ordering matters: crop coordinates stay relative to the original, and tone runs on the small image, not the 4x one.

A few things that were not obvious going in:

Tile it. A 512 px tile is about 70 MB of activations; a whole 2048 px photo would not be. Each tile is cut with a 16 px overlap on every side (the source is mirrored past its edges so border tiles get context too), run through the model, and the padded border is cropped off the 4x result before stitching with Vips::Image.arrayjoin. No visible seams.

Skip the Ruby arrays. My first version marshalled tensors as nested Ruby arrays. A 512 px tile is 800k input floats and 12M output floats, and building those objects cost more than the inference itself. Now the tile is cast to float, split into three planar bands, and the raw bytes are written straight into an OrtValue:

scaled = tile.cast(:float).linear(1.0 / 255, 0)
planes = (0...3).map { |band| scaled[band].write_to_memory }.join

input = OnnxRuntime::OrtValue.from_shape_and_type([1, 3, height, width], :float)
input.data_ptr.put_bytes(0, planes)

output = model.run_with_ort_values(nil, { "input" => input }).first

The output planes come back the same way and get read into vips as single-band float images, joined, scaled by 255 and cast to uchar (which also clamps the model's slight overshoot).

Tag the colourspace. Single-band buffers join up as b-w, and the JPEG encoder then happily writes greyscale. One copy(interpretation: :srgb) fixed a very confusing afternoon.

Cap the threads. The job queue runs inside the web container on a six-core box. An upscale saturating every core stalls guest uploads for its whole duration. Four intra-op threads is about 3x faster than one, and six gains nothing over four, so four it is.

What it costs

On an M-series Mac with four threads, a 1080x810 booth photo takes 18 seconds end to end: download, model, JPEG encode, upload. Model time scales with input pixels, roughly 10-15 s per megapixel. The production box is a Hetzner CPU instance, so slower, and I'm reading real numbers off the job logs now.

That's why the feature has guardrails rather than being a default:

• Only photos with a longest edge at or below 2048 px are eligible. Above that the photo already prints well and a minute of inference buys nothing visible. The UI reports "already high resolution" and the tile shows what it would become ("1080 x 810 becomes 4320 x 3240") before you commit.
• Output is capped at 4096 px on the longest edge, the same cap the photobooth uses.
• Bulk selection is limited to 50 photos per request. The image editing lane runs one job at a time; 500 photos at 30 s each is four hours of a blocked queue.

The takeaway for me: for a narrow, well-defined image task, a 5 MB model on the CPU you already own beats an API call on privacy, cost and predictability. The engineering is mostly plumbing, and the plumbing is worth it.

If you're curious about the product side, the feature is part of Keepsake at knipsmig.com.