Kestra 2.0: a new engine, workers anywhere, and still Apache 2.0

In February 2022 I published my first post here, called Kestra, infinitely scalable open source orchestration and scheduling platform. I described an orchestrator using Kafka as both queue and database, Elasticsearch as the repository behind the UI, and I was proud of the 350,000 executions a month it was running at Leroy Merlin.

Two things about that post did not survive contact with reality. The word "infinitely", which I have since retired. And the architecture, which people told me was far too heavy to even try. This is the follow-up, four years and one engine rewrite later.

What the feedback changed first

The most common reply to that 2022 post, in one form or another: requiring a Kafka cluster and an Elasticsearch cluster to schedule a job is absurd. That was fair. Four months later we shipped a JDBC backend, and a single Postgres or MySQL was enough to run everything, queue and state included.

That set the pattern for the whole project. Someone tells us the deployment story is bad, we fix the deployment story. The years in between, quickly:

• Subflows and fan-out over files, so a large pipeline stopped being one giant YAML file.
• Task runners, so the same script task runs as a Docker container, a local process or a Kubernetes pod by changing one property instead of rewriting the task.
• Git sync, a browser editor that autocompletes from the real plugin schemas, then a CLI and GitHub Actions for deploying flows from CI.
• Unit tests for flows with fixtures, so a change can be checked in CI without touching the production warehouse.
• A playground to run a flow task by task while you are still writing it.
• 1.0 in September 2025, our first release with long-term support.

And 2.0 shipped this month, with most of the engine replaced.

Why the engine had to be rewritten

In 1.x, every worker needed a connection to the central database. That single design decision dictated where you were allowed to run tasks. If a security team would not open a route from one network to the Postgres in another, you had two options: deploy a full Kestra per site, or give up and leave scripts on a local box with cron and no orchestration at all. I watched teams pick the second option more than once.

2.0 splits Kestra into a control plane and a data plane. The executor, the scheduler, the webserver, the indexer and a new worker controller live in the control plane, and none of them run user code. Workers are the data plane, and they are where your code executes.

A worker now opens a single persistent gRPC stream to the worker controller. The connection is always made by the worker, never in the other direction. Jobs travel out on that stream, and results, logs and metrics come back on the same one. The channel can be encrypted with TLS, and each worker can be required to present a client certificate or a JWT before any job is dispatched to it.

The practical consequence: a worker holds no database credentials, and nothing has to connect inbound to it. It can run in another cloud, in another region, on-prem next to the data, or inside a network that only allows outbound traffic. The control plane stays where you want it, and the data plane stays where your rules say it must.

The second piece of debt was internal. In 1.x the queue and the repository came as a fixed pair: JDBC for both, or Kafka plus Elasticsearch. That meant two engine implementations, every bug fixed twice, and every behavioural difference between the two paths ours to explain. 2.0 has one executor, one scheduler, one worker, and you pick the queue and the repository independently. The Kafka Streams engine is deleted, and I am not sorry to see it go.

What else changed in 2.0

• Loop replaces ForEach and ForEachItem. Each iteration is its own sub-execution, so a runaway loop can no longer take the executor down. That happened.
• Trigger conditions become one when expression, like your CI config.
• Quotas cap executions per flow or namespace, so one team's mistake stops being everyone's incident.
• Any flow can be an MCP tool. One trigger, and an agent can call it. It gets a normal execution with the same permissions as a person clicking Run, labelled system.from: mcp. Steps you cannot undo can wait for a human.
• AI is optional. The AI Agent task can point at Ollama on your own hardware. The Copilot can be switched off with kestra.ai.enabled: false, after which no AI endpoint exists on the server.
• New UI, canvas, Drafts. Canvas and YAML stay in sync. It is still always the YAML underneath, in your Git repo, with no code that imports us.

What did not change

Declarative YAML. Any language, each script in its own container, nothing imported from Kestra inside your code. Move to another orchestrator and you rewrite the flows, not the business logic. That is on purpose.

The license. We know a major version is where projects usually change it. We did not, and we do not plan to. The engine, the UI and all the plugins stay Apache 2.0. Open core is the model, we say so on the pricing page feature by feature, and the open source edition is a complete, production-grade orchestrator, not a demo.

2.0 is an LTS. Since the first release, more than four billion executions have run on Kestra, most of them on the open source edition, on hardware we will never see. The least we owe those people is a boring upgrade, so tell us where the migration hurts.

Still open source, and still the same license

The engine, the UI, the no-code editor and all the plugins are Apache 2.0, in the public repository. In 2022 I wrote a section in that first post titled "Kestra is Open Source", and I would rather it stay boring than have to write a follow-up explaining a license change. We know a major version is exactly where projects usually make that move. We did not, and we are not planning to.

To be straight about the boundary, since open core deserves a clear answer: RBAC, SSO, audit logs, multi-tenancy, worker groups, Policies, Cases and the cloud VM task runners are paid. Everything needed to actually run flows in production is not.

The fastest way to look at it, with an embedded database and nothing else to install:

docker run --pull=always --rm -it -p 8080:8080 --user=root \
  --name kestra \
  -v kestra_data:/app/storage \
  -v kestra_db:/app/data \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp:/tmp \
  -e KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true \
  kestra/kestra:latest-slim server local

What I would like back

In that 2022 post I bragged about 350,000 executions a month. Since then more than four billion executions have run on Kestra, mostly on machines I will never see. Which mainly means we owe the people running it an upgrade that does not ruin their week.

So the useful thing you can do is break it and tell me where. The engine rewrite is the biggest change we have ever shipped, and release candidates only find the obvious problems.

• GitHub: https://github.com/kestra-io/kestra
• The 2.0 release notes: https://kestra.io/blogs/release-2-0
• The 2022 post, for contrast: https://dev.to/kestra/kestra-infinitely-scalable-open-source-orchestration-and-scheduling-platform-6p2