Road to 1.0.0

Fleans 1.0.0 is the release where throughput grows with the cluster: every shared point on the hot path — the single PostgreSQL primary, cluster-wide singleton grains, one reminder per timer — is removed, so adding silos and storage nodes adds capacity.

Note

This is a living plan, not a promise. The current release is v0.11.0; each step below links to its GitHub issue, and the plan changes as the benchmarks come in. Until 1.0.0, Fleans has no upgrade path between releases — schemas and events change in place, and a fresh database is the upgrade. Comments and pushback are welcome on the linked issues.

Goals

Scalability here means three independent axes. Each scales by its own mechanism, and a bottleneck on any one of them caps the whole system.

Axis	Metric	Level 1	Level 2 (1.0.0 goal)
Start rate	completed instances per second (10-step process)	1,000 / s	10,000 / s
Latency	p99 process duration, no waits	under 1 s	under 1 s
Waiting instances	live instances (awaiting a message, an approval, a timer)	10 million	100 million
Timers	active timers at the same time	1 million	10 million

For reference, Zeebe’s engineering team targeted around 500 process instances per second at p99 under one second in 2023 (Camunda). The targets are proposals: they will be confirmed or revised by real workloads.

Assumptions:

• Self-hosted Kubernetes, no managed cloud services required.
• .NET 10 and Orleans stay the foundation; BPMN semantics and the journal contract (IEventStore) do not change.
• The journal store is Citus first, with YugabyteDB as the alternative when built-in Raft failover matters more than the smallest code change. The choice does not affect the other layers.

Where Fleans stands today

Today Fleans hits PostgreSQL before it hits silo CPU: in the v0.1.0 load test two silos saturate the database (Postgres CPU above 600 %, connection-pool exhaustion) — see Load testing. Behind the database sit shared points inside the cluster, found by the October 2026 code audit.

Bottleneck	Where	Why it does not scale	Issue
One PostgreSQL primary	event store and all grain storage	Every silo writes the journal, projections and grain state to one node	#879
Projection on every append	workflow state projection	Loads and diffs the instance’s projected rows synchronously after each write	#861
No retention	—	Completed instances accumulate in the journal and the read model forever	#862
One evaluator for the cluster	script and condition evaluator grains	[WorkerPlacement] overrides [StatelessWorker], leaving a single activation	#857
One grain per start event	start-event listener grains	Fixed: every silo caches which processes an event starts, refreshed on deploy, enable and disable; a start goes straight to the new WorkflowInstance. A silo that misses a refresh is at most 30 s stale	#858
One grain per signal name	signal correlation grain	Fixed: a name’s subscriptions are spread over 64 bucket grains by instance id, and a broadcast calls the buckets in parallel; each write touches only its own rows	#859
Global conditional-start registry	conditional start registry grain	Fixed: cached in every silo with the start listeners; the grain is read only on a cache miss	#858
One reminder per timer	timer callback grain	Each silo holds every reminder of its range in memory and re-reads the range every 5 minutes; Redis keeps all reminders in one key	#886
A definition copy per instance	WorkflowInstance	Fixed: one [Immutable] definition per silo, from a cache; completed instances deactivate at once. Definition lookups are still list scans	#860
A start is 3–4 calls and writes	instance start path	Extra round-trips on the most frequent operation	#863
Double write when an append fails	journal append	A non-idempotent append corrupts the journal when Orleans retries it	#877

Already in good shape: message correlation grains are keyed by message name plus correlation key, so correlation is naturally sharded; and InstanceId is a random Guid v4, so it creates no hot spots.

Target architecture

Two links stay on the hot path — the instance grain and the sharded journal. Both scale by adding nodes; everything else receives data asynchronously through Kafka.

Target architecture. The highlighted arrow is the write path; Kafka, the projector and the read model lag by seconds and never slow an instance down.

A process step is one append to its journal shard and one notification to Kafka. The projector builds the read model from the journal on its own, and external workers receive tasks over gRPC and report each result with one call.

Journal

The journal is the only source of truth, so it lives in a store sharded by InstanceId: writes then grow with storage nodes the same way compute grows with silos.

1. A separate connection for the journal (Persistence:Journal:Provider, ConnectionStrings:fleans-journal) — #879. The journal and snapshots move to their own store; the read model and other grain state stay on PostgreSQL.
2. Citus as the first sharded provider. Only the journal-batch and snapshot tables are distributed by InstanceId, with snapshots colocated with their journal. Every journal query already filters by InstanceId, so each becomes a single-shard query with plain PostgreSQL semantics. The shard count is set with headroom (128).
3. YugabyteDB as the alternative when built-in Raft failover matters more than the smallest change; it needs retries on Snapshot-isolation conflicts and squashed migrations.
4. Idempotent append through a batch id. An id is generated before the retry loop and stored with the batch. If a retry finds a batch with the same id at that version, the append succeeded. This closes #877 on any store.
5. Compact payloads. Events no longer carry a full snapshot of the variables (#867); a snapshot every 100 events bounds replay.
6. Contract tests for IEventStore that every provider passes: contiguous versions, conflict returns false, a snapshot exactly at its version, an idempotent retry.

Two invariants make sharding possible:

• Journal tables are only ever accessed by InstanceId — no joins or queries across instances.
• InstanceId stays a random Guid v4, never a time-ordered id: a monotonic key would put every new instance on the last shard under range sharding. People see the BusinessId.

Read model

The read model leaves the hot path: the grain writes only the journal, and a separate projector builds projections asynchronously, seconds behind.

All six points are in place (#861): the grain publishes InstanceAdvanced on the existing stream provider (Redis by default, Kafka optional), and a projector grain builds the read model on the worker silos ([WorkerPlacement]), so it scales with fleans-worker instead of needing a deployment of its own. Each silo writes the projections of many instances in one batch (up to 64 per upsert, Fleans:Projector): about 0.05 write transactions per instance on the read-model primary, down from 4 when each instance was written on its own. Every read-model query goes through ConnectionStrings:fleans-query, and the admin UI shows the version each view reflects. In the target design:

1. A notification instead of a write. After a successful append the grain publishes a small InstanceAdvanced(InstanceId, Version) event to Kafka, partitioned by InstanceId so per-instance order holds.
2. A projector service consumes it. It reads the journal tail by InstanceId, applies events through the same replay code the grain uses, and upserts the read model in batches.
3. Coalescing. Ten notifications for one instance in a window become one projection up to the latest version — the main win for fan-out and multi-instance.
4. Recovery without scanning the journal. Kafka delivers at least once and each projected row stores its version, so a replay is idempotent. Catch-up when a grain activates stays as a safety net.
5. Reads come from replicas. ConnectionStrings:fleans-query points at a read replica; the admin UI, MCP and REST lists never touch the primary.
6. The UI shows the lag honestly — “state as of version N”. The exact state of one instance still comes from its grain.

Retention (#862):

• Read-model tables are partitioned by completion month; old partitions are archived and dropped whole (a partition drop is far cheaper than DELETE). Done for the instance details on PostgreSQL (#980): entries, variable scopes and condition results; the instance rows stay unpartitioned.
• A retention worker selects completed instances from the read model and deletes their journal and snapshots by InstanceId, keeping the journal invariant intact.
• Optionally, an instance’s journal is compressed into S3-compatible storage before deletion, for audit.

Messages, signals and start events

Every grain that the whole stream of one event passes through is replaced by either a sharded group of grains or a read-only cache in every silo.

Mechanism	Today	Target
Message correlation	one grain per message name + correlation key	unchanged — already sharded by key
Message / signal start	subscribed processes cached in every silo, refreshed on deployment; the start goes straight to a new WorkflowInstance (#858)	unchanged — done
Message-start deduplication	a send with an IdempotencyKey runs once, in a grain per message name + key; a retry within 24 h returns the first outcome. Sends without a key pay nothing (#743)	done; POST /Instances/purge removes expired records in bounded batches (#977)
Signal subscriptions	done: B bucket grains per signal keyed by name + hash(InstanceId) mod B, B = 64 by default (Fleans:Signals:BucketCount), stored and written per row (#859)	B buckets per signal keyed by name + hash(InstanceId) mod B, B = 64 by default
Signal broadcast	done: parallel fan-out to the B buckets; each bucket queues its subscribers in pages on its own stream and returns, and one consumer per bucket delivers them at Fleans:Signals:DeliveryConcurrency (default 1), so a broadcast’s load is bounded however many instances wait. 20 000 subscribers on the laptop stack: delivered in ~20 s, linear-10 kept its start rate (#976). A transient failure keeps the subscription for the next broadcast (#742)	parallel fan-out to B buckets; large buckets page subscribers through a stream
Conditional-start registry	cached in every silo, like start listeners (#858)	unchanged — done

Rules for every sharded grain: state is stored per row (add or remove one subscription), never rewritten whole, so subscribe cost does not grow with N; every method reachable from a stream stays idempotent; the bucket count is configuration and changes only on an empty cluster before 1.0.

Timers

Timers scale in two steps: first, upgraded Orleans reminders on PostgreSQL with an index; then, only if the timers-1m benchmark fails, P scheduler shard grains with one reminder each — making the reminder count a constant (P = 256) instead of one per timer.

Today every BPMN timer is a timer-callback grain with its own Orleans reminder. Orleans documents no limit on the number of reminders, but in Orleans 10.0.1 (pinned until #886 moved Fleans to 10.4) their cost grows with the number of timers, not with how many fire — and boundary timers on long user tasks easily reach millions. How it works (checked against the Orleans source):

• The grain ring is split between silos. Each silo loads every reminder of its range into memory and runs a timer for each one.
• Every 5 minutes (RefreshReminderListPeriod) each silo re-reads its whole range. At 10 million timers the cluster reads about 33 thousand rows per second even if none fires.
• The Redis provider (the Fleans default) keeps every reminder of the cluster in one sorted set. One key lives on one node, so Redis Cluster cannot spread it.
• The PostgreSQL provider reads ranges by grain hash, but the bundled schema has no index on that column, so every range read scans all reminders.
• Orleans 10.3.0 adds ReminderLoadingWindow (10 minutes by default): only reminders due within the window stay in memory. That fixes memory, not the full-range re-read or the single Redis key.

Step 1 (#886): upgrade Orleans to 10.3 or later, move reminders to the PostgreSQL provider in their own database, add a (ServiceId, GrainHash) index, then run timers-1m. Step 2, only if that run fails — the scheduler shards:

1. A timers table Timers(Partition, DueAt, InstanceId, TimerId) in the same sharded store as the journal, distributed by Partition = hash(InstanceId) mod P.
2. Registration is an idempotent upsert by (InstanceId, TimerId); cancellation deletes the row.
3. A timer shard grain per partition reads its “due now or within 60 s” window with one range query by (Partition, DueAt) every few seconds, holds it in memory and fires on grain timers.
4. Firing calls the WorkflowInstance and deletes the row. A repeat is safe: the engine already ignores stale timer fires.
5. Self-healing. On activation an instance re-registers its pending timers from state, so a timer row lost after an append comes back.
6. Timer start events (one scheduler grain per process) stay on reminders — there are only a handful.

Firing precision is about a second, enough for BPMN SLA timers.

Task execution

The most frequent work — evaluating conditions and taking transitions — runs inside the instance grain with no network hop; only heavy tasks go over the network, and they scale on a separate pool of silos.

1. Compiled expressions, cached per silo. Done. Conditions and script statements are parsed once per silo and the compiled delegate is reused, keyed by the expression text — so every version of a definition that keeps an expression shares it. A deploy compiles every condition and script first: one that does not compile rejects the deploy instead of failing its first instance, and the compiled delegates are already cached on the deploying silo.
2. Conditions run in process. Done. Gateway, complex-gateway activation, multi-instance completion and conditional-event conditions are evaluated inside the instance grain, in the call that reaches them — no stream and no grain call. An exclusive gateway step went from about 100 ms to about 1 ms (TestingHost, idle cluster).
3. Scripts go straight to a [StatelessWorker]. Done. The instance dispatches a script task one-way to the script executor — local on a Combined silo, on a Worker silo from Core — and the executor calls back with the result, so neither a stream nor the instance’s turn waits for the script (#857 removed the placement conflict). A lost dispatch or result is re-sent after Fleans:Worker:ScriptResultTimeout and on the instance’s activation; only the first result is applied. A script step went from about 100 ms to 2–4 ms (TestingHost, idle cluster).
4. One process definition per silo. The definition is [Immutable] and comes from the cache instead of being copied into every activation (#860): less memory per instance, faster activation.
5. A fast start path. The API calls the new WorkflowInstance once and it reads the definition from the cache; 3–4 calls and writes become one append (#863).
6. Custom tasks on their own pool. Done (#1013). Custom tasks run on gRPC workers outside the cluster and scale independently of it.
7. External workers over gRPC. Done (#979). For tasks executed by someone else’s code — in the spirit of Zeebe job workers and Camunda 7 external tasks — a worker process subscribes to the API over gRPC, receives the tasks of its types on that stream and reports each outcome with one call; it needs no broker and joins no cluster. See External workers (gRPC). The first version used a Kafka topic per task type with completion over REST (150 instances/s of load-one-task on the laptop stack, below the plugin host’s 340/s); tests/load/docker-compose.yml measures the gRPC worker on the same stack.

Cluster and infrastructure

Each kind of load gets its own deployment with its own autoscaling, and an overloaded system answers 429 instead of timing out.

Deployment	Hosts	Scales on
fleans-api	REST, an Orleans client with no grains	requests per second, CPU
fleans-core	WorkflowInstance, correlation and timer shards	CPU, activation count
fleans-worker	scripts, the InstanceAdvanced consumer and read-model upserts	CPU, task queue length, Kafka consumer lag
fleans-management, fleans-mcp	admin UI and MCP, reading replicas	manual

• API split from the silos. Done: Fleans:Role=Api runs fleans-api as an Orleans client with no grains (the Helm chart’s api.enabled), so REST ingress scales independently of activations. Without it the core silo still serves REST.
• Streams on Kafka with 64–128 partitions instead of the Redis default. Redis streams remain for development and small installations.
• Membership tied to pods. The load test showed a window of about a minute where a dead silo stays in the cluster table and still receives calls. The Orleans Kubernetes hosting package marks silos without a pod as dead at startup and has two silos watch pods at runtime; together with graceful shutdown on SIGTERM this closes the window. It complements the clustering provider rather than replacing it (#885).
• Membership on PostgreSQL. The table load is tiny (a row per silo, a heartbeat every 30 seconds); atomic writes and high availability are what matter. PostgreSQL gives transactions and Patroni; Redis replicates asynchronously and can lose an acknowledged write on failover, so it stays for development and small installations.
• Backpressure at the edge. Done: an adaptive admission limit in every fleans-api process (Fleans:Admission, on by default) shrinks when the p95 latency of starts, messages and task operations passes its target and answers 429 with Retry-After for the rest (#922). On the laptop stack, one-task offered at 500/s — past the CPU ceiling of about 400/s — completed 272/s with a p99 of 1.2 s and was back to normal the next minute; without the limit it completed 34/s and was still working off the backlog minutes later. The first collapse the load test showed was not CPU: custom-task handler grains stayed activated for 15 minutes after their task and filled the plugin host’s memory; they now deactivate once the task is done. Orleans load shedding (Fleans:LoadShedding, off by default; the API answers 503) and the fixed ASP.NET Core rate limiter stay as opt-in extras. The limit does not react to projector lag, by design (#978). Execution, including user-task operations, never waits on the read model, so a slow projector should not throttle the write path. Alert on fleans.projection.lag instead; the risk of losing trimmed stream notifications is tracked in #850.
• Connection pools. PgBouncer in front of the read-model PostgreSQL; for the journal, a pool per Citus node or YugabyteDB’s connection manager.
• Scale metrics in the Fleans meter: append p99, projector lag, timer lag (actual fire time minus due time), Kafka consumer lag. Autoscaling is tuned on these. Shipped as fleans.journal.append.duration, fleans.projection.lag, fleans.timer.lag and fleans.kafka.consumer.lag (catalog). The Helm chart autoscales Core and Worker silos on CPU (core.autoscaling, worker.autoscaling), and with KEDA also on these metrics (autoscaling.keda, Prometheus triggers).

Where each kind of data lives:

Data	Store	Why	Issue
Journal, snapshots (later the timers table)	Citus or YugabyteDB	writes shard by InstanceId	#879
Cluster membership	PostgreSQL + Orleans Kubernetes hosting	atomicity and HA matter more than throughput	#885
Reminders	PostgreSQL, own database, (ServiceId, GrainHash) index, Orleans 10.3+	cost grows with the number of timers	#886
Read model	PostgreSQL primary + replicas	ad-hoc admin queries	#861
Stream pub-sub state	none: implicit subscriptions only	nothing to store or clean up	#911
Streams	Kafka	partitions by InstanceId, visible lag	—
Redis	development and small installations only	optional: no host connects to it when clustering, reminders and streams use other providers	#911

How it is measured

Scalability is proven by a curve: doubling silos and storage nodes nearly doubles throughput while p99 stays flat.

• The metric is completed instances per second and p99 process duration, not HTTP requests per second or /Execution/start latency. Both come from the Fleans meter: fleans.workflow.terminated{result="completed"} for the rate and the fleans.workflow.duration histogram for p99.
• Steady state: 5 minutes of warm-up, 30 minutes measured.
• No published number with an error rate above 1 %.
• Hardware and configuration in every report: silo vCPU and memory, storage nodes, shard and Kafka partition counts.

Scenario	What it checks	Pass
linear-10	10 script tasks in a row	the level’s start rate and p99
parallel	fork/join over 3 branches	p99 no worse than linear-10 × 1.5
events	start, wait for a message, correlate	no lost correlations; p99 wait under 1 s after the message
timers-1m	1 million timers due within a minute	timer lag p99 under 2 s
long-lived	10 million waiting instances, then linear-10	start rate drops by no more than 10 %
signal-fanout	one signal to 100 thousand subscribers	everyone receives it; starts are unaffected

Scaling matrix: silos 2 → 4 → 8 → 16 and journal nodes 1 → 2 → 4 → 8. Scaling efficiency is the throughput gain divided by the resource gain; the target is 0.8 or higher at every step. Results are published on Load testing for the release they were measured on.

First run on one laptop (October 2026, pre-release)

The load-test stack in tests/load/ is this target architecture scaled down to one laptop (10 CPU Docker VM): REST on its own Api-role clients, the journal on Citus (coordinator + 2 workers), the read model on a primary with a streaming replica, a dedicated projector silo, Kafka streams and no Redis. With the roadmap pull requests of October 2026 applied:

Process	Sustained (p99 < 1 s)	p99	CPU per instance (laptop)
linear-10 (10 script tasks)	150–200 instances/s	0.13 s at 100/s, 0.19 s at 150/s	29 ms
one external task (Zeebe’s benchmark process)	340 instances/s	0.84 s	16 ms

The laptop-scale scaling matrix gives an efficiency of 0.87–0.88 for Core silos 1 → 2 → 3 and 0.87 for journal nodes 1 → 2; past the ceiling the adaptive admission limit answers 429 and the cluster keeps completing near capacity. Scaled linearly, that is roughly 45 vCPU with high availability for 500 instances/s of linear-10 at p99 under 1 s — Zeebe’s 2023 target used 64 broker vCPU — and about 90 vCPU for level 1. The estimate, its assumptions and the raw numbers are in target-arch-2026-10-24/report.md in fleans-load-results; it is a laptop shape, not a published benchmark — the full scaling matrix still needs a multi-node run.

Roadmap

The work runs in five phases, and each one starts only after the previous phase’s benchmark has hit its bottleneck. Phase 0 changes no storage and is the cheapest gain; phase 1 is the main architectural step, after which writes stop hitting a single node. In keeping with the project’s keep-it-simple rule, phases 3 and 4 shrink if real workloads are served earlier.

Road to 1.0.0 — five phases, with a measured gate between each two.

Risk	Consequence	Mitigation
Asynchronous projection	Lists in the admin UI lag reality by seconds	The UI shows the projection’s version; one instance’s exact state is read from its grain
Open-source Citus has no automatic failover	A journal node failure stops writes to its shards	Patroni on every node, or YugabyteDB
A second database to run	More backups, monitoring and on-call	On Citus the read model can live in local tables on the coordinator — one cluster
Timer and signal shards are harder than singleton grains	New failure classes: a lost timer, a double delivery	Timer self-healing on activation, idempotency, timers-1m and signal-fanout scenarios
Knowledge concentrated in few people	The architecture lives in the authors’ heads	A design document per phase, invariants written down next to the code

Sources

• Load testing
• Zeebe, or How I Learned to Stop Worrying and Love Batching
• Orleans — Timers and reminders · Cluster management · Kubernetes hosting
• Orleans source: LocalReminderService (v10.0.1) · RedisReminderTable (v10.0.1) · ReminderOptions (v10.3.0)
• Issues: #857, #858, #859, #860, #861, #862, #863, #877, #879, #885, #886