Performance Benchmarks

Throughput and latency of the /decide-gateway and /routing/evaluate endpoints under sustained load

Overview

The decision engine contributes ~6 ms (server-side average) to a payment routing decision at typical load. A single 2-core pod sustains 835 req/s peak with zero errors, and throughput scales linearly with additional pods — both routing modes are stateless and share no per-request mutable state.

These figures are from k6 load tests against a single containerised pod. All latency numbers in the tables below represent the server-side processing time reported in each /decide-gateway response — the time the pod spends on the routing decision itself, excluding network distance to and from your application.

Endpoint	Algorithm	Use case
POST /decide-gateway	SR_BASED_ROUTING	Success-rate driven gateway selection
POST /routing/evaluate	RULE_BASED_ROUTING	Priority-rule evaluation using the active configured routing rule

---

Test Environment

Cluster node

Property	Value
vCPU capacity	8
Memory capacity	15.41 GiB
Architecture	linux/amd64
Kernel	Amazon Linux 2023 (6.12.x)

Pod provisioning

Resource	Request	Limit
CPU	1000m	2000m
Memory	256 MiB	512 MiB

Log level: INFO

---

Test Methodology

Tool: k6 — 30s steady load per VU step.

Metrics

• Round-trip (RT) — total time measured by the load generator, including local network.
• Server-side — processing time reported by the pod in the /decide-gateway response latency field, isolating pure compute. Not available for /routing/evaluate.
• HTTP overhead — round-trip minus server-side: HTTP framing, request routing, and response serialisation outside the handler’s timer. Add your production network RTT on top of this.

---

SR_BASED_ROUTING — /decide-gateway

Payload: 3 card variants rotated across VUs (VISA Debit/AED, Mastercard Credit/USD, Amex Debit/EUR), 2–3 eligible gateways each.

Throughput and latency

VUs	Throughput	RT avg	RT p50	RT p95	Server avg	Server p50	Server p95	HTTP avg	HTTP p95
5	38 req/s	8 ms	7 ms	15 ms	5 ms	4 ms	10 ms	3 ms	6 ms
20	147 req/s	10 ms	8 ms	20 ms	6 ms	5 ms	14 ms	4 ms	9 ms
50	345 req/s	15 ms	12 ms	33 ms	10 ms	8 ms	23 ms	6 ms	13 ms

Every /decide-gateway response includes a latency field (milliseconds) reporting the pod’s own processing time. Use this in your monitoring to separate compute time from HTTP stack overhead in production.

Key observations

• Server-side p95 holds at 14 ms up to 147 req/s (20 VUs) — gateway outages and payment-flow config are served from in-process caches, eliminating per-request DB queries.
• At 50 VUs (345 req/s), server p95 is 23 ms.
• The HTTP overhead column (round-trip − server-side) reflects HTTP framing and serialisation cost, not network distance. Add your production RTT on top of the round-trip figures.

These figures model realistic client behaviour with a 100 ms pause between requests. They reflect sustainable throughput for a typical integration, not raw server capacity. See the saturation section below for the throughput ceiling.

Feedback path — /update-gateway-score

Every /decide-gateway call is followed by a fire-and-forget feedback write to /update-gateway-score. This is what keeps SR scores current — it records the outcome of each routing decision back into Redis so the next decision reflects real gateway performance. It runs outside the routing response path and does not block the caller, but it does consume pod resources and contributes to Redis load.

VUs	Feedback avg	Feedback p95
5	2 ms	4 ms
20	3 ms	7 ms
50	3 ms	6 ms

Feedback latency stays flat under load because it is a single Redis write with no read dependencies. At 345 req/s (50 VUs) this means ~345 score updates per second — well within the Redis pool capacity at pool_size = 5 with auto-pipelining.

---

Saturation point

To find the raw throughput ceiling, the same endpoint was driven with no sleep between requests — each VU fires as fast as the server can respond.

Test conditions: release build (LTO, codegen-units=1), 2-CPU Docker limit, 0% error rate throughout. CPU% captured via docker stats in parallel; 200% = 2 cores fully utilized.

VUs	Throughput	CPU%
5	678 req/s	151%
10	816 req/s	190%
20	835 req/s	195%
30	831 req/s	198%
50	815 req/s	198%
75	792 req/s	196%
100	789 req/s	198%
150	779 req/s	197%
200	741 req/s	197%
300	725 req/s	199%

Interpretation

• Peak: 835 req/s at 20 in-flight requests. CPU crosses 190% at 10 VUs and stays there — both cores are fully loaded from that point on.
• Sustained ceiling: 725–835 req/s from 20 to 300 VUs with zero errors. The pod does not shed requests under overload; it queues them.
• The sleep-based figures (345 req/s at 50 VUs) represent 41% of this ceiling. The gap is client think-time, not server capacity.

bash scripts/load-test/benchmark.sh --saturation --max-vus 300 --step-dur 20s

---

RULE_BASED_ROUTING — /routing/evaluate

Payload: same 3 payment attribute variants (amount + authentication_type), evaluated against an active priority rule: checkout → stripe → adyen.

Throughput and latency

VUs	Throughput	RT avg	RT p50	RT p95	Error rate
5	39 req/s	6 ms	4 ms	17 ms	0.06%
20	153 req/s	7 ms	4 ms	17 ms	0.01%
50	389 req/s	6 ms	4 ms	14 ms	0.01%

/routing/evaluate does not include a latency field in the response, so server-side and network latency cannot be separated for this endpoint.

Key observations

• p95 stays flat at 14–17 ms across all concurrency levels and does not grow under load.
• Throughput scales to 389 req/s on a single 2-core pod — rule evaluation is pure in-memory computation with no Redis or DB I/O.

---

SR vs Rule-based

VUs	SR throughput	SR server p95	Rule RT p95
5	38 req/s	10 ms	17 ms
20	147 req/s	14 ms	17 ms
50	345 req/s	23 ms	14 ms

SR server-side p95 climbs at 50 VUs as Redis round-trips grow under contention. This is well below the pod’s CPU saturation point (~835 req/s peak with no-sleep load); the sleep-based test reaches Redis contention before it saturates CPU. Rule-based p95 stays flat because it issues no external I/O.

---

Scaling Guide

SR_BASED_ROUTING

Ceiling per 2-core pod: ~725–835 req/s (835 req/s peak at 20 in-flight requests, sustained above 725 req/s all the way to 300 in-flight with zero errors).

Target throughput	Pods	CPU per pod	Notes
< 100 req/s	1	2 cores	Comfortable headroom — well below saturation
100 – 400 req/s	1	2 cores	Pod within capacity; latency remains flat
400 – 800 req/s	1	2 cores	Approaching ceiling; monitor server p95
> 800 req/s	2+	2 cores	Add pods; raise Redis pool_size proportionally

RULE_BASED_ROUTING

Target throughput	Pods	CPU per pod	Notes
< 400 req/s	1	2 cores	Single pod handles comfortably
> 400 req/s	2+	2 cores	Linear scale-out; p95 remains flat

Do not set CPU requests below 1000m. The Tokio async runtime sizes its worker thread pool based on available CPUs — a very small quota results in a single worker thread and elevated latency under concurrency.

---

What Affects Performance

Log level — largest single factor

Running at DEBUG emits ~55 JSON-serialised log lines per request. At 345 req/s (50 VUs) this means ~19 000 extra serialisations per second and raises server-side p95 significantly. Always set INFO or higher outside of active debugging.

[log.console]
level = "INFO"

Redis round-trips (SR routing only)

Each SR routing decision reads per-gateway SR scores from Redis, checks in-flight keys, and writes feedback scores. Gateway outage lists and per-merchant payment-flow config are served from in-process caches (30 s and 60 s TTL respectively) so only the SR score lookups reach Redis on the hot path. With pool_size = 5 and auto-pipelining this handles the full ~835 req/s pod ceiling on a single 2-core pod. Raise pool_size proportionally when adding pods.

[redis]
pool_size = 10          # default 5; raise when adding pods beyond the first
auto_pipeline = true    # keep enabled

CPU allocation

The runtime sizes its worker pool based on available CPUs. With a 2-core limit the pod runs 2 concurrent async workers, which is the binding constraint for SR routing throughput.

---

Reproducing These Results

# SR + Rule-based routing (sleep-based, realistic think-time)
bash scripts/load-test/benchmark.sh

# Saturation test — find raw throughput ceiling (no sleep, ramps to max VUs)
bash scripts/load-test/benchmark.sh --saturation
bash scripts/load-test/benchmark.sh --saturation --max-vus 300 --step-dur 20s

# Results are saved to scripts/load-test/profile_results/ after each run

Related Docs

• Configuration — log level and Redis pool settings referenced above.
• Local Setup Guide — bringing up the stack these benchmarks run against.
• API Guide — /decide-gateway and /routing/evaluate request examples.